Fixing news.hwmoon.com deployment outages with self-healing agent healthchecks

This incident review covers the repeated publishing and deployment failures that stopped scheduled updates on news.hwmoon.com.

At first glance, the Slack alerts looked like a Key Vault or token expiration problem. The actual logs pointed somewhere else.

The direct cause was not Key Vault expiration.
The self-hosted Azure Pipelines agent runtime was unstable.
az, npm, npx, AzureKeyVault, Checkout, and Bash task wrappers all failed with exit 139.

That distinction mattered. Extending a token lifetime would not fix a process-level segmentation fault.

Symptoms

All times below are KST.

Time	Pipeline	Run	Failing area	Observation
2026-06-14 22:00	article-generation	#2213	AzureCLI secret load	az crashed with exit 139
2026-06-15 08:00	article-generation	#2215	SWA CLI deploy	npx @azure/static-web-apps-cli crashed
2026-06-15 10:00	article-generation	#2218	npm ci	repeated SIGSEGV under the Node/npm runtime
validation	news-cd-static-web-app	#2226	Checkout task	Agent.PluginHost checkout plugin crashed
validation	news-cd-static-web-app	#2229	AzureKeyVault@2	task Node24 wrapper crashed

The failures were not isolated to one command. The crash pattern moved across Azure CLI, npm, npx, Azure Pipelines task wrappers, and checkout.

Authentication failures usually produce 401, 403, forbidden, or secret-not-found errors. This incident produced process crashes.

Why It Was Not Key Vault Expiration

Key Vault and PAT expiration were ruled out for three reasons.

First, service principal login and Key Vault reads succeeded in runs where the local task process did not crash.

Second, the Azure DevOps PAT path worked during the incident. One generation run created and completed a pull request before the later deploy step failed.

Third, AzureKeyVault@2 itself crashed with exit 139. That points to the agent task runtime, not a secret value.

The operating conclusion was:

Question	Answer
Was the key expired?	Not the direct cause
Should the key be set to never expire?	No
What failed?	self-hosted agent Node/task runtime
What should be fixed?	runtime health, task avoidance, and pre-slot repair

Long-lived tokens increase security risk without addressing this failure mode.

Remediation

The fix was designed to avoid the unstable layers instead of simply retrying the same path.

1. Replace runtime Key Vault task usage

The old pipeline fetched secrets through AzureCLI@2 or AzureKeyVault@2. During this incident, those tasks were part of the failing runtime surface.

The pipeline now consumes Key Vault values through an Azure DevOps variable group:

Key Vault
  -> Azure DevOps variable group
  -> pipeline variables
  -> Bash/SWA CLI environment

The variable group is vg-hwmoon-global-insightlab-dev.

2. Bypass the default checkout task

The default checkout: self path also crashed through Agent.PluginHost. The hardened pipelines now use manual git fetch with explicit authentication and timeouts.

checkout: none
  -> git fetch with System.AccessToken
  -> PAT fallback
  -> timeout
  -> no interactive prompt

System.AccessToken is preferred. PAT is only a fallback.

3. Remove NodeTool dependency

NodeTool@0 reused a broken cache and later failed while extracting downloaded Node archives. The hardened path installs Node with Bash plus Python standard libraries.

download Node archive
  -> download SHASUMS256.txt
  -> verify sha256
  -> extract with Python tarfile
  -> prepend PATH

The app build runtime is pinned to Node.js 22.22.3. The agent task wrapper runtime is repaired to Node.js 24.16.0.

4. Make Static Web Apps deployment explicit

Static Web Apps deployment now uses the SWA CLI directly with retries and smoke checks.

npx @azure/static-web-apps-cli@2.0.9 deploy dist
  -> retry up to 3 attempts
  -> cleanup helper containers
  -> verify public routes

This makes the failure boundary clearer: npm install, static build, SWA upload, and public route validation are separate signals.

Verification

The news hardening work was merged through PR #498; the main merge commit was 5c6a694.

Run	Branch	Result	Meaning
#2235	PR branch	succeeded	hardened CD path validated
#2240	main	succeeded	production deployment validated
public check	https://news.hwmoon.com/kr/	HTTP 200	site route updated
archive check	/kr/archive/	latest article visible	content published

The public site showed the 2026-06-15 08:00 article after deployment.

Scheduled Self-Healing Healthcheck

A separate pipeline now repairs and warms the self-hosted agent before operating slots:

agent-runtime-healthcheck

It runs ten minutes before each two-hour publishing window.

KST	Purpose
07:50	prepare for 08:00 slot
09:50	prepare for 10:00 slot
11:50	prepare for 12:00 slot
13:50	prepare for 14:00 slot
15:50	prepare for 16:00 slot
17:50	prepare for 18:00 slot
19:50	prepare for 20:00 slot

Azure Pipelines schedules use UTC:

schedules:
  - cron: "50 23,1,3,5,7,9,11 * * *"
    displayName: Repair and warm news deployment agent before operating slots
    branches:
      include:
        - main
    always: true

The healthcheck performs:

• repair agent Node24 runtime to 24.16.0
• prepare app Node runtime 22.22.3
• verify Azure Repos authentication
• verify npm and npx execution
• verify SWA CLI 2.0.9
• alert Slack if the check fails

Manual validation run #2244 succeeded.

Agent Node24 runtime is already v24.16.0.
node v22.22.3
npm 10.9.8
Repository authentication check succeeded.
SWA CLI 2.0.9
Agent runtime healthcheck completed.

Queue and Concurrency

The deployment queue was also checked. cd-console-static-web-app ran before the news deployment, so news waited in the self-hosted agent queue and then started normally.

That behavior was queue serialization, not simultaneous workspace corruption.

Still, queue delay can become publishing delay. This is why the healthcheck runs ten minutes before the scheduled publishing slot instead of at the exact same minute.

Operating Rule

For future incidents, the first decision is the failing layer.

Failure signal	First layer to inspect
AzureKeyVault@2, AzureCLI@2, or task wrapper exit 139	agent task runtime
npm ci, npx, or node SIGSEGV	Node cache/runtime
401, 403, forbidden, secret not found	authentication or Key Vault
build succeeds but deploy fails	SWA token or SWA CLI
deploy succeeds but public site is stale	smoke check, cache, route, or dist path
pipeline waits in notStarted	busy/offline agent or queue contention

The key signal is exit 139. In this environment, it should be treated as a runtime crash first, not as a secret expiration.

Remaining Risk

The automation does not remove every risk.

Risk	Response
NAS host is down	Slack healthcheck failure and manual NAS check
Azure DevOps outage	wait/retry after service recovery
Node download fails	checksum retry and next scheduled healthcheck
PAT actually expires	System.AccessToken first, PAT fallback failure visible
Key Vault access is removed	variable group secret download fails early

The durable fix was not an unlimited key. It was better failure classification, fewer fragile task dependencies, self-healing runtime preparation, and public route verification.