Preventing another news.hwmoon.com article publishing outage

This incident review covers a scheduled publishing failure on news.hwmoon.com. The first signal looked like a deployment problem, but the failure actually had two layers: a build-time parsing failure and a later Rollup stability issue during article preflight.

The key conclusion was:

The direct failure was not Key Vault.
The build validation failed before Static Web Apps token usage became relevant.

That distinction matters because debugging the wrong layer wastes recovery time.

Timeline

All times are KST.

Time	Run	Result	Meaning
2026-06-05 08:01	cd-dev-static-web-app #1741	failed	Astro build stopped on an invalid token
2026-06-05 09:03	article-generation #1747	failed	generated candidate hit Rollup deopt behavior and retry ended with SIGSEGV
2026-06-05 09:19	PR #381	merged	pinned Astro/Rollup behavior and isolated candidates
2026-06-05 09:20-09:25	#1751-#1754	succeeded	CI/CD verified main
2026-06-05 09:25-09:28	article-generation #1755	succeeded	generation, preflight, and deployment recovered

Why It Looked Like Key Vault

Static Web Apps deployment uses a deployment token, so failures near deployment often smell like Key Vault or secret retrieval problems. In this case, the failing step was earlier.

The run reached:

content safety
content check
content audit
Astro build

The build failed before the deployment token mattered. That changed the recovery path from “inspect secrets” to “inspect generated content, dependencies, and build tooling.”

First Fix: Stronger Preflight

The first fix made validation harder to bypass:

• check generated markdown for control characters
• validate frontmatter fences and replacement characters
• run content safety before static build
• use a build wrapper that cleans generated output before retry
• keep deployment preflight consistent between CI and article generation
• use clean checkout on the self-hosted agent

This made build failures more explicit and reduced the chance that a partial generated file would poison future runs.

Second Fix: Dependency Locking

The next scheduled run revealed a different failure:

expressionsToBeDeoptimized is not iterable
SIGSEGV

The generated article passed basic text safety, but the build toolchain was unstable under the current dependency combination. Retrying the same candidate with the same dependency graph was not enough.

The dependency policy became:

{
  "devDependencies": {
    "astro": "6.3.5"
  },
  "overrides": {
    "rollup": "4.43.0"
  }
}

The lockfile is part of the fix. Without it, npm ci in Azure DevOps can still drift through transitive dependency resolution.

Candidate Isolation

The old model allowed one bad generated candidate to fail the entire scheduled run.

candidate -> article file -> preflight fail -> pipeline fail

The new model tries candidates independently:

candidate A -> preflight fail -> remove file -> try candidate B
candidate B -> preflight pass -> publish

If all candidates fail, the run fails visibly. That is intentional. Silent content corruption is worse than a clear failed run.

Verification

Local checks included:

node --check app/scripts/run-article-generation.mjs
node --check app/scripts/check-source-text-safety.mjs
node --check app/scripts/run-build-with-retry.mjs
npm --prefix app ci --ignore-scripts
npm --prefix app run article:check
npm --prefix app run content:safety
npm --prefix app run deploy:preflight

Azure DevOps confirmed recovery with successful CI/CD and article-generation runs.

The public site was also checked with HTTP requests. A pipeline is not fully recovered until the public route is updated.

Operating Rule

When a publishing run fails:

1. identify the exact failing step
2. decide whether the failure is before or after secret usage
3. inspect generated content before blaming deployment
4. inspect dependency drift before retrying blindly
5. isolate bad generated candidates
6. verify the public URL after deployment

The durable fix was not “run it again.” The durable fix was better failure classification, candidate isolation, dependency pinning, and production route verification.