Automating NAS agent operations with Slack-based AIOps

The Slack AIOps work was not about adding one more notification. The goal was to change the operating loop from “a person checks the system in the morning” to “the system detects, classifies, remediates when safe, and reports the result.”

The self-hosted agent solved the hosted-minute problem, but it introduced a new operating surface. If the NAS agent is offline, if a scheduled run does not start, or if a deploy finishes without updating the public route, someone needs to know quickly.

Slack is the right surface because it is already where operational attention lives.

Operating Problem

The publishing system touches several layers:

Azure DevOps schedule
  -> NAS self-hosted agent
  -> article generation
  -> validation
  -> Astro build
  -> Azure Static Web Apps deploy
  -> public route smoke check

When something fails, the message “pipeline failed” is not enough. The useful question is:

Which layer failed, what did the system do, and does a human need to decide anything?

That is where AIOps becomes useful. In this context, AIOps does not mean an autonomous system changing production without limits. It means event collection, state classification, safe runbook execution, and concise human-readable reporting.

Target Loop

The target operating loop is:

Detect
  -> Classify
  -> Remediate when safe
  -> Notify
  -> Escalate only when human judgment is required

The boundary between safe automation and human approval matters.

Safe actions:

• recheck public routes
• rerun a failed content generation job
• restart a known maintenance pipeline
• clean exited helper containers below a defined threshold
• report agent status
• collect logs and summarize failure context

Human approval required:

• credential rotation
• DNS changes
• Terraform apply
• paid resource creation
• Slack app permission expansion
• destructive NAS maintenance

This boundary keeps AIOps from becoming a remote shell with a friendly UI.

Why Slack

Slack provides three useful patterns:

Pattern	Use
incoming webhook	structured status reports
slash command	request a safe runbook
thread	keep incident context together

The goal is not to make Slack the source of truth. Azure DevOps remains the execution and audit layer. Slack is the control and notification surface.

Architecture

The safer structure is to keep the NAS agent outbound-only.

watchdog schedule
  -> Azure DevOps REST API
  -> NAS self-hosted agent checks
  -> Slack webhook report

Slack slash command
  -> HTTPS command endpoint
  -> verify Slack signing secret
  -> queue Azure DevOps maintenance pipeline
  -> NAS self-hosted agent executes
  -> Slack response

Slack should not SSH into the NAS. It should queue approved automation through Azure DevOps so logs, permissions, and history remain visible.

Health Checks

The watchdog should inspect:

Layer	Check	Healthy state
Schedule	expected publication slot	schedule exists and is enabled
Pipeline	queue state	enabled and able to run
Agent	self-hosted runner	online and not stuck
Content	latest generated article	expected time window satisfied
Build	latest static build	succeeded
Deploy	public route	HTTP 200
Cleanup	helper containers	below threshold
Cost	agent pool	self-hosted path used intentionally

The message should say which layer failed. That avoids the common operational trap of checking Key Vault, DNS, and build logs in the wrong order.

Slack Message Design

A useful status message is short but specific:

[HWMOON Ops] 2026-06-01 09:05 KST

Schedule: OK
Pipeline: OK
NAS Agent: Online
Latest generation: 08:00 KST
Build: Succeeded
Deploy: 200 OK
Action: None required

An incident message should include:

Layer: Deploy
Symptom: public route returned 404
Last good run: #1754
Failed run: #1755
Automated action: CD rerun queued
Human action: verify if route still fails after rerun

This makes the Slack message an operational breadcrumb, not just an alarm.

Expansion Path

The same framework can grow carefully:

• summarize Azure DevOps logs
• classify recurring failure signatures
• detect dependency drift
• compare expected article slots to actual public routes
• run safe cleanup tasks
• generate weekly reliability reports

The important rule is that every new automation must declare whether it is informational, safe remediation, or approval-required.

Decision

Slack is a good AIOps surface for this project because it reduces context switching and keeps operational events visible. Azure DevOps remains the execution layer. The NAS agent remains outbound. Automation is allowed only inside defined runbook boundaries.

That is the shape I want for a small production publishing system: practical, observable, and not pretending that AI should own every decision.