Incident response
Route the right alert, preserve the evidence, and close only what you verified.
Notification rules decide who should be contacted. Incident actions record the operator decision without rewriting the underlying run or fabricating recovery.
Create and verify a notification channel
- Open Notification channels → Add channel. Choose Email, Slack, Webhook, PagerDuty, or Telegram and enter the provider destination.
- Scope the rule. Match any monitor or one monitor, any incident type or one type, and any severity or warning/critical.
- Set response timing. Configure cooldown, escalation delay, quiet hours in the workspace timezone, and whether recovery notifications should be sent.
- Simulate the route. Review the visual routing preview before creating the channel. Simulation does not contact the provider.
- Send a test alert. Provider acceptance proves the configured route accepted the message; it does not prove inbox delivery, human receipt, or reading.
Scheduled canaries
Opt in to provider-path checks from every 15 minutes through every 7 days. The channel shows due time, last verification, and canary state.
Provider evidence
The attempt ledger keeps bounded status, timing, response classification, retry state, and acceptance evidence without exposing channel secrets or full provider payloads.
Work the incident queue
The incident list separates open, critical, unassigned, and muted occurrences. Filter by status or severity, open the exact incident, and record a reason for every response-state change.
Assign
Set yourself as the visible responder without changing incident or monitor health.
Acknowledge
Record that a responder owns the incident. Escalation delivery can stop while the incident remains open.
Silence
Suppress notifications for a bounded duration. Monitoring and evidence collection continue.
End silence
Resume eligible notification routing for the still-open occurrence.
Verify recovery
Queue a real uptime probe when available. Heartbeat recovery still requires fresh evidence from the external workload.
Close as operator override
Close the occurrence with a reason when recovery cannot be proved. Monitor health becomes unknown and customer state remains monitoring until a fresh successful run arrives.
Bulk acknowledge and bulk silence support up to 100 selected incidents and require one reason for the complete action.
Use the evidence before changing state
Incident evidence
Review the occurrence type, severity, monitor, opening run, current health, dependency or suppression state, and type-derived safe summary.
Run and log evidence
Inspect lifecycle timestamps, business-output results, semantic-check results, and bounded logs. Private failure details are not copied into public or management summaries.
Delivery evidence
Confirm which operator channels were matched, scheduled, attempted, accepted, retried, suppressed, or failed.
Response history
Use assignment, comments, runbook steps, reasoned actions, and status-page updates to preserve who decided what and when.
Understand recovery and repeat occurrences
- A fresh successful production run after the incident opened can close the active occurrence and send an eligible recovery notification.
- An operator override records a human decision; it does not turn an unknown target into healthy.
- A later outage creates a new occurrence and a new alert cycle instead of reopening historical delivery evidence.
- An active dependency incident can hold duplicate child delivery while child health remains visible. If the child is still failing after the dependency resolves, its delivery is reconsidered.
- Flapping opens a distinct warning occurrence and suppresses ordinary incident delivery for the configured window without hiding the monitor's health changes.