Incident response

Severity

Examples:

Anything touching billing correctness or tenant isolation starts at SEV1 and is downgraded only after it is understood, never before.

Roles

For SEV3 and SEV4 one person holds all four. For SEV1 they are four people, and the IC is explicitly not the person who knows the system best — that person should be fixing it.

Timeline

T+0     Alert fires or a report arrives
T+5     Acknowledged. IC assigned. Incident channel opened.
T+10    Severity set. Status page posted if customer-visible.
T+15    Hypothesis. Mitigation started.
        SEV1: secondary paged, leadership notified.
T+30    Status page updated. Escalate if no progress.
T+60    Escalate again if unmitigated. Consider rollback over diagnosis.
Ongoing Status page every 30 min for SEV1, every 60 for SEV2, even with no news.
Mitigated Status page updated. Monitoring period begins.
Resolved Status page final update. Postmortem scheduled.

Mitigate before diagnosing. A rollback that restores service in two minutes beats a root cause found in forty. The cause can be found afterwards, with the system healthy and nobody watching the clock.

Communication

Status page

Investigating   We are aware and looking into it.
Identified      We know the cause and are working on a fix.
Monitoring      A fix is applied; we are watching.
Resolved        Confirmed healthy.

Rules, learned the usual way:

• Post before you understand it. "Investigating elevated tier 2 start latency in bom1" at T+10 is worth more than a precise statement at T+45.
• Name the affected region and tier. A global-sounding post makes unaffected customers investigate their own systems for no reason.
• Say what still works. "Running sessions are unaffected; new session creation is failing" tells customers whether to act.
• Never say "no customer impact" during an incident. You do not know that yet.
• Update on schedule even with no news. Silence reads as abandonment.

Customer notification

Security contacts are maintained separately from billing contacts precisely so a breach notification does not go to accounts payable.

Postmortems

Required for every SEV1 and SEV2, and for any SEV3 that recurs. Published internally within five business days; shared with affected customers on request.

Blameless

The question is never "who ran the command". It is "what let a single command do that", "why did nothing catch it", and "why did it take 40 minutes to notice". A human error that a system permitted is a system finding.

Template

# INC-412 — Tier 2 start latency degradation in bom1

## Summary
Two sentences: what broke, who it affected, how long.

## Impact
Affected organizations, sessions, duration, SLO consequence, credits owed.

## Timeline
Every action and observation, with timestamps, from the scribe's log.

## Root cause
The actual cause. Not "a bug was deployed" — what the bug was, why review
did not catch it, why staging did not catch it.

## Contributing factors
What made it worse, slower to detect, or harder to fix.

## What went well
Genuinely. If detection was fast, say so — it is a control that worked.

## Action items
| Action | Owner | Due | Type |
|---|---|---|---|
| … | … | … | prevent / detect / mitigate |

Action items are classified as prevent, detect or mitigate. A postmortem with only prevent items has not taken seriously that the next failure will be different.

Action items have owners and dates, are tracked to completion, and are reviewed monthly. Unreviewed action items are how the same incident happens twice.

Severity-specific procedures

Security incident

1. Preserve evidence first. Snapshot affected hosts; do not reboot anything.
2. Security lead becomes IC, regardless of who was on call.
3. Assess scope before containing, when containment would destroy evidence — but contain immediately if exposure is ongoing.
4. Legal and compliance are notified at T+0, not after the technical work.
5. Customer notification within 24 hours for a confirmed breach, 72 hours for a suspected one.
6. No public statement until the security lead and legal approve.

Billing incident

1. Stop the pipeline rather than let it produce more wrong data.
2. Quantify which organizations, which meters, which direction, how much.
3. Correct before customers notice. Credit notes for overbilling, issued proactively.
4. Decide on under-billing deliberately and record the decision. Usually: do not back-bill below a threshold.
5. Root-cause the pipeline defect before closing. A mismatch that recurs is systematic.

Capacity incident

1. Reduce pool targets to free capacity for real sessions. Start latency degrades; availability recovers. That is the right trade in a shortage.
2. Contact the organizations driving demand. Often a misconfigured loop that they will stop.
3. Lower quotas temporarily if one organization is consuming a region — with a call, not silently.
4. Escalate to capacity planning. There is no operational fix for not enough machines.

On-call

Sustained paging above the load target is itself an incident. The fix is not a tougher rotation; it is fewer alerts that page. Alerts for conditions a reconciler resolves within its loop interval open a ticket instead.

Every on-call engineer has access to every runbook, the infra console, and break-glass request rights — approval still requires a second person.