01 — The best engineering teams still have incidents. They respond to them better.
This is a guide for engineering leaders who want to build a calmer, faster, more humane incident response practice — whether you’re running a 10-person startup or a 500-person engineering org.
Every meaningful software system fails. What matters is what happens in the 90 minutes after an incident fires. The difference between teams that treat 3 AM pages as routine and teams that treat them as catastrophes comes down to process, culture, and a handful of decisions made long before the alert fires.
What follows is a synthesis of what the best teams — Google SRE, Datadog, Monzo, Stripe — have published about how they actually work. We’ve kept the operational substance. You’ll find frameworks you can copy, templates you can adapt, and benchmarks you can measure yourself against.
Read it end-to-end in about twelve minutes. Skim it for the templates. Send it to your on-call rotation. Either way, the goal is the same: by the end of the next quarter, your team should be measurably better at the thing that keeps you up at night.
“Low MTTR does not mean you never have incidents. It means your team has the tooling, processes, and confidence to recover quickly.” — DORA research, 2025
02 — Five things that separate elite from average.
If you read nothing else in this playbook, read these five findings. They’re the through-line for everything that follows.
- The gap between elite and average is enormous. Elite performers recover from incidents 6,570 times faster than low performers, according to DORA’s State of DevOps research. A gap that size reflects a different operating model.
- Alert fatigue is the silent MTTR killer. 67% of engineers admit to ignoring or dismissing alerts without investigating. 85% of teams report that most of their alerts are false positives. The fix is to delete alerts.
- Operational toil is rising. Despite the AI boom, operational toil rose from 25% to 30% of engineering time in 2025 — the first increase in five years. Tools without process change the shape of the work and leave its volume unchanged.
- Downtime is now a board-level risk. 68% of organizations report losing more than $300,000 per hour during IT incidents. 8% lose more than $1 million per hour. “We’ll fix it when we have time” is no longer a defensible posture.
- The endgame is agentic resolution. The next frontier is autonomous systems that close the loop from stack trace to deployed fix, with engineers reviewing the changes the system writes. More on this in Section 09.
03 — The five stages of incident response — and where teams actually fail.
Most teams are competent at two or three of these stages and quietly broken on the rest. The fix is knowing exactly where your weak link is.
Stage 01 — Detect
Something is wrong. How fast do you know? Elite teams structure observability so that real problems surface within minutes, through carefully curated signals tied to user impact.
The tactical move: Apply the 30-day rule: if no one has acted on an alert in 30 days, delete it. This is the single highest-leverage hour your on-call lead will spend this quarter.
Stage 02 — Triage
Is this a SEV1 or a SEV3? Who owns it? Do we wake someone up? Triage is the stage where judgment compounds most — a 30-second wrong call here costs hours later. Elite teams rely on a rubric.
Stage 03 — Diagnose
Why is this happening? This is the single longest stage for most teams — and the one most amenable to automation. Diagnosis is pattern-matching against logs, recent deploys, dependency graphs, and historical incidents.
Stage 04 — Resolve
Roll back or fix forward? Write the patch, run it through CI, ship it. Resolution is where engineering craft matters most — but also where automation is moving fastest. A growing share of scoped, stack-trace-anchored fixes can now be generated, tested in your pipeline, and deployed without human authorship.
Stage 05 — Learn
A post-mortem that changes behavior. PagerDuty’s 2026 data shows that organizations which turn incidents into structured learning cycles are significantly more likely to see resilience improvements year over year.
04 — The severity classification matrix.
Copy this. Adapt the thresholds to your product. Pin it in your on-call channel. The single most common cause of a botched incident is ambiguity about how bad it is.
A note on standards: Google SRE, Stripe, Atlassian, and PagerDuty each publish their own severity matrices — none of them identical. Some use S0–S2, some SEV1–SEV5. The structure below is a reasonable default for a 10–500 person engineering org; adapt the thresholds to your revenue model and SLOs.
| Severity | Criteria | Response |
|---|---|---|
| SEV 1 | Revenue-impacting. Customer-visible. Affects >10% of users OR any data loss or security breach. | Page on-call immediately. Incident commander assigned within 5 min. Public status page updated within 15 min. All hands if unresolved in 30. |
| SEV 2 | Major feature broken. Affects 1–10% of users. Revenue at risk but not halted. No data loss. | Page on-call. Response within 15 min. Status page updated within 30 min. Target resolution within 4 hours. |
| SEV 3 | Degraded experience. Workaround exists. Affects <1% of users. No revenue impact. | Logged during business hours. Assigned to service owner. Target resolution within 2 business days. |
| SEV 4 | Minor issue. Cosmetic, edge case, or known limitation. No user impact. | Filed as a ticket. Prioritized in normal backlog. |
Rule one: err up. When in doubt between SEV2 and SEV3, call it SEV2. The cost of over-responding is an hour of one engineer’s evening. The cost of under-responding is a customer-facing outage that you didn’t know about until Monday.
Rule two: severity is set by impact. A one-line config fix for a SEV1 is still a SEV1. A week-long refactor for a SEV3 is still a SEV3. Keep impact and effort separate or your priorities will drift.
05 — The on-call rotation that doesn’t burn people out.
Your on-call system is a load-bearing wall for your whole engineering culture. Get it wrong and you’ll lose your best engineers quietly, over 18 months, to competitors whose rotations feel humane.
1. Shift length follows load — Datadog and Monzo both document the same pattern: 8-hour shifts for high-pager-load rotations, 12-hour shifts to minimize handoff risk when load is lower. One-week rotations are the ceiling. Anything longer breaks recovery cycles.
2. Primary, secondary, shadow — Every rotation has three roles. Primary takes the page. Secondary covers if primary can’t respond within ~10 minutes. Shadow is learning the system — they watch the primary’s incidents in real time and gradually earn the pager themselves. This is how institutional knowledge actually transfers.
3. Compensation is not optional — Either pay for on-call time directly, or compensate with equivalent time off. The specific structure matters less than the principle: on-call is work. Teams that treat it as “part of being an engineer” see the highest attrition.
4. Focus time during on-call weeks — The engineer holding the pager should not also be on the critical path for a sprint commitment. Their job that week is on-call work: responding, refining runbooks, tuning alerts, doing the post-mortems. This improves both service reliability and team velocity — because the rest of the team isn’t waiting on a distracted engineer.
5. Track after-hours pages as a leading indicator — Track the distribution as well as the count. If the same two engineers take 80% of the weekend pages, you have a rotation problem or a knowledge-concentration problem. Both predict attrition.
6. Quarterly on-call health review — Thirty minutes. Four questions: How many pages did we take? How many were actionable? Who took the most? What’s the one alert we should delete or rewrite? This is the highest-ROI meeting on your engineering calendar.
06 — Writing runbooks that actually get used.
Most runbooks rot. They’re written by someone who’s leaving, in a wiki nobody visits, during the week of the incident — and never touched again. Here’s the pattern that works.
Write them in the post-mortem. The best runbook is written by the person who just resolved the incident, while their memory is fresh. Make it a required output of every SEV1 and SEV2 post-mortem.
Test them quarterly. Pick one runbook per quarter. Have a different engineer execute it in a staging environment. You’ll find that roughly half are already out of date. That’s the point.
Keep them in the repo. Runbooks are code. They live next to the service they describe. They get reviewed in PRs. They’re versioned with the service. Wikis are where runbooks go to die.
Write for 3 AM. The audience is a half-asleep engineer. Short sentences. Copy-paste commands. No narrative. If you want to explain the reasoning, link to a design doc.
07 — The blameless post-mortem, done right.
Most post-mortems fail in one of three ways: they assign blame in passive voice, they produce action items that never ship, or they produce no follow-up at all. Here’s a template and a ritual that addresses all three.
Writing them for executives. The audience is the engineering team. If your post-mortem reads like a board report, you’ve lost the room. Technical detail belongs in it.
Treating blamelessness as politeness. Blameless is a structural commitment that the post-mortem will not be used in performance reviews, that root causes are systemic, and that engineers who caused incidents are the ones most likely to prevent the next one. The moment an engineer starts filtering what they say, you’ve lost the signal.
Not enforcing action item follow-through. Review every open post-mortem action item at the start of each month’s engineering leadership meeting. Items older than 60 days either get deadlined, re-assigned, or explicitly killed. Ghost action items are a cultural signal that the whole ritual is theater.
08 — Four metrics, honestly measured.
Teams that measure well improve faster. Teams that measure badly optimize for the wrong things. Here’s the short list. Later in the document we’ll look at what an agent changes about each of them.
| Metric | Definition | Elite benchmark |
|---|---|---|
| MTTD (Mean time to detect) | From incident start to first signal captured. | < 5 minutes |
| MTTA (Mean time to acknowledge) | From signal captured to triage begun. | < 5 minutes, 24/7 |
| MTTR (Mean time to restore) | From incident start to service restored. | < 1 hour |
| CFR (Change failure rate) | % of deploys that cause an incident. | 0–15% |
Measuring MTTR from acknowledgment. If your monitoring has a 30-minute blind spot, that blind spot is part of your MTTR. Elite teams are honest about this.
Tracking incident count alone. An engineering org that had ten SEV3s and zero SEV1s last quarter is performing wildly better than one that had two SEV1s. Raw counts obscure this. Track by severity, always.
“Elite performers are 973 times more likely to deploy on demand and 6,570 times faster at recovering from incidents than low performers.” — DORA State of DevOps research
09 — The agentic turn: from monitoring to resolution.
Something structural is changing in how incident response works: a shift from tools that tell you about problems to agents that resolve them — through your CI, into your repo, with your review rules intact.
For twenty years, the incident response tooling market has been organized around observation. Monitoring tells you what’s broken. Alerting tells you it’s urgent. On-call tools route the page to the right human. Every one of these categories ends at the same place: a tired engineer opening a laptop.
That architecture made sense when the bottleneck was information — when engineers needed to be pointed at the problem. It makes less sense now, when the bottleneck is human cognition at 3 AM and the systems available can, for a growing class of stack-trace-anchored errors, handle the entire loop: reading the error, pulling the relevant files, forming a hypothesis, writing a scoped fix, running it through your CI pipeline, and opening a pull request.
What agentic resolution actually means
The word “agent” gets used loosely. Here’s the precise definition worth holding in mind: an agentic system observes, decides, acts, and verifies — iteratively, without a human in each loop. Applied to incident response, that means:
- Observe — the agent captures a production error with full stack trace, request context, and environment metadata. No human triage.
- Decide — the agent pulls the relevant source files, forms a hypothesis about the root cause, and determines a scoped fix. No human diagnosis.
- Act — the agent writes the fix, creates a branch, and opens a pull request that runs through your existing CI pipeline. No human authorship.
- Verify — the agent checks that CI passes, the fix addresses the original error, and the change is scoped to the minimum necessary. A human reviews the PR before merge.
The critical design constraint: the human stays in the loop at the review stage, and the agent handles the stages before it. This is what distinguishes an agentic system from a chatbot or a code suggestion tool.
What this means for MTTR
When the diagnostic, authoring, and testing stages are handled by an agent, the time from error detection to a reviewable fix can shrink from hours to minutes. The human contribution shifts from doing the work to reviewing the work — a fundamentally different cognitive load at 3 AM.
This is one part of what bugstack does. bugstack is an autonomous maintenance department for your software: it watches your app and your repository, puts specialist AI agents on what breaks, and sends you the fix as a pull request you control. The next section says exactly what it covers today.
10 — What this does and doesn’t cover.
An honest benchmark discloses its scope. Here’s exactly what bugstack covers today, and what falls outside the frame.
In scope
| Problem type | Description |
|---|---|
| Uncaught server-side errors | TypeError, ReferenceError, NullPointerException and similar runtime errors with a clear stack trace, captured through an SDK for JavaScript/TypeScript, Python, Ruby or Go |
| Failed deploys | Deploys that fail on a connected host. Render is the only host today |
| Known-vulnerable packages | Found by a repository scan that runs every four hours |
| Committed secrets | Found by a repository scan that runs nightly |
| Outdated packages | Found by a repository scan that runs daily |
Out of scope
- Infrastructure outages (cloud provider failures, DNS, hardware)
- Performance degradation without error signals
- Browser-side errors and silent failures that never throw
- Business logic disputes (the code works as written, but the spec was wrong)
- Security vulnerabilities requiring adversarial analysis
- Data pipeline and ML model issues without stack traces
This distinction matters because it defines the boundary of automation. The crash fixes in scope are stack-trace-anchored, code-level, reproducible, and testable, which is where agentic resolution is strongest. bugstack keeps a person on the larger ones: a crash fix over 3 files or 30 lines always waits for review.
11 — bugstack vs. industry benchmarks.
With scope defined and metrics established, here’s how bugstack’s agentic approach relates to DORA’s elite-performer benchmarks, and what we have measured so far.
| Metric | Industry elite | With bugstack | What changes |
|---|---|---|---|
| MTTD (Mean time to detect) | <5 min | Captured by the SDK when the error is thrown | Uncaught server-side errors are captured at the point of failure. |
| MTTA (Mean time to acknowledge) | <5 min | No human triage step | An agent picks up the error. Nobody has to be paged for the work to start. |
| MTTR (Mean time to restore) | <1 hour | No published figure | The fix arrives as a pull request. Time to restore still includes your CI, your review and your deploy. |
| Severity classification | Manual, per-incident | Unchanged | Severity is still set by impact, using your matrix. What changes is the work needed to respond. |
| Response rate | Business hours + on-call | Around the clock | Errors in scope go to an agent whenever they happen. Not every fix holds: 74% of crash fixes held in our benchmark. |
The severity collapse
For errors within bugstack’s scope, the operational side of the severity hierarchy changes. A SEV2 that would have cost an on-call engineer hours — including context-switching, diagnosis, fix, and testing — can arrive as a pull request to review. The severity is still SEV2 by impact criteria. But when the fix holds, the operational burden drops to a PR review. When it does not, your normal process takes over.
Response rate: the overlooked metric
Many production errors receive no engineering response. They’re logged, maybe triaged, and deprioritized into a backlog that never gets worked.
An error in scope goes to an agent. The fix runs through your own CI and reaches you as a pull request you control.
bugstack does not promise a working fix for every error. What changes is the default. The question shifts from “did anyone respond?” to “do we approve this fix?”
12 — The 90-day implementation roadmap.
You don’t need to overhaul your incident response practice in a single sprint. Here’s a phased approach you can adapt to the size of your team.
Week 1: Automate first, then build around it
Most roadmaps start with process and push automation to month two. This one starts with automation. bugstack has three autonomy levels per repository: “Draft fixes for me”, “Open PRs, I merge” and “Fix & merge for me”. New accounts start at the review level, so nothing merges without your approval, and you immediately get data on what’s fixable vs. what needs human attention. Start here.
- Set up bugstack at the review level — fixes reach you for review and nothing merges on its own.
- Connect your highest-traffic repo first, and install the SDK in the service behind it.
- Write or update severity definitions (Section 04 template)
- Audit your alert inventory — delete anything no one has acted on in 30 days
Weeks 2–4: Foundation + Measurement
- Review the first 10–20 agent-generated PRs to calibrate trust and scope
- Document your on-call rotation structure and compensation model
- Pick one critical service and write a runbook using the Section 06 template
- Instrument MTTD, MTTA, MTTR, and CFR for the past quarter (even rough estimates are useful)
- Identify your top 3 recurring incident types by severity-weighted impact
- Run a tabletop exercise using one of your new runbooks — note what breaks
Weeks 5–8: Expand + Trust
- Review the autonomy level on each repository and raise it only where the fixes have earned it
- Establish CI gate requirements for agent-authored changes
- Add remaining production repos to bugstack
- Hold your first quarterly on-call health review
- Compare time to restore for agent-drafted vs. manually written fixes — and write down what you find
Weeks 9–12: Calibration
- Review MTTR trends against your Week 4 baseline
- Adjust autonomy levels based on how many fixes held and how many did not
- Consider “Fix & merge for me” on low-risk repositories. Automatic merging needs a reproduction that failed before the fix and passed after
- Run a second tabletop exercise — compare response quality to Week 2
- Write the post-mortem on the rollout itself: what worked, what didn’t, what to change
The best time to start was yesterday. The second best time is now.
Every day without automated fixes is another night where a null-check bug pages someone at 2am for a fix an agent could have drafted. The process improvements in this roadmap are valuable on their own. Automation adds to them: install the SDK, connect a repo, and review the first fix when it arrives.
Sign up with GitHub and start with $100 of free credit, or book a call and we set it up with you. Get started, or book a demo to see how it would handle your stack.
Footnotes
- DORA — Accelerate State of DevOps Report (2024–2025). Google Cloud. Published annually; the 6,570× recovery-speed ratio and elite CFR range (0–15%) are drawn from the 2024 edition, which surveyed ~36,000 professionals globally. https://dora.dev/research
- OpsRamp — State of Alert Fatigue Report (2025). Surveyed 500 IT operations professionals. Key findings: 67% routinely ignore alerts; 85% report majority of alerts are false positives; 74% say on-call engineers feel overwhelmed. https://www.opsramp.com
- Google — DevOps Productivity Report: Taming Toil (2025). Documented the rise in operational toil from 25% to 30% of engineering time — the first year-over-year increase since 2020. 78% of developers report spending ≥30% of time on manual toil. Introduced the “30-day rule” for alert hygiene. https://cloud.google.com/devops
- PagerDuty — State of Digital Operations (2026). Annual survey of 1,000+ operational leaders. 68% report >$300K/hr downtime costs; 8% exceed $1M/hr. 42% cite incidents as a direct driver of developer burnout. Organizations with structured learning cycles report 2.4× higher resilience improvement rates. https://www.pagerduty.com/resources
- Haystack Analytics — Developer Burnout Index (2025). Longitudinal study tracking developer wellbeing metrics across 2,000+ engineering organizations. 65% burnout figure represents a 12-point increase from the 2023 baseline. https://www.usehaystack.io
- Datadog Engineering Blog — “On-Call at Datadog” (2024); Monzo Engineering Blog — “How We Handle On-Call” (2024). Both organizations publicly document 8- and 12-hour shift structures, mandatory secondary rotation, and shadow programs for new on-call participants. https://www.datadoghq.com/blog | https://monzo.com/blog
- Atlassian — Incident Management Handbook (2025). Comprehensive guide covering MTTR measurement methodology, severity frameworks, and post-mortem best practices. Emphasizes measuring MTTR from first user impact, not from acknowledgment. https://www.atlassian.com/incident-management