When production goes down, the first 60 minutes follow a precise protocol: observe and qualify the incident (T+0), appoint a single incident commander and open communication (T+5), work through hypotheses in order of probability (T+10), choose between mitigation and correction (T+30), then restore service or escalate (T+60). This framework avoids the two most expensive traps: disorderly agitation and blind restarts that destroy the evidence.
T+0: observe and qualify, touching nothing
Before acting, you must know what is actually broken. Three questions in under five minutes:
- What is the scope? Is everything down, or only one feature, one region, one type of user? A test from an external network avoids confusing a service outage with an outage of your own access.
- Since when? The start time, cross-referenced with monitoring graphs, immediately narrows the possible causes.
- What is the business impact? Blocked sales, data at risk, or mere slowness: the answer determines the urgency level and who must be informed.
Absolute rule at this stage: restart nothing, change nothing. A restarted service wipes its memory, its running processes and often the explanation of the outage.
T+5: a single commander and paced communication
The factor that distinguishes well-handled incidents is not technical — it is the incident commander: one person who coordinates, decides and speaks. Without one, three engineers modify the same configuration in parallel and management interrupts the experts every five minutes to ask for a status.
- The commander opens a dedicated incident channel where everything is logged: actions, timestamps, findings. This thread becomes the incident's memory and the raw material for the post-mortem.
- They publish a first status message, however minimal: incident in progress on X, impact Y, investigation under way, next update in 20 minutes. Paced communication removes 80 percent of interruptions.
- They shield the technical responders: questions go through the commander, not to the people investigating.
T+10: hypotheses in order of probability
You do not search at random: you work through causes in the statistical order in which they actually occur.
- The last change. A deployment, migration, configuration or DNS change in the past hours: the most frequent cause. Reflex question: what changed?
- Expired certificates. A sudden outage at a round hour, TLS errors in the logs: a thirty-second check, and a recurring oversight even in mature teams.
- A full disk. Logs, local backups or temporary files saturating a volume: the database or the service stops dead.
- The database. Exhausted connections, a query locking a table, broken replication: the symptoms are timeouts everywhere while the application servers look healthy.
- An external dependency. Payment gateway, third-party API, CDN, cloud provider: check their status pages before searching your own systems for an hour.
- Load. A legitimate traffic spike, a forgotten marketing campaign, or a denial-of-service attack.
T+30: mitigate first, fix later
At the half-hour mark, a decision is due: favor restoring service, not the elegance of the solution. Restoring through a workaround — rolling back the last deployment, failing over to a healthy server, disabling the faulty feature, serving a degraded page — and then fixing the root cause calmly afterwards is almost always the right call.
Rollbacks deserve their own rules: verify that database migrations are reversible before going back, and record the current state (logs, metrics, a dump if possible) before any manipulation. A hasty rollback over a modified data schema can turn an outage into data loss.
T+60: restored, or deliberate escalation
At the hour, two outcomes. If service is restored: watch the metrics closely, keep the channel open and schedule the post-mortem. If the incident persists: escalate without ego — the vendor's expert, the cloud provider's support, specialized external reinforcement. The classic mistake is waiting until the third hour to call for help, when fatigue has already degraded decisions and muddied the trails.
The mistakes that make an outage worse
- Blind restarts: the symptom sometimes disappears, the cause remains, and the evidence is destroyed. The outage returns, with no explanation available.
- Touching everything in parallel: several simultaneous changes make it impossible to know what helped or harmed.
- Failing to preserve logs: rotation, restarts or reinstalls that erase the only material for analysis.
- Communicating too late: silence breeds panic, interruptions, and sometimes business decisions made on false information.
- Hunting for a culprit in the heat of the moment: fear pushes people to withhold information, exactly what the incident does not need.
The blameless post-mortem, and preparing beforehand
In the following days, a blameless post-mortem reconstructs the timeline, identifies the causes — there are almost always several — and produces concrete, dated actions: missing monitoring, an alert to create, a procedure to write. The founding principle: you fix systems, not people; someone who made a mistake within a system that allowed it is not the problem, the system is.
Finally, the 60 minutes are won before the incident: monitoring with alerts on vital signs, a written runbook accessible outside the failing system, up-to-date emergency contacts, a tested rollback, and regular drills. Teams that recover in 40 minutes are not smarter: they had prepared. For an SMB without an internal 24/7 on-call rotation, the equivalent of that preparation is knowing precisely whom to call in the first minute — a partner who already has this protocol in muscle memory.