The 2:00 AM Crisis That Almost Destroyed Everything
It was 2:00 AM on a Wednesday in January 2023 at Exclusive IT Solutions. Our production environment for a major e-commerce client went dark.
Not slowed. Not degraded. Dark. Complete outage.
I received the alert. My heart jumped. I scrambled to my computer.
I checked:
- Load balancer: Healthy
- Application servers: Healthy
- Database: Healthy
- But customer-facing site: Down completely
Everything looked fine on paper. But users couldn't access the site.
I started investigating. DNS was working. Firewalls were passing traffic. SSL certificates were valid.
I spent 30 minutes going through standard troubleshooting. Everything checked out fine.
I was starting to panic. This was critical. Every minute of downtime was costing this client thousands of pounds. Their peak shopping season was about to start.
Then I made a mistake that almost made things worse.
In my panic, I started changing things. Restarting services. Clearing caches. Tweaking configurations.
I had no evidence any of this would help. I was just... doing something. Anything.
Fortunately, my team lead called me.
"Omeiza," he said calmly, "what's your evidence that these changes will help?"
"I don't have any," I admitted. "I'm just trying things."
"Stop," he said. "You're making this worse. Let's use our incident severity framework."
The Metaphor: The Firefighter vs. The Incident Commander
Let me explain with a metaphor that transformed how I approach production incidents.
Rushing to fix things is like a firefighter trying every door in a burning building at once.
You see flames everywhere. People are screaming. Everything feels urgent. You try every door simultaneously. You waste energy. You spread your efforts. You might even make things worse by opening wrong doors and spreading fire.
Time-sensitive resolution is like an incident commander who assesses first, then acts with purpose.
The commander doesn't just start spraying water everywhere. They:
- Assess the situation
- Identify the most dangerous areas
- Prioritize responses based on severity and impact
- Direct resources effectively
- Communicate clearly
The firefighter feels like they're doing more. The incident commander actually does more good.
The Struggle: The "Fast Isn't Good Enough" Mindset
I'll be honest: in that 2:00 AM crisis, I felt like I wasn't trying hard enough if I wasn't changing things immediately.
Every second the site was down felt like failure. Every minute of investigation felt like wasting time.
I equated speed with effectiveness. Rushing with helping.
But here's the truth: in production incidents, rushing without direction is worse than not rushing at all.
The Transformation: The Severity-Based Response Framework
That morning changed everything. We implemented a severity classification system that changed how we respond to incidents.
The "Severity-First" Framework
Principle 1: Severity Determines Speed, Not Panic
Severity 1: Critical
- Definition: Complete system outage affecting all users or critical business function
- Response time: Immediate (within 5 minutes)
- Action: All hands on deck. Drop everything else.
- Example: Complete e-commerce site down during peak shopping
Severity 2: High
- Definition: Major system degradation or partial outage affecting significant user base
- Response time: Within 15 minutes
- Action: Assign senior resources. Monitor closely.
- Example: Checkout system not working, affecting 30% of orders
Severity 3: Medium
- Definition: Minor degradation or limited outage affecting subset of users
- Response time: Within 1 hour
- Action: Assign appropriate resources. Standard SLA.
- Example: Search functionality not working, but browsing and checkout work
Severity 4: Low
- Definition: Cosmetic issue or non-critical bug with workaround available
- Response time: Within 24 hours
- Action: Add to backlog. Schedule appropriately.
- Example: Display issue on one non-critical page
Principle 2: Diagnosis Before Action
Severity 1-2:
- 5 minutes for initial assessment
- 15 minutes for diagnosis
- Then action with clear plan
Severity 3-4:
- 30 minutes for initial assessment
- 1 hour for diagnosis
- Then action with clear plan
Why: Rushing to action without diagnosis means you might be "fixing" the wrong thing, or making things worse.
Principle 3: Evidence-Based Decisions
Every action during incident response must be based on evidence:
Evidence sources:
- Monitoring data and logs
- Error patterns and frequency
- User reports and symptoms
- Recent changes and deployments
No evidence? No action. Investigate until you have evidence.
Principle 4: Clear Communication
Stakeholders need to know:
- What happened?
- What's the impact?
- What's being done?
- When will there be an update?
- What's the estimated resolution time?
Not technical details. Business impact and clear expectations.
Principle 5: Document as You Go
Everything during incident response gets documented:
- What you tried
- What worked and what didn't
- Root cause when identified
- Resolution implemented
- Lessons learned
Why: You learn from incidents. Documentation is how you capture that learning.
The Real-World Resolution of Our 2:00 AM Crisis
Let me share how we resolved that incident using this framework.
Initial Assessment (2:05 AM)
- Severity: 1 - Complete outage affecting all users during pre-peak season
- Evidence: Load balancer healthy, app servers healthy, database healthy, but site down
- Impact: Estimated £8,000 per hour of lost sales
Diagnosis (2:05 AM - 2:20 AM)
What we checked:
- DNS: Working correctly
- Firewall: All rules correct, traffic passing
- SSL certificates: Valid and not expired
- CDN: Content Delivery Network caching... status: Unknown
Discovery: We checked our CDN provider's status page.
The Issue: Our CDN provider was experiencing a global outage. All cached content was unavailable. Our origin servers were fine, but the CDN wasn't serving content.
Why our monitoring didn't catch it: Our monitoring checked our origin servers directly. It didn't check what users actually experienced (through the CDN).
Resolution (2:20 AM - 2:35 AM)
Action taken: We bypassed the CDN. We updated DNS to point directly to our origin servers.
Impact of bypass:
- Site came back online immediately
- No performance degradation (origin servers were sized to handle the load)
- Temporary solution until CDN resolved their outage
Root cause: Complete CDN provider outage (external dependency)
Post-Incident Actions
Immediate (2:35 AM):
- Communicated resolution to stakeholders
- Updated monitoring to check CDN status
- Documented incident fully
Follow-up (within 1 week):
- Added CDN status check to our monitoring (not just origin servers)
- Implemented CDN failover mechanism (automatic bypass if CDN is down)
- Reviewed all external dependencies for similar risks
- Updated incident response plan for CDN-specific scenarios
The Results: Measurable Impact
This Incident
- Downtime: 35 minutes total
- Revenue impact: Estimated £4,700 (not ideal, but much better than it could have been)
- Communication: 5 status updates provided at appropriate intervals
- Stakeholder confidence: Maintained throughout (clear communication)
Before This Framework
Looking back at previous incidents:
- Average resolution time: 2.5 hours
- Incidents made worse by rushing: 15% of incidents
- Stakeholder confusion: High (inconsistent communication)
- Post-incident learning: Minimal (documented haphazardly)
After This Framework
Looking at incidents in the 6 months after implementing this:
- Average resolution time: 45 minutes (70% faster)
- Incidents made worse by rushing: 0% (none)
- Stakeholder confusion: Low (clear, consistent communication)
- Post-incident learning: High (systematic documentation, actions taken)
The Metaphor: The ER Triage
I like to explain this with another metaphor:
Production incident response is like an Emergency Room.
When patients arrive, the first thing that happens is triage. Not everyone gets treated immediately.
The most critical patients go first. The less critical patients wait. This might feel unfair if you're the less critical patient, but it's how you save the most lives.
Severity classification is triage.
Severity 1 is like cardiac arrest. Immediate, all hands, life-or-death.
Severity 4 is like a broken finger. Important, treat it, but not at the expense of cardiac arrest.
And diagnosis is like the doctor's assessment. They don't just start cutting. They assess, they diagnose, then they treat.
Rushing to treatment without diagnosis might kill the patient.
Common Mistakes to Avoid
Mistake 1: Acting Without Evidence
The Problem: Let's try restarting the database. Maybe that'll help?
The Fix: What evidence do you have that the database is the problem? No evidence? Don't restart it.
Mistake 2: Equating Speed with Effectiveness
The Problem: I need to fix this NOW. I'll change everything quickly!
The Fix: Speed without direction is chaos. Directional, measured action is effectiveness.
Mistake 3: Treating All Incidents as Critical
The Problem: Everything is urgent! Drop everything for every incident!
The Fix: Classify severity. Respond appropriately. Not everything is Severity 1.
Mistake 4: Poor Communication
The Problem: We're working on it. We'll tell you when it's fixed.
The Fix: Clear, regular communication. What's happening? What's the impact? When's the next update?
Mistake 5: Not Documenting Lessons
The Problem: It's fixed! Let's move on.
The Fix: Document everything. Why did it happen? What did we learn? What changes are we making?
The Hard Truth
Fast isn't always best. Appropriate is always better.
I've seen incidents where rushing made things worse—longer outages, more confusion, more mistakes.
I've seen incidents where measured, severity-based, evidence-driven response resolved problems faster and more reliably.
The best incident response isn't the fastest. It's the most appropriate.
A Framework for Your Incident Response
If you're managing production incidents:
Step 1: Classify Severity
What level is this? Critical? High? Medium? Low?
Step 2: Set Response Expectations
Based on severity, communicate when updates will come.
Step 3: Diagnose Before Acting
What evidence do you have? What does it tell you?
Step 4: Act with Clear Plan
What are you doing? Why? What's the expected outcome?
Step 5: Communicate Progress
Regular updates. Clear expectations. Honest about unknowns.
Step 6: Document Everything
What happened? Why? What did we learn?
Quick Incident Response Template
INCIDENT RESPONSE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
INCIDENT DETAILS
Date/Time: ________________________
Incident ID: ________________________
Severity: [ ] 1-Critical [ ] 2-High [ ] 3-Medium [ ] 4-Low
Affected Systems: ________________________
Impact Assessment: ________________________
RESPONSE EXPECTATIONS
Target diagnosis time: ________________________
Target resolution time: ________________________
Update frequency: ________________________
DIAGNOSIS
Evidence Gathered:
- ________________________
- ________________________
- ________________________
Diagnosis: ________________________
ACTION PLAN
What are we doing? ________________________
Why will this fix it? ________________________
Expected outcome? ________________________
COMMUNICATION LOG
Time | Message | To Whom
-----|--------|--------
____ | ________________________ | _____________
____ | ________________________ | _____________
RESOLUTION
What was the root cause? ________________________
What did we do to fix it? ________________________
Is the issue resolved? [ ] Yes [ ] No
POST-INCIDENT
What did we learn? ________________________
What changes are we making? ________________________
Preventative actions: ________________________
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
How do you handle production incidents? Do you rush, or do you respond strategically?
#IncidentManagement #ProductionSupport #ITOperations #CrisisManagement #IncidentResponse