The earlier archive Written for a different audience — operations teams, business stakeholders, new joiners. It is kept unchanged, with its original date, because a dated body of work says something a rewritten one cannot.

Production Incidents: What Time-Sensitive Resolution Really Means

It was 2:00 AM on a Wednesday in January 2023 at Exclusive IT Solutions. Our production environment for a major e-commerce client went dark.


The 2:00 AM Crisis That Almost Destroyed Everything

It was 2:00 AM on a Wednesday in January 2023 at Exclusive IT Solutions. Our production environment for a major e-commerce client went dark.

Not slowed. Not degraded. Dark. Complete outage.

I received the alert. My heart jumped. I scrambled to my computer.

I checked:

  • Load balancer: Healthy
  • Application servers: Healthy
  • Database: Healthy
  • But customer-facing site: Down completely

Everything looked fine on paper. But users couldn't access the site.

I started investigating. DNS was working. Firewalls were passing traffic. SSL certificates were valid.

I spent 30 minutes going through standard troubleshooting. Everything checked out fine.

I was starting to panic. This was critical. Every minute of downtime was costing this client thousands of pounds. Their peak shopping season was about to start.

Then I made a mistake that almost made things worse.

In my panic, I started changing things. Restarting services. Clearing caches. Tweaking configurations.

I had no evidence any of this would help. I was just... doing something. Anything.

Fortunately, my team lead called me.

"Omeiza," he said calmly, "what's your evidence that these changes will help?"

"I don't have any," I admitted. "I'm just trying things."

"Stop," he said. "You're making this worse. Let's use our incident severity framework."

The Metaphor: The Firefighter vs. The Incident Commander

Let me explain with a metaphor that transformed how I approach production incidents.

Rushing to fix things is like a firefighter trying every door in a burning building at once.

You see flames everywhere. People are screaming. Everything feels urgent. You try every door simultaneously. You waste energy. You spread your efforts. You might even make things worse by opening wrong doors and spreading fire.

Time-sensitive resolution is like an incident commander who assesses first, then acts with purpose.

The commander doesn't just start spraying water everywhere. They:

  1. Assess the situation
  2. Identify the most dangerous areas
  3. Prioritize responses based on severity and impact
  4. Direct resources effectively
  5. Communicate clearly

The firefighter feels like they're doing more. The incident commander actually does more good.

The Struggle: The "Fast Isn't Good Enough" Mindset

I'll be honest: in that 2:00 AM crisis, I felt like I wasn't trying hard enough if I wasn't changing things immediately.

Every second the site was down felt like failure. Every minute of investigation felt like wasting time.

I equated speed with effectiveness. Rushing with helping.

But here's the truth: in production incidents, rushing without direction is worse than not rushing at all.

The Transformation: The Severity-Based Response Framework

That morning changed everything. We implemented a severity classification system that changed how we respond to incidents.

The "Severity-First" Framework

Principle 1: Severity Determines Speed, Not Panic

Severity 1: Critical

  • Definition: Complete system outage affecting all users or critical business function
  • Response time: Immediate (within 5 minutes)
  • Action: All hands on deck. Drop everything else.
  • Example: Complete e-commerce site down during peak shopping

Severity 2: High

  • Definition: Major system degradation or partial outage affecting significant user base
  • Response time: Within 15 minutes
  • Action: Assign senior resources. Monitor closely.
  • Example: Checkout system not working, affecting 30% of orders

Severity 3: Medium

  • Definition: Minor degradation or limited outage affecting subset of users
  • Response time: Within 1 hour
  • Action: Assign appropriate resources. Standard SLA.
  • Example: Search functionality not working, but browsing and checkout work

Severity 4: Low

  • Definition: Cosmetic issue or non-critical bug with workaround available
  • Response time: Within 24 hours
  • Action: Add to backlog. Schedule appropriately.
  • Example: Display issue on one non-critical page

Principle 2: Diagnosis Before Action

Severity 1-2:

  • 5 minutes for initial assessment
  • 15 minutes for diagnosis
  • Then action with clear plan

Severity 3-4:

  • 30 minutes for initial assessment
  • 1 hour for diagnosis
  • Then action with clear plan

Why: Rushing to action without diagnosis means you might be "fixing" the wrong thing, or making things worse.

Principle 3: Evidence-Based Decisions

Every action during incident response must be based on evidence:

Evidence sources:

  • Monitoring data and logs
  • Error patterns and frequency
  • User reports and symptoms
  • Recent changes and deployments

No evidence? No action. Investigate until you have evidence.

Principle 4: Clear Communication

Stakeholders need to know:

  • What happened?
  • What's the impact?
  • What's being done?
  • When will there be an update?
  • What's the estimated resolution time?

Not technical details. Business impact and clear expectations.

Principle 5: Document as You Go

Everything during incident response gets documented:

  • What you tried
  • What worked and what didn't
  • Root cause when identified
  • Resolution implemented
  • Lessons learned

Why: You learn from incidents. Documentation is how you capture that learning.

The Real-World Resolution of Our 2:00 AM Crisis

Let me share how we resolved that incident using this framework.

Initial Assessment (2:05 AM)

  • Severity: 1 - Complete outage affecting all users during pre-peak season
  • Evidence: Load balancer healthy, app servers healthy, database healthy, but site down
  • Impact: Estimated £8,000 per hour of lost sales

Diagnosis (2:05 AM - 2:20 AM)

What we checked:

  1. DNS: Working correctly
  2. Firewall: All rules correct, traffic passing
  3. SSL certificates: Valid and not expired
  4. CDN: Content Delivery Network caching... status: Unknown

Discovery: We checked our CDN provider's status page.

The Issue: Our CDN provider was experiencing a global outage. All cached content was unavailable. Our origin servers were fine, but the CDN wasn't serving content.

Why our monitoring didn't catch it: Our monitoring checked our origin servers directly. It didn't check what users actually experienced (through the CDN).

Resolution (2:20 AM - 2:35 AM)

Action taken: We bypassed the CDN. We updated DNS to point directly to our origin servers.

Impact of bypass:

  • Site came back online immediately
  • No performance degradation (origin servers were sized to handle the load)
  • Temporary solution until CDN resolved their outage

Root cause: Complete CDN provider outage (external dependency)

Post-Incident Actions

Immediate (2:35 AM):

  • Communicated resolution to stakeholders
  • Updated monitoring to check CDN status
  • Documented incident fully

Follow-up (within 1 week):

  • Added CDN status check to our monitoring (not just origin servers)
  • Implemented CDN failover mechanism (automatic bypass if CDN is down)
  • Reviewed all external dependencies for similar risks
  • Updated incident response plan for CDN-specific scenarios

The Results: Measurable Impact

This Incident

  • Downtime: 35 minutes total
  • Revenue impact: Estimated £4,700 (not ideal, but much better than it could have been)
  • Communication: 5 status updates provided at appropriate intervals
  • Stakeholder confidence: Maintained throughout (clear communication)

Before This Framework

Looking back at previous incidents:

  • Average resolution time: 2.5 hours
  • Incidents made worse by rushing: 15% of incidents
  • Stakeholder confusion: High (inconsistent communication)
  • Post-incident learning: Minimal (documented haphazardly)

After This Framework

Looking at incidents in the 6 months after implementing this:

  • Average resolution time: 45 minutes (70% faster)
  • Incidents made worse by rushing: 0% (none)
  • Stakeholder confusion: Low (clear, consistent communication)
  • Post-incident learning: High (systematic documentation, actions taken)

The Metaphor: The ER Triage

I like to explain this with another metaphor:

Production incident response is like an Emergency Room.

When patients arrive, the first thing that happens is triage. Not everyone gets treated immediately.

The most critical patients go first. The less critical patients wait. This might feel unfair if you're the less critical patient, but it's how you save the most lives.

Severity classification is triage.

Severity 1 is like cardiac arrest. Immediate, all hands, life-or-death.

Severity 4 is like a broken finger. Important, treat it, but not at the expense of cardiac arrest.

And diagnosis is like the doctor's assessment. They don't just start cutting. They assess, they diagnose, then they treat.

Rushing to treatment without diagnosis might kill the patient.

Common Mistakes to Avoid

Mistake 1: Acting Without Evidence

The Problem: Let's try restarting the database. Maybe that'll help?

The Fix: What evidence do you have that the database is the problem? No evidence? Don't restart it.

Mistake 2: Equating Speed with Effectiveness

The Problem: I need to fix this NOW. I'll change everything quickly!

The Fix: Speed without direction is chaos. Directional, measured action is effectiveness.

Mistake 3: Treating All Incidents as Critical

The Problem: Everything is urgent! Drop everything for every incident!

The Fix: Classify severity. Respond appropriately. Not everything is Severity 1.

Mistake 4: Poor Communication

The Problem: We're working on it. We'll tell you when it's fixed.

The Fix: Clear, regular communication. What's happening? What's the impact? When's the next update?

Mistake 5: Not Documenting Lessons

The Problem: It's fixed! Let's move on.

The Fix: Document everything. Why did it happen? What did we learn? What changes are we making?

The Hard Truth

Fast isn't always best. Appropriate is always better.

I've seen incidents where rushing made things worse—longer outages, more confusion, more mistakes.

I've seen incidents where measured, severity-based, evidence-driven response resolved problems faster and more reliably.

The best incident response isn't the fastest. It's the most appropriate.

A Framework for Your Incident Response

If you're managing production incidents:

Step 1: Classify Severity

What level is this? Critical? High? Medium? Low?

Step 2: Set Response Expectations

Based on severity, communicate when updates will come.

Step 3: Diagnose Before Acting

What evidence do you have? What does it tell you?

Step 4: Act with Clear Plan

What are you doing? Why? What's the expected outcome?

Step 5: Communicate Progress

Regular updates. Clear expectations. Honest about unknowns.

Step 6: Document Everything

What happened? Why? What did we learn?


Quick Incident Response Template

INCIDENT RESPONSE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

INCIDENT DETAILS
Date/Time: ________________________
Incident ID: ________________________
Severity: [ ] 1-Critical [ ] 2-High [ ] 3-Medium [ ] 4-Low
Affected Systems: ________________________
Impact Assessment: ________________________

RESPONSE EXPECTATIONS
Target diagnosis time: ________________________
Target resolution time: ________________________
Update frequency: ________________________

DIAGNOSIS
Evidence Gathered:
- ________________________
- ________________________
- ________________________

Diagnosis: ________________________

ACTION PLAN
What are we doing? ________________________
Why will this fix it? ________________________
Expected outcome? ________________________

COMMUNICATION LOG
Time | Message | To Whom
-----|--------|--------
____ | ________________________ | _____________
____ | ________________________ | _____________

RESOLUTION
What was the root cause? ________________________
What did we do to fix it? ________________________
Is the issue resolved? [ ] Yes [ ] No

POST-INCIDENT
What did we learn? ________________________
What changes are we making? ________________________
Preventative actions: ________________________

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

How do you handle production incidents? Do you rush, or do you respond strategically?

#IncidentManagement #ProductionSupport #ITOperations #CrisisManagement #IncidentResponse

The archive

Every article, 2013–2026

Thirty dated articles on application support, delivery, infrastructure and data — plus the recent writing on CI/CD reproducibility and offline-first design.

The recent work is written for a specialist audience, and it is where the measurement discipline is set out in full.

Recent writingOpen Source Impact