The earlier archive Written for a different audience — operations teams, business stakeholders, new joiners. It is kept unchanged, with its original date, because a dated body of work says something a rewritten one cannot.

Uptime Monitoring: Your Early Warning System

It was 2:00 AM. I was at home, asleep. Our main application server crashed.


The Alert That Didn't Alert

It was 2:00 AM. I was at home, asleep. Our main application server crashed.

But I didn't find out until 6:30 AM when the operations manager called me, angry.

"Why is the system down? Customers can't place orders. This is ridiculous."

I scrambled to my computer. I checked logs. The server had crashed at 2:00 AM. Our monitoring system... hadn't sent an alert.

I checked the monitoring configuration. Alert thresholds were set too high. The server was at 98% CPU for 2 hours before it crashed. But the alert was configured for "99% for 10 minutes."

We never got the warning because our monitoring was configured to tell us when something WAS broken, not when something was GETTING broken.

The system died before we knew it was sick.

The Problem: Alert Fatigue vs Early Warning

At i4cus Nigeria Limited, we had learned the hard way about monitoring.

Initially, we monitored everything. Every metric. Every threshold. Every anomaly.

Result? Alert fatigue.

Our team received dozens of alerts every day:

  • "CPU is above 70%"
  • "Disk is 60% full"
  • "Response time is 1.5 seconds"
  • "Error rate is 0.5%"

Most of these were normal variations. After a few weeks, we stopped looking at alerts. We assumed they were just more noise.

And that's exactly when something went wrong.

The Struggle: Finding the Right Balance

I spent weeks experimenting with monitoring configurations. Every time I thought I had it right, something happened that proved I was wrong.

Too many alerts → Team ignores everything Too few alerts → We miss critical problems Wrong metrics → We're not monitoring what matters Delayed alerts → We respond too late

It felt like an impossible balance.

Then I had a conversation with a seasoned systems engineer who changed my approach.

"Omeiza," he said, "you're thinking about this wrong. Monitoring isn't about alerts. It's about early warning. You want to know something's wrong BEFORE it affects users—not after."

That insight transformed my approach.

The Framework: Monitoring That Predicts Business Impact

I developed a monitoring framework that focused on metrics that predicted problems, not just problems themselves.

The "Early Warning" Monitoring Framework

1. Monitor What Users Experience (Not Just What Systems Do)

Wrong approach: Monitor server CPU, memory, disk usage

Right approach: Monitor response time, error rate, transaction success rate

Why: Users don't care if CPU is 90%. They care if pages load slowly. Server metrics help you diagnose, but user metrics help you know when to act.

Wrong approach: Alert when CPU > 90%

Right approach: Alert when CPU has increased 20% over baseline for 3 hours

Why: A sudden jump from 30% to 50% is more significant than a steady 90%. Trends reveal emerging problems before they become crises.

3. Monitor the Leading Indicators

Wrong approach: Alert when system is down

Right approach: Alert when error rate is increasing or response time is degrading

Why: By the time a system is down, it's too late. Increasing error rate or degrading response time are leading indicators of failure.

4. Monitor Business Impact, Not Just Technical Status

Wrong approach: "Server is down" alert

Right approach: "Order placement failure rate is 5% - affecting customer experience"

Why: Status alerts tell you what's broken. Business impact alerts tell you what matters.

The Implementation: What We Changed

At i4cus, we completely overhauled our monitoring approach. Here's what we did:

Step 1: Identify Critical User Journeys

We stopped monitoring everything. We started monitoring what mattered:

Critical journeys identified:

  • Customer placing an order
  • User logging in
  • Admin viewing reports
  • Integration with payment processor

Why: These are the user experiences that, if broken, significantly impact business.

Step 2: Define Meaningful Metrics for Each Journey

User JourneyMetrics MonitoredLeading Indicators
Order Placement- Success rate<br>- Response time<br>- Error type distribution- Response time increasing 20%<br>- Error rate above 1%
User Login- Success rate<br>- Time to authenticate<br>- Failed login attempts- Failure rate increasing<br>- Time to authenticate slowing
Report Viewing- Load time<br>- Data retrieval time<br>- Timeout rate- Load time > 5 seconds<br>- Timeout rate > 0.5%
Payment Integration- Transaction success rate<br>- API response time<br>- Error codes- Success rate below 95%<br>- Response time > 3 seconds

Step 3: Implement Trend-Based Alerting

Instead of threshold alerts ("CPU > 90%"), we implemented trend alerts ("CPU increasing 20% over baseline for 1 hour").

Example:

  • Baseline CPU for application server: 40%
  • Trend alert triggers if: CPU has increased to 50% and stayed there for 1 hour

Result: We caught problems 30-60 minutes before they became incidents.

Step 4: Prioritize Alerts by Business Impact

Every alert included business impact:

Before: "Application server CPU is 95%"

After: "Order placement response time has increased from 2 seconds to 5 seconds. Customer experience is degrading. Investigate within 15 minutes."

Result: Team knew priority and urgency. No more wondering "how bad is this?"

Step 5: Regularly Review and Adjust

Every month, we reviewed:

  • Which alerts were useful?
  • Which were noise?
  • Which problems did we miss?
  • Which metrics were misleading?

Result: Monitoring got smarter over time. We reduced alert volume by 70% while increasing incident detection by 40%.

Real-World Example: The Early Warning That Saved Us

Three months after implementing this approach, we had an incident that proved its value.

The Situation

At 10:00 AM, we received a trend alert: "Order placement response time has increased 30% over baseline for 45 minutes. Customer experience is degrading."

What We Found

After investigation, we discovered:

  • A recent deployment had introduced a database query performance regression
  • Query that used to take 200ms was now taking 600ms
  • Load was gradually increasing as backlog accumulated

What We Did

  • We rolled back the deployment
  • Response time returned to normal within 15 minutes
  • No users experienced significant disruption

What Would Have Happened Without Early Warning

If we had waited for threshold-based alerts:

  • Alert would have triggered at 2:00 PM when response time hit 10 seconds
  • By then, system would have been overwhelmed
  • Rollback would have taken longer due to system load
  • Users would have experienced 2+ hours of degraded service

The early warning alert prevented 2 hours of customer experience degradation.

The Results: Measurable Impact

Incident Detection Time

Before: Average 45 minutes from issue start to detection After: Average 12 minutes from issue start to detection Improvement: 73% faster detection

User Impact Duration

Before: Average incident affected users for 2 hours After: Average incident affected users for 35 minutes Improvement: 71% shorter impact duration

Alert Quality

Before: 80 alerts per day, 15% actionable (12 useful alerts) After: 25 alerts per day, 60% actionable (15 useful alerts) Result: Fewer alerts, same useful information, less alert fatigue

Team Response Time

Before: Average 25 minutes from alert to investigation After: Average 8 minutes from alert to investigation Improvement: 68% faster response (because alerts had clear business impact)

Common Mistakes to Avoid

Mistake 1: Monitoring Technical Metrics Instead of User Experience

The Problem: Alerts about server status when users are unaffected

The Fix: Monitor what users experience. Server metrics help diagnosis, user metrics alert you to problems.

The Problem: Alerting only when something crosses a threshold

The Fix: Monitor trends and leading indicators. Problems don't happen suddenly—they develop over time.

Mistake 3: Alerting on Everything

The Problem: Too many alerts leads to alert fatigue and ignored alerts

The Fix: Alert on what matters. Use severity levels. Make high-severity alerts rare and actionable.

Mistake 4: Alerts Without Context

The Problem: "CPU is 95%" doesn't tell you what to do

The Fix: Include business impact, severity, and recommended action time in every alert.

Mistake 5: Never Reviewing or Adjusting

The Problem: Monitoring setup once and never touched again

The Fix: Regularly review and adjust. Your systems change. Your monitoring should too.

The Hard Truth

Good monitoring is an early warning system. It tells you something's wrong before users notice.

Bad monitoring is a noise generator. It alerts you constantly but doesn't actually help.

I've seen organizations with thousands of metrics and hundreds of alerts that still miss critical incidents because they're monitoring the wrong things.

And I've seen organizations with 20 carefully selected metrics that catch 90% of incidents before they affect users.

Monitoring isn't about how many metrics you have. It's about having the right ones.

A Framework for Your Monitoring

If you're setting up or improving monitoring:

Step 1: Identify Critical User Journeys

What does your business actually do? What would hurt if it stopped working?

Step 2: Define Metrics for Each Journey

Monitor success rate, response time, and error distribution for each journey.

Step 3: Implement Trend-Based Alerting

Don't just alert when something crosses a threshold. Alert when it's trending wrong.

Step 4: Include Business Impact

Every alert should answer: "What does this mean for users? How urgent is this?"

Step 5: Review and Adjust Regularly

Your systems evolve. Your monitoring should evolve too.

The Inspiration

Six months after we implemented this monitoring approach, the operations manager who had called me angry that 6:30 AM came to me.

"You know what I realized?" he said. "Good monitoring isn't about catching problems. It's about preventing them."

That's exactly right.

When you have early warning, you're not reacting to problems. You're preventing them from becoming incidents.

That's the power of good monitoring.


Quick Monitoring Checklist

ComponentWhat to MonitorAlert ConditionSeverity
Order PlacementSuccess rate, response timeSuccess rate < 95% OR response time > 5 secondsHigh
User LoginSuccess rate, auth timeSuccess rate < 98% OR auth time > 3 secondsMedium
Payment IntegrationTransaction success, API responseSuccess rate < 95% OR response time > 3 secondsCritical
DatabaseQuery performance, connection poolQuery time > 2x baseline OR connections > 90%High
External APIsResponse time, error rateResponse time > 2x baseline OR error rate > 5%Medium
System ResourcesCPU, memory, disk (trend-based)Any metric > 2x baseline for 1 hourMedium

What's your experience with monitoring? Do you have early warning systems or just alert generators?

#Monitoring #SystemReliability #ITOperations #Uptime #ProactiveMonitoring #EarlyWarning

The archive

Every article, 2013–2026

Thirty dated articles on application support, delivery, infrastructure and data — plus the recent writing on CI/CD reproducibility and offline-first design.

The recent work is written for a specialist audience, and it is where the measurement discipline is set out in full.

Recent writingOpen Source Impact