The earlier archive Written for a different audience — operations teams, business stakeholders, new joiners. It is kept unchanged, with its original date, because a dated body of work says something a rewritten one cannot.

Why Regular System Health Checks Prevent Big Problems

It was 4:45 PM on a Friday. I was packing up, mentally already at home, when the monitoring system went dark.


The Friday Afternoon Surprise

It was 4:45 PM on a Friday. I was packing up, mentally already at home, when the monitoring system went dark.

Not the monitored systems—the monitoring system itself.

I scrambled to investigate. Turns out, the disk that hosted our monitoring logs was 98% full. When the monitoring system tried to write a log entry, it couldn't. When it couldn't write logs, it crashed. When it crashed, we lost visibility into all our systems.

This wasn't a sudden problem. The disk had been filling up for months. 90%. 95%. 97%. 98%.

But we never noticed until it was too late.

The irony wasn't lost on me: The system designed to detect problems had a problem we didn't detect.

The Expensive Lesson

That Friday afternoon cost us:

  • 4 hours of emergency work (overtime pay)
  • Weekend disruption (I was on call)
  • Lost visibility into system health for 2 days
  • A very embarrassed conversation with my manager

But the real cost was what could have happened: What if a customer-facing system had failed that weekend? What if a security vulnerability had gone undetected?

We got lucky. The monitoring system went down on a Friday afternoon, not a Monday morning.

But luck isn't a strategy.

The "I'll Fix It When It Breaks" Mindset

I see this everywhere. Organizations wait for systems to fail before they address underlying issues.

  • "The server is slow, but it's still working. We'll upgrade when it crashes."
  • "The backup is failing sometimes, but most of the time it works. We'll fix it when it actually fails."
  • "The error logs show some warnings, but nothing's broken yet. We'll investigate if it becomes a problem."

This is like waiting until your car's engine seizes before changing the oil. Sure, you saved money on oil changes. But now you need a new engine.

The Simple Solution That Changed Everything

After that Friday afternoon, I proposed something radical at Auto360:

Every Monday morning, I'll spend 30 minutes checking system health.

My manager was skeptical. "We have monitoring tools. They alert us when something's wrong."

"Monitoring alerts us when something IS wrong," I explained. "Health checks tell us when something's GETTING wrong. There's a difference."

He agreed to a 4-week trial.

What I Built: The 30-Minute Health Check

I didn't create a complex system. I created a simple, repeatable checklist that I ran every Monday morning.

The Monday Morning Checklist

1. Disk Space (5 minutes)

  • Check all critical servers for disk usage
  • Flag anything above 75%
  • Note growth trends (is it growing faster than normal?)

2. System Performance (5 minutes)

  • Check CPU, memory, and disk usage averages
  • Compare to historical baselines
  • Look for anomalies (sudden spikes, gradual increases)

3. Application Logs (5 minutes)

  • Review error logs from the past week
  • Count error types (how many of each type?)
  • Look for patterns (same error appearing repeatedly?)

4. Backup Integrity (5 minutes)

  • Confirm backups completed successfully
  • Test a random restore (just verify it works)
  • Check backup storage capacity

5. Security Patches (5 minutes)

  • Check if any critical security patches are missing
  • Review vulnerability scan results
  • Note any new security alerts

6. Integration Health (5 minutes)

  • Test critical API endpoints
  • Verify data synchronization between systems
  • Check external dependencies (are they still available?)

That's it. 30 minutes. Every Monday morning.

The Results: Problems I Found

In the first 12 weeks of running this simple health check, I found issues that, if left unaddressed, would have caused significant problems:

Week 2: Disk Heading Toward 90%

Found: Application server disk at 82%, growing 3% per week If ignored: Would have crashed within 3 weeks Action taken: Cleared old logs and increased monitoring frequency Impact: Prevented midday system crash during peak usage

Week 4: Database Performance Declining

Found: Database query times increasing by 15% each week If ignored: Would have caused system-wide slowdown within 2 months Action taken: Added indexes to slow queries Impact: Prevented user complaints about performance

Week 6: Backup Storage Nearly Full

Found: Backup server at 88% capacity, would be full in 4 weeks If ignored: Would have stopped backups, risking data loss Action taken: Cleaned up old backups and expanded storage Impact: Prevented catastrophic data loss scenario

Week 8: Security Patch Missed

Found: Critical security patch not applied to web server If ignored: Would have left system vulnerable to known exploit Action taken: Applied patch immediately Impact: Prevented potential security breach

Week 10: External API Degrading

Found: Third-party payment API response times increasing If ignored: Would have caused payment failures during busy period Action taken: Implemented retry logic and contacted vendor Impact: Prevented payment processing failures

The ROI Calculation

Let's do the math on these 12 weeks:

Time Invested

  • 12 weeks × 30 minutes = 6 hours total

Problems Prevented

  • 1 potential system crash
  • 1 significant performance degradation
  • 1 backup failure scenario
  • 1 security vulnerability
  • 1 payment processing failure

Estimated Cost If These Had Happened

  • System crash: 4 hours downtime × £6,000/hour = £24,000
  • Performance degradation: 2 weeks degraded UX × lost productivity = £8,000
  • Backup failure: Data recovery + reputation = £15,000
  • Security breach: Incident response + potential fines = £50,000
  • Payment failures: Lost transactions + customer trust = £12,000

Total potential cost: £109,000

Time invested: 6 hours

ROI: 1,716% per hour

But here's the thing that really matters: These weren't hypothetical risks. These were problems that WOULD have happened. I saw the trends. I knew the math. I caught them before they became incidents.

Why This Works So Well

1. Early Warning, Not Just Detection

Monitoring tells you something IS wrong. Health checks tell you something's GETTING wrong. The difference is the time you have to respond.

2. Pattern Recognition

When you check the same things consistently, you develop intuition. "This disk growth rate seems faster than normal" catches problems before they reach thresholds.

3. Knowledge Accumulation

Every health check adds to your understanding of normal system behavior. When something changes, you notice immediately—not because an alert triggered, but because it doesn't match your mental model of "normal."

4. Proactive, Not Reactive

You're fixing problems before they cause incidents. You're not scrambling in crisis mode. You're calmly addressing issues before they become urgent.

5. It Builds Trust

When stakeholders see you catching and fixing problems before they impact business, trust grows. You're not just fixing things—you're preventing them from breaking.

Making It Sustainable

Here's how to make health checks stick (I learned from mistakes):

Mistake 1: Making Them Too Complex

The Problem: I initially tried to check everything, created a 2-hour checklist, and abandoned it after 3 weeks.

The Fix: Focus on 5-7 critical items only. 30 minutes maximum.

Mistake 2: Not Following a Schedule

The Problem: "I'll do it when I have time" turned into "I never have time."

The Fix: Calendar it. Same day, same time. Make it non-negotiable.

Mistake 3: Not Acting on Findings

The Problem: I found issues but didn't prioritize fixing them. The health check became a paperwork exercise.

The Fix: Every issue found goes on a todo list with a due date. No exceptions.

Mistake 4: Doing It Alone

The Problem: When I was on vacation, health checks didn't happen.

The Fix: Document the process. Rotate responsibility. Make it team-owned, not person-owned.

A Framework for Your Health Checks

Not every organization needs the same checks. Here's how to customize:

For Small Businesses (1-5 systems)

Focus on:

  • Backups (critical for any size business)
  • Disk space (first thing to cause failures)
  • Basic performance (is the system usable?)

For Medium Businesses (5-20 systems)

Add:

  • Security patches (you're a bigger target)
  • Application logs (patterns indicate problems)
  • External dependencies (are vendors reliable?)

For Large Enterprises (20+ systems)

Add:

  • Capacity planning (are you growing out of your infrastructure?)
  • Compliance checks (are regulations being met?)
  • SLA monitoring (are you meeting commitments to stakeholders?)

The Hard Truth I Learned

The most expensive support incidents are the ones that could have been prevented.

Every major incident I've experienced in my career—from healthcare system outages to commercial application failures—had early warning signs. The problem wasn't that we didn't detect them. The problem was that we didn't detect them early enough.

Prevention isn't about spending massive amounts of time or money. It's about consistency. It's about checking the same things regularly, tracking trends, and addressing small issues before they become big ones.

The Call to Action

Start today. Not tomorrow. Not next week. Today.

Create your health check checklist. Block 30 minutes on your calendar. Make it non-negotiable.

Your future self—the one who's not dealing with a preventable system failure at 4:45 PM on a Friday—will thank you.


Simple Health Check Template

System Health Check - [Date]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Checked by: _______________
Time spent: ___ minutes

DISK SPACE
[ ] Critical servers reviewed
[ ] Any disk > 75%: ___________
[ ] Growth rate concerns: ___________

SYSTEM PERFORMANCE
[ ] CPU/Memory/Disk usage reviewed
[ ] Anomalies detected: ___________

APPLICATION LOGS
[ ] Error logs reviewed (past 7 days)
[ ] Repeated errors: ___________

BACKUP INTEGRITY
[ ] All backups completed successfully
[ ] Random restore tested: [ ] Yes [ ] No
[ ] Backup storage capacity: ___________

SECURITY PATCHES
[ ] Critical patches missing: ___________
[ ] Vulnerability scan results reviewed: [ ] Yes [ ] No
[ ] New security alerts: ___________

INTEGRATION HEALTH
[ ] Critical APIs tested: [ ] Yes [ ] No
[ ] Data sync verified: [ ] Yes [ ] No
[ ] External dependencies available: [ ] Yes [ ] No

ISSUES TO ADDRESS
1. ___________ (Priority: ___ , Due: ___ )
2. ___________ (Priority: ___ , Due: ___ )
3. ___________ (Priority: ___ , Due: ___ )

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Do you run regular health checks? What's saved you from disaster?

#SystemMaintenance #PreventiveMaintenance #ITOperations #HealthChecks #ProactiveMonitoring #BusinessContinuity

The archive

Every article, 2013–2026

Thirty dated articles on application support, delivery, infrastructure and data — plus the recent writing on CI/CD reproducibility and offline-first design.

The recent work is written for a specialist audience, and it is where the measurement discipline is set out in full.

Recent writingOpen Source Impact