Case study 01

The pipeline went red because the bill was due.

How 28 red runs in a row turned out to be a billing decision, and what a local gate is actually for when the runner never starts

In plain terms

The build machine ran out of its monthly allowance, so the pipeline went red. Nothing was wrong with the code — the jobs simply never started, and the failure looked exactly like a broken build. This runs the same checks on the developer's own laptop, so the answer arrives in about a hundred seconds and does not depend on anybody's budget.

  • Before: the account used all 2,000 of its included build minutes, with 11 days of the month still to go
  • 28 builds in a row failed, none of them on code — the jobs were never started
  • After: the same check runs locally, takes about 100 seconds, and says why it failed
  • The checks that moved to the laptop now cost no build minutes at all
  • It also found three real defects the missed builds had been hiding

On 21 October 2026 the account this project lives on used the last of its 2,000 included Actions minutes, with eleven days of the month still to go. Twenty-eight runs out of the last thirty failed, and not one of them failed on code.

RoleDesign, implementation, evidence
Reading time9 min read
ProjectCI Kit · MIT · Shell

01

The problem

A red build is not always a red build

On 21 October 2026, the account running this workspace ran out of its included GitHub Actions minutes. The notification said it plainly:

You have used 100% of the Actions minutes included for the omeiza-workspace account

2,000 min used / 2,000 min included

You'll be billed for usage beyond the included amount — unless a $0 budget is set for Actions, which blocks further usage until it resets. Your usage will reset in 11 days on November 01, 2026.

Eleven days of pipeline left to run, and no minutes to run it on. Every job from that point either cost money per minute or did not start at all.

The largest repository in the workspace, bytesickle-web, had 28 failing runs out of the last 30. Every one failed in about six seconds, with the same annotation:

The job was not started because recent account payments have failed or your spending limit needs to be increased. Please check the 'Billing & plans' section in your settings

The cause was the bill, not the code. The included minutes were gone, the spending limit that would have allowed paid overage had been reached, and the jobs were never allocated a runner at all.

Taken at face value a red build is a red build. The developer assumes their change broke something, and either fixes a problem that does not exist or pushes again and waits. What actually happened was that nothing was verified at all, and the reason was one line of text in a log nobody reads.

The failure was not a flake and it was not a regression. It was an infrastructure ceiling, and it looked exactly like a broken build — which is the part that makes it worth writing down.

The pipeline went red because the bill was due. Twenty-eight runs in a row, and not one of them failed on code.

What it looked like day to day

  • 28 of the last 30 runs failed, none of them on code
  • Every failure arrived in about six seconds — far too fast for a test or a compile
  • The account's 2,000 included minutes were exhausted with 11 days left in the cycle
  • Until the reset on 1 November, every job either cost money per minute or did not start
  • A stale secret-scan allowlist entry and 93 dependency advisories, 8 of them critical, were waiting behind the same wall

02

The approach

Stop renting a runner to find out whether your code is broken

The fix is not to buy more minutes. It is to notice that most of what the pipeline does every single commit does not need a shared machine at all: the lint, the type check, the unit tests, and the workflow files themselves. Those can run on the laptop in front of the developer, in about a hundred seconds, for nothing.

CI Kit gives a repository a local gate that mirrors what the GitHub Actions workflow does: the same installs, the same linters, the same type checks, the same tests. The design principle is a single copy of the setup. The gate scripts live in the repository under scripts/ci/, and both the Makefile and the workflow call them. One copy of the setup, so it cannot disagree with itself.

That matters more than it sounds. The usual reason a local gate and a CI run disagree is that they are two implementations of the same intent that drift. Here they are the same files, so the only thing that can differ is the hardware.

The gates are driven by what the repository actually contains, so the tool works with any language rather than being one language's plugin. It also runs the workflow files themselves locally through act, so the YAML is checked before the push rather than after it.

For the eleven days until the reset, this was the difference between shipping and not shipping. The checks that no longer need a runner no longer need minutes, and the account's allowance stopped being the constraint on whether the work could move.

03

The architecture

Four parts

   developer                repository                  runner
      |                        |                           |
      |   make ci              |                           |
      |----------------------->|                           |
      |                        |  scripts/ci/*.sh          |
      |                        |  the setup and the gates  |
      |                        |         |                 |
      |                        |         |                 |
      |                        |         +---------------->|
      |                        |            the same script|
      |                        |                           |
      |   the same script, so the local run and the runner
      |   cannot disagree about what the code needs        |
  1. The shared setup

    scripts/ci/*.sh holds every install and every gate. The Makefile calls them. The workflow calls them. There is no second implementation anywhere, so the local verdict and the CI verdict agree by construction rather than by luck.

  2. The gate targets

    make ci runs the full gate, make ci-fast runs lint and type checks only. The targets are generated from the repository's contents, so a TypeScript repository gets vue-tsc and a Go repository gets go vet, without configuration.

  3. make doctor

    A separate check for whether the local toolchain is the one CI pins. This is the diagnostic that tells you the two environments differ, before a failing test makes you guess.

  4. make act

    Runs the GitHub Actions workflow files locally through act, so the YAML itself is validated on the laptop. A malformed workflow is caught before it is pushed, not after it is queued.

04

The results

The honest measurement, including the part that is not a win

Before

  • 4–12 min, billedFeedback on a push that breaks the build
  • 6 s, if you read the logFeedback when the runner never starts
  • NeverFeedback when the runner never starts and you are not reading the log
  • NoneCommits verified during a billing block

After

  • ~100 s, freeFeedback on a push that breaks the build
  • ~100 s, and it says whyFeedback when the runner never starts
  • ~100 sFeedback when the runner never starts and you are not reading the log
  • All of themCommits verified during a billing block
  • The local gate is not faster than a CI runner. On identical work — npm ci, two type checks, vitest, the build — it measured 99 s on a 2022 Intel laptop against 43–48 s on a GitHub runner. A modern runner has more cores and faster storage, and for work that is mostly npm ci and TypeScript, it wins. The local gate on that machine is roughly twice the wall clock.
  • The first attempt at that measurement was discarded: the machine had a load average of 212 from an unrelated build, and the same gate took 194 s under that load against 105 s without it. Both numbers are published because the first one is what a measurement looks like when nobody checks the load.
  • No runner-minute savings are claimed. Where CI does run, a local gate reduces the number of red builds, not the number of green ones. Green builds still belong on the runner. The value is a guarantee of feedback, not a speedup.
  • The tool is not a replacement for CI. It does not test the runner, the matrix, the scheduled jobs, or the deployment. It is a faster answer to 'did I break something?'
  • The numbers come from one machine and one workspace. They are a starting point for your own measurements, not a benchmark.

05

What it found

What the gate found that the CI could not

Found by running the local gate against repositories whose CI was in the state described above. All three were found in under four minutes on a laptop, with no runner minutes spent.

A stale secret-scan allowlist

.gitleaksignore named LOCALDEVELOPMENTGUIDE.md, but the file had moved to docs/product/LOCALDEVELOPMENTGUIDE.md. The fingerprint never matched, so a tracked documentation file — a curl sample containing the literal string YOUR_TOKEN — was failing the secret scan. The file is tracked, so the CI scan was failing on it too. The CI had been red for the same reason the jobs were not starting, and nobody could tell the two apart.

Ninety-three dependency advisories

npm audit --audit-level=high is in the workflow. The local gate ran it and it failed: 93 advisories, 53 high, 8 critical. The same command CI runs.

Build artefacts tripping a working-tree scan

Playwright traces under test-results/ contain a token, and a working-tree secret scan reads them. They are build output and belong in the allowlist.

06

Lessons

What I would tell my past self

A red build is not always a red build

The instinct on seeing red is to assume your change broke it. The cheapest diagnostic is the timing: six seconds is not a test failure. Six seconds is a job that never started. Reading the annotation instead of the code would have cost ten seconds and saved a week.

Publish the measurement you threw away

The 194-second run under a load average of 212 is in the evidence document. Discarding it silently would have been the normal choice. Publishing it is the reason the 105-second number can be believed at all — a reader knows the spread, and knows the author looked for it.

Green builds still belong on the runner

The honest framing of a local gate is a guarantee, not a replacement. Claiming the local gate is faster would have been false on this hardware, and a single person checking it would have made everything else in the document untrustworthy.

One copy of the setup is the whole trick

Every drift problem I have seen in CI came from two implementations of the same intent. The fix was not a cleverer reconciliation layer. It was deleting one of the two copies and having the Makefile and the workflow call the same files.

Keep reading

Four containers, 250 MB, and a proxy that never settled.

Documentation in the repository

Indexed rather than copied, so the repository stays the single source of truth.