Case study 03
The controller was the problem, not the automation.
An air-gapped Windows Service that runs enterprise workflows from JSON blueprints — and records every attempt, including the ones it had to retry
In plain terms
A platform that can only be changed by editing the program that runs it will eventually be edited in a hurry, by somebody who did not design it. This moves the business rules out of the program and into ordinary data files, so a manager can read a workflow, a reviewer can check it, and a new system plugs in without touching the engine.
- Before: a workflow was a method, and changing it needed a code change and a deployment
- After: a workflow is a file, and changing it needs a file review
- Adding a system means dropping in a plugin file — the engine is never edited
- Every attempt, including the retries, is written down
A platform that can only be extended by editing the thing that runs it will eventually be edited in a hurry, by someone who is not the person who designed it.
01
The problem
Three couplings, each of which is a separate way to lose control
Enterprise automation usually arrives as a controller application. A workflow is a method. Changing a workflow means changing code, which means a build, which means a deploy, which means the change is gated on a pipeline rather than on the person who understands the business rule.
That is the first coupling: logic and orchestration are tangled together. A step that shells out to Artifactory and a step that calls Jira live in the same method as the code that decides what runs and in what order, and there is no way to review the first without reading the second.
The second coupling is to the network. A controller that calls a cloud AI endpoint to classify an ambiguous request has a hard dependency on somebody else's uptime, and in an air-gapped environment it has none at all. When the endpoint is unreachable, the sensible thing — dropping the transaction, or guessing — are both bad, and the system has to know which one it did.
The third coupling is to the engine. Every new integration — Kubernetes, CyberArk, another Artifactory, a tracking system — arrives as an edit to a core module. Core modules accumulate edits. Eventually the core is the thing nobody dares touch, and every integration is a fork of it.
None of these are exotic failures. They are the ordinary, predictable consequence of the shape of the code, and they are all fixable by changing the shape rather than by adding more care.
A workflow that can only be changed by editing the thing that runs it will eventually be edited in a hurry, by someone who is not the person who designed it.
What it looked like day to day
- Adding a step to a pipeline required a code change, a review and a deployment window
- The engine module was edited for every new integration, and had stopped being safe to touch
- Retry state lived in memory, so an interrupted run left no recoverable record of what had happened
- An unreachable dependency left the platform with no defined behaviour — it either blocked the pipeline or guessed
- The audit trail was reconstructed from log text after the fact rather than written as it happened
02
The approach
Three separations, and a defined answer for every failure
Workflows are data. A pipeline is a JSON blueprint: a list of steps, each with an identifier, a task, an optional retry count and an input map whose values may interpolate the outputs of earlier steps. It can be diffed, reviewed, approved and replayed without a build. A business rule changes by editing a file, which is the right tool for a business rule.
The engine core is immutable. It lives in its own module scope with a composition root that dot-sources the internal components — the database engine, the rules engine, the orchestrator and the router. The engine does not know what any particular workflow does. It knows how to walk a blueprint, resolve an input, run a task, and record the outcome.
Extensions are discovered, not registered. A new integration is a file named *.plugin.ps1 dropped into the plugins directory. It is pure, it takes strict parameters, it does not mutate globals, and it returns dictionaries or hash tables. The engine finds it. The core is never edited.
And for the failure case there is a defined answer rather than a hopeful one. Ambiguous intent is classified by a local model on the host, and if that model is unreachable the classifier falls back to a deterministic regular-expression matrix before dropping the transaction. Either way, the choice made is written to the audit trail — so the system can say not just that it degraded, but exactly which path it took.
03
The architecture
The layout is the argument
secure-api/
├── config.json settings, feature toggles, RBAC tokens
├── server.ps1 async runspace-pool socket loop
├── register-service.ps1 service deployment
├── backup-database.ps1 transaction-aware online backup
├── workflows/ pure JSON blueprints (data, not code)
├── tests/ Pester harness — every toggle, offline
├── plugins/ auto-discovered *.plugin.ps1
├── Modules/PipelineEngine/ the immutable engine core
│ └── src/ database · rules · orchestrator · router
└── public/ the control console, served locally
Immutable engine core
Modules/PipelineEngine is a separate module scope with a manifest, a composition root and four internal components: the transactional SQLite database engine, the rules engine that validates and shields against command injection, the orchestrator that resolves variable interpolation and drives the pipeline loop, and the router that performs zero-trust header checks and matches routes by regular expression.
Declarative workflow blueprints
The workflows directory holds pure JSON — no code, no PowerShell, no side effects at load time. A blueprint declares steps, tasks, inputs and an optional retry count. Because it is data, the same blueprint can be validated by the Pester harness offline, which is what the test suite does with every toggle line.
Plug-in auto-discovery
The plugins directory is the extension surface. Storage, DevOps and tracking plugins ship as examples, and a new one needs nothing but a file with the right suffix. A plugin that breaks the rules — mutating globals, or returning the wrong shape — is rejected by the contract rather than by convention.
Thread-safe shared caches
A synchronised hashtable is injected across the runspace pool. The plugin cache compiles script structures on first use and then serves them without touching the filesystem again inside the hot loop. A second, separate cache holds just-in-time secrets generated through the host's DPAPI, with a TTL so a rotated credential cannot outlive its rotation.
Transactional persistence and safe backup
The database engine writes every step attempt — Retrying, Success or Failed — to SQLite inside a transaction. The backup script performs an online snapshot through SQLite's own backup mechanism, so a copy can be taken without dropping active workers or refusing an inbound webhook.
Local-first interface layer
The router serves a control console straight out of a public directory on the same socket, with the Vue runtime cached locally rather than pulled from a CDN. There is no build step, no bundler and no node_modules in the deployment — upgrading the console is editing layout tags in a file.
04
The results
What changed, structurally
Before
- In controller codeWhere a workflow lives
- A build and a deploymentTo change a workflow
- An edit to the engine coreTo add an integration
- In memoryWhere retry state lives
- Requires a cloud endpointIntent classification
After
- A JSON blueprintWhere a workflow lives
- A reviewed file changeTo change a workflow
- A plugin fileTo add an integration
- Transactional SQLiteWhere retry state lives
- A local model, with a deterministic fallbackIntent classification
- No wall-clock benchmark is claimed for this platform, and none should be inferred from one. The claims here are architectural and are checkable by reading the layout: the engine core is a separate module scope, the workflows are data, and the plugins are discovered rather than registered.
- The retry policy is explicit. A step declares an optional retryCount, and the delay escalates as 2^(attempt - 1) — two seconds, then four, then eight. Each of those attempts writes its own row, so the audit trail shows the transient failure, the recovery delay and the eventual outcome as three separate facts rather than one summary.
- Every cross-cutting concern is a toggle on disk rather than a branch in code: MaintenanceMode returns 503 for executions while leaving the audit read endpoints live, EnableRBAC degrades to a single master token, EnableApprovalGates permits a hotfix to bypass an approval checkpoint, and EnableAIFallback drops an unstructured task rather than guessing. Flipping any of them does not require a restart.
- The platform has no external dependencies and no build step. It is registered as a service and it runs — which is the entire point in an environment where the public internet is not available.
05
The public record
The air gap is only as strong as its weakest supply chain
A platform that refuses network traffic is only meaningfully air-gapped if the artefacts it runs were verified before they arrived. That is the part of this design most likely to be got wrong, and it is worth being explicit about it.
A downloader that greps for a keyword is not a verifier
The asset hydrator fetches the Vue runtime on a connected machine and writes it into the project for transfer to the air-gapped host. If the integrity check is a regular expression over the first line of the downloaded file, then it verifies nothing: it proves a string appears somewhere, not that the bytes are the expected bytes. The check has to be a hash of the whole file against a pinned expected digest, and the digest has to be pinned from a source that is not the same host that served the file.
Verify the archive, not just the payload
The right order is: fetch the published artefact, verify the signature or digest of the archive, extract, then verify the extracted file's digest against the version manifest. Checking only the extracted file means the extraction step is trusted blindly.
Fail with the real reason
When verification fails, the error must say that the digest did not match the pinned value. A generic 'possible network interception' sends the operator looking at the network, when the network was never the thing at fault.
06
Lessons
What I would tell my past self
Configuration is data, not code
Every hour spent moving a business rule out of a controller and into a reviewable file is an hour of the reviewer's attention that used to go to reading code. It is the same lesson as the mock engine's zero-dependency rule, at a different scale: the cheapest thing to review is data, because there is nothing in it that can surprise you.
An extension point that requires editing the core is not an extension point
The plugin directory only works because the engine never learns what a plugin does. The moment a plugin needs something from the core that the core does not already expose, the boundary leaks — and it leaks exactly once, in a hurry, with a deadline.
Define the degraded path before you need it
The local model with a regular-expression fallback was designed as a pair, not as a fallback added later. When the primary path is unavailable, the system already knows what it will do and the audit trail already knows how to record it.
Audit the attempts, not the outcome
Recording only the final state of a retried step erases the information an auditor most wants: that the system hit a transient fault and recovered on its own. Writing one row per attempt is barely more work and turns 'it worked eventually' into a legible sequence.
Keep reading