The stopwatch in Meera’s hand read eleven seconds. That was the interval between her colleague typing the halt command and the runaway data-export job actually stopping, during the team’s first AutomationOps stop-lever drill at a mid-sized heart hospital group this spring. Eleven seconds felt excellent until the team found an older job — one running outside the runner, reachable only by editing a file on the host itself — whose real stop time was closer to four minutes, most of it spent finding the file. Evidence, not opinion: that is the difference between believing you govern your automation and actually doing it. This is the diary of that quarter, told with the numbers the team collected and read through AutomationOps, the governance discipline the Atlas curates. The entry frames it plainly: AutomationOps — documented by The Ops Atlas as a production discipline. Cardiology turns out to be a natural lens for it. Both fields grade severity, both trust measurement over impression, and both have learned that a clean-looking chart can hide a sick patient.
Week One of the AutomationOps Diary: Counting What Runs
The quarter opened with an inventory, because nobody can govern what they cannot list. Meera and her colleague Arvind walked the departments, grepped crontabs, and asked the unglamorous question — what fires without a human present? The count landed at fifty-eight scheduled or triggered actions: restarts, log rotations, database refreshes, cleanup sweeps, certificate renewals, one mystery script uploading billing extracts to a server nobody could identify. Twenty-two had no recorded owner. Nine ran under a single shared account whose password four people knew. The inventory itself became the first artefact of governance: a register where each job got a line — what it does, what it may touch, whose identity it borrows, and when somebody will next look at it. Every line answered the same four questions:
- What the job does, phrased so a non-engineer could read it aloud.
- Which machines and data it may touch, with nothing implied beyond.
- Whose identity it runs under — shared accounts retired job by job.
- When its next review falls due, as a calendar date rather than a promise.
Written as code and reviewed like code, the register turned a folktale estate into a countable one. Fifty-eight sounds small. It is precisely the size at which most organisations stop counting and start assuming.
A Grading Scale Borrowed From Cardiology
Clinicians will recognise the move the team made next. Cardiology grades heart failure by what the heart can manage — from no limitation through to the class where symptoms arrive at rest, as the NYHA Class IV definition lays out — and treatment intensity follows the grade. The team graded every job the same way, by consequence of failure. Read-only checks became grade zero: safe to run anywhere, no ceremony. Reversible actions against one machine were grade one, needing an owner and a log line. Reversible actions across many machines were grade two, and they earned the full restraint package: a preview run first, small batches of targets, health checks between waves. Grade three was reserved for actions that erase data or can never be walked back, and the rule there was absolute — a second person approves every run, the destructive path prefers soft deletion where the storage layer permits, and rollout happens in stages even when everyone is impatient. The Atlas maps that top grade onto formal two-person authorisation controls from recognised security catalogues, which made the conversation with the audit committee refreshingly short.
The Empty List That Educated an Industry
Why all the ceremony? Because the failure mode is documented, named and instructive. Google engineers once published the anatomy of an incident in which a maintenance tool designed to wipe specific storage volumes asked an inventory service for its target list, received nothing back, and treated the nothing as a licence to proceed. The tool erased data belonging to customers across the cloud platform. Empty never means all. After that post-mortem circulated, target validation hardened across the industry into a simple refusal: a job handed an empty target set stops and raises a hand. Meera’s team implemented it in an afternoon and watched it fire for the first time three weeks later, when a misconfigured filter silently produced zero machines and the job aborted instead of “succeeding” catastrophically. The prevention cost one if-statement. The incident it mirrors cost a trust dividend nobody has fully paid back.
Guards That Live in the Pipeline
Governance that depends on memory decays; governance that lives in the delivery pipeline does not. Each register entry in Meera’s setup carries its tier, limits and identity as structured configuration, and approval routing reads that configuration — a grade-three job cannot even reach production without the recorded second approver. Guard tests run alongside unit tests in CI: try to register a job without an owner and the pipeline rejects it; try to widen a target scope beyond the entry’s limits and it rejects that too. Repeatability got the same treatment. Jobs are written to declare outcomes rather than perform gestures, so a rerun on an already-correct system is a no-op rather than a second dose. Where a gesture is unavoidable — a payment, an outbound message — the caller carries a unique reference the system records on arrival, making duplicates refusable. The Atlas notes that a major engineering organisation built exactly this property into its repair tooling, which is why those repairs could safely run on a fifteen-minute schedule around the clock. Arvind’s summary became the team’s favourite slide: a job you are afraid to rerun is a job you do not really control.
The Quarter’s AutomationOps Evidence
Numbers, as promised. After registration and controls, the flagship cleanup job — forty executions a month, each sparing a technician roughly twenty minutes of manual work, after subtracting maintenance time — returned on the order of 7.83 net hours monthly. Build effort broke even in 5.1 months. Completed runs sat at ninety-five percent, with the remaining five percent halting safely into a human queue, which the team counts as the system working rather than failing. Ownership coverage, their favourite number, climbed from thirty-eight percent in week one to ninety-one percent by the quarter’s close. No other single figure moved the audit conversation the way that one did. Two other measurements earned their keep. Guard-trip rate showed the brakes engaging twice in the quarter, both times correctly. And stop time — drill-measured — fell from that first four-minute outlier to under fifteen seconds estate-wide, once every job was folded into the runner with a reachable halt. The team also studied the incident report from 2025 in which an automated change emptied a DNS record inside a major cloud provider’s database service and, crucially, nothing caught the emptiness — dependency after dependency failed for hours because a name pointed nowhere. The takeaway went straight into a guard test: certain systems must never be able to write a blank record without an alarm. Automation that watches for absence, not just presence, is the difference between those two worlds.
When the Chart Looks Clean but the Patient Is Unwell
One more cardiology parallel, and it is the subtle one. Cardiac syndrome X describes genuine chest pain with arteries that angiography shows as clear — the standard test says fine while the person still hurts. Automation has its own version: dashboards green, success rates high, and yet clinicians quietly keeping manual workarounds alive because they trust the workaround more than the job. The team found two of those in week nine and treated them as defects in trust, not in staff — the jobs were rebuilt with visible previews and honest logs until people actually used them. Measurement, like murmur grading scales, exists to make the invisible comparable; a number nobody believes is decoration. And some complaints stay unexplained even after every test, the way syndrome X remains a puzzle — which is the honest argument for keeping humans in the loop with real authority, including the standing right to halt any automated action without asking permission first. Eleven seconds, by the end of the quarter. The stopwatch was the whole story.
