Diagnosing Repeat IT Failures: An ITOps Case for Evidence First

Diagnosing Repeat IT Failures: An ITOps Case for Evidence First

A digital health service that goes down twice a month is not unlucky; it has a cause nobody has isolated yet. Operations teams approach that situation like any careful investigator: define the symptom, gather evidence, test explanations, and fix the one that holds up. This article applies that logic to ITOps, the function responsible for keeping IT services running, and it stays strictly in the territory of administration and data, not clinical care.

The symptom: the same trouble returns

Consider a generic scenario. A patient-portal login slows to a crawl on the first working day of each week. Tickets pile up, a technician restarts a component, and by Wednesday the complaints fade. Everyone concludes the issue is resolved. It is not. According to the Ops Atlas reference on ITOps, this is the point where problem management takes over from incident handling: the aim is to remove the cause of repeat failures rather than reset the symptom each time.

Step one: confirm it really is a pattern

Evidence starts with records. If tickets are logged inconsistently, any pattern you find may be an artefact of how people write. Before drawing conclusions, check three things: do the tickets share a time pattern, do they mention the same service, and were they opened through the same channel? The Atlas warns about shadow spreadsheets, private lists kept outside the official tool. They hide recurrences, because the official count looks lower than reality.

As a comparison from another field, we have written about how clinicians group patients by functional class to keep communication consistent; our overview of a classification system used in cardiology conversations is purely a reference piece. The operational equivalent is the priority scheme on a service desk: shared labels make patterns visible.

Step two: look at what changed

The most common cause of a new fault is a recent change. So ask what changed the previous Friday. The Atlas lists practical mechanics for a change process: a written risk score with weights, a single forward schedule that flags conflicts, and a rule that suspends a change model after too many failures. One example in the entry limits a pre-approved model to no more than one failure in its last twenty uses, and suspends it otherwise.

Two details from the same entry matter for diagnosis. First, change records created by a deployment pipeline should be created, not approved, by the pipeline. Second, every failed or emergency change deserves a review. If your weekly slowdown appears right after a scheduled maintenance job, the schedule and the review notes are where the trail starts.

Step three: check the map is honest

A diagnosis fails when the inventory is wrong. Say the slow component is believed to run on one server, but a duplicate record exists, or the server was retired and an orphaned entry remains. The Atlas walks through a toy audit of ten register entries. Nine are fully filled in, yet two have sat untouched for over two months, one belongs to nothing, and one reuses a serial number seen elsewhere. The sample is tiny on purpose: even a tidy-looking register hides quiet errors that mislead investigators.

The same source also warns against swinging the other way. A register trying to hold everything is hard to keep accurate. Track what decisions need, and measure its health the way you would any other data set.

Step four: do not trust a score you cannot interrogate

Teams sometimes buy a prediction tool and hope it will tell them which change caused the trouble. The Atlas includes a sober example. A bank’s preprint, still awaiting peer review, tested a model against its own risk questionnaire. Among changes that later triggered a major incident, the model caught 63 percent and the questionnaire about a fifth. Precision came out at 0.55, measured over 203 changes where 35 had caused such an incident.

Then comes the part analysts should memorise. The Atlas does the base-rate sum: suppose failures are one in a hundred and the tool screens ten thousand changes. Only about six flags in every hundred would point at a genuine failure. Rare events make flags noisy. The practical response is to use such scores to prioritise review time, not to approve or reject changes on their own.

Step five: choose the smallest fix that removes the cause

Once the evidence points somewhere, resist the urge to add process everywhere. Piling on new steps after every failure is a classic pitfall, says the Atlas. Delivery research it cites found nothing showing that sign-off from an outside body reduces failed changes, and it favours peer review. Regulation still matters, though: the EU digital resilience law expects approvers to be independent of the people making a change, and expects urgent changes to be signed off once they are live, with lighter duties for firms under the simplified framework.

So a fix might be one of these: move the heavy job away from the Monday peak, add a pre-change conflict check, or retire the duplicate record that kept sending engineers to the wrong host.

Step six: prove the fix worked

  1. Record the baseline: how many tickets the pattern produced over the previous eight weeks.
  2. Apply the fix through a normal change, with its own record.
  3. Watch the same measure for at least four cycles.
  4. Close the problem record only when the pattern stays gone.

If you want more context on how the discipline sits in a wider digital programme, our write-up of engineering discipline across a digital ecosystem may help.

The broader lesson is that recurring faults are a data problem before they are a technical one. Count them honestly, trace the changes, trust the map only after checking it, and treat any prediction as a pointer rather than a verdict. Do that, and the Monday slowdown becomes a closed record rather than a ritual.