Book Appointment

IncidentOps Compared: Three Ways to Run a Bad Day at Hospital IT

Healthcare Careers

Let’s say you run the systems behind a busy outpatient department, and the online booking portal stops accepting requests on a Monday at nine. What happens next depends far less on how clever your engineers are than on which of three styles of response your organisation has quietly adopted. That is the core idea behind IncidentOps, and in this piece we line those styles up side by side to help you spot the one your own department really lives in. The discussion is about administration and information systems only; nothing here touches clinical practice.

The three contenders are the improvised scramble, the written-down playbook, and the coordinated team with practised roles. Think of them as three kitchens during a dinner rush: one where everybody shouts, one where there is a laminated card on the wall, and one where somebody is actually calling the orders. Each deserves a fair hearing, including the situations where it genuinely pays its way.

Style one: the improvised scramble

This is how most small teams begin, and the Ops Atlas page on IncidentOps gives the stage a deliberately unflattering label, heroic. The Atlas places it at the bottom of its proposed five-step maturity ladder. Outcomes hinge on who is physically nearby, usually the same two or three exhausted people who know where everything is.

The upsides are real. Nothing needs maintaining, there is no paperwork, and a gifted problem-solver can restore service quickly on a good day. The downsides show up on the bad ones. Several people start poking at the same system without telling each other. Front-desk staff get contradictory answers. Nobody records what was tried, so the next occurrence starts from zero. And because the hero is also the commander, the person who should be watching the whole picture is buried in a console.

Style two: the written playbook

The next step up, which the Atlas calls defined, adds paper: a severity scale, an on-call rota and a short template. This is a big improvement for very little cost. Staff know whom to call, and the argument about whether something is serious enough gets settled by a table rather than by whoever shouts loudest.

Still, a playbook alone has a ceiling. Documents do not command anything. When the portal fails, the card says “notify stakeholders,” but it does not say who writes the message, how often, or in what words. Playbook-only teams tend to fail on communication and handover, the two areas where roles matter most. You also get a subtle problem with the rota: one worked example in the Atlas has six engineers covering a two-slot weekly round-the-clock schedule end up on call roughly a third of the time, beyond the 25 percent ceiling Google recommends. A rota can exist on paper and still wear people out.

Style three: the coordinated team

At the third rung, the team keeps a steady rhythm of updates and has real people wearing real hats. A commander decides and coordinates without touching the controls. A separate group is the only one allowed to change production. A communications lead posts at a regular interval. A scribe keeps the timeline, and a deputy waits in reserve in case the commander needs relief.

According to the Atlas proposal, reaching this stage means the update rhythm is actually kept in nine of every ten serious incidents. The cost is rehearsal time and a willingness to rotate the commander role instead of handing it to the most senior person. The payoff is predictability. Everyone outside the response, including front-desk staff and department heads, knows when the next update arrives, which removes the constant interruptions that slow responders down.

A quick side-by-side

  • Speed to first action: the scramble is fast when the right hero is present and slow otherwise; the playbook is steady; the coordinated team is consistently quick because the first minutes follow a known pattern.
  • Communication: the scramble improvises, the playbook says “notify,” the coordinated team names a person and a schedule.
  • Burnout risk: highest in the scramble, hidden in the playbook if the rota is thin, lowest when roles rotate.
  • Learning afterwards: the scramble forgets, the playbook records, the coordinated team turns a review into a few owned actions.
  • Cost to maintain: nothing, a little, and a recurring rehearsal.

Choosing a size that fits you

The Atlas offers a tier framework that helps you avoid over-building. At the lightest tier, you simply open tickets. The next tier adds one rota and a template. Roles such as a commander, a communications lead and a hosted status page become worthwhile once an issue regularly crosses team boundaries or you see two or more serious events a month. Deputies, liaisons and sub-teams only earn their keep when more than five responders turn up often. The guideline behind that last number is borrowed from emergency services: one supervisor per five people, and split into sub-groups beyond that.

Here is a practical way to read that framework. Count how many people normally join when something breaks, and how often the trouble spills into another team’s territory. If the answer is usually two people and almost never, a ticket queue and a rota are plenty. If the answer is a handful of people from three departments, with the front desk calling every ten minutes for news, you have outgrown the light tiers and a named commander will save you real time. Tiers are not a ranking of seriousness; they are simply a match between the amount of structure and the amount of coordination your incidents demand.

For a typical hospital administrative department, that probably means the middle tiers. You do not need a command post, but you do need to know who is in charge on Monday at nine.

Where the comparison gets interesting: mitigation

One more difference separates the styles, and it concerns the order of work. Scramble teams tend to hunt for the cause first, because understanding feels like progress. Coordinated teams apply the Atlas principle of mitigating before explaining. Undoing the latest release, steering users away from one misbehaving server, disabling a fresh option, or moving to a standby copy: every one of those moves is possible once the problem area is pinned down, and none has to wait for a full explanation. In the Atlas’s account of a large cloud-cluster failure, a rollback of images once the faulty area was identified would have ended the trouble around ten in the morning rather than about noon.

A note on messaging and independence

The Atlas names two architecture rules that apply to every style. First, the channel you use to page people and the page that shows service status must not depend on the systems they describe. Its cautionary tale dates from early 2017, when a big cloud provider’s service-health console sat unusable for hours because the console itself depended upon the failed storage layer. If your patient-facing status note is hosted on the same server as the portal, it will vanish just when you need it.

Second, keep a single written account of the event and recycle it, instead of retelling the saga separately in chat, in a ticket and in an email. Both rules are cheap to follow and expensive to ignore.

What our own readers might do next

If you are curious about how an organisation decides when specialist support is needed, our earlier piece on recognising the point at which back trouble calls for a dedicated spine centre follows a similar logic from the patient side: define the trigger first, then act. A companion read, exploring non-surgical routes before anyone mentions an operation, makes the same case for trying proportionate steps first, which is also the spirit of the lighter IncidentOps tiers.

Verdict

Before you decide, ask three plain questions of your last bad day. Did anyone outside the response know when they would next hear something? Could a newcomer have taken over halfway without a long briefing? And did the review end with a few jobs that someone actually owns? Three yeses suggest you are already close to the third style, whatever the documents say. Three noes are not a verdict on your people; they are a map of where a little structure would help most.

None of the three styles is shameful, and the scramble is a perfectly sensible starting point for a team of three. The mistake is staying there once incidents become routine. If you recognise your Monday morning in the first description, borrow one habit from the second and one from the third: write a severity table this week, and name a commander and a communications lead for the next outage, even if it is only a rehearsal. Small steps up the ladder beat a grand programme that never starts.