Book Appointment

A Spine Department’s Monday Table: A Made-Up DataOps Story

Healthcare Careers

Consider an invented tale, with no real hospital behind it, about a small operations analytics group that supports a spine surgery department’s scheduling office. It is meant to show how DataOps habits arrive in a team one irritation at a time, which is how they usually do arrive. Nothing here concerns clinical decisions; the group only handles the figures that planners use to staff theatres and book follow-up slots.

The Monday nobody trusted

The group’s morning table listed next-day theatre sessions, the clerical staff assigned to each, and a count of pending pre-admission paperwork. It was rebuilt overnight from three source systems. On one Monday the table showed a suspiciously light day, a planner rang the analyst to ask whether sessions had been cancelled, and the analyst discovered that one of the three feeds had delivered only its Friday slice. The overnight job had finished without a complaint.

Nobody was hurt by it. The planner simply double-checked by phone, which is exactly the problem: a report that people verify by phone has stopped saving anyone time. The analyst, who had been reading about DataOps, opened the Ops Atlas entry and recognised the symptom. The Atlas describes the discipline as the practice of treating each data load as something to inspect, not just something to run, and it separates changes to code from changes to data that arrive every night without any release.

Week one: write down what “on time” means

The first conversation was not technical at all. The team asked the planners when they actually opened the table. The answer was seven in the morning on working days. The Atlas offers a promise of just this kind as its freshness illustration: ready before seven, nineteen working days out of twenty. Spread over roughly three months of working days, the remaining twentieth is a small stock of permitted misses, a handful of mornings. With a number on the wall, a late table stopped being a quarrel and became a spent allowance.

The group also learned that an objective only helps if something independent measures it. Reading the job’s own “done” message would never reveal a table that finished on time but empty. They added a small separate check that looked at row counts per source and ran at 06:30, early enough to leave a buffer.

Week three: stop publishing into the live table

The analyst’s pipeline used dbt for its transformations, and a build step there made the team feel safe, since a failing test halts everything downstream. A caveat in the Atlas caught the team off guard. A failing test does skip the models further along, yet the one that failed has already been rebuilt where it lives, so its flawed rows sit in the destination schema. Anyone querying that table directly is reading the bad version.

The remedy the Atlas suggests is to build into an audit schema, run the tests there, and only then promote the result to the schema that planners read. It is the write-audit-publish pattern in a small, practical form. The team adopted it for the one morning table and left their experimental sheets alone, since the same entry warns that the approach adds delay and extra writes and belongs on the few outputs that truly matter.

Week five: replacing, not appending

A second incident came from a rerun. After the Friday-only delivery was fixed, somebody reran the job, and the table doubled up the Friday rows because the load appended instead of replacing. The fix was dull and effective: each load now overwrites the full partition for the day it covers, which makes a repeat run harmless. The Atlas, quoting Airflow’s guidance, prefers upserts over blind inserts for the same reason.

For the arrival problem itself, the team borrowed the lookback idea. Each night the job re-reads the previous three days and reconciles partition counts with the source, so a slice that shows up late is picked up automatically on the following run. The Atlas mentions a dbt incremental example that does just this with a three-day window.

A word on the incident log, which mattered more than any tool. Every entry recorded four facts: what was wrong, who noticed it, how long the fault ran before anyone noticed, and how long it took to repair. Within a few weeks the pattern was obvious. The repairs were quick; the noticing was slow. That one observation redirected the whole effort from heroic fixes toward earlier detection.

Week eight: a plain tool for a plain habit

None of this needed a purchase. The team already ran their pipeline code through a shared repository, and a colleague elsewhere in the same network had written up the basics of how a GitHub workflow runs checks on every proposed change, which became the template for their pull-request tests. For visibility they reused the dashboards already running for the hospital’s IT estate, in the manner described in a short piece on using Grafana for monitoring, adding two panels: arrival time against the 07:00 target, and rows received per source.

One warning from the Atlas shaped how they set alerts. Flat limits drawn around an average only make sense when the data has one steady pattern. Theatre sessions rise and fall by weekday, so the group built separate baselines for working days and weekends, instead of one band that was too loose to catch a partial load.

How the quarter ended

The ending is modest. The table was late on two mornings, within the allowance of three. Planners stopped ringing to check. Of the four incidents logged in the quarter, the group found three themselves; before the changes they would have heard about most of them from a planner. The Atlas puts this same ratio forward, the share of incidents discovered by consumers, as the headline measure, and the team simply noticed it going down.

They also wrote their first small contract with the clerical team that sends one of the three feeds: the columns expected, what each status value means, the arrival time, and a named contact. It took an afternoon, and it ended an argument about what “cancelled” meant that had run for months.

Reading the entry for yourself

If you would like the source for the ideas above, the Ops Atlas page on DataOps lays them out in order, with an honest remark that the vocabulary remains unsettled and no formal standard exists. Take that as permission to start small. One table, one promise about timing, one independent check, and a log of who noticed each fault: that is a complete first iteration, and it is the shape of the story above.