Priya joined the IT desk of a spine hospital group on a Monday and inherited the delivery setup on the Tuesday. Two appointment-booking portals, a patient-education site, one self-managed GitLab box in a rack room, and a binder — an actual paper binder — titled “How We Release”, last updated the previous winter. The binder’s key instruction, in bold: deploy from Anand’s laptop on Friday evening, because that is when traffic dips. Nobody asked what happened when Anand was on leave, because so far he had not been. GitLabOps was not a word anyone at the desk used yet, but the month that followed was, in hindsight, a compressed course in it — and in the specific order its repairs should be done.
What the Desk Actually Inherited
The first week was spent writing down what existed, which is less boring than it sounds and considerably more useful. The inventory, in no particular order:
- Three projects carried almost identical pipeline files, each with small local “improvements” that quietly contradicted the others, so identical pushes behaved differently depending on which folder they landed in.
- One runner, registered months earlier and never revisited, served every kind of job — public merge requests, internal builds, production deploys — with nothing separating the trustworthy from the merely submitted.
- A cloud access key with broad permissions sat in a CI variable, copied from a setup guide at some unrecorded date, older than the binder itself.
- Production deployments had no approval step and a fondness for colliding with each other whenever two people pushed the same afternoon.
- The GitLab install itself had not been touched in the better part of a year, and nobody could say aloud what version it was on without checking.
None of this was exotic. That is the point worth underlining for any small team: broken delivery setups are almost never broken in novel ways. They fail in the same handful of places, which means a short, practical list of repairs covers most of the damage.
The GitLabOps Repairs, in the Order That Paid
The desk worked through three repairs in sequence, ordered by how much day-to-day pain each removed.
Repair one: stop letting one machine carry everything
The single runner was split into two groups with a simple rule behind the split: work from untrusted forks runs on machines that are created fresh for the job and destroyed afterwards, configured to serve exactly one assignment before being discarded. Anything touching production requires a runner that only protected branches can reach. Shell-based executors were barred from anything untrusted — a job on a shared shell can read everything the previous job left behind, which is a gift to nobody. The rule of thumb that stuck on a sticky note: trust level first, speed second.
Repair two: stop carrying long-lived keys at all
The ancient cloud key was deleted rather than rotated, which felt alarming for about a day. In its place, deploy jobs receive short-lived identity tokens that the cloud provider verifies directly, so nothing durable sits in a variable waiting to leak. The job token that GitLab hands to each pipeline was restricted to an explicit allowlist of the two registries the desk actually uses. The security review that had been postponed twice was rescheduled and passed in forty minutes, mostly because there was suddenly very little to review.
Repair three: give production a gate and a lane
Production became a protected environment: the deploy job refuses to run unless it arrives from a protected branch, and one named person signs off before it proceeds. Each release now waits its turn in a single-file queue, so same-afternoon pushes line up politely instead of tangling mid-flight — which, given the desk’s history, was the single most satisfying change of the month. Outdated deploy jobs are refused instead of silently shipping something stale on top of a newer release. The approval takes the signer under a minute, which is roughly what it used to take to discover, mid-incident, that nobody had looked at the change at all.
The GitLabOps Upgrade That Kept Slipping
The self-managed instance was the least glamorous item on the list and the one with a deadline attached. Version lines contain releases where you must pause: background maintenance has to finish before the next hop, and skipping the pause corrupts the quiet parts of the database rather than the loud ones. The desk scheduled the hop for a Sunday morning, waited for the background work to drain, and moved up several minor versions to within two of current. It took ninety minutes and produced exactly one gray hair. The recurring item now on the desk calendar says what it says so it never again depends on someone remembering.
One Decision Reversed When the Constraint Vanished
A rounding error of the month: the desk had assumed their runner machines needed to sit inside the hospital network, and had scored a self-scaling Docker arrangement accordingly. When it turned out no job actually required private-network reachability, the scoring was redone and hosted machines came out ahead — roughly 4.30 to the old setup’s 3.85. The lesson was not about the numbers. It was that a runner strategy is an answer to a set of constraints, and when a constraint is discovered to be imaginary, the answer deserves a second look.
What Changed That Anyone Can Feel
Four weeks in: merge requests get feedback in minutes instead of afternoons, nothing has queued mysteriously since the runner split, production carries an approval trail an auditor could follow, and Friday-evening laptop deploys are retired — the binder went into a drawer. Colleagues in clinical work describe something similar in triage: you sort out what is actually causing the trouble before choosing any treatment, exactly as this guide to treatment options to weigh before considering surgery lays out. And like knowing when a problem has crossed the line where a specialist should take over, half the value is knowing which repairs are beyond a two-person desk’s spare hours.
For the fuller reference the desk worked from — pipeline structure, runner trust, deployment gates, upgrade rhythm — the Atlas’s GitLabOps entry covers the discipline end to end. Priya’s summary for the next hire, scribbled on the inside cover of a fresh binder, fits in a sentence: find the causes first, fix them in the order they hurt, and put the calendar entries in before you trust anyone’s memory.