Last winter a friend of mine took over the overnight on-call rota for a small clinic network’s computer systems. On her first shift she texted me five questions in a row, each starting with “wait, why”. Her list is the backbone of this piece, because the answers to those five questions are most of what SysOps, the craft of keeping individual servers well, really teaches. Nothing here concerns treatment or patient care; it is all about the ordinary machinery behind a website or booking tool.
Why does the server say “busy” when the processors look idle?
This was her opening complaint. A dashboard showed a load figure of twelve on a machine whose processors were nearly asleep. She assumed the monitoring was broken. It was not. According to the SysOps page in the Ops Atlas, that number includes jobs parked in an uninterruptible wait for input and output, which means a sluggish drive can make the whole box seem overrun while the processors loaf.
The practical lesson is to treat the figure as a prompt, not a verdict. Ask what the machine is waiting for before you decide it needs more horsepower. She wrote that sentence on a sticky note and put it on her monitor.
Is it a problem that almost no memory is free?
On her second night the free-memory line sat at a few percent and she nearly paged the whole team. Linux keeps spare memory busy as a file cache, which is a feature, not a leak. The reading worth trusting, says the Atlas, is the line labelled available, because it estimates what could be handed to a program on demand.
Picture a waiting room where unused chairs get lent to a neighbouring office, then returned when patients arrive. Empty-looking chairs are not a sign of trouble; unreturned ones are. The available figure is simply the count of chairs you could reclaim in a minute.
How long until the disk fills up?
This question produced the most satisfying arithmetic of her first week. Take a 500 GiB volume that holds 410 GiB. It grew by 63 GiB over seven days, so about nine gibibytes a day. That leaves 90 GiB, and ten days sounds right. But the Atlas example reminds readers that a root reserve of five percent, about 25 GiB, is not available to ordinary programs. Applications hit the wall near 475 GiB, which gives a runway of roughly 7.2 days.
She found it striking that a ten-day answer and a seven-day answer can both come from the same honest ledger. The gap is the sort of detail that turns a calm Friday into a weekend callout. She now charts runway in days rather than percent full, and she has stopped trusting round numbers.
Should I just restart it?
Every instinct on a quiet night says yes. A restart is quick, it often clears the symptom, and it feels productive. The Atlas names restart-first as a trap, because the memory state, open connections and half-finished work that would explain the fault vanish at the moment of the restart.
Her compromise was a small routine. Before touching anything she records what the machine is doing: running programs, recent log lines, disk and network figures. It takes about a minute and it has already saved her from repeating the same fix twice. If a restart really is needed, she has evidence in hand to explain why the problem happened.
Why does the dashboard say 40% processor use while users complain of slowness?
This was her toughest night. The entry on the Atlas page lists three culprits. First, averages over five minutes can flatten short bursts that users feel keenly. Second, on virtual machines the hypervisor may be lending your slice of the processor to someone else, a pattern called steal time. Third, the real bottleneck may sit somewhere else entirely, perhaps a disk array or a backed-up queue.
She added a finer-grained chart and a steal-time line, and the mystery dissolved within a day. A reading is only as honest as its sampling interval.
Why did the logs eat the disk overnight?
Week three brought a quieter puzzle. A volume that had looked comfortable at supper time was nearly full by dawn, and the culprit was the system journal. The Atlas explains that the journal daemon has built-in ceilings: by default it may use up to a tenth of the filesystem, it aims to leave fifteen percent free, and no single journal file grows past 4 GiB. Those limits sound generous until several services chatter at once.
The fix she chose was to state her storage preference explicitly in the configuration instead of relying on whatever default applied, then cap the size to something she could defend. Two lines of settings replaced a recurring surprise. She also learned that rotating a file by copying and truncating it can drop a very small slice of lines written during the copy, so she picked a rotation method that did not need that trick for the services she cared about.
Why did a busy program suddenly refuse new connections?
Week four delivered a stranger symptom: a booking tool that worked all morning and then began rejecting visitors. The cause turned out to be a ceiling on open files. According to the entry, a service started by systemd begins with a soft cap of 1024, with a hard limit far above it, and the shortcut directive most people reach for carries a “do not use” note in the manual. She raised the limit for that one service through the proper drop-in file and wrote the reason beside it.
Bonus questions that came after the first week
Do we still need staff if everything is hosted in the cloud?
Yes, with a caveat. The Atlas does not soften this: when you rent raw virtual machines, the operating system inside, updates included, stays on your side of the fence. Someone still has to watch time, disks, logs and permissions on those virtual machines.
Can a snapshot stand in for a backup?
No. The Atlas lists confusing the two as a common error. A point-in-time copy kept beside the original data tends to share its fate, while a backup has to be restorable somewhere else. She schedules a small restore test every quarter so the claim “we have backups” is something she has actually seen work.
Is it safe to switch off the security module until things work?
The sensible answer is to relax a narrow rule, not the entire guard. The Atlas recommends keeping the mandatory access control system enforcing and loosening only the specific domain or profile that blocks a legitimate action, then tightening it again once the cause is fixed.
What she keeps on her desk now
- A card with three questions for any alert: what is waiting, what is full, and what changed recently.
- A running notebook of hand edits, with the date and reason beside each one.
- A weekly check of disk runway measured in days.
- A calendar reminder to test a restore, not merely confirm that a backup job ran.
Where to read next
If your own back trouble brought you to this site rather than an interest in servers, the guide to the choices people explore ahead of any operation for back pain is a far better starting place than anything above. For readers deciding when a problem has outgrown general advice, the article about signs that point toward specialist spine care frames the question calmly.
What SysOps looks like at three in the morning
Her last text to me that week was short: “The job is mostly asking better questions at three in the morning.” That is a fair summary of SysOps. Learn what each number really counts, record what you touch, rehearse recovery, and the nights get quieter every month.
