Logs & Troubleshooting

Everything before this page was about making things work. This page is about what you do when they don’t — methodically, under pressure, at an hour you’d rather be asleep. The skill isn’t memorizing commands; it’s having a system so you investigate instead of flail. The manual, error-prone step we’re removing here is guessing — randomly restarting things and hoping. We replace it with a repeatable triage routine that turns “it’s broken” into “here’s exactly what’s wrong.”

Where the logs are

Two places, depending on the era of the software:

• The journal (systemd services) — query with journalctl, as in systemd & Services.
• /var/log — flat text files, the classic home for logs (and where plenty of software still writes).

/var/log/
├── syslog  /  messages    general system messages (Debian / RHEL respectively)
├── auth.log / secure      logins, sudo, SSH  → who did what
├── kern.log               kernel messages
├── nginx/                 per-service directories: access.log, error.log
└── dpkg.log / dnf.log     package install/upgrade history

journalctl -u myapp -p err --since "1 hour ago"   # service errors, last hour
tail -f /var/log/nginx/error.log                  # follow a flat-file log live
grep -i "out of memory" /var/log/syslog           # search the system log

The reflex when something breaks: read the logs first. Most incidents are explained by a line in a log you haven’t read yet.

dmesg: the kernel’s own log

dmesg shows the kernel ring buffer — messages from the kernel itself, below the application layer. This is where you find hardware errors, disk failures, and — critically — the OOM killer.

sudo dmesg -T | tail -50              # -T gives human-readable timestamps
sudo dmesg -T | grep -i "oom\|killed process"

When a Linux box runs out of memory, the kernel doesn’t politely ask — it picks a process and kills it (the OOM killer), and the only record is in dmesg and the journal. “My app keeps dying for no reason” is, astonishingly often, “the kernel OOM-killed it and nothing in the app’s own logs would ever show that.” Knowing to check dmesg saves hours.

The four-tool triage: top, df, free, ss

When a box is unhealthy, the cause is almost always one of four exhausted resources: CPU, disk, memory, or network/connections. One tool each. Run them in order and you’ve localized most problems in under a minute.

CPU and processes — top

top                 # live; 'M' sorts by memory, 'P' by CPU, '1' shows per-core, 'q' quits

Read the load average (top-right): three numbers for the last 1, 5, and 15 minutes. As a rule of thumb, compare to your core count (nproc) — a load of 8 on an 8-core box is fully busy; 30 on 8 cores is badly overloaded. Then scan for the process eating CPU. Is one process pinned at 100%? Is load high but no single process busy (often I/O wait — check wa% in the CPU line)?

Disk space — df (and du)

df -h               # disk space per filesystem, human-readable
df -i               # INODES — you can run out of these even with space free
du -sh /var/log/*   # what's eating space in a directory

A full disk is one of the most common and most disruptive failures: databases stop accepting writes, apps crash, logs can’t be written (so the logs about the problem fail too). df -h is often the very first thing to check. Watch for 100% on / or /var. And don’t forget df -i: millions of tiny files can exhaust inodes while gigabytes of space sit free — a baffling failure until you know to look.

Memory — free

free -h             # total / used / free / available, human-readable

Read the available column, not free. Linux deliberately uses spare RAM for disk cache, so free looks alarmingly low on a healthy system — that cache is reclaimable. available is the honest estimate of what a new process could actually get. If available is near zero and swap is filling, you’re memory-starved and the OOM killer is circling (back to dmesg).

Network and ports — ss

ss -tlnp            # TCP (t) listening (l) sockets, numeric (n), with the owning process (p)
ss -tn state established   # established connections — are you leaking them?
ss -s               # summary counts by socket state

ss (the modern replacement for netstat) answers “is my service actually listening on the port I think?” and “how many connections are open?” A service that “won’t accept traffic” often isn’t listening where you expect — bound to 127.0.0.1 instead of 0.0.0.0, or not bound at all. This is the bridge into Ports & Firewalls.

The 60-second triage sweep

When paged, run these before forming a theory: uptime (load), df -h (disk), free -h (memory), ss -tlnp (is it even listening?), then journalctl -u <svc> -p err --since "15 min ago". Four resources plus the logs — you’ve ruled out the common causes before you start guessing.

The methodical mindset

Tools are half of it; method is the other half. Flailing under pressure makes outages longer. A disciplined loop:

   1. OBSERVE   What exactly is wrong? Read the logs. Quantify it
               (error rate? since when? which endpoint?). Don't guess yet.
   2. HYPOTHESIZE  Form ONE testable theory. ("Disk is full," not "something's off.")
   3. TEST     Check it with a tool (df -h). Confirm or kill the theory fast.
   4. CHANGE   Make ONE change. Just one — so you know what fixed it.
   5. VERIFY   Did the symptom actually clear? Watch the metric, not your hope.
              ↺ If not, back to step 2 with what you learned.

Two rules that separate calm responders from panicked ones:

• Change one thing at a time. Change five and something improves — you’ll never know which, and you can’t write a useful postmortem.
• Read before you act. The error message, the log line, the exit code — the answer is usually already written down. Restarting blindly can erase the evidence you needed.

The restart trap

Restarting often “fixes” things — by clearing the symptom and destroying the cause. The OOM-killed process restarts and runs fine… until memory fills again in two hours. If you restart, capture the state first (journalctl, dmesg, top output, a heap dump), so the incident becomes a root-caused fix instead of a recurring 3 a.m. page. A restart that hides a cause is a manual step you’ll pay for repeatedly.

Where this goes next

This page is reactive triage on one box, by hand. It works, but it doesn’t scale: you can’t SSH into a hundred servers, and top can’t tell you what happened last Tuesday at 2 a.m. The discipline of shipping logs, metrics, and traces to a central system — so you can answer questions across your fleet and back in time, ideally before users notice — is observability, the whole of Part 7. The mindset you build here, by hand on one machine, is exactly what that tooling automates and scales. Learn it manually first; you’ll trust the dashboards more when you know what they’re standing in for.

Under the hood — how the OOM killer picks its victim

When memory runs out, the OOM killer isn’t random. The kernel keeps an oom_score for every process — driven mainly by how much memory it uses — and kills the highest scorer, because freeing the biggest process recovers the most memory fastest. That’s usually your main application (the fattest target), occasionally an innocent bystander. You can bias the choice with oom_score_adj (roughly -1000 to +1000): nudge a critical process down, or volunteer a disposable one to die first.

This matters doubly in containers. Each container runs in a cgroup with its own memory limit, and exceeding that limit triggers a cgroup-local OOM kill even when the host has memory to spare. The tell-tale sign is exit code 137 — that’s 128 + 9, i.e. killed by signal 9 (SIGKILL). So “my pod keeps dying with exit 137” almost always means “it blew its memory limit and got OOM-killed.” Knowing the score and the exit code turns a baffling restart loop into a one-line diagnosis — the manual guessing this page exists to kill. (cgroup limits come from namespaces & cgroups.)

Check your understanding

1. An app “keeps dying for no reason” and its own logs show nothing. Where do you look, and what kernel behavior are you hunting for?
2. free -h shows very little free memory on a healthy server. Why isn’t that alarming, and which column should you actually read?
3. Map each of the four triage tools (top, df, free, ss) to the resource it checks, and give one failure each reveals.
4. Why is “change one thing at a time” a hard rule during an incident? What does violating it cost you afterward?
5. Restarting a service often clears the symptom. Explain the “restart trap” and what you should capture before restarting — and connect this page’s manual triage to observability.

Show answers

1. Look in dmesg (the kernel ring buffer) and the journal, hunting for the OOM killer. When a box runs out of memory the kernel picks a process and kills it, and the only record is in dmesg/the journal — the app’s own logs would never show it. sudo dmesg -T | grep -i "oom\|killed process" is the move.
2. It’s not alarming because Linux deliberately uses spare RAM for disk cache, which is reclaimable — so free looks low on a perfectly healthy system. Read the available column instead: it’s the honest estimate of what a new process could actually get.
3. top → CPU/processes (one process pinned at 100%, or high load from I/O wait); df → disk (a full / or /var, or exhausted inodes via df -i); free → memory (available near zero with swap filling, OOM circling); ss → network/sockets (a service not listening where expected — bound to 127.0.0.1 instead of 0.0.0.0).
4. Because if you change five things and something improves, you’ll never know which one fixed it — you can’t reproduce the fix, can’t trust it didn’t mask the real cause, and can’t write a useful postmortem. One change at a time keeps cause and effect legible.
5. The restart trap: restarting often “fixes” things by clearing the symptom while destroying the cause (the OOM-killed process restarts and runs fine… until memory fills again in two hours). Capture state first — journalctl, dmesg, top output, a heap dump — so it becomes a root-caused fix, not a recurring 3 a.m. page. This manual, single-box triage is what observability automates and scales across the fleet and back in time.