systemd & Services

Back in the process model we hit a wall: starting your app with python app.py & doesn’t survive logout, doesn’t restart when it crashes, and doesn’t come back after a reboot. Running real software that way is the manual, fragile habit this page kills. The fix is a service manager — software whose entire job is to start, supervise, restart, and stop your processes for you. On modern Linux that manager is systemd, and it’s PID 1 — the very root of the process tree.

What systemd is, and why it’s PID 1

systemd is the first thing the kernel starts at boot, so it becomes the ancestor of everything else. It brings the machine up (mounting filesystems, starting networking, launching services in dependency order) and then stays running to supervise those services for the life of the box. Because it’s always there, it can do the things your shell can’t: notice a process died and restart it, hold a process’s logs, and bring it back automatically after a reboot.

systemd manages units — a general word for “a thing systemd looks after.” The kind we care about is the service unit (a .service file), which describes a long-running process.

   kernel boots
      │
      ▼
   systemd (PID 1) ──reads unit files──► starts services in dependency order
      ├── nginx.service        (restart on failure)
      ├── postgresql.service   (start after network)
      └── myapp.service        (your app — managed, supervised, logged)

A service unit, line by line

A unit is a plain text file (text again — readable, diffable, version-controllable). Here’s a minimal one for your app, at /etc/systemd/system/myapp.service:

[Unit]
Description=My web application
# don't start until networking is up
After=network.target

[Service]
ExecStart=/usr/bin/python3 /opt/myapp/app.py
WorkingDirectory=/opt/myapp
# run as an unprivileged user, NOT root
User=myapp
Group=myapp
# inject config via the environment…
Environment=NODE_ENV=production
# …or load many vars from a file
EnvironmentFile=/etc/myapp/env
# bring it back if it crashes, waiting 5s between attempts
Restart=on-failure
RestartSec=5

[Install]
# start at boot, in normal multi-user mode
WantedBy=multi-user.target

(One syntax gotcha, since you’ll copy files like this: unit files do not allow inline # comments — a comment must be on its own line, or systemd reads it as part of the value.)

Look at how much of Part 1 this one file pulls together: it runs the process (process model), as a dedicated unprivileged User (least privilege), with configuration passed through the environment (the shell). The unit is the operational knowledge about your service, written down instead of living in someone’s head.

systemctl: the control panel

systemctl is how you talk to systemd. The verbs you’ll use daily:

sudo systemctl daemon-reload         # re-read unit files after editing one (don't forget this!)
sudo systemctl start myapp           # start it now
sudo systemctl stop myapp            # stop it now (sends SIGTERM, then SIGKILL after a timeout)
sudo systemctl restart myapp         # stop then start
sudo systemctl reload myapp          # ask it to reload config without a full restart (if supported)
sudo systemctl enable myapp          # start automatically at BOOT
sudo systemctl disable myapp         # don't start at boot
sudo systemctl enable --now myapp    # enable AND start in one step
systemctl status myapp               # is it running? recent logs, PID, memory
systemctl is-active myapp            # just the state, scriptable

start vs enable — the distinction that trips everyone up

start runs the service right now, once. enable wires it to launch automatically at every boot. They’re independent: a service can be running now but not enabled (gone after a reboot), or enabled but not yet started. For anything you want to survive a reboot, you need both — hence the handy enable --now. Forgetting enable is the classic “it was working, then the box rebooted and the app never came back” incident.

Notice that systemctl stop does the graceful thing from the process model: it sends SIGTERM, waits (TimeoutStopSec, default 90s), and only then SIGKILLs. Your app’s clean SIGTERM handling is what makes a systemctl restart during a deploy lose zero requests.

Restart policies: self-healing, declared once

Restart=on-failure is small but profound. It means: if my app crashes, bring it back automatically. You’ve replaced a human (“page someone, they SSH in, they restart it”) with a policy the machine enforces in milliseconds, every time, forever. The common values:

Restart=	Restarts when…
no (default)	never
on-failure	the process exits non-zero or is killed by a signal
on-abnormal	killed by a signal or timeout (not on a non-zero exit code)
always	it stops for any reason, even a clean exit

Don’t restart-loop blindly

Restart=always with no backoff can hammer a broken service into a tight crash loop, burning CPU and flooding logs. Pair it with RestartSec= (a delay between attempts) and StartLimitIntervalSec / StartLimitBurst (give up after N failures in a window) so a genuinely broken service stops and pages a human instead of thrashing. Self-healing is great; an infinite loop pretending to self-heal is not.

This is the same idea you’ll meet at a higher altitude as the reconciliation loop in Kubernetes: declare the desired state (“this should be running”), and let the system continuously make reality match. systemd is that loop for one machine.

Logs: journald

systemd captures everything your service writes to stdout and stderr and stores it in a structured log called the journal (managed by journald). You don’t have to configure log files — just write to stdout and systemd captures it. That convention (“a service logs to stdout, the platform handles the rest”) carries straight into containers and the twelve-factor world.

journalctl -u myapp                  # all logs for this unit
journalctl -u myapp -f               # follow live (like tail -f)
journalctl -u myapp -n 100           # last 100 lines
journalctl -u myapp --since "10 min ago"
journalctl -u myapp -p err           # only error-priority and worse
journalctl -u myapp -b               # only since the last boot

Because the journal is structured, you can filter by unit, time, priority, and boot — far more powerful than grepping a flat file. We go deep on this in Logs & Troubleshooting.

Running your app as a managed service: the full loop

Putting it together — this is the recipe you’ll repeat for every service you own:

sudo useradd --system --no-create-home myapp    # 1. a dedicated, unprivileged user
sudo nano /etc/systemd/system/myapp.service     # 2. write the unit (see above)
sudo systemctl daemon-reload                     # 3. make systemd aware of it
sudo systemctl enable --now myapp                # 4. start it AND survive reboots
systemctl status myapp                           # 5. confirm it's running
journalctl -u myapp -f                            # 6. watch its logs

Compare that to python app.py &. The service version survives logout, restarts on crash, comes back after reboot, runs as a least-privilege user, takes config from the environment, shuts down gracefully, and keeps structured logs — all declared in one reviewable file. That is the manual, error-prone “babysit a background process” step, eliminated. The machine does the babysitting now, the same way every time.

Under the hood — ordering is not requirement

The unit above says After=network.target, and it’s tempting to read that as “this service needs the network.” It doesn’t — and the distinction trips up almost everyone. systemd splits two ideas that English jams together:

• Ordering (After=, Before=) — when to start relative to another unit. On their own, these do not pull that unit in.
• Requirement (Wants=, Requires=) — the other unit should also be started. Requires= is strict (if it fails, your unit fails too); Wants= is the softer “start it if you can,” and is usually what you want.

A robust dependency uses both: Wants=foo.service and After=foo.service. A second trap: network.target is reached early and only means networking is being set up, not that an IP is configured — if your app truly needs a working network at start, order on network-online.target instead. Writing these down correctly is the point: the start-up dependencies that used to live as “remember to bring up X before Y” in someone’s head become declared, enforced fact.

The architect’s lens

systemd is a tool you choose to put every service under, so weigh it with the five questions:

• Why does it exist? Because python app.py & doesn’t survive logout, doesn’t restart on crash, and doesn’t come back after a reboot — you need software that is always there (PID 1, the root of the process tree) to start, supervise, and resurrect processes for the life of the box.
• What problem does it solve? It captures the operational knowledge about a service — run as an unprivileged User, config via Environment, Restart=on-failure, a graceful SIGTERM→(90s)→SIGKILL stop, structured journald logs — in one reviewable .service file instead of someone’s head.
• What are the trade-offs? The declarative unit adds ceremony (write it, daemon-reload, enable --now) and hides footguns: forgetting enable means the app silently vanishes on reboot, and Restart=always with no backoff hammers a broken service into a tight crash loop unless you add RestartSec= and StartLimitBurst.
• When should I reach past it? When you must schedule and heal services across many machines — that’s the Kubernetes reconciliation loop; systemd is the same “declare desired state, make reality match” loop for one machine.
• What breaks if I remove it? You’re back to babysitting background processes by hand — no auto-restart, no boot persistence, no graceful shutdown — so a crash or deploy at 3 a.m. pages a human instead of self-healing in milliseconds.

Check your understanding

1. Why can systemd restart a crashed service when your shell can’t? What’s special about being PID 1?
2. A service is started but not enabled. What happens after a reboot, and why is this a common incident?
3. Trace what systemctl restart myapp does to the running process in terms of signals. Why does your app’s SIGTERM handling matter for a zero-downtime deploy?
4. What’s the risk of Restart=always with no other settings, and which directives tame it?
5. How does a systemd restart policy foreshadow the Kubernetes reconciliation loop? State the shared idea in one sentence.

Show answers

1. systemd is PID 1 — the first process the kernel starts and the ancestor of everything — so it’s always running for the life of the box. That lets it do what your shell can’t: notice a process died and restart it, hold its logs, and bring it back after a reboot. Your shell exits on logout and takes its background jobs with it.
2. After a reboot the service does not come back — start runs it once, now; enable is what wires it to launch automatically at boot, and they’re independent. It’s a classic incident (“it was working, then the box rebooted and the app never returned”) precisely because forgetting enable is silent until a reboot exposes it. You need both — hence enable --now.
3. systemctl restart does the graceful thing: it sends SIGTERM, waits (TimeoutStopSec, default 90s), and only then SIGKILLs. Your app’s clean SIGTERM handling — finishing in-flight requests, flushing, closing connections before exiting — is what lets a restart during a deploy lose zero requests, which is the basis of a zero-downtime/rolling deploy.
4. Restart=always with no backoff can hammer a broken service into a tight crash loop — burning CPU and flooding logs while pretending to self-heal. Tame it with RestartSec= (a delay between attempts) and StartLimitIntervalSec / StartLimitBurst (give up after N failures in a window) so a genuinely broken service stops and pages a human.
5. Both declare a desired state and let the system continuously make reality match it: systemd (“this should be running” → restart on failure) is that loop for one machine; the Kubernetes reconciliation loop is the same idea at cluster scale.