Boot, systemd and services¶
Status: AI draft, 2026-11-16. Not yet verified by me. Commands are from the official documentation and were not run; output is illustrative.
In one line¶
systemd is PID 1 on most Linux servers: it starts services from unit files, restarts them when they fail, captures their logs in the journal, and orders startup by dependencies. systemctl status and journalctl -u are the first two commands for any service problem.
The plain idea¶
When a server boots, something has to start the database, the web server, the log shipper and the SSH daemon, in a sensible order, and keep them running. On most current distributions that something is systemd.
You describe each service in a small text file: what command to run, as which user, what it depends on, and what to do if it dies. systemd reads these files, starts everything, watches every process it started, restarts what crashes, and keeps each service's output in one searchable log. One tool, systemctl, starts, stops and inspects them all.
How it works¶
Boot, briefly¶
flowchart TD
A["Firmware: BIOS or UEFI"] --> B["Bootloader: GRUB"]
B --> C["Kernel and initramfs"]
C --> D["systemd as PID 1"]
D --> E["default.target, usually multi-user.target"]
E --> F["Services that target wants, in dependency order"]
- The firmware finds a bootable disk and starts the bootloader (GRUB on most servers).
- GRUB loads the kernel and a small temporary filesystem, the initramfs, which holds the drivers needed to find the real root disk.
- The kernel mounts the root filesystem and starts PID 1, which is systemd.
- systemd starts the default target and everything it needs.
Units¶
Everything systemd manages is a unit, described by a unit file. The suffix says the type:
| Type | What it manages | Example |
|---|---|---|
.service |
A process or a one-off command | nginx.service |
.timer |
A schedule that starts a service (Cron and timers) | backup.timer |
.socket |
A listening socket that starts a service on the first connection | docker.socket |
.target |
A named group of units, used as a synchronisation point | multi-user.target, network-online.target |
.mount |
A mounted filesystem | data.mount |
Packaged unit files live in /usr/lib/systemd/system/ (or /lib/systemd/system/). Your own, and your overrides, go in /etc/systemd/system/, which takes precedence.
Anatomy of a service unit¶
[Unit]
Description=Sample API
After=network-online.target
Wants=network-online.target
[Service]
User=app
WorkingDirectory=/opt/sample-app
Environment=PORT=8080
EnvironmentFile=-/etc/sample-app/env
ExecStart=/opt/sample-app/bin/server
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
| Line | Meaning |
|---|---|
After= |
Ordering only: start after these units if they are being started. It does not pull them in. |
Wants= |
A soft dependency: start these too. Requires= is the hard version: if the required unit fails, this one is stopped. |
User= |
Run as this user instead of root. |
Environment=, EnvironmentFile= |
Set environment variables. The - before the path means "ignore the file if it is missing". |
ExecStart= |
The command, with a full path. systemd does not use a shell, so pipes and $VAR expansion need /bin/sh -c. |
Restart=on-failure |
Restart if the process exits non-zero or is killed by a signal. always restarts even after a clean exit. |
RestartSec= |
Wait this long before restarting. |
WantedBy= |
What systemctl enable hooks this unit into, so it starts at boot. |
Service types. The default, Type=simple, assumes the process started by ExecStart is the service and stays in the foreground. A program that forks into the background needs Type=forking. A command that runs and finishes uses Type=oneshot. Most modern programs should simply stay in the foreground.
Restart limits. If a service fails too often in a short window (by default 5 starts within 10 seconds), systemd gives up and leaves it failed until you reset it. StartLimitIntervalSec= and StartLimitBurst= in [Unit] change the window.
Stopping¶
systemctl stop sends SIGTERM to the service's processes, waits (90 seconds by default, TimeoutStopSec=), then sends SIGKILL (Processes and signals). Every process the service started is in the service's own cgroup, so none are left behind.
supervisord compared¶
Some applications, especially containers that run several processes, use supervisord instead: a Python program that starts and restarts processes listed in its own config file, controlled with supervisorctl.
| systemd | supervisord | |
|---|---|---|
| Runs as | PID 1 on the host | An ordinary process, often started by systemd or as a container's entrypoint |
| Scope | The whole system: boot, mounts, timers, sockets | Only the programs listed in its config |
| Logs | Journal | Its own log files per program |
| Typical use | Every service on a server | Several processes inside one container, or older setups |
A concrete example¶
A new service, sample-api, will not stay up after a deploy.
systemctl status sample-apishowsActive: activating (auto-restart) (Result: exit-code), with the last few log lines underneath. It is crashing and being restarted.journalctl -u sample-api -n 50 --no-pagershowsError: open /etc/sample-app/config.yaml: permission denied.systemctl cat sample-apishowsUser=app. The config file was copied as root with mode600, so theappuser cannot read it (Users, groups and permissions).- Fix the ownership, then
systemctl restart sample-api. - If it had stopped restarting and shown
start request repeated too quickly, also runsystemctl reset-failed sample-apito clear the start limit.
Why not just...?¶
- ...start it with
nohup ./server &from a shell? Nothing restarts it when it crashes, it does not start after a reboot, its output goes to a stray file, and stopping it means hunting for its PID. - ...edit the packaged unit file in
/usr/lib/systemd/system? The next package upgrade overwrites it. Usesystemctl edit <unit>, which writes an override in/etc/systemd/system/<unit>.d/that survives upgrades. - ...put
After=network.targetand assume the network is ready?network.targetmeans the network stack has started, not that an address is configured. For a service that must connect out at start, usenetwork-online.targetwith bothAfter=andWants=, or better, make the service retry.
Commands to know¶
Status, the first command for any service:
$ systemctl status sample-api
● sample-api.service - Sample API
Loaded: loaded (/etc/systemd/system/sample-api.service; enabled; preset: enabled)
Active: active (running) since Mon 2026-11-16 09:12:04 UTC; 3min ago
Main PID: 2311 (server)
Tasks: 9 (limit: 4598)
Memory: 41.2M
CPU: 1.204s
CGroup: /system.slice/sample-api.service
└─2311 /opt/sample-app/bin/server
Loaded shows the unit file path and whether it is enabled (starts at boot). Active shows the state and since when. CGroup lists every process belonging to the service.
| Command | What it does |
|---|---|
systemctl start, stop, restart <unit> |
Change the state now. |
systemctl reload <unit> |
Ask the service to reload its config without restarting, if the unit supports it. |
systemctl enable --now <unit> |
Start at boot, and start now. disable reverses it. |
systemctl daemon-reload |
Re-read unit files after you change one. Without it, systemd keeps using the old version. |
systemctl cat <unit> |
The unit file and every override, as systemd sees them. |
systemctl edit <unit> |
Create an override file. |
systemctl list-units --failed |
Every unit in the failed state. |
systemctl is-active <unit>, is-enabled <unit> |
One-word answers, for scripts. |
journalctl -u <unit> -f |
Follow the unit's logs (Logs). |
systemd-analyze blame |
How long each unit took during boot. |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
activating (auto-restart) in a loop |
The program | journalctl -u <unit> -n 50 |
It crashes at start: bad config, permissions, a missing dependency |
start request repeated too quickly |
Start limit | systemctl status |
Restarted too often; fix the cause, then systemctl reset-failed |
| A change to the unit file has no effect | Unit cache | Did you run daemon-reload? |
systemd still uses the old definition |
| Service not running after a reboot | Enablement | systemctl is-enabled <unit> |
Started by hand but never enabled |
status=203/EXEC |
ExecStart |
systemctl cat; ls -l the binary |
Wrong path, no execute bit, or a script with a bad shebang |
| Works from a shell, fails as a service | Environment | Compare User=, WorkingDirectory=, environment |
systemd does not load your shell profile; relative paths and variables differ |
Active: active (exited) for a long-running program |
Service type | Does the program fork into the background? | Type=simple with a program that daemonises itself |
Interview questions¶
1. What is systemd, and what is a unit?
Answer
- The init system and service manager, running as PID 1 on most Linux distributions.
- It starts, stops, supervises and logs services, and orders the boot by dependencies.
- A unit is anything it manages, described by a file: services, timers, sockets, targets, mounts.
2. What is the difference between After= and Requires=?
Answer
After=is ordering only: if both start, this one starts later.Requires=(and the softerWants=) pulls the other unit in.- They are usually used together:
Wants=orRequires=to start it,After=to wait for it.
3. You edited a unit file and restarted the service, but nothing changed. Why?
Answer
- systemd caches unit definitions.
- Run
systemctl daemon-reloadafter editing, then restart. systemctl catshows the definition systemd is actually using, including overrides.
4. A service keeps restarting. Walk me through your investigation.
Answer
systemctl statusfor the state, exit code and last log lines.journalctl -u <unit>around the restarts for the real error.systemctl catto checkExecStart,User, environment and working directory.- Reproduce as that user with the same environment; fix the cause, then
reset-failedif the start limit was hit.
5. How do you change a setting of a packaged service without your change being lost on upgrade?
Answer
systemctl edit <unit>creates a drop-in override in/etc/systemd/system/<unit>.d/.- Overrides survive package upgrades and take precedence.
daemon-reloadand restart to apply.
6. What happens when you run systemctl stop on a service?
Answer
- systemd sends SIGTERM to the service's processes (all of its cgroup by default).
- It waits up to
TimeoutStopSec(90 seconds by default). - Anything still running then gets SIGKILL.
7. When would you choose supervisord over systemd?
Answer
- Inside a container that must run several processes, where systemd is not PID 1.
- On older setups that already use it.
- On a normal server, systemd is already there and does more: boot ordering, journal, cgroups, timers.
Try it¶
Untested: written from documentation. Run this on a Linux virtual machine with systemd, not in a plain container.
Step 1. Create a service that writes a line every 5 seconds.
sudo tee /etc/systemd/system/hello.service >/dev/null <<'EOF'
[Unit]
Description=Hello loop
[Service]
ExecStart=/bin/sh -c 'while true; do echo hello; sleep 5; done'
Restart=on-failure
EOF
sudo systemctl daemon-reload
sudo systemctl start hello
Step 2. Inspect it, and follow its log.
systemctl status hello
journalctl -u hello -f
Step 3. Kill its main process with SIGKILL and predict what systemd does, then check status again.
sudo systemctl kill -s KILL hello
systemctl status hello
Step 4. Clean up.
sudo systemctl stop hello
sudo rm /etc/systemd/system/hello.service
sudo systemctl daemon-reload
Zoom out¶
systemd keeps services alive on a single server; Kubernetes does the same job across many machines, with the same ideas of desired state, restart policies and graceful stops (Architecture and the reconcile loop). Without a supervisor, a crash at 3 a.m. stays down until someone notices. The principle to carry: status for what happened, the journal for why, cat for how it is configured.