Logs¶
Status: AI draft, 2026-11-16. Not yet verified by me. Commands and output were run in a throwaway container; addresses are rewritten to example ranges. A plain container has no systemd journal, so
journalctlappears as commands only, without output.
In one line¶
Logs live in three places: the systemd journal (journalctl), files under /var/log, and a container's stdout. Read them by narrowing the time window first, then finding the first error, and keep them from filling the disk with rotation.
The plain idea¶
A log is a program's diary: one line per event, with a time. When something breaks, the log is usually the only record of what the program saw just before. Two skills matter. Knowing where a given program writes, because there are several possible places, and knowing how to read a large log under pressure without drowning in it.
The third skill is housekeeping. Logs grow forever unless something trims them, and a full disk is one of the most common self-inflicted outages (Disks and filesystems).
How it works¶
Where a log line can end up¶
flowchart TD
A["Program writes a line"] --> B{"How was it started?"}
B -->|"systemd service, writes to stdout"| C["systemd journal: journalctl -u name"]
B -->|"writes its own file"| D["/var/log/name/app.log"]
B -->|"inside a container"| E["Container runtime: docker logs or kubectl logs"]
C -.->|"optionally forwarded"| F["rsyslog: /var/log/syslog"]
D --> G["logrotate trims and compresses"]
| Source | How to read it | Notes |
|---|---|---|
| systemd journal | journalctl |
Binary, indexed store. Captures each service's stdout and stderr, plus kernel messages. May be lost on reboot unless stored on disk (/var/log/journal). |
/var/log files |
less, tail -f, grep |
Written by the program itself or by rsyslog. Common files: syslog or messages (general), auth.log or secure (logins and sudo), kern.log, dpkg.log or the dnf log (package changes). |
| Container stdout | docker logs, kubectl logs |
The runtime saves what the container prints (How Kubernetes logging works). |
Severity¶
Syslog defined eight severity levels, which journald also uses. From most to least severe: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7). journalctl -p err shows level 3 and anything more severe.
Rotation¶
logrotate runs daily (from a cron job or a systemd timer), renames the current file, optionally compresses old ones, and deletes the oldest. The hard part is the program that is still writing:
| Method | How it works | Trade-off |
|---|---|---|
create with a postrotate signal |
Rename app.log to app.log.1, create a new app.log, then tell the program to reopen its file (often SIGHUP or a reload) |
Clean, but the program must support reopening. |
copytruncate |
Copy app.log to app.log.1, then empty app.log in place |
Works with any program, but lines written between the copy and the truncate are lost. |
Without either, the program keeps writing to the renamed file (it holds the file open, not the name), and the new app.log stays empty.
Structured logs¶
A line such as level=error user=42 order=9001 msg="payment failed", or the same as JSON, can be filtered by field instead of by guessing at text. Log pipelines can index each field, and jq can query JSON logs on the command line (jq and yq).
A concrete example¶
At 14:05 users report that checkout fails. The service runs under systemd.
- Narrow the window.
journalctl -u checkout --since "14:00" --until "14:10". Ten minutes instead of ten days. - Find the first error, not the loudest.
journalctl -u checkout --since "14:00" -p err, then read upwards from the first one. Later errors are often consequences of the first. - Follow a request. If the error line carries a request ID,
grepfor that ID across this log and the upstream service's log to see the whole path. - Check the neighbours.
journalctl -k --since "14:00"for kernel messages such as an OOM kill.journalctl --since "14:00" -p warningwithout-u, to see what else happened on the machine. - Watch live while you test a fix:
journalctl -u checkout -f.
Why not just...?¶
- ...
catthe whole log and scroll? A busy service writes thousands of lines a minute. Narrow by time first, then by severity, then by text. - ...delete old logs by hand when the disk fills? A deleted file the program still has open keeps its space, and the next time nobody remembers. Configure rotation once, with a size or age limit.
- ...log everything at debug level, just in case? Debug logs cost disk, network and money in a log pipeline, and bury the lines that matter. Log at info, and raise the level temporarily when investigating.
Commands to know¶
Reading the journal (no output shown, because a plain container has no journal):
| Command | What it shows |
|---|---|
journalctl -u nginx |
Everything the nginx unit logged. |
journalctl -u nginx -f |
Follow new lines as they arrive. |
journalctl -u nginx --since "1 hour ago" |
A time window. Also --since "2026-11-16 14:00" --until "14:10". |
journalctl -p err -b |
Errors and worse since this boot. -b -1 is the previous boot. |
journalctl -k |
Kernel messages, the same as dmesg. |
journalctl -u nginx -o json-pretty -n 1 |
The last entry with every field the journal stored. |
journalctl --disk-usage |
How much space the journal uses. --vacuum-size=500M trims it. |
Reading files:
| Command | What it does |
|---|---|
tail -n 100 app.log |
The last 100 lines. |
tail -F app.log |
Follow the file, and keep following after it is rotated (capital F). |
less +F app.log |
Follow inside less. Ctrl+C stops following so you can scroll and search with /. |
zgrep ERROR app.log.2.gz |
Search a compressed rotated log without unpacking it. |
grep -c ERROR app.log |
Count error lines (Text tools). |
What /var/log holds in a minimal Ubuntu container:
# ls /var/log
README
alternatives.log
apt
bootstrap.log
btmp
dpkg.log
faillog
journal
lastlog
private
wtmp
A logrotate rule, run three times by force:
# cat /etc/logrotate.d/sample-app
/var/log/sample-app/*.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
copytruncate
}
# logrotate -f /etc/logrotate.d/sample-app
# ls -l /var/log/sample-app
-rw-r--r-- 1 root root 0 Nov 16 03:01 app.log
-rw-r--r-- 1 root root 21 Nov 16 03:01 app.log.1
# logrotate -f /etc/logrotate.d/sample-app
# echo "x" >> /var/log/sample-app/app.log
# logrotate -f /etc/logrotate.d/sample-app
# ls -1 /var/log/sample-app
app.log
app.log.1
app.log.2.gz
| Directive | Meaning |
|---|---|
daily |
Rotate once a day. Also weekly, monthly, or size 100M. |
rotate 7 |
Keep 7 old files; delete older ones. |
compress, delaycompress |
Compress old files, but leave the newest old one (app.log.1) uncompressed for one more cycle. |
missingok, notifempty |
No error if the log is missing; skip it if it is empty. That is why the second forced run, on an empty file, did nothing. |
copytruncate |
Copy then empty the live file, instead of renaming it. |
logrotate -d <file> shows what it would do without doing it.
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
The new app.log stays empty after rotation |
Rotation | lsof <old file> shows the program still writing to it |
Neither copytruncate nor a reopen signal is configured |
| Logs from before a reboot are gone | Journal storage | ls /var/log/journal; journalctl --list-boots |
The journal is kept only in memory |
| Disk full of logs | Rotation | du -sh /var/log/*; logrotate -d |
No rule for that file, or a size far too large between daily runs |
journalctl -u app is empty |
Where it logs | systemctl cat app; the app's config |
The program writes to its own file, not stdout |
| Timestamps do not line up between two services | Time | timedatectl on both hosts |
Clocks out of sync, or one logs local time and the other UTC |
Interview questions¶
1. Where do logs go on a modern Linux server?
Answer
- The systemd journal, which captures services' stdout and stderr and kernel messages; read with
journalctl. - Files under
/var/log, written by programs themselves or by rsyslog. - For containers, the runtime's log of each container's stdout, read with
docker logsorkubectl logs.
2. How do you see only errors from one service in the last hour?
Answer
journalctl -u <service> --since "1 hour ago" -p err.-p errincludes everything more severe than err as well.- Add
-fto keep following.
3. What is the difference between copytruncate and rotating with a reopen signal?
Answer
- With a signal, the file is renamed, a new one created, and the program told to reopen; nothing is lost, but the program must support it.
copytruncatecopies the file then empties it in place; it works with any program, but lines written in between are lost.- Without either, the program keeps writing to the renamed file.
4. You are paged at 3 a.m. for a failing service. How do you read its logs efficiently?
Answer
- Narrow to the time window around the first alert.
- Filter by severity, then find the first error and read the lines just before it.
- Follow a request ID across services if there is one.
- Check kernel messages and other services on the host for OOM kills, disk errors or restarts.
5. After logrotate ran, the application's log file is empty but the application is clearly busy. Why?
Answer
- The program holds the old file open and keeps writing to it under its new name,
app.log.1. - logrotate created a new empty
app.logthat nobody writes to. - Fix: add
copytruncate, or apostrotatecommand that makes the program reopen its log.
6. Why are structured logs better than free-text logs?
Answer
- Each value has a named field, so you can filter and aggregate by field instead of guessing at text patterns.
- Log pipelines can index fields, which makes searches fast and reliable.
- Tools such as
jqcan query JSON logs directly.
Try it¶
Step 1. Start a container, install logrotate and create a log.
docker run --rm -it ubuntu:24.04 bash
apt-get update -qq && apt-get install -y -qq logrotate
mkdir -p /var/log/sample-app && seq 1 3 > /var/log/sample-app/app.log
Step 2. Write a rule, then ask logrotate what it would do, without doing it.
printf '/var/log/sample-app/*.log {\n rotate 2\n compress\n delaycompress\n copytruncate\n}\n' > /etc/logrotate.d/sample-app
logrotate -d /etc/logrotate.d/sample-app
Step 3. Predict the file names after each forced run, then run it three times, adding a line in between.
logrotate -f /etc/logrotate.d/sample-app; echo x >> /var/log/sample-app/app.log; ls /var/log/sample-app
Step 4. Type exit. The container is deleted.
Zoom out¶
On a server, logs come through the journal and /var/log; in a cluster, the same lines are collected by an agent and shipped to a central store (Fluent Bit, OpenSearch). Without logs, an incident is guesswork. The principle to carry: narrow the time window first, then find the first error, not the loudest one.