systemd Services and journalctl

Every modern mainstream distribution boots with systemd, which starts services in the right order, restarts them when they crash, and collects their output in a structured log. Turning your application into a systemd service is how it survives reboots and how operators expect to manage it. This lesson covers systemctl for controlling services, the unit file format for defining your own, and journalctl for reading the logs — the three tools you will use on every server.

Controlling Services with systemctl

bash

systemctl status nginx              # running? since when? last log lines
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx        # stop then start
sudo systemctl reload nginx         # re-read config without dropping connections (if supported)
sudo systemctl enable nginx         # start at boot
sudo systemctl disable nginx
sudo systemctl enable --now nginx   # enable and start in one step
systemctl is-active nginx           # prints active/inactive; exit code for scripts
systemctl is-enabled nginx
systemctl list-units --type=service --state=running
systemctl list-unit-files --type=service     # everything installed and whether enabled
systemctl --failed                  # what is broken right now

enable does not start a service and start does not enable it — a common source of "it stopped working after reboot".

Anatomy of a Unit File

Custom services live in /etc/systemd/system/. Package-provided units are in /lib/systemd/system/ and should not be edited directly. A service file has three sections.

bash

# /etc/systemd/system/api.service
[Unit]
Description=Orders API
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=api
Group=api
WorkingDirectory=/srv/api
EnvironmentFile=/etc/api/env
Environment=NODE_ENV=production
ExecStart=/usr/bin/node /srv/api/server.js
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

| Directive | Meaning | |---|---| | After= / Before= | ordering only; does not start the other unit | | Wants= / Requires= | start the other unit too; Requires fails if it fails | | Type=simple | ExecStart is the main process (default) | | Type=oneshot | runs to completion; for scripts and timers | | Type=notify | the service signals readiness (nginx, many Go apps) | | User=, Group= | drop privileges; never run apps as root | | EnvironmentFile= | KEY=value lines, keeps secrets out of the unit | | Restart= | no, on-failure, always | | RestartSec= | delay before restarting | | WantedBy=multi-user.target | which target pulls it in when enabled |

ExecStart needs an absolute path and does not go through a shell, so ~, pipes and && do not work. Wrap complex commands in a script or use /bin/bash -c '...'.

Installing and Updating a Service

bash

sudo useradd --system --no-create-home --shell /usr/sbin/nologin api
sudo nano /etc/systemd/system/api.service
sudo systemctl daemon-reload            # required after creating or editing any unit
sudo systemctl enable --now api
systemctl status api

After every edit to a unit file, run daemon-reload before restarting, otherwise systemd keeps using the old definition. To override a packaged unit without editing it:

bash

sudo systemctl edit nginx        # creates /etc/systemd/system/nginx.service.d/override.conf

bash

[Service]
LimitNOFILE=65536

Hardening Options

A few directives limit the damage a compromised service can do and cost nothing to add.

bash

[Service]
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/srv/api/uploads

systemd-analyze security api scores a unit and lists which options would improve it.

Reading Logs with journalctl

The journal captures stdout, stderr and syslog from every unit, indexed by unit, time and priority.

bash

journalctl -u api                     # everything from one unit
journalctl -u api -f                  # follow, like tail -f
journalctl -u api -n 100 --no-pager   # last 100 lines, plain output
journalctl -u api --since "1 hour ago"
journalctl -u api --since today --until "14:00"
journalctl -u api -p err              # priority err and above (emerg..debug)
journalctl -u api -o json-pretty      # structured fields
journalctl -b                         # since this boot; -b -1 for the previous boot
journalctl -k                         # kernel messages (like dmesg)
journalctl --disk-usage
sudo journalctl --vacuum-size=500M    # trim old logs

Combine filters: journalctl -u nginx -u api -p warning --since yesterday. For services that write their own files (nginx access logs, application log files), the journal only has what went to stdout/stderr — check both.

Common Mistakes

Forgetting daemon-reload after editing a unit, then debugging the old version.
enable without start (or vice versa).
A relative path or shell syntax in ExecStart.
Restart=always on a service that fails instantly, producing a crash loop; add StartLimitIntervalSec and check journalctl -u.
Running the app as root because it needs port 80; put nginx in front, or grant AmbientCapabilities=CAP_NET_BIND_SERVICE.

Key Takeaways

systemctl start|stop|restart|reload|enable|disable|status controls services; enable --now does both enable and start.
Unit files have [Unit], [Service] and [Install]; ExecStart needs an absolute path, User= drops privileges, Restart=on-failure recovers from crashes.
Run daemon-reload after any unit change; use systemctl edit to override packaged units.
journalctl -u name -f, --since, -p err and -b cover almost every log query; --vacuum-size keeps the journal small.
systemd-analyze security scores a unit's sandboxing and suggests hardening directives.

Next lesson: Logs and System Monitoring — read /var/log, watch CPU, memory, disk and network in real time, and spot trouble before users do.