Background Jobs, nohup and Signals

The process management lesson showed how to see and stop processes. This one is about controlling the ones you start yourself: running a command in the background while you keep typing, pausing and resuming it, keeping it alive after you log out, and understanding the signals that kill, Ctrl+C and the system send. These skills matter most over SSH, where closing a laptop lid can otherwise terminate a two-hour import.

Foreground and Background

By default a command runs in the foreground: the shell waits for it to finish and your prompt is unavailable. Appending & starts it in the background and returns the prompt immediately. Background jobs still write to your terminal, so redirect their output.

bash

./build.sh &                          # prints [1] 4821 — job number and PID
./build.sh > build.log 2>&1 &         # quieter
sleep 300 &
jobs                                  # list this shell's jobs

bash

[1]-  Running    ./build.sh > build.log 2>&1 &
[2]+  Running    sleep 300 &

+ marks the current job (the default for fg and bg), - the previous one. Jobs are referred to as %1, %2, and so on; %% is the current job.

Suspending, Resuming and Switching

Ctrl+Z sends SIGTSTP, which pauses the foreground program and returns you to the prompt. From there you can resume it in the background or bring it back.

bash

vim notes.md      # press Ctrl+Z inside vim
jobs              # [1]+  Stopped   vim notes.md
bg                # continue it in the background (not useful for vim, useful for builds)
fg %1             # bring it to the foreground again
kill %1           # terminate a job by job number
wait %1           # block until job 1 finishes; wait with no argument waits for all

A common workflow is to start a server, Ctrl+Z, bg, run a quick test in the same shell, then fg to watch the logs again.

Surviving Logout: nohup and disown

When your terminal closes, the shell sends SIGHUP (hang up) to its jobs and they exit. Two tools prevent that.

bash

nohup ./long-task.sh > task.log 2>&1 &     # immune to SIGHUP from the start

nohup ignores the hangup signal and, if you do not redirect output, writes it to nohup.out. For a job you already started without it:

bash

./long-task.sh &
disown -h %1        # mark the job so the shell does not send it SIGHUP

For anything that must keep running across reboots, or that needs restarts on failure, use a systemd service (a later lesson) rather than nohup. For interactive work that must survive disconnects, use tmux, which has its own lesson.

Signals: How Processes Are Told to Do Things

A signal is a numbered notification the kernel delivers to a process. The process can catch most of them and react — save state, reload config, clean up — but two cannot be caught at all.

| Signal | Number | Sent by | Default effect | |---|---|---|---| | SIGHUP | 1 | terminal closing | terminate; daemons reinterpret as "reload config" | | SIGINT | 2 | Ctrl+C | terminate | | SIGQUIT | 3 | Ctrl+\ | terminate with core dump | | SIGKILL | 9 | kill -9 | terminate immediately, cannot be caught | | SIGTERM | 15 | kill (default) | terminate gracefully | | SIGSTOP | 19 | kill -STOP | pause, cannot be caught | | SIGCONT | 18 | kill -CONT, fg, bg | resume | | SIGTSTP | 20 | Ctrl+Z | pause (catchable) | | SIGUSR1/SIGUSR2 | 10 / 12 | applications | user-defined |

bash

kill 4821                 # SIGTERM: the process may finish requests and exit cleanly
kill -15 4821             # same
kill -9 4821              # SIGKILL: last resort; no cleanup, possible corrupted files
kill -l                   # list all signal names
kill -HUP $(cat /run/nginx.pid)     # reload nginx without downtime (signal the master, not workers)
killall -TERM node        # every process named node
kill -STOP 4821; kill -CONT 4821    # freeze and thaw

Always try SIGTERM first and wait a few seconds. Databases, web servers and anything writing files rely on it to flush and close cleanly. Reach for -9 only when a process ignores TERM.

Watching What Is Running: htop

top ships everywhere, but htop (sudo apt install htop) is easier to read: colour bars for CPU cores and memory, a tree view (F5), sorting by column (F6), filtering by name (F4), and you can send a signal to the highlighted process with F9 without knowing its PID. Space tags several processes to act on at once, and q quits.

Exit Status After a Signal

A process killed by a signal exits with status 128 + signal number. 130 therefore means "interrupted with Ctrl+C" and 137 means "killed with SIGKILL" — a code you will see when a container exceeds its memory limit and the kernel's OOM killer intervenes.

bash

sleep 100      # press Ctrl+C
echo $?        # 130

Common Mistakes

Starting a server with & over SSH, logging out and finding it dead. Use nohup, disown, tmux or systemd.
Using kill -9 by default, leaving lock files and half-written data behind.
Forgetting to redirect output of a background job, so it scribbles over your prompt.
Confusing job numbers (%1) with PIDs (4821); kill 1 would target PID 1, which is init.

Key Takeaways

& runs a command in the background; jobs, fg, bg and %n manage the jobs of the current shell.
Ctrl+Z suspends, bg resumes in the background, fg brings a job back.
nohup cmd & or disown -h keeps a job alive after logout; systemd and tmux are the robust alternatives.
SIGTERM (15) asks a process to exit cleanly; SIGKILL (9) forces it and cannot be caught; SIGHUP often means reload.
Exit status 128 + n means the process died from signal n.

Next lesson: Package Managers: apt and yum — install, update and remove software from your distribution's repositories.