Capstone: Set Up a Production Server from Scratch

This chapter is a project. Starting from a blank Ubuntu server with only a root password or cloud key, you will end with a hardened machine that serves an application over HTTPS, restarts it on failure, backs itself up nightly, and can be rebuilt from a script. Every step uses a command from an earlier lesson; the value here is the order, the checks between steps, and the habit of writing down what you did. Budget an hour, use a throwaway cloud VM, and destroy it when you are done.

Phase 0: Plan and Record

Before touching the server, write a short runbook — a text file in a repository — containing the hostname, IP, provider, purpose, the user you will create, and the ports you intend to expose. Add every command you run to it as you go. Six months from now this file is the difference between a five-minute rebuild and a lost weekend.

Phases 1–3: Base System

Phase 1: First Login and Updates

bash

ssh root@203.0.113.10            # or ubuntu@ with a cloud key
cat /etc/os-release && uname -r
apt update && apt -y upgrade
[ -f /var/run/reboot-required ] && reboot
hostnamectl set-hostname web-01
timedatectl set-timezone UTC

Check: hostname prints web-01, date shows UTC, apt list --upgradable is empty.

Phase 2: Your Own User and SSH Keys

bash

adduser alice
usermod -aG sudo alice
rsync --archive --chown=alice:alice ~/.ssh /home/alice     # copy root's authorized key

From your laptop, confirm the new account works before locking down SSH:

bash

ssh alice@203.0.113.10 'sudo -n true && echo sudo-ok'

Then, on the server:

bash

sudo nano /etc/ssh/sshd_config.d/hardening.conf

bash

PasswordAuthentication no
PermitRootLogin no
KbdInteractiveAuthentication no
MaxAuthTries 3

bash

sudo sshd -t && sudo systemctl reload ssh

Check: a second terminal can still log in as alice; ssh root@... is refused.

Phase 3: Firewall, fail2ban and Automatic Updates

bash

sudo apt install -y ufw fail2ban unattended-upgrades
sudo ufw default deny incoming && sudo ufw default allow outgoing
sudo ufw limit OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable
sudo ufw status verbose

printf '[DEFAULT]\nbantime = 1h\nfindtime = 10m\nmaxretry = 5\n\n[sshd]\nenabled = true\n' | sudo tee /etc/fail2ban/jail.local
sudo systemctl enable --now fail2ban
sudo dpkg-reconfigure -plow unattended-upgrades

Check: sudo ss -tulpn shows only 22 listening (and later 80/443); sudo fail2ban-client status sshd reports the jail active.

Phases 4–5: Application and Backups

Phase 4: The Application

Follow the previous lesson: install Node and Nginx, create the app system user, clone the code into /srv/app/current, write app.service, enable it, configure the Nginx site, run certbot. Then verify from outside:

bash

curl -sI https://app.example.com | head -1        # HTTP/2 200
curl -s https://app.example.com/health             # {"status":"ok"}
sudo systemctl is-active app nginx                 # active active

Deliberately break something to see the safety net work: sudo kill -9 $(systemctl show -p MainPID --value app) and watch journalctl -u app -f show the restart within five seconds.

Phase 5: Backups and Scheduled Jobs

bash

sudo mkdir -p /backups && sudo chown alice:alice /backups
cat > /home/alice/bin/backup.sh <<'EOF'
#!/usr/bin/env bash
set -Eeuo pipefail
today=$(date +%F)
dest=/backups
rsync -a --delete --link-dest="$dest/latest" /srv/app/current/ "$dest/$today/app/"
ln -sfn "$dest/$today" "$dest/latest"
find "$dest" -maxdepth 1 -type d -name '20*' -mtime +14 -exec rm -rf {} +
echo "backup $today ok"
EOF
chmod +x /home/alice/bin/backup.sh

Schedule it with a systemd timer at 02:30 with Persistent=true (cron works too), run it once by hand, and restore one file from /backups/latest to /tmp to prove it. Then sync /backups off the machine with rsync to a second host or object storage.

Phase 6: Rebuild Test

The proof that the runbook is complete is a second server built from it. Turn Phases 1–3 into bootstrap.sh (see the sample above), create a fresh VM, run the script, and time how long it takes to reach a green health check. Anything you had to do by hand is a missing line in the script. Destroy both VMs when finished.

Acceptance Checklist

| Check | Command | Expected | |---|---|---| | Root SSH blocked | ssh root@host | Permission denied | | Password SSH blocked | ssh -o PubkeyAuthentication=no alice@host | Permission denied | | Only 22/80/443 open | sudo ss -tulpn | three listeners | | HTTPS valid | curl -sI https://app.example.com | HTTP/2 200 | | App restarts on crash | kill the PID, systemctl status app | active within 5 s | | Backup exists and restores | ls /backups/latest, copy a file | matches source | | Updates automatic | cat /etc/apt/apt.conf.d/20auto-upgrades | both "1" | | fail2ban active | sudo fail2ban-client status sshd | jail listed |

Common Mistakes

Locking down SSH before verifying the new user can log in and use sudo.
Enabling ufw before allowing SSH.
Running the app or the deploy from /root or /home/alice instead of /srv/app.
Backups that were never restored, or stored only on the same disk.
Doing it all interactively and never writing the runbook, so the second server takes as long as the first.

Key Takeaways

Order matters: updates, then a sudo user with keys, then SSH lockdown, then firewall and fail2ban, then the app.
Verify between phases and keep a second SSH session open while changing SSH or the firewall.
Run the app as its own user under systemd behind Nginx with HTTPS from certbot; prove the restart by killing it.
Back up with rsync --link-dest, schedule it, restore a file, and copy backups off the machine.
Write the runbook as you go and turn it into bootstrap.sh; the rebuild test is the real acceptance criterion.

Next lesson: Linux Command Line Cheat Sheet and Interview Questions — consolidate the whole course into a one-page reference and practise the questions interviewers ask.