Pipes, Redirection and Command Chaining

This is the lesson where the command line stops being a list of separate tools and becomes a construction kit. Unix commands are designed to read a stream of text in and write a stream of text out; redirection points those streams at files, and pipes connect one command’s output to another’s input. Together they let you build data pipelines in a single line.

The Three Standard Streams

Every process starts with three streams:

stdin (0) – standard input, normally your keyboard
stdout (1) – standard output, normally your screen
stderr (2) – standard error, also your screen, but a separate channel

Keeping errors on their own stream means you can save results to a file while still seeing problems—or silence one without the other.

Redirecting Output: > and >>

bash

ls -l > listing.txt        # write stdout to a file (overwrites!)
date >> events.log         # append to a file
echo "# My Notes" > notes.md
wc -l < notes.md           # < feeds a file to stdin

One character difference, big consequence: > truncates the file first, >> adds to the end. A useful idiom: > file on its own empties a file without deleting it.

Redirecting Errors

bash

find / -name "*.conf" 2> errors.txt      # errors to a file
find / -name "*.conf" 2> /dev/null       # discard errors entirely
make > build.log 2>&1                     # both streams into one file

/dev/null is the system’s black hole—anything written there vanishes. 2>&1 means "send stderr wherever stdout is going" and must come after the > redirection.

Pipes: Connecting Commands

The pipe | feeds one command’s stdout into the next command’s stdin, with no temporary file:

bash

ls -l | less                        # page through a long listing
grep "GET" access.log | wc -l       # count GET requests
history | tail -n 20                # your last 20 commands
ps aux | grep nginx                 # find a process

Pipeline Workhorses: sort, uniq, cut, tr

A few small filters appear in almost every pipeline:

bash

sort names.txt              # alphabetical
sort -n sizes.txt           # numeric sort
sort -rn | head             # biggest first
uniq -c                     # collapse adjacent duplicates, with counts
cut -d: -f1 /etc/passwd     # field 1 of colon-separated lines
tr "a-z" "A-Z"              # translate characters

uniq only removes adjacent duplicates, so the classic pattern is sort | uniq.

A Real Pipeline, Built Step by Step

Question: which IP addresses hit our web server most often?

bash

cut -d" " -f1 access.log \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -n 10

Read it left to right: extract the first field (the IP), sort so duplicates sit together, count each run, sort counts descending, keep the top ten. This is how experts build pipelines too—one stage at a time, checking output after each | before adding the next.

Chaining Commands: ;, && and ||

Separate from pipes, you can sequence whole commands:

bash

cd /tmp; ls                        # run both, regardless of success
mkdir build && cd build            # run 2nd only if 1st succeeded
ping -c1 example.com || echo down  # run 2nd only if 1st FAILED
make && echo OK || echo FAILED     # common success/failure pattern

&& is the safety belt of one-liners: sudo apt update && sudo apt upgrade never upgrades from a stale package list.

tee: Watch and Save at Once

bash

./deploy.sh | tee deploy.log       # show output AND write it to a file
echo "opt=1" | sudo tee -a /etc/app.conf   # append to a root-owned file

The second form solves a classic puzzle: sudo echo x > file fails because the shell (not sudo) opens the file—tee runs under sudo and does the writing.

Practice Session

bash

cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn
ls /etc | wc -l
dmesg 2>/dev/null | tail -n 5
echo "hello pipeline" | tr "a-z" "A-Z" | tee shout.txt

Key Takeaways

> overwrites, >> appends, 2> redirects errors, 2>&1 merges streams.
| connects commands into pipelines without temporary files.
sort, uniq -c, cut and head combine into powerful text analytics.
&& and || chain commands on success or failure; tee saves while displaying.

Next lesson: Editing Files with nano and vim Basics — finally, changing files in place.