tee, xargs and Command Substitution

The pipes lesson connected the output of one command to the input of the next. This lesson adds the tools that fill the gaps: tee when you need output in two places, xargs when a command wants arguments rather than standard input, command substitution when you need a command's result inside another command, and here-documents when you need to feed multi-line text. Together they let you compose almost any one-off task from existing tools.

tee: Split Output in Two

tee reads standard input, writes it to standard output and to one or more files at the same time. Use it to watch a long process while keeping a log, or to write a file that needs root permissions.

bash

npm run build 2>&1 | tee build.log          # watch and save
./test.sh | tee -a results.log              # -a appends instead of overwriting
echo "127.0.0.1 app.local" | sudo tee -a /etc/hosts

The last example is the standard idiom for appending to a protected file. sudo echo ... >> /etc/hosts fails because the redirection is performed by your unprivileged shell, not by sudo; with tee the privileged process does the writing.

xargs: From Lines to Arguments

Many commands ignore standard input and only act on arguments: rm, mkdir, chmod, gzip, ls. xargs reads whitespace-separated items from stdin and appends them to a command line.

bash

cat packages.txt | xargs sudo apt install -y
find . -name "*.bak" | xargs rm
ls *.jpg | xargs -n 1 echo "processing"     # -n 1: one item per command

Options that matter in practice:

| Option | Purpose | |---|---| | -0 | items are separated by null bytes (pair with find -print0) | | -I {} | substitute each item where {} appears, one command per item | | -n N | at most N items per command | | -P N | run up to N commands in parallel | | -r | do nothing if input is empty (GNU) | | -t | print each command before running it | | -p | prompt before each command |

bash

find . -type f -name "*.png" -print0 | xargs -0 -P 4 -n 10 optipng -quiet
cat urls.txt | xargs -I {} curl -sS -o /dev/null -w "%{http_code} {}\n" {}
git ls-files "*.md" | xargs -r wc -l

-P gives you a simple parallel job runner without any extra software. -0 is not optional when filenames may contain spaces — without it My File.txt becomes two arguments.

Command Substitution

$(command) runs the command and replaces itself with the command's output, minus trailing newlines. The older backtick form `command` does the same but cannot be nested cleanly, so prefer $( ).

bash

today=$(date +%F)
files=$(ls *.log | wc -l)
echo "There are $files log files on $today"

kill $(pgrep -f "node server.js")
cd "$(git rev-parse --show-toplevel)"        # jump to repo root
export PATH="$(npm prefix -g)/bin:$PATH"
for f in $(cat list.txt); do echo "$f"; done  # fine for simple single-word lines

Quote substitutions that produce paths: cd "$(dirname "$0")". Without quotes the shell splits the result on spaces.

Process Substitution

<(command) makes a command's output look like a file, so tools that only accept filenames can read from a pipeline. It is a bash and zsh feature (not POSIX sh).

bash

diff <(sort a.txt) <(sort b.txt)
comm -12 <(sort list1) <(sort list2)          # lines common to both
while read -r line; do echo ">> $line"; done < <(grep ERROR app.log)

The last form is preferable to grep ERROR app.log | while read ... because the loop runs in the current shell, so variables set inside it survive.

Here-Documents and Here-Strings

A here-document feeds a block of text as standard input. Quote the delimiter ('EOF') to prevent variable expansion inside the block.

bash

cat <<EOF > /etc/nginx/conf.d/app.conf
server {
    listen 80;
    server_name $DOMAIN;
}
EOF

sudo tee /etc/systemd/system/app.service > /dev/null <<'EOF'
[Unit]
Description=Demo app
EOF

grep -o "a" <<< "banana" | wc -l   # here-string: a single line as stdin, prints 3

<<-EOF strips leading tab characters so the block can be indented inside a script.

Common Mistakes

sudo echo x > protected-file — the redirect happens as you, not root. Use sudo tee.
xargs rm on find output containing spaces; always -print0 | xargs -0.
ls | xargs ... when the list is empty runs the command with no arguments, sometimes on the current directory. Add -r.
Unquoted $(...) in a path context, which breaks on spaces.
An unquoted here-doc delimiter when the content contains $ you want kept literal.

Key Takeaways

tee duplicates a stream to the screen and a file; sudo tee writes protected files.
xargs converts input lines into arguments; use -0 with find -print0, -I {} for per-item commands and -P for parallelism.
$(command) embeds output in another command; quote it when the result is a path.
<(command) feeds command output to tools that expect filenames.
Here-docs (<<EOF) supply multi-line input; quote the delimiter to stop expansion.

Next lesson: Text Processing: cut, sort, uniq, tr and wc — slice columns, sort, deduplicate and count with the classic Unix text tools.