Pipes and Redirection: The Unix Grammar

The Unix philosophy in one line: programs that do one thing, connected by pipes, each reading stdin and writing stdout. Master that grammar and the shell becomes a programming language you improvise in.

The three streams

• stdin (fd 0) — input. Default: keyboard. From a file: < file.
• stdout (fd 1) — normal output. To a file: > file (overwrite), >> file (append).
• stderr (fd 2) — errors. Redirect separately: 2> errors.log, merge into stdout: 2>&1, discard: 2>/dev/null.

Pipes compose

# find the 10 most common error messages in today's logs
cat app.log | grep ERROR | cut -d' ' -f4- | sort | uniq -c | sort -rn | head -10

Each stage reads the previous stage's output — sort knows nothing about grep. That's the power: wc -l counts lines of anything, head/tail slice any stream, tee splits it (write to file AND keep piping).

The essential redirections

cmd > out.txt 2>&1        # stdout and stderr both to file
cmd >> log.txt 2>&1       # append mode — never overwrite
cmd < input.txt           # feed a file to stdin
cmd | other_cmd           # stdout of cmd → stdin of other_cmd
cmd && next               # run next only if cmd succeeded
cmd || fallback           # run fallback only if cmd failed
cmd1; cmd2                # run both regardless

::lab(pipeline-drill)

Field notes

• > truncates before the command runs — cat file > file gives you an empty file. Use sponge (from moreutils) or write to a temp file.
• xargs bridges pipes to commands that take args, not stdin: find . -name "*.log" | xargs rm.
• A pipeline is only as fast as its slowest stage — sort buffers everything before emitting; for huge streams, use sort -S to control buffer size or restructure to avoid sorting.