Lesson 2 of 9 · 22m

Text Processing: grep, sed, awk, jq

The four text tools that cover 90% of data wrangling — pattern matching, stream editing, field extraction, and JSON querying.

Most log files, CSVs, and JSON payloads are answerable with one of four tools. Knowing which one — and the one or two flags that matter — is the difference between 30 seconds and 30 minutes.

grep: find lines

grep -r "TODO" src/           # recursive
grep -i "error" app.log       # case-insensitive
grep -v "debug" app.log       # invert — lines NOT matching
grep -c "GET" access.log      # count matches
grep -A 3 "Exception" log     # 3 lines of context after
grep -P 'user_id=\d+' log     # Perl regex

rg (ripgrep) is grep's modern replacement — faster, .gitignore-aware, defaults you'll never fight.

sed: stream edit

sed 's/foo/bar/g' file        # substitute — s/old/new/g
sed -i 's/foo/bar/g' file     # in-place (dangerous — test first)
sed -n '10,20p' file          # print lines 10-20
sed '/^#/d' config.conf       # delete comment lines

awk: fields and columns

awk '{print $1, $3}' file                 # columns 1 and 3
awk -F, '{sum += $2} END {print sum}' csv # sum column 2
awk '$3 > 100' data.txt                   # filter rows where col 3 > 100

awk is a tiny language — BEGIN, main pattern-block per line, END. The pattern is condition { action }.

jq: JSON that doesn't require Python

jq '.items[] | .name' data.json                    # extract a field
jq '[.users[] | select(.active)] | length' data   # filter + count
jq -r '.id' file.json | xargs -I{} echo {}        # raw strings for piping

::exercise(count-by-field)

Field notes

less +F log follows a file like tail -f but with search (/ to find patterns) — the best log pager you already have.
column -t aligns whitespace-separated output into readable columns; csvlens or csvlook for CSVs.
The tool that knows one trick well beats the tool that knows everything badly — jq for JSON, yq for YAML, xsv/csvkit for CSV.