Bash and the command line

Eleven ideas that the rest of the document assumes. Read these once and most bash behaviour stops being arbitrary.

What the shell actually is

Three things people call 'the terminal', and why only one of them runs your code

Three separate programs are involved every time you type a command, and confusing them is the source of a surprising number of stuck moments.

The terminal emulator — iTerm, GNOME Terminal, the VS Code panel — is a window that draws characters and captures keystrokes. It knows nothing about bash. Its ancestor was a physical device with a keyboard and a screen wired to a serial line, and the emulation is faithful enough that terminal control codes, window size signals and the idea of a 'tty' all survive from that era.

The shell — bash, zsh, fish — is an ordinary program running inside that window. It reads a line of text, transforms it according to a fixed set of rules, and asks the kernel to run programs on your behalf. That is the whole job.

Bash has no special privileges; it is a userspace binary at /usr/bin/bash like any other.

The kernel is what actually creates processes, opens files and moves bytes between them. The shell cannot do any of those things itself. It can only ask, via system calls, and report back what happened.

So the shell is a read-eval-print loop where 'eval' means 'expand this text, then ask the kernel to execute the result'. Everything in this document is either a rule about the expansion, or a program the shell can be asked to run.

Builtins, functions, and real programs

Most commands are separate executables on disk. ls is /usr/bin/ls; the shell finds it, asks the kernel to run it, and waits. But some commands are builtins, implemented inside bash itself, and the reason is usually that they need to change the shell's own state.

cd is the classic example. Directory is a property of a process. If cd were an external program, the kernel would start it, it would change its own directory, and then it would exit — leaving your shell exactly where it was.

So cd has to be a builtin. The same logic explains export, read, source, trap, set and exit.

type -a echo
# echo is a shell builtin
# echo is /usr/bin/echo

type -a cd
# cd is a shell builtin

type -a ll
# ll is aliased to `ls -lah`

Which shell, which config file

Bash reads different startup files depending on how it was started, and this catches everyone eventually. A login shell (SSH session, console login) reads /etc/profile then the first of ~/.bash_profile, ~/.bash_login, ~/.profile. A non-login interactive shell (a new terminal tab) reads ~/.bashrc.

A non-interactive shell (running a script) reads neither, unless BASH_ENV is set.

The alias you added to .bashrc works in your terminal but not over ssh host 'command'. Many .bash_profile files end with a line that sources .bashrc, to paper over the distinction.

Processes, and why children cannot change their parents

fork, exec, and the one-way flow of environment

When bash runs an external command it does two things. First it forks: the kernel makes a near-identical copy of the bash process, with a new process ID. Then, in the copy, it execs: the process replaces its own program image with the new binary, keeping its PID, its open file descriptors and its environment.

The parent waits for the child to finish and collects its exit status.

Almost everything surprising about shell scope falls out of that one mechanic. The child got a copy of the parent's environment. Nothing the child does can reach back up.

# In the child, x is set. In the parent, it never existed.
( x=5 )
echo "${x:-unset}"      # unset

# Same story with cd
( cd /tmp; pwd )        # /tmp
pwd                     # unchanged

An ordinary variable lives only in the current shell. export exists to mark it for copying into the environment of any child the shell creates. There is no reverse operation, because there is no mechanism for one: a child cannot write into a parent's memory.

Where subshells hide

You get a subshell explicitly with ( ... ), but bash also creates them silently in several places, and that silence is where the bugs live:

• Every stage of a pipeline runs in its own process. a | b | c is three children.
• Command substitution $(...) runs in a subshell whose stdout is captured.
• Backgrounding with & creates one.
• So does a command group that is piped as a whole. Redirection alone does not: { ...; } > file and while ...; done < file both run in the current shell, which the < <(...) fix below relies on.

Combine the first of those with a loop and you get the most famous bash surprise there is:

count=0
grep ERROR app.log | while read -r line; do
  (( count++ ))
done
echo "$count"     # 0. Always 0.

The fix is not to make the loop smarter; it is to stop putting the loop in a child. Process substitution feeds the same data in through a redirect instead of a pipe, which keeps the loop in the current shell:

count=0
while read -r line; do
  (( count++ ))
done < <(grep ERROR app.log)
echo "$count"     # correct

source versus execute

./script.sh forks and execs: a new bash reads the file, and whatever it sets dies with it. source script.sh (or . script.sh) reads the file into the current shell, as if you had typed it. Environment setup files must be sourced for the change to survive, and a script that ends in exit will close your terminal if you source it instead of running it.

Everything is a file descriptor

The table behind redirection, pipes, and the ordering rule people get wrong

Every process has a small numbered table of open things. Entry 0 is standard input, 1 is standard output, 2 is standard error. A process reads from 0 and writes to 1 and 2 without knowing or caring what is on the other end — a terminal, a file, a socket, another process.

That indifference is what makes redirection possible at all.

Redirection is the shell rearranging that table in the forked child, in the instant between fork and exec. The program itself is never told about it. sort < data.txt does not pass a filename to sort; it opens the file, wires it to descriptor 0, and then runs a program that reads from descriptor 0 exactly as it always does.

Why 2>&1 has to come last

Redirections are processed strictly left to right, and 2>&1 means make descriptor 2 point at whatever descriptor 1 currently points at. It copies a destination; it does not create a link. Read it that way and the classic confusion resolves itself:

cmd > out.log 2>&1
# 1. stdout -> out.log
# 2. stderr -> wherever stdout is now = out.log      both captured

cmd 2>&1 > out.log
# 1. stderr -> wherever stdout is now = the terminal
# 2. stdout -> out.log                               stderr still on screen

The second form is not a mistake in all cases — it is how you send errors to the screen while piping clean output onward — but it is almost never what someone typing it meant.

Note also that the shell opens the target file before the command runs. sort file.txt > file.txt truncates the file to zero bytes and then asks sort to read it. Write output to a new name and move it into place.

A pipe is a buffer with two ends

a | b asks the kernel for a pipe: a small in-memory buffer, typically 64KB, with a write end and a read end. The shell wires a's descriptor 1 to the write end and b's descriptor 0 to the read end, then starts both processes at the same time. They are not sequential.

If b is slow, the buffer fills and a blocks on write; if a is slow, b blocks on read. That is backpressure, for free.

It also explains why head on a huge file is instant: once head has its ten lines it exits, the read end closes, the writer gets SIGPIPE on its next write, and the whole thing stops.

Process substitution

<(cmd) runs a command and hands you a filename — usually /dev/fd/63 — that reads from its output. This lets you pass a command's output to a program that insists on file arguments, which is most of them.

diff <(ssh prod cat /etc/app.conf) <(cat local.conf)
comm -23 <(sort a.txt) <(sort b.txt)

The expansion pipeline

The most useful thing to understand about bash

Before bash runs anything, it rewrites the line you typed. It does this in a fixed order, and nearly every bash bug that feels like witchcraft is a misunderstanding of that order. The stages are:

• Brace expansion — {1..5}, {a,b}c
• Tilde expansion — ~, ~user
• Parameter expansion — $var, ${var#pat}, ${var:-default}
• Command substitution — $(cmd)
• Arithmetic expansion — $(( 2 + 2 ))
• Word splitting — the result is chopped at every character in IFS
• Filename expansion (globbing) — any word containing *, ? or [ is matched against the filesystem
• Quote removal — the quotes themselves are stripped off before exec

Stages 6 and 7 are the dangerous ones, and they happen to the result of the earlier stages. Bash does not know that the text now sitting there came from a variable. It just sees words.

file="my report.txt"
rm $file
# after stage 3:  rm my report.txt
# after stage 6:  rm  [my] [report.txt]      two arguments, two wrong filenames

rm "$file"
# stages 6 and 7 skipped:  rm [my report.txt]   one argument, correct

That is the entire purpose of double quotes: they turn off word splitting and globbing for whatever is inside. The rule that follows is to quote every expansion, not just the ones that might contain a space. You rarely know in advance, and the failure is silent.

Ordering consequences

Because brace expansion is stage 1 and parameter expansion is stage 3, {1..$n} cannot work: by the time bash knows what $n is, the brace stage is long over. Use a C-style loop or seq.

Because word splitting is stage 6, unquoted command substitution gets shredded the same way variables do. for f in $(ls) breaks on any filename with a space, which is one of several reasons not to parse ls.

Because globbing is stage 7 and happens against the real filesystem, a pattern that matches nothing is, by default, passed through literally. for f in *.log on a directory with no logs runs the loop once with f set to the string *.log. shopt -s nullglob makes non-matching globs vanish instead, which is almost always what you wanted.

The four kinds of quoting

• No quotes — everything expands, then splits and globs.
• Double quotes "..." — $, backticks and \ still work; splitting and globbing do not.
• Single quotes '...' — nothing at all is special. You cannot put a single quote inside.
• $'...' — no expansion, but backslash escapes are interpreted, so $'\n' is a real newline and $'\t' a real tab.

Exit status is the type system

One integer, and every control structure built on it

Every process that ends returns a number between 0 and 255 to its parent. Zero means success. Anything else means a failure, and the specific value is the program's way of saying which one.

This is inverted relative to boolean logic in every programming language you know, and it is inverted for a good reason: there is exactly one way to succeed and many ways to fail, so success gets the single value.

Bash's control flow reads nothing but this integer. if cmd; then runs the body when cmd exits 0. while cmd; do loops while it keeps exiting 0.

a && b runs b only if a exited 0; a || b runs b only if a did not. ! inverts it.

There is no boolean type hiding underneath. [[ -f x ]] is a command that exits 0 or 1, and if treats it like any other.

if grep -q ERROR app.log; then
  echo 'found errors'
fi

# same thing, no if:
grep -q ERROR app.log && echo 'found errors'

curl -fsS "$url" || die 'request failed'

Conventions worth knowing

• 0 success
• 1 generic failure
• 2 misuse — bad arguments, usually
• 126 found but not executable
• 127 command not found
• 128+N killed by signal N, so 130 is Ctrl-C (SIGINT is 2) and 137 is SIGKILL (9)

Seeing 137 in a container's logs is not a crash; it is the kernel or the OOM killer telling you the process was killed outright. 143 is a clean SIGTERM. Reading these saves a lot of guessing.

What set -e really does

set -e tells bash to exit as soon as a command returns non-zero. The rule has an important exemption: a command whose status is already being examined does not trigger it. That covers the condition of if, while and until, anything to the left of && or ||, and anything negated by !.

The exemption is what makes set -e usable at all. Without it, every if grep -q would kill your script. But it also creates the trap that catches people:

set -e

# Neither of these exits. set -e ignores the LEFT side of &&.
(( VERBOSE )) && set -x
if (( VERBOSE )); then set -x; fi

# THIS exits when VERBOSE is 0 — bare, with nothing examining it.
(( VERBOSE ))

The second trap is that a function's failure is only seen where the function is called. And the third is that local x=$(cmd) always succeeds, because the status you get back is local's, not the command's. Declare first, assign on the next line, when the status matters.

Pipelines and pipefail

A pipeline's exit status is, by default, the status of its last command. So curl bad-url | jq . exits 0, because jq successfully processed nothing. set -o pipefail changes this to the rightmost non-zero status, which is what almost everyone assumed was happening.

${PIPESTATUS[@]} holds the individual statuses if you need to know which stage broke.

Text as the universal interface

Why so few tools compose so widely, and where the model gives out

Unix made an unusual bet: instead of rich data types passed between programs, everything is a stream of bytes, and by convention those bytes are lines of text. Any tool can therefore be attached to any other.

There are no adapters, no schemas, no serialisation format to agree on. sort does not know or care whether it is sorting hostnames, log lines or CSV.

The price is that structure is not carried by the format. It is carried by convention instead, and conventions can be violated. That is where the sharp edges are.

Where the model breaks

The record separator is the newline and the field separator is whitespace. Filenames on Linux may contain both. So the moment a filename has a space in it, whitespace-separated tools mis-split it; and a filename containing a newline can break a line-oriented pipeline entirely.

This is not theoretical. It is what happens when someone drags a file called flight test 2026.csv onto a server.

The escape hatch is the null byte, which is the one character a filename cannot contain. Tools that deal in filenames grew matching options for it:

find . -name '*.log' -print0 | xargs -0 gzip
grep -rlZ 'TODO' src/ | xargs -0 sed -i 's/TODO/DONE/'

while IFS= read -r -d '' f; do
  echo "$f"
done < <(find . -type f -print0)

The tool ladder

The classic text tools form a ladder of increasing power, and the skill is knowing when to stop climbing:

• cut / tr — fixed columns and character swaps. No logic. Fast to type, fails on aligned whitespace.
• grep — select lines by pattern. Answers 'which lines'.
• sed — a line editor as a stream. Answers 'change these lines'.
• awk — a small programming language with automatic field splitting, variables, arrays and arithmetic. Answers 'compute something per line, then summarise'.
• Python / a real program — when you need data structures, error handling, or anything you will still be maintaining in six months.

A useful heuristic: if the awk program has grown past one line, or you have started nesting sed expressions, you have climbed one rung too high and should write a script.

Regex is three languages wearing a trenchcoat

BRE (basic) is what grep and sed use by default, and it requires backslashes for +, ?, ( and {. ERE (extended) is what grep -E, sed -E and awk use, where those characters work as you expect. PCRE (grep -P, most languages) adds lookarounds, lazy quantifiers and \d.

rg does not belong in that group by default. Its Rust engine has no lookaround or backreferences at all, and needs rg -P to get them. When a pattern mysteriously fails, dialect is the first thing to check.

Structured data needs a structured tool

JSON is not line-oriented, and every attempt to parse it with grep and sed works right up until a field contains a brace, a value spans lines, or the key order changes. jq exists because it understands the grammar: it parses JSON into a real tree, applies a filter language to it, and emits JSON or raw text. The same argument gives you yq for YAML.

Reach for them from the first attempt, not after the regex has failed twice.

Files, names, and permissions

Inodes, links, and why a deleted file can still fill your disk

A file's data and metadata live in an inode. A filename is not part of the file: it is an entry in a directory that points at an inode.

A directory is just a file containing a list of names and inode numbers. This indirection is not a detail; it explains most filesystem behaviour that seems strange.

A hard link is a second directory entry pointing at the same inode. Both names are equally real; neither is the original. The inode keeps a link count, and the data is freed only when that count reaches zero.

A symlink is a different thing entirely: a tiny file whose contents are a path, resolved at access time. It can cross filesystems, can point at nothing, and breaks if the target moves.

Deleting an open file

The kernel frees an inode's blocks when both the link count is zero and no process has it open. rm on a 40GB log that nginx still has open removes the name, so du can no longer see it. But the space is not returned until nginx closes the descriptor or restarts.

df -h /var          # 100% full
du -sh /var/log     # only 2G accounted for

sudo lsof -nP | grep '(deleted)'
# nginx  1204  www-data  2w  REG  8,1  41231872000  /var/log/nginx/access.log (deleted)

Rename is atomic, and that is a deployment strategy

Renaming within a filesystem is a single atomic operation: readers see either the old name or the new one, never a half-written state. The safe way to write a config file follows from this: write config.new, then mv it into place. The symlink-swap release pattern relies on the same guarantee: you point current at a fully-built directory in one instant.

ln -sfn /srv/releases/2026-08-20 /srv/current
# -n stops ln following the existing symlink and creating
#    /srv/current/2026-08-20 instead

Reading permission bits

-rwxr-x--- is a type character followed by three triplets: owner, group, everyone else. Each is read, write, execute, numbered 4, 2, 1, so 750 means owner gets 7 (rwx), group gets 5 (r-x), others get 0.

On a directory, the bits mean something different and it trips people up. Read means you may list the names. Execute means you may traverse into it and access things by name.

A directory with r-- lets you see the names but stat nothing; one with --x lets you open a file you already know the path of, but not list the contents. Execute is the one that matters.

Signals, sessions, and staying alive

What kills your job when you close the laptop, and the three ways to stop it

A signal is a small asynchronous notification the kernel delivers to a process. Most can be caught and handled; two cannot. SIGTERM (15) is a polite request to shut down: the process can flush buffers, close connections and exit cleanly.

SIGKILL (9) is not delivered to the process at all; the kernel destroys it outright, with no chance to clean up. SIGINT (2) is what Ctrl-C sends. SIGHUP (1) originally meant the modem hung up.

The order to try is always TERM, wait a few seconds, then KILL. Reaching for kill -9 first is how you end up with corrupt files and stale lock files.

Why Ctrl-C stops the whole pipeline

Your terminal has a foreground process group, and Ctrl-C sends SIGINT to every process in it. So a | b | c dies as a unit instead of leaving orphans behind.

When your SSH connection drops, the kernel sends SIGHUP to the session's processes, and the default action for SIGHUP is to terminate. That is the mechanism behind 'my long job died when my laptop slept'.

Three answers, for three different problems

• nohup / disown / setsid — detach the process from the terminal so SIGHUP never reaches it. Cheap, but you cannot reattach, and you get whatever you redirected to a file and nothing more.
• tmux / screen — the session lives in a server process that owns its own pseudo-terminals. Your client attaches and detaches at will; the programs inside never notice. This is what you want for interactive work, or when you might need to look at something and type a response.
• systemd unit — the process is owned by the init system, starts at boot, restarts on failure, logs to the journal, and has resource limits. This is what you want for anything that should still be running next month.

The choice is really about who should own the process. If the answer is 'me, for the next hour', use tmux. If it is 'the machine, indefinitely', write a unit file.

# right now, and I might want to watch it
tmux new -d -s import './import.sh > import.log 2>&1'
tmux attach -t import

# forever, and it should come back after a reboot
sudo systemctl enable --now app.service

Trapping signals in a script

trap installs a handler. Handling INT and TERM lets a script kill its children and remove its lock file before dying, instead of leaving debris. You cannot trap KILL or STOP: that is the point of them.

trap 'echo "shutting down" >&2; kill 0; exit 130' INT TERM
trap 'rm -rf -- "$TMP_DIR"' EXIT

Where a command comes from

PATH resolution, and the reason your cron job cannot find anything

When you type a name, bash resolves it in a strict order: aliases first, then shell functions, then builtins, then a search along PATH. PATH is a colon-separated list of directories, searched left to right, first match wins. It is not recursive and it does not look in the current directory.

Bash caches the results in a hash table, so installing a new binary sometimes does not take effect until you run hash -r or open a new shell.

type -a python3          # every meaning, in resolution order
command -v jq            # the path bash would actually use
echo "$PATH" | tr ':' '\n'

The environment is inherited, not global

There is no global environment on a Unix system. Each process has its own copy, received from its parent at exec time.

So an environment variable is only visible to things started after it was exported, by that shell or its descendants. Setting something in one terminal has no effect on another, and a running service will not pick up a change to /etc/environment until it restarts.

Why it works in your shell and fails in cron

This is the most common Linux support question, and it has one answer. Cron does not run your shell's startup files.

A cron job gets a minimal environment — often just PATH=/usr/bin:/bin, HOME, SHELL and LOGNAME. No nvm, no virtualenv, no direnv, none of your aliases, and a PATH that is missing /usr/local/bin.

The same applies, for the same reason, to systemd units, Docker CMD entries, and ssh host 'command'. The fix is always one of three things: use absolute paths, set the environment explicitly, or source the profile you depend on.

# fragile
*/5 * * * * sync-logs.sh

# survives
*/5 * * * * /usr/local/bin/sync-logs.sh >> /var/log/sync.log 2>&1

# or, in the script itself
export PATH="/usr/local/bin:/usr/bin:/bin"
source /srv/app/.venv/bin/activate

Debugging a machine, in layers

A fixed order of questions, so you stop guessing

When something is down, the useful skill is not knowing more commands. It is asking the questions in an order that halves the search space each time.

A request to your service passes through a stack of layers, and each layer has one tool that tests it in isolation. Work outward from the process, or inward from the client, but do work in order.

Is the process even alive?

systemctl status app answers this and shows the last few log lines. If it is restarting in a loop, journalctl -u app -n 100 shows why. Exit code 137 means it was killed: check dmesg -T | grep -i 'killed process' for the OOM killer before assuming a crash.

Is it listening, and on what?

ss -tulpn shows every listening socket and the process that owns it. The common finding here is a service bound to 127.0.0.1 when it needed to be on 0.0.0.0, which looks completely healthy from the box and completely dead from anywhere else.

Can I reach the port?

nc -zv host port from the client side tests reachability with no application logic involved. A hang means a firewall is dropping packets; an immediate 'connection refused' means the packet arrived and nothing was listening. Those are different problems with different owners.

Is the name resolving to what I think?

dig +short name from the machine that is failing. Stale DNS, split-horizon resolvers and an /etc/hosts entry someone added during an incident three months ago all show up here.

Is TLS the problem?

openssl s_client -connect host:443 -servername host shows the certificate actually being served, its chain and its dates. Expired certificates and a missing intermediate are the two findings, and the second one is invisible in a browser that has the intermediate cached.

What does the application say?

Only now do you look at HTTP. curl -v shows the full exchange; curl -o /dev/null -sw '%{http_code}\n' gives you just the status. A 502 from nginx means nginx is fine and the upstream is not — which sends you back to the first question, one layer down.

systemctl is-active app          # 1. alive?
ss -tulpn | grep 8000            # 2. listening?
nc -zv 10.0.4.13 8000            # 3. reachable?
dig +short api.example.com       # 4. resolving?
curl -sI https://api.example.com # 5. answering?
journalctl -u app -p err -n 50   # 6. complaining?

The resource questions, in parallel

• Disk — df -h for space, df -i for inodes, du -xh --max-depth=2 | sort -h to find the culprit. If df and du disagree, look for deleted-but-open files.
• Memory — free -h, and read the available column, not free. Linux fills spare RAM with page cache on purpose; that is not a leak.
• CPU — uptime load average compared against nproc. Load counts processes waiting on disk too, so high load with idle CPU means I/O.
• Descriptors — ulimit -n and /proc/PID/limits. 'Too many open files' under load is almost always this.

The shell as an agent's hands

Why CLI is the interface, and what changes when the operator is a model

An AI agent needs a way to act on the world, and the command line turns out to be close to ideal for it. It is text in and text out, so it needs no adapter. It is composable, so small capabilities combine into large ones without anybody writing integration code.

It is auditable: every action is a command you can read, log and replay. And it already has forty years of tools, so the agent inherits git, rg, docker and curl without any of them being designed for it.

The consequence is that agent design is largely shell design. The same instincts that make a good deploy script — fail loudly, be idempotent, support a dry run, keep the blast radius small — are what make an agent's tools safe to hand over.

Headless mode makes the model a pipeline stage

claude -p runs non-interactively: prompt in, result out, exit. That single property lets a language model sit inside a pipeline next to grep and jq, instead of being a place you go and paste things into.

git diff main...HEAD | claude -p 'review for auth and injection risks'

journalctl -u app -p err --since '1 hour ago' \
  | claude -p 'group these by root cause, most frequent first'

claude -p 'list exported functions in src/api.ts as JSON' \
  --output-format json | jq -r '.result'

Read and write are different risks

Reading is cheap and reversible; writing is neither. It is worth splitting an agent's capabilities along that line and granting them separately — read tools freely, write tools with a confirmation step or a scoped allowlist. --allowedTools 'Bash(npm test:*)' 'Read' 'Edit' is a meaningfully different grant from --allowedTools 'Bash', and the specificity is the whole point.

Idempotency and dry runs

An agent will retry. A tool that is safe to run twice is a tool you do not have to reason about carefully. mkdir -p is idempotent; mkdir is not.

rsync is; cp -r into a moving target is not. Where a command is destructive, give it a preview mode and make the agent use it first — rsync -n, git clean -nd, find ... -print before -delete.

git is the undo layer

The reason it is reasonable to let a model edit your files at all is that git makes every change reviewable and reversible. Work on a branch, keep commits small enough to revert individually, and read the diff instead of the model's description of the diff.

For parallel work, git worktree gives each agent its own directory and branch backed by one repository, so two of them are not fighting over the same index and the same dev server port.

for t in graph auth export; do
  git worktree add -b "feat/$t" "../app-$t" main
done
# one agent per directory, one branch each, one shared history

Containers for blast radius

Permission prompts are a control that assumes a human is reading them. When you skip them for an unattended run, the container becomes the control instead: mount only the directory that matters, drop the network if the task does not need it, and treat the whole thing as disposable. The question to answer before starting is not 'will it behave' but 'what is the worst thing that survives if it does not'.

The master script

Below is a single script that uses almost every pattern in the scripting section at once. It is not a toy: strip the deploy-specific parts out of main and you have a skeleton worth starting every real script from.

Read it in five passes.

The preamble sets strict mode and resolves the script's own directory, so the script behaves the same whether it is run from its folder or from /. It does not resolve a symlink to the script itself: for that, run readlink -f on ${BASH_SOURCE[0]} first.

The output helpers send every diagnostic to stderr, which keeps stdout clean for piping, and run gives the whole script a dry-run mode for free by routing every side effect through one function.

The trap block guarantees the temp directory is removed on every exit path, including Ctrl-C and a mid-script failure, because EXIT fires for all of them.

The argument parser uses getopts with a leading colon so it reports its own errors instead of bash's.

main does the actual work last, after the whole file has parsed.

Two details are easy to miss and worth stealing.

The verbose flag is applied with an if instead of (( VERBOSE )) && set -x, purely for readability. Contrary to a widespread belief, the && form is safe: set -e ignores any command that is not the last in an && list. What does kill a script is a bare (( VERBOSE )) on its own line.

And the deploy does not finish at the restart. It polls the health endpoint, so a service that comes back broken fails the deploy instead of passing it.

#!/usr/bin/env bash
#
# deploy.sh - copy a build to a host and restart the service
#
# Usage: deploy.sh [-e ENV] [-n] [-v] TARGET
#

set -euo pipefail
IFS=$'\n\t'

SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT_NAME="${0##*/}"
TMP_DIR="$(mktemp -d)"
readonly SCRIPT_DIR SCRIPT_NAME TMP_DIR

# --- configuration: environment wins, defaults fill the gaps ----------------
ENVIRONMENT="${ENVIRONMENT:-staging}"
DRY_RUN=0
VERBOSE=0

# --- output -----------------------------------------------------------------
# Everything diagnostic goes to stderr so stdout stays pipeable.
_ts()  { date +%H:%M:%S; }
log()  { printf '%s       %s\n' "$(_ts)" "$*" >&2; }
warn() { printf '%s WARN  %s\n' "$(_ts)" "$*" >&2; }
die()  { printf '%s FATAL %s\n' "$(_ts)" "$*" >&2; exit 1; }

# run: echo a command, then run it - or just echo it during a dry run.
run() {
  if (( DRY_RUN )); then
    log "DRY  $*"
  else
    log "RUN  $*"
    "$@"
  fi
}

need() { command -v "$1" >/dev/null 2>&1 || die "missing dependency: $1"; }

usage() {
  cat <<EOF
$SCRIPT_NAME - copy a build to a host and restart the service

Usage: $SCRIPT_NAME [options] TARGET

Options:
  -e ENV   environment to deploy to (default: $ENVIRONMENT)
  -n       dry run: print commands without running them
  -v       verbose: trace every command
  -h       show this help
EOF
}

# --- cleanup runs on every exit path, including errors and Ctrl-C -----------
cleanup() {
  local code=$?
  rm -rf -- "$TMP_DIR"
  if (( code != 0 )); then
    warn "exited with status $code"
  fi
  return "$code"
}
trap cleanup EXIT
trap 'die "interrupted"' INT TERM

# --- arguments ---------------------------------------------------------------
while getopts ':e:nvh' opt; do
  case "$opt" in
    e) ENVIRONMENT="$OPTARG" ;;
    n) DRY_RUN=1 ;;
    v) VERBOSE=1 ;;
    h) usage; exit 0 ;;
    :) die "option -$OPTARG requires a value" ;;
    \?) usage >&2; die "unknown option: -$OPTARG" ;;
  esac
done
shift $(( OPTIND - 1 ))

# Either form works: set -e ignores the left side of &&. What WOULD
# exit is a bare `(( VERBOSE ))` on its own line.
if (( VERBOSE )); then
  PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: '
  set -x
fi

if (( $# != 1 )); then
  usage >&2
  die "expected exactly one TARGET"
fi
readonly TARGET="$1"

# --- the actual work ---------------------------------------------------------
main() {
  need rsync
  need ssh

  [[ -d "$SCRIPT_DIR/dist" ]] || die "no build found at $SCRIPT_DIR/dist"

  log "deploying $TARGET to $ENVIRONMENT"

  run rsync -az --delete \
    --exclude '.DS_Store' \
    "$SCRIPT_DIR/dist/" "deploy@${TARGET}:/srv/app/"

  run ssh "deploy@${TARGET}" 'sudo systemctl restart app'

  # Wait for the service to answer before declaring victory.
  local i
  for (( i = 1; i <= 10; i++ )); do
    if run ssh "deploy@${TARGET}" 'curl -fsS localhost:8000/health >/dev/null'; then
      log "healthy after ${i}s"
      return 0
    fi
    sleep 1
  done

  die "service did not come back healthy"
}

main "$@"

Bash scripting

The language itself: quoting, expansion, control flow, and the safety rails that stop a script from doing half a job and calling it success.

Bash is not a general-purpose language and it is not trying to be. It is a language for gluing programs together, and it is excellent at exactly that: launching processes, wiring their inputs and outputs, branching on whether they worked, and passing text between them. Everything it does well is in service of that. Everything it does badly — arithmetic, data structures, error handling, anything with a float in it — is a sign you have wandered outside its purpose.

The practical consequence is a rule of thumb about size. Under fifty lines and mostly calls to other programs, bash is the right choice and any alternative is overhead. Past a few hundred lines, or the moment you find yourself wanting a dictionary of lists, you are writing the wrong thing in the wrong language and should move to Python before it gets worse. Most scripts that became a maintenance problem crossed that line gradually and nobody noticed.

The other thing to internalise is that bash defaults are wrong for scripts. They were designed for an interactive shell, where a failed command should not close your terminal and an unset variable expanding to nothing is convenient. In a script those same defaults mean a failure in step two is invisible and steps three through nine run anyway, on incomplete state. set -euo pipefail is not paranoia; it is switching from interactive defaults to programming defaults.

So the material below is ordered accordingly: the safety preamble first, then the expansion rules that quoting protects you from, then control flow, then the plumbing — redirection, traps, arguments — that turns a sequence of commands into something you can hand to someone else.

Skeleton & safety

The first ten lines of a script decide whether its failures are loud or silent. Strict mode, a resolved script directory, dependency checks and a cleanup trap cost nothing and remove whole categories of bug.

#!/usr/bin/env bash

Finds bash on PATH instead of hardcoding a location. Portable across Linux, macOS and containers.

#!/usr/bin/env bash
set -euo pipefail

set -e

Aborts the script the moment any command exits non-zero.

set -e
cp missing.txt /tmp/   # script stops here
echo 'never printed'

set -u

Treats reading an unset variable as a fatal error, so typos surface instead of silently expanding to nothing.

set -u
rm -rf "$BULID_DIR/"   # typo -> hard error, not `rm -rf /`

set -o pipefail

Makes a pipeline fail if any stage fails, not just the final one.

set -o pipefail
curl -sf https://api.example.com/x | jq .id   # curl 404 now fails the line

set -euo pipefail

The standard opening line: stop on error, stop on unset variables, stop on broken pipelines.

#!/usr/bin/env bash
set -euo pipefail

IFS=$'\n\t'

Restricts word-splitting to newlines and tabs so filenames with spaces stop shattering into pieces.

IFS=$'\n\t'
for f in $(find . -name '*.log'); do echo "$f"; done

SCRIPT_DIR=...

Resolves the directory the script lives in, so it works no matter where it is called from.

SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/lib/common.sh"

command -v

Checks whether a dependency exists before you rely on it.

need() { command -v "$1" >/dev/null || { echo "missing: $1" >&2; exit 1; }; }
need jq; need rsync

readonly

Locks a variable after assignment so later code cannot clobber it.

readonly CONFIG="/etc/app/config.yml"
readonly -a REGIONS=(us-west us-east)

mktemp -d

Creates a uniquely named temp directory, safely, with no race against other processes.

TMP="$(mktemp -d)"
trap 'rm -rf -- "$TMP"' EXIT

source vs ./script

source runs the file in the current shell (so it can set your variables); executing it spawns a child that cannot.

source ./env.sh     # exports land in this shell
./env.sh            # exports die with the child

set -a; source .env; set +a

Loads a .env file and exports every variable it defines.

set -a
source .env
set +a
python -m app

exec > >(tee -a log) 2>&1

Redirects everything the script prints from that point on into a log file and the terminal at once.

exec > >(tee -a "/var/log/deploy-$(date +%F).log") 2>&1
echo 'this is captured'

main "$@"

Wraps the body in a function and calls it last, so the whole file parses before anything executes.

main() {
  log 'starting'
  deploy "$1"
}
main "$@"

Variables & expansion

Parameter expansion is bash's standard library. Prefix and suffix stripping, defaults, and case conversion all happen in the shell with no process launched, which is both faster and less fragile than piping through sed.

VAR=value

Assigns a variable. No spaces around the equals sign, ever.

count=3
name="flight test"
path="$HOME/logs"

"$var"

Quoting stops word-splitting and glob expansion. This is the single highest-value habit in bash.

file="my report.txt"
rm "$file"     # removes one file
rm $file       # tries to remove 'my' and 'report.txt'

${var:-default}

Uses a fallback when the variable is unset or empty, without assigning it.

PORT="${PORT:-8000}"
echo "listening on ${HOST:-0.0.0.0}:$PORT"

${var:=default}

Same idea, but also assigns the default back to the variable.

: "${LOG_LEVEL:=info}"
echo "$LOG_LEVEL"   # info, and LOG_LEVEL is now set

${var:?message}

Aborts with your message if the variable is missing. Cheap required-argument validation.

API_KEY="${API_KEY:?set API_KEY before running}"

${var:+alt}

Expands to the alternate value only when the variable is set AND non-empty. Drop the colon — ${var+alt} — for the set-at-all form. Useful for optional flags.

VERBOSE_FLAG="${DEBUG:+--verbose}"
myapp $VERBOSE_FLAG

${#var}

Length of a string. For an array you must subscript it — ${#arr[@]} is the element count, while a bare ${#arr} gives the length of element zero.

s="HLX-003"
echo "${#s}"        # 7

${var#pat} / ${var##pat}

Strips a prefix: # removes the shortest match, ## the longest.

p="/var/log/app/error.log"
echo "${p##*/}"     # error.log
echo "${p#*/}"      # var/log/app/error.log

${var%pat} / ${var%%pat}

Strips a suffix: % shortest match, %% longest.

f="backup.tar.gz"
echo "${f%.gz}"      # backup.tar
echo "${f%%.*}"     # backup

${var/old/new}

Replaces the first occurrence; double slash replaces all.

s="a-b-c"
echo "${s/-/_}"      # a_b-c
echo "${s//-/_}"     # a_b_c

${var^^} / ${var,,}

Upper-cases or lower-cases a string in place.

env="Staging"
echo "${env,,}"      # staging
echo "${env^^}"      # STAGING

${var:offset:length}

Substring by position.

sha="9f3c1d8a2b"
echo "${sha:0:7}"    # 9f3c1d8

$(( expr ))

Integer arithmetic. No $ needed on variable names inside.

total=$(( count * 2 + 1 ))
pct=$(( done * 100 / total ))

(( expr ))

Arithmetic as a statement or a test.

(( retries++ ))
if (( retries > 3 )); then die 'giving up'; fi

$(command)

Captures a command's stdout into a value.

branch="$(git rev-parse --abbrev-ref HEAD)"
now="$(date -u +%FT%TZ)"

${!var}

Indirect expansion: use the value of one variable as the name of another.

env=PROD_URL
PROD_URL="https://api.example.com"
echo "${!env}"

declare -i / -r / -A

Types a variable as integer, read-only, or associative array.

declare -i attempts=0
attempts+=1        # arithmetic, not string concat

printf -v var

Formats directly into a variable with no subshell.

printf -v padded '%04d' 42
echo "$padded"      # 0042

$$ $! $? $0 $LINENO

Current PID, last background PID, last exit status, script name, current line.

long_task &
child=$!
wait "$child" || echo "failed: $?"

Arrays

Arrays exist so that a list of things stays a list of things. The dominant use is building command arguments: appending optional flags to an array and expanding it with "${args[@]}" is the only approach that survives values containing spaces.

arr=(a b c)

Creates an indexed array.

hosts=(hlx-002 hlx-003 hlx-004)
echo "${hosts[0]}"   # hlx-002

"${arr[@]}"

Expands to every element as a separate quoted word. This is what you almost always want.

for h in "${hosts[@]}"; do ssh "$h" uptime; done

arr+=(item)

Appends to an array.

args=()
args+=(--verbose)
[[ -n $CFG ]] && args+=(--config "$CFG")
myapp "${args[@]}"

${#arr[@]}

Number of elements.

if (( ${#hosts[@]} == 0 )); then die 'no hosts'; fi

mapfile -t arr < file

Reads a file into an array, one line per element, without the trailing newlines.

mapfile -t lines < hosts.txt
printf '%s\n' "${lines[@]}"

declare -A map

Associative array, i.e. a string-keyed dictionary.

declare -A ip
ip[hlx-003]=10.0.4.13
for k in "${!ip[@]}"; do echo "$k -> ${ip[$k]}"; done

${arr[@]:1:2}

Slices an array by offset and count.

echo "${hosts[@]:1}"      # everything after the first

local -n ref=arr

Nameref: lets a function operate on a caller's array by name.

push() { local -n a="$1"; shift; a+=("$@"); }
push hosts hlx-005

Tests & conditionals

[[ ]] is a bash keyword, not a command, which is why it can skip word splitting and support regex. Use it and treat [ ] as a POSIX compatibility artefact you will meet in other people's scripts.

[[ ... ]]

Bash's conditional. Safer than [ ] because it does not word-split or glob unquoted variables.

if [[ -n $name && $count -gt 0 ]]; then echo ok; fi

[[ $a == $b ]]

String equality. The right side is a glob pattern unless you quote it.

if [[ $file == *.log ]]; then echo 'a log'; fi
if [[ $file == "$other" ]]; then echo 'literal match'; fi

[[ $s =~ regex ]]

Regex match, with captures landing in BASH_REMATCH.

if [[ $tag =~ ^v([0-9]+)\.([0-9]+) ]]; then
  echo "major=${BASH_REMATCH[1]} minor=${BASH_REMATCH[2]}"
fi

-f -d -e -s -r -w -x

File tests: regular file, directory, exists, non-empty, readable, writable, executable.

[[ -f $cfg ]] || die "no config at $cfg"
[[ -d $out ]] || mkdir -p "$out"

-z / -n

True when a string is empty / non-empty.

if [[ -z ${API_KEY:-} ]]; then die 'API_KEY is required'; fi

-eq -ne -lt -le -gt -ge

Numeric comparison inside [[ ]]. Inside (( )) use the normal < > == symbols.

if [[ $code -ge 500 ]]; then echo 'server error'; fi
if (( code >= 500 )); then echo 'same thing'; fi

file1 -nt file2

True when file1 is newer than file2. Poor man's make.

if [[ src/app.ts -nt dist/app.js ]]; then npm run build; fi

cmd && cmd || cmd

Runs the second only on success, the third only on failure.

mkdir -p build && cd build || die 'cannot enter build'

cmd || true

Swallows a non-zero exit so set -e does not kill the script.

grep -q WARN app.log || true

case

Pattern-matching branch. Cleaner than a stack of elifs.

case "$env" in
  prod|production) url=$PROD ;;
  stag*)           url=$STAGING ;;
  *)               die "unknown env: $env" ;;
esac

Loops

Two rules cover most loop bugs. Read files with while IFS= read -r line, never for line in $(cat file). And feed the loop with < <(cmd) instead of cmd |, so it runs in your shell and variables set inside it survive.

for x in ...

Iterates over a list of words.

for env in dev staging prod; do
  echo "deploying $env"
done

for i in {1..10}

Brace expansion generates sequences, with an optional step.

for i in {1..5}; do curl -sf "http://host/$i" || break; done
echo {0..20..5}      # 0 5 10 15 20

for ((i=0; i<n; i++))

C-style loop when you need a real counter.

for (( i=0; i<${#hosts[@]}; i++ )); do
  echo "$i: ${hosts[i]}"
done

while IFS= read -r line

The correct way to read a file line by line, preserving whitespace and backslashes.

while IFS= read -r line; do
  echo "got: $line"
done < hosts.txt

done < <(cmd)

Process substitution feeds a command into the loop without a subshell, so variables set inside survive.

count=0
while IFS= read -r l; do (( count++ )); done < <(grep ERROR app.log)
echo "$count"   # correct

read -r -d ''

Reads null-delimited records, which is the only fully safe way to handle arbitrary filenames.

while IFS= read -r -d '' f; do
  echo "$f"
done < <(find . -name '*.log' -print0)

for f in *.log

Globbing beats parsing ls output.

shopt -s nullglob
for f in logs/*.log; do gzip "$f"; done

shopt -s globstar

Enables ** for recursive matching.

shopt -s globstar nullglob
for f in src/**/*.ts; do echo "$f"; done

break / continue

Exit a loop, or skip to the next iteration. Both take a level count for nested loops.

for h in "${hosts[@]}"; do
  ping -c1 -W1 "$h" >/dev/null || continue
  echo "$h is up"
done

ssh -n inside a loop

Stops ssh from swallowing the loop's stdin and eating the rest of your input.

while read -r h; do ssh -n "$h" uptime; done < hosts.txt

Functions & scope

Functions in bash return a status, not a value; anything you want back has to be printed and captured. local is what stops a helper quietly overwriting a caller's variable, and forgetting it is the usual cause of a script that works alone and breaks when sourced.

name() { ...; }

Defines a function. Arguments arrive as $1, $2, "$@".

log() { printf '%s %s\n' "$(date +%T)" "$*" >&2; }
log 'starting deploy'

local

Scopes a variable to the function. Without it, everything is global.

retry() {
  local n=0 max=$1; shift
  until "$@"; do (( ++n >= max )) && return 1; sleep $(( n * 2 )); done
}

return vs echo

return sets an exit status (0-255); to hand back a value, print it and capture with $( ).

current_sha() { git rev-parse --short HEAD; }
sha="$(current_sha)"

"$@" vs "$*"

"$@" preserves each argument separately; "$*" flattens them into one string.

run() { echo "+ $*" >&2; "$@"; }
run rsync -a "my dir/" server:/srv/

shift

Drops $1 and slides the rest down. Standard in argument parsers.

cmd=$1; shift
case $cmd in deploy) deploy "$@" ;; esac

export -f

Makes a function visible to child bash processes, including xargs and find -exec.

process() { echo "handling $1"; }
export -f process
find . -name '*.log' -exec bash -c 'process "$0"' {} \;

Redirection & I/O

Redirection is fd-table manipulation, applied left to right, before the program starts. Heredocs are worth particular attention: whether you write <<EOF or <<'EOF' decides whether variables expand locally or on the remote host, and getting it backwards is a rite of passage.

> and >>

Overwrite or append stdout to a file.

date > run.log
echo 'second line' >> run.log

2>&1

Points stderr at wherever stdout is currently going. Order matters.

cmd > out.log 2>&1     # both to out.log
cmd 2>&1 > out.log     # stderr to terminal, stdout to file

&> file

Bash shorthand for sending stdout and stderr to the same place.

npm run build &> build.log

2>/dev/null

Discards error output only.

find / -name core 2>/dev/null

>/dev/null 2>&1

Silences a command entirely; you only care about its exit status.

if command -v docker >/dev/null 2>&1; then echo present; fi

tee

Writes to a file and passes the stream onward.

make 2>&1 | tee build.log | grep -i error

tee >(cmd)

Forks a stream into another command as well as onward.

generate | tee >(gzip > archive.gz) | wc -l

<<EOF

Heredoc: a multi-line literal on stdin. Variables expand.

cat <<EOF > /tmp/nginx.conf
server { server_name $DOMAIN; }
EOF

<<'EOF'

Quoted heredoc: nothing expands. Use for scripts, awk programs and anything containing $.

ssh host bash <<'EOF'
cd /srv/app && git pull && systemctl restart app
EOF

<<-EOF

Strips leading tabs so a heredoc can sit at the indentation of its block.

if true; then
	cat <<-EOF
		indented in source, flush in output
	EOF
fi

<<< "string"

Herestring: feeds a single string to a command's stdin.

jq -r .name <<< "$json"
read -r a b <<< "$line"

<(cmd)

Process substitution: makes a command's output look like a file.

diff <(ssh prod cat /etc/app.conf) <(cat local.conf)

printf

Predictable formatted output. Unlike echo, it behaves the same everywhere.

printf '%-12s %s\n' "$host" "$status"
printf '%s\n' "${arr[@]}"

read -p / -s / -t

Prompts for input, hides it, or times out.

read -rp 'Deploy to prod? [y/N] ' ans
[[ $ans == [yY] ]] || exit 0

[[ -t 0 ]]

Detects whether stdin is a terminal, so a script can behave differently when piped.

if [[ -t 0 ]]; then read -rp 'confirm: ' a; else a=yes; fi

Traps, exits & jobs

Traps are the closest bash has to finally. An EXIT trap that removes the temp directory runs on success, on error, and on Ctrl-C, which means cleanup stops depending on the script reaching its last line.

trap ... EXIT

Runs cleanup no matter how the script ends: success, error, or exit.

TMP=$(mktemp -d)
trap 'rm -rf -- "$TMP"' EXIT

trap ... ERR

Fires on any command that would trip set -e. Ideal for a one-line error report.

trap 'echo "failed at line $LINENO: $BASH_COMMAND" >&2' ERR

trap ... INT TERM

Handles Ctrl-C and kill so long jobs shut down cleanly.

trap 'echo interrupted >&2; kill -- $(jobs -p) 2>/dev/null; exit 130' INT TERM

exit codes

0 means success. 1 is generic failure, 2 is usage error, 126/127 mean not executable / not found, 128+N means killed by signal N.

exit 2   # bad arguments
# 130 = Ctrl-C (128 + SIGINT 2)

cmd & / wait

Backgrounds a job and waits for it. wait -n returns as soon as any one finishes.

for h in "${hosts[@]}"; do deploy "$h" & done
wait
echo 'all done'

timeout

Kills a command that runs too long.

timeout 30s curl -sf "$url" || echo 'timed out or failed'
timeout -k 5s 30s ./long-job.sh

flock

Stops two copies of the same script running at once.

exec 9>/var/lock/deploy.lock
flock -n 9 || { echo 'already running'; exit 1; }

Arguments & debugging

A script that supports -h and validates its arguments is one someone else can use. When something misbehaves, bash -n catches syntax, set -x with an informative PS4 shows execution, and shellcheck catches the quoting bugs that neither will.

while [[ $# -gt 0 ]]

Hand-rolled parser. The only practical way to support --long-options in bash.

while [[ $# -gt 0 ]]; do
  case $1 in
    -e|--env)   ENV=$2; shift 2 ;;
    -n|--dry-run) DRY=1; shift ;;
    --)         shift; break ;;
    -*)         die "unknown option: $1" ;;
    *)          ARGS+=("$1"); shift ;;
  esac
done

getopts

Built-in parser for short options. Compact, but no long-option support.

while getopts ':e:nvh' opt; do
  case $opt in
    e) ENV=$OPTARG ;;
    n) DRY=1 ;;
    v) VERBOSE=1 ;;
    h) usage; exit 0 ;;
    :) die "-$OPTARG needs a value" ;;
    \?) die "unknown option -$OPTARG" ;;
  esac
done
shift $(( OPTIND - 1 ))

usage() heredoc

Every script deserves a -h. A heredoc keeps it readable.

usage() {
  cat <<EOF
${0##*/} - deploy an artifact

Usage: ${0##*/} [-e ENV] [-n] TARGET
EOF
}

bash -n

Parses the script without running it. Catches syntax errors before they hit production.

bash -n deploy.sh && echo 'syntax ok'

bash -x / set -x

Traces every command as it executes, with variables expanded.

bash -x ./deploy.sh staging
# or scope it:
set -x; risky_bit; set +x

PS4

Controls the trace prefix. Adding file and line turns -x output into something you can actually read.

PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: '
set -x

shellcheck

Static analysis for shell. It catches quoting bugs you will otherwise find in production.

shellcheck deploy.sh
shellcheck -S warning -x scripts/*.sh

declare -p

Prints a variable's real definition, including type and array structure.

declare -p hosts
# declare -a hosts=([0]="hlx-002" [1]="hlx-003")

Text & data

Finding things, then reshaping them. grep, sed, awk and jq cover most of what people reach for Python to do.

This section is one idea applied repeatedly: a stream of lines goes in, a transformed stream comes out, and the tools chain end to end because they all agree on that shape. Once you see the ladder — select with grep, edit with sed, compute with awk — most data wrangling stops being a question of which incantation and becomes a question of which rung.

Two habits separate people who are fast at this from people who are not. The first is composing incrementally: run the first stage, look at the output, add the next stage, look again. A ten-stage pipeline written in one go and debugged as a unit is miserable; the same pipeline grown one stage at a time is easy. Pipe to head while you are building so you are not waiting on the full dataset.

The second is knowing when the text model no longer fits. JSON, YAML and anything with nested structure are not line-oriented, and every regex you write against them is a bet that the formatting will not change. jq and yq parse the real grammar and cost nothing to reach for. The rule is to use them from the first attempt rather than the third.

Finally, filenames deserve their own paranoia. They may contain spaces and, on Linux, newlines. Any pipeline that moves filenames between tools should use null separation — find -print0, xargs -0, read -d '' — and any variable holding one should be quoted. This is the single most common source of scripts that work for a year and then destroy something.

Finding files

find walks the tree and evaluates an expression on each entry; the order of tests matters for speed, since -prune and -maxdepth cut whole subtrees. Preview with -print before you use -delete, every time.

find . -name

Walks a tree matching filenames. -iname is case-insensitive.

find . -name '*.log'
find /srv -iname 'readme*'

find -type

Restricts to files (f), directories (d), or symlinks (l).

find . -type d -name node_modules -prune -o -type f -name '*.ts' -print

find -mtime / -mmin

Filters by modification age: days or minutes. + is older than, - is newer than.

find /var/log -type f -mtime +30
find . -type f -mmin -15

find -size

Filters by size. Suffixes: k, M, G.

find /var -type f -size +100M -exec ls -lh {} +

find -exec {} +

Runs a command on the results. The + form batches them into few invocations; ; runs one per file.

find . -name '*.pyc' -exec rm -f {} +
find . -name '*.conf' -exec grep -l listen {} +

find -print0 | xargs -0

Null-delimits results so filenames with spaces or newlines survive.

find . -name '*.log' -print0 | xargs -0 gzip

find -maxdepth

Limits recursion depth. Put it before other tests for speed.

find . -maxdepth 2 -name '*.json'

find -delete

Deletes matches directly. Run the find first without it, every time.

find /tmp/cache -type f -mtime +7 -delete

fd

Modern find: sane defaults, respects .gitignore, much faster to type.

fd '\.log$'
fd -e ts -E node_modules
fd -H -t d cache

Searching content

grep answers 'which lines match'. Learn -r, -n, -C, -l, -q and -F and you have covered most daily use. rg is the same idea with better defaults — recursive, gitignore-aware, and fast enough to change what you bother searching.

grep -rn

Recursive search with line numbers. The workhorse.

grep -rn 'TODO' src/
grep -rni 'api_key' --include='*.py' .

grep -i -w -v

Case-insensitive, whole-word only, or inverted match.

grep -w port config.yml
grep -v '^#' /etc/nginx/nginx.conf | grep -v '^$'

grep -E / -F / -P

Extended regex, fixed strings (fast, no escaping), or Perl-compatible regex.

grep -E 'ERROR|FATAL' app.log
grep -F '[2026-08-20]' app.log
grep -Po '(?<=user_id=)\d+' app.log

grep -A -B -C

Prints lines after, before, or around each match. Essential for reading stack traces.

grep -C5 'Traceback' app.log
journalctl -u api | grep -A20 'panic'

grep -l / -c / -o

List matching filenames only, count matches, or print just the matched part.

grep -rl 'deprecated' src/ | wc -l
grep -o 'HTTP/1.1" [0-9]*' access.log | sort | uniq -c

grep -q

Silent: exits 0 on first match. Built for if statements.

if grep -q 'ready' /var/log/app.log; then echo up; fi

grep --exclude-dir

Skips directories you never want to search.

grep -rn 'apiKey' . --exclude-dir={node_modules,.git,dist}

rg

ripgrep: recursive by default, gitignore-aware, and fast enough to change how you work.

rg 'useEffect' -t ts
rg -i 'timeout' -g '!*.lock'
rg --files | rg config

sed

A stream editor: it applies commands to each line as it passes. Substitution is ninety percent of what anyone uses it for. Beyond a couple of expressions chained together, switch to awk or a script — long sed programs are write-only.

sed 's/old/new/'

Substitutes the first match per line; add g for every match.

sed 's/localhost/api.internal/g' config.yml

sed -i

Edits the file in place.

sed -i 's/DEBUG/INFO/' app.conf
sed -i.bak 's/8000/8080/' app.conf

sed 's|old|new|'

Any character can be the delimiter, which saves escaping slashes in paths.

sed 's|/usr/local|/opt|g' paths.txt

sed -n 'Np'

Prints only selected lines. -n suppresses the default echo.

sed -n '5p' file.txt
sed -n '10,20p' file.txt
sed -n '/START/,/END/p' log.txt

sed 'Nd' / '/re/d'

Deletes lines by number or pattern.

sed '1d' data.csv                 # drop the header
sed '/^#/d; /^$/d' config.conf   # strip comments and blanks

sed -E

Extended regex, so you can use + ? ( ) without backslashes, plus \1 backreferences.

sed -E 's/([a-z]+)-([0-9]+)/\2_\1/' names.txt

sed 'i\' / 'a\'

Inserts a line before or appends after a match.

sed '/^\[server\]/a\    timeout = 30' app.ini

awk

awk splits each line into fields automatically and gives you variables, arrays and arithmetic. That combination makes group-by-and-count a one-liner, which is why it survives despite everything newer. NR, NF, $1, $NF and END will carry you a long way.

awk '{print $1}'

Splits each line into fields and prints one. $0 is the whole line, $NF is the last field.

ps aux | awk '{print $2, $11}'
awk '{print $NF}' access.log

awk -F

Sets the field separator.

awk -F: '{print $1}' /etc/passwd
awk -F, '{print $3}' data.csv

awk '/re/ {action}'

Only acts on lines matching a pattern.

awk '/ERROR/ {print $1, $2, $NF}' app.log
awk '$9 >= 500 {print $7}' access.log

awk BEGIN/END

Runs setup before the first line and a summary after the last.

awk '{sum += $2} END {print "total:", sum}' sizes.txt
awk 'BEGIN {OFS=","} {print $1, $3}' data.tsv

awk NR / NF

NR is the current line number, NF the field count on this line.

awk 'NR > 1' data.csv          # skip the header
awk 'NF != 5 {print NR": "$0}' data.tsv   # find malformed rows

awk counting with arrays

Awk has associative arrays, which makes group-by a one-liner.

awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn | head

awk printf

Formats numbers and columns properly.

df -h | awk 'NR>1 {printf "%-20s %s\n", $6, $5}'

awk averages

Sum, count and divide in one pass.

awk -F' ' '{t+=$10; n++} END {printf "avg %.1f bytes over %d reqs\n", t/n, n}' access.log

Slicing, sorting, counting

The connective tissue. sort | uniq -c | sort -rn is the most reusable pipeline in Unix — it answers 'what are the most common values' for any column of anything, and it turns up in log analysis constantly.

cut

Pulls out fields or character ranges. Simpler than awk when the delimiter is single-character.

cut -d: -f1,7 /etc/passwd
cut -c1-8 log.txt

tr

Translates or deletes characters.

tr 'A-Z' 'a-z' < names.txt
tr -d '\r' < win.txt > unix.txt
tr -s ' ' < spaced.txt

sort

Sorts lines. -n numeric, -r reverse, -u unique, -h human sizes, -k by column.

sort -rn sizes.txt
sort -t, -k3 -n data.csv
du -sh * | sort -h

uniq -c / -d

Counts or shows duplicates. Requires sorted input.

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

wc -l

Counts lines, words or bytes.

rg --files -t ts | wc -l
wc -l *.log | sort -n

head / tail

First or last N lines. tail -n +2 skips a header.

head -50 app.log
tail -n 200 app.log
tail -n +2 data.csv

tail -f / -F

Follows a growing file. -F keeps following across log rotation.

tail -F /var/log/nginx/error.log
tail -F app.log | grep --line-buffered ERROR

column -t

Aligns whitespace-separated output into readable columns.

mount | column -t
cat data.csv | column -t -s,

paste / join / comm

Merge files side by side, join on a key, or compare two sorted lists.

paste a.txt b.txt
comm -23 <(sort a.txt) <(sort b.txt)   # only in a

diff -u

Unified diff between two files, the same format git uses.

diff -u old.conf new.conf
diff -ru dirA/ dirB/

xargs

Turns stdin into command arguments.

rg -l 'TODO' | xargs wc -l
cat urls.txt | xargs -n1 -P8 curl -sfO

xargs -I{}

Places each input at a chosen spot, one invocation per item.

cat hosts.txt | xargs -I{} ssh {} 'uptime'

xargs -0 -P

Null-safe input and parallel execution across N processes.

find . -name '*.png' -print0 | xargs -0 -P4 -n1 optipng

JSON & YAML

jq parses JSON properly and applies a filter language to the result. Use -r when feeding other commands, --arg to pass shell variables in safely, and // for defaults. Never build a jq program by string-concatenating shell variables into it.

jq .

Pretty-prints and validates JSON.

curl -s https://api.example.com/status | jq .

jq -r

Raw output: strings without quotes, which is what you want when feeding other commands.

jq -r '.items[].name' data.json | sort

jq .a.b[0]

Navigates nested structures. .[] iterates an array.

jq '.data.results[0].id' resp.json
jq '.[] | .name' list.json

jq select()

Filters by predicate.

jq '.pods[] | select(.status != "Running") | .name' pods.json

jq map / building objects

Reshapes data into exactly the structure you need.

jq '[.items[] | {id, host: .metadata.host}]' data.json

jq --arg

Passes a shell variable in safely, without string-concatenating into the program.

jq --arg h "$HOST" '.hosts[] | select(.name == $h)' inv.json

jq -c / @csv / @tsv

Compact one-line output, or convert to delimited text.

jq -c '.[]' big.json > lines.jsonl
jq -r '.rows[] | [.id, .name] | @csv' data.json

jq // default

Supplies a fallback when a key is missing or null.

jq -r '.config.timeout // 30' app.json

jq -s / -n

Slurps multiple inputs into one array, or starts with no input at all.

jq -s 'add' part*.json
jq -n --arg v "$VER" '{version: $v}'

yq

jq syntax for YAML. Ideal for docker-compose and Kubernetes manifests.

yq '.services.api.image' docker-compose.yml
yq -i '.replicas = 3' deploy.yml

HTTP & encoding

curl in a script needs -fsS: fail on HTTP errors, stay quiet, but still show real errors. Without -f, a 500 response exits 0 and your script carries on with an error page in its variable, which is a bug that can survive for months.

curl -sS

Silent, but still shows errors. The correct default for scripts.

curl -sS https://api.example.com/health

curl -f

Makes curl exit non-zero on HTTP 4xx/5xx instead of cheerfully printing the error page.

curl -fsS "$url" | jq . || die 'request failed'

curl -H -d -X

Sets headers, sends a body, chooses a method.

curl -fsS -X POST https://api.example.com/jobs \
  -H 'Content-Type: application/json' \
  -d '{"target":"hlx-003"}'

curl -o / -O / -L

Saves to a named file, keeps the remote name, or follows redirects.

curl -fsSL -o app.tar.gz "$RELEASE_URL"

curl -w

Prints transfer metrics after the request. Great for spotting slow DNS or TLS.

curl -o /dev/null -sS -w 'dns %{time_namelookup}s tls %{time_appconnect}s total %{time_total}s\n' "$url"

curl -I / --retry

Headers only, or automatic retry with backoff.

curl -I https://example.com
curl -fsS --retry 5 --retry-delay 2 "$url"

curl -w '%{http_code}'

Grabs just the status code for a health check.

code=$(curl -o /dev/null -sw '%{http_code}' "$url")
[[ $code == 200 ]] || die "got $code"

base64

Encodes or decodes. -w0 stops GNU base64 wrapping lines.

printf '%s' "$user:$pass" | base64 -w0
base64 -d <<< "$encoded"

sha256sum / md5sum

Checksums a file or verifies one.

sha256sum firmware.bin
sha256sum -c checksums.txt

file / stat

Identifies a file's real type, or shows its metadata.

file mystery.bin
stat -c '%s bytes, modified %y' app.log

xxd / strings

Hex dump a binary, or pull readable text out of one.

xxd -l 64 firmware.bin
strings -n 8 firmware.bin | grep -i version

envsubst

Substitutes environment variables into a template file.

export DOMAIN=api.example.com
envsubst < nginx.conf.tpl > /etc/nginx/conf.d/api.conf

dos2unix / tr -d '\r'

Strips Windows line endings, the cause of many mysterious 'command not found' errors.

tr -d '\r' < script.sh > fixed.sh

Servers

Getting onto a box, working out what it is doing, and fixing it. Ordered roughly by what you reach for first when something is broken.

Operating a server is mostly a diagnostic activity, and the ordering matters more than the tool inventory. The layered approach from the foundations applies directly here: process, port, network path, name resolution, TLS, then application. Each layer has one command that tests it without involving the others, and going in order means you narrow the problem instead of accumulating theories about it.

Underneath that, three subsystems account for most of what you will actually touch. systemd owns process lifetime: what starts at boot, what restarts on failure, what the resource limits are. journald owns the record of what happened, which is why journalctl -u service --since replaces reading four different log files in three formats. And the filesystem is where the surprises live — a full disk that du cannot explain, an inode exhaustion that reports as 'no space' with 40GB free, a permission that is really a group membership from a session that predates the change.

Containers add a layer rather than replacing one. A container is a normal Linux process with a restricted view of the filesystem, network and process table. docker logs is reading a file, docker exec is entering namespaces, and a container killed with exit code 137 was sent SIGKILL (128+9) — often the OOM killer, but just as often docker stop running past its grace period, which docker inspect -f '{{.State.OOMKilled}}' will settle. When Docker behaves strangely, the underlying Linux behaviour is usually the explanation.

One habit worth building before you need it: on a machine you did not set up, spend two minutes running systemctl list-units --failed, df -h, free -h and journalctl -p err -b before you change anything. It is a cheap baseline, and it frequently finds the actual problem before you have finished forming a hypothesis about it.

SSH & transfer

Most SSH friction disappears into ~/.ssh/config. Host aliases, jump hosts and ControlMaster connection reuse turn a paragraph of flags into a word, and make repeated commands to the same host effectively instant.

~/.ssh/config

Stores per-host settings so you type ssh prod instead of a paragraph of flags.

Host prod
  HostName 10.0.4.13
  User deploy
  IdentityFile ~/.ssh/id_ed25519
  ServerAliveInterval 30

ssh-keygen -t ed25519

Generates a modern keypair.

ssh-keygen -t ed25519 -C "reuben@$(hostname)"

ssh-copy-id

Installs your public key on a remote host's authorized_keys.

ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected]

ssh -J jumphost

Hops through a bastion to reach a host you cannot route to directly.

ssh -J bastion.example.com [email protected]

ssh -L local:host:remote

Local port forward: exposes a remote service on your own machine.

ssh -N -L 5432:localhost:5432 prod
# now psql -h localhost hits the remote database

ssh -R

Reverse forward: exposes a local port on the remote host.

ssh -N -R 8080:localhost:3000 prod

ssh -D

SOCKS proxy through the remote host, for reaching an internal network from a browser.

ssh -N -D 1080 bastion

ControlMaster

Reuses one TCP connection for every later ssh to the same host. Makes repeated commands instant.

Host *
  ControlMaster auto
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 10m

ssh host 'cmd'

Runs one command and exits. Quote it so it expands on the remote side.

ssh prod 'systemctl is-active app && journalctl -u app -n20 --no-pager'

rsync -avz

Copies only what changed, preserving permissions and timestamps.

rsync -avz --progress ./dist/ deploy@prod:/srv/app/

rsync --dry-run --delete

Mirrors a directory, removing extras on the far side. Preview it first, always.

rsync -avzn --delete ./dist/ prod:/srv/app/   # -n = dry run
rsync -avz --delete ./dist/ prod:/srv/app/

rsync --exclude

Skips paths. Repeatable, or read from a file.

rsync -avz --exclude '.git' --exclude 'node_modules' ./ prod:/srv/app/

scp

Quick one-off copy. rsync is better for anything repeated.

scp prod:/var/log/app.log ./
scp -r ./configs prod:/etc/app/

systemd

systemd owns process lifetime. The four commands that matter are status, restart, daemon-reload after editing a unit, and enable --now to start something and keep it starting. systemctl edit creates overrides that survive package upgrades.

systemctl status

Shows whether a unit is running, its PID, and the last few log lines.

systemctl status app.service
systemctl status nginx --no-pager -l

systemctl start/stop/restart

Controls a service now.

sudo systemctl restart app
sudo systemctl reload nginx   # reload config without dropping connections

systemctl enable --now

Starts a service and sets it to start at boot, in one step.

sudo systemctl enable --now app.service

systemctl daemon-reload

Re-reads unit files after you edit one. Required before a restart picks up changes.

sudo systemctl daemon-reload && sudo systemctl restart app

systemctl cat / edit

Shows a unit's effective definition, or creates a drop-in override that survives package upgrades.

systemctl cat app.service
sudo systemctl edit app.service

systemctl list-units --failed

The first thing to run on a box that is misbehaving.

systemctl list-units --failed
systemctl list-timers --all

a minimal unit file

The shape of a service definition in /etc/systemd/system/.

[Unit]
Description=App API
Wants=network-online.target
After=network-online.target

[Service]
User=app
WorkingDirectory=/srv/app
ExecStart=/srv/app/.venv/bin/uvicorn main:app --port 8000
Restart=always
RestartSec=3
EnvironmentFile=/etc/app/env

[Install]
WantedBy=multi-user.target

systemctl --user

Manages per-user services, no root required.

systemctl --user status syncthing
loginctl enable-linger "$USER"   # keep user services alive after logout

systemd-run

Runs a command as a transient unit, so it survives your shell and gets logged.

systemd-run --user --unit=import ./long-import.sh

Logs

journald indexes by unit, time and priority, so the useful queries are filters rather than greps: -u for a service, --since for a window, -p err for severity, -b -1 for the boot before the reboot nobody can explain.

journalctl -u

Logs for a single unit.

journalctl -u app.service -n 200 --no-pager

journalctl -f

Follows the journal live.

journalctl -u app -f
journalctl -f -u nginx -u app   # two units at once

journalctl --since

Time windows, in plain English.

journalctl --since '1 hour ago'
journalctl -u app --since '2026-08-20 09:00' --until '09:30'

journalctl -p err

Filters by priority: emerg, alert, crit, err, warning, notice, info, debug.

journalctl -p err -b
journalctl -p warning --since today

journalctl -b

This boot only. -b -1 is the previous boot, which is where you look after an unexplained reboot.

journalctl -b -1 -p err

journalctl -k

Kernel messages, the same content as dmesg but with the journal's filters available.

journalctl -k -b | grep -i 'usb\|oom'

journalctl -o json

Structured output you can pipe into jq.

journalctl -u app -o json --since '10 min ago' | jq -r '.MESSAGE'

journalctl --vacuum-time

Trims the journal when it has eaten the disk.

journalctl --disk-usage
sudo journalctl --vacuum-time=7d

dmesg -T -w

Kernel ring buffer with human timestamps, optionally following. Where hardware and USB problems show up.

sudo dmesg -T | tail -50
sudo dmesg -TW

Processes

Find it, understand it, then signal it. pgrep -af before pkill -f so you can see exactly what you are about to kill, and lsof when the question is which process holds a port or a deleted file.

ps aux

Every process with user, CPU, memory and full command line.

ps aux | grep -v grep | grep uvicorn
ps aux --sort=-%mem | head

pgrep / pkill

Finds or signals processes by name, without the grep dance.

pgrep -af uvicorn
pkill -f 'python -m worker'

kill / kill -9

Sends SIGTERM (polite, default) or SIGKILL (immediate, no cleanup).

kill 4821          # TERM: let it shut down
kill -9 4821       # KILL: last resort

htop / top

Live process view. htop is interactive and sortable.

htop -u app
top -b -n1 | head -20   # batch mode, scriptable

nohup / disown / setsid

Detaches a job so it survives your session closing.

nohup ./import.sh > import.log 2>&1 &
disown

lsof -i

Shows what is listening on or connected to a port.

sudo lsof -i :8000
sudo lsof -i -P -n | grep LISTEN

lsof +D / deleted files

Finds open files under a directory, or space held by files that were deleted but not closed.

sudo lsof +D /var/log
sudo lsof -nP | grep '(deleted)'

nice / renice

Lowers a job's CPU priority so it does not starve the things that matter.

nice -n 19 ./reindex.sh
sudo renice -n 10 -p 4821

/proc/PID

Live per-process detail from the kernel.

cat /proc/4821/cmdline | tr '\0' ' '; echo
ls -l /proc/4821/cwd
cat /proc/4821/limits

Networking

One tool per layer, used in order: ss for listening sockets, nc for reachability, dig for resolution, openssl s_client for certificates, curl -v for HTTP. Testing them out of order is how a DNS problem becomes an afternoon.

ss -tulpn

Every listening TCP/UDP socket with the owning process. Replaces netstat.

sudo ss -tulpn
sudo ss -tp state established

ip a / ip r

Interface addresses and the routing table.

ip -brief a
ip r get 10.0.4.13

dig

DNS lookup. +short gives you just the answer.

dig +short api.example.com
dig @1.1.1.1 example.com MX

curl -I / -v

Checks response headers, or shows the full TLS and request handshake.

curl -sI https://example.com | head -1
curl -v https://example.com 2>&1 | grep -i 'subject\|issuer'

nc -zv

Tests whether a TCP port is reachable, with no payload.

nc -zv 10.0.4.13 8000
nc -zv -w2 db.internal 5432

mtr / traceroute

Traces the path to a host; mtr keeps updating so you can spot the hop that loses packets.

mtr -rwc 20 api.example.com

tcpdump

Captures packets. -w writes a file you can open in Wireshark.

sudo tcpdump -i any -n port 8000
sudo tcpdump -i any -w /tmp/cap.pcap host 10.0.4.13

ping / arping

Basic reachability. -W sets a short timeout so scripts do not hang.

ping -c3 -W1 10.0.4.13

ufw

Simple firewall front-end on Debian/Ubuntu.

sudo ufw status numbered
sudo ufw allow from 10.0.0.0/8 to any port 8000 proto tcp

hostname -I / getent

Local addresses, and name resolution the way the system actually does it.

hostname -I
getent hosts api.example.com

Disk & filesystem

Space, inodes, and who is holding what. When df says full and du disagrees, look for deleted-but-open files. When there is space but writes fail, check df -i — millions of small files exhaust inodes long before bytes.

df -h

Free space per mounted filesystem.

df -h
df -h --output=source,pcent,target | sort -k2 -r

df -i

Inode usage. A disk can be 'full' with plenty of bytes left if something created millions of tiny files.

df -i

du -sh * | sort -h

What is actually taking the space in this directory.

du -sh -- * | sort -h
sudo du -xh /var --max-depth=2 | sort -h | tail -20

ncdu

Interactive disk usage browser. The fastest way to find the culprit.

sudo ncdu -x /

lsblk / findmnt

Block device tree, and what is mounted where.

lsblk -f
findmnt /srv

ln -s / readlink -f

Creates a symlink, or resolves one to its final target.

ln -sfn /srv/releases/2026-08-20 /srv/current
readlink -f /srv/current

tar czf / xzf / tf

Create, extract, or list an archive.

tar czf backup.tar.gz -C /srv app
tar tzf backup.tar.gz | head
tar xzf backup.tar.gz -C /tmp/restore

chmod

Sets permissions, numerically or symbolically.

chmod 750 deploy.sh
chmod +x script.sh
chmod -R g+rX /srv/app

chown

Changes owner and group.

sudo chown -R app:app /srv/app

ls -l decoded

drwxr-xr-x is type, then owner/group/other triplets of read, write, execute.

ls -l app.log
# -rw-r----- 1 app app 4096 Aug 20 09:14 app.log

umask / sticky bit

Default permission mask for new files, and the +t bit that stops users deleting each other's files in shared dirs.

umask 027
chmod 1777 /tmp/shared

id / groups / usermod

Who you are, what you are in, and how to add a group membership.

id
sudo usermod -aG docker "$USER"   # log out and back in to take effect

Resources & packages

Read free -h by the available column and load average against nproc. High load with idle CPU means processes are waiting on disk, not compute, and that sends you somewhere completely different.

free -h

Memory usage. Look at the 'available' column, not 'free'.

free -h

uptime / load

Load average over 1, 5 and 15 minutes. Compare against your core count.

uptime
nproc

vmstat / iostat

Sampled CPU, memory and disk throughput. Ignore the first row; it is an average since boot.

vmstat 2 5
iostat -xz 2

OOM killer check

Confirms whether the kernel killed your process for using too much memory.

sudo dmesg -T | grep -i 'killed process'
journalctl -k | grep -i oom

ulimit -n

Open file descriptor limit. A busy server hitting it reports 'too many open files'.

ulimit -n
cat /proc/$(pgrep -f uvicorn | head -1)/limits

apt

Install, search and inspect packages on Debian/Ubuntu.

sudo apt update && sudo apt install -y jq ripgrep
apt policy nginx
apt list --installed | grep openssl

dpkg -S / -L

Which package owns this file, and what files did that package install.

dpkg -S /usr/bin/jq
dpkg -L nginx-common | grep conf

crontab -e

Per-user scheduled jobs. Fields: minute hour day-of-month month day-of-week.

*/15 * * * * /srv/app/bin/sync.sh >> /var/log/sync.log 2>&1
0 3 * * * /usr/bin/find /tmp/cache -mtime +7 -delete
@reboot /srv/app/bin/warm-cache.sh

systemd timers

The modern alternative to cron: real logs, dependencies, and missed-run catch-up.

systemctl list-timers --all
systemctl --user enable --now backup.timer

watch

Re-runs a command every couple of seconds and highlights what changed.

watch -n2 -d 'ss -tn state established | wc -l'
watch -n5 'docker ps --format "{{.Names}}\t{{.Status}}"'

Docker

A container is a process with a restricted view. Most debugging is therefore ordinary Linux debugging done through logs, exec and inspect. Two things bite in production: unbounded json-file logs filling the disk, and down -v deleting the volume holding your database.

docker ps -a

Running and stopped containers. --format trims the noise.

docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

docker logs -f --tail

Follows a container's output from the last N lines.

docker logs -f --tail 100 api
docker logs --since 15m api 2>&1 | grep ERROR

docker exec -it

Opens a shell inside a running container.

docker exec -it api bash
docker exec -it api sh   # alpine images have no bash

docker run --rm -it

Runs a throwaway container interactively. --rm cleans up on exit.

docker run --rm -it -v "$PWD:/w" -w /w python:3.12 bash

docker compose

Manages a multi-container stack from a compose file.

docker compose up -d
docker compose logs -f api
docker compose down -v

docker inspect

Full JSON metadata. Pipe to jq to find one field.

docker inspect api | jq -r '.[0].NetworkSettings.IPAddress'
docker inspect -f '{{.State.Health.Status}}' api

docker stats

Live CPU, memory and I/O per container.

docker stats --no-stream

docker system df / prune

Shows what Docker is storing, then reclaims it.

docker system df
docker system prune -a --volumes

log rotation caps

Docker's default json-file driver grows without limit until it fills the disk.

# /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "3" }
}

docker cp

Moves files in or out of a container.

docker cp api:/app/config.yml ./
docker cp ./patch.py api:/app/patch.py

nginx & TLS

nginx -t before every reload, without exception. The trailing slash on proxy_pass changes the upstream path, and a 502 means nginx is healthy and the thing behind it is not.

nginx -t

Validates config before you reload. Never reload without it.

sudo nginx -t && sudo systemctl reload nginx

reverse proxy block

The standard shape for putting nginx in front of an app server.

location /api/ {
  proxy_pass http://127.0.0.1:8000/;
  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
}

access & error logs

Where nginx tells you what actually happened.

sudo tail -F /var/log/nginx/error.log
awk '$9 ~ /^5/ {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

openssl x509 -dates

Reads a certificate's validity window.

openssl x509 -in cert.pem -noout -subject -dates

openssl s_client

Inspects the certificate a live server is serving, including the chain.

openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null | openssl x509 -noout -dates

Dev & agents

Sessions that outlive your connection, git moves that get you out of trouble, and running Claude Code as a component in a pipeline rather than a chat window.

This section is about two things that turn out to be the same thing: keeping work alive longer than a connection, and handing work to something that is not you.

The first is a solved problem with three answers, and the choice is really about ownership. If the work should outlive your SSH session but not your interest in it, use tmux. If it should outlive the machine's next reboot, write a systemd unit. nohup is for the case where you want neither and have thirty seconds.

The second is newer and less settled. Running a model headlessly — prompt in, result out, exit — makes it a component rather than a destination, and components compose. But delegation only works when the results are reviewable, and in practice that means git is doing the heavy lifting: a branch per task, commits small enough to revert individually, and a diff you read yourself rather than a summary you trust. git worktree extends this to parallel work, giving each task its own directory and branch over a single repository.

The git material below is weighted towards recovery for a reason. Once something else is making commits on your behalf, the commands that matter most are the ones that undo — reflog, reset, revert, restore — and the distinction between --force and --force-with-lease stops being pedantry. It is worth being fluent in those before you need them at speed.

tmux

tmux keeps a session alive in a server process, so your programs never learn that your connection dropped. Learn four things — new -s, attach -t, Ctrl-b d, Ctrl-b z — and the rest is optional for a long time.

tmux new -s name

Starts a named session that keeps running after you disconnect.

tmux new -s flighttest

tmux attach -t / ls

Lists sessions and reattaches to one.

tmux ls
tmux attach -t flighttest
tmux a          # most recent

Ctrl-b d

Detaches, leaving everything running.

# Ctrl-b then d

tmux new -d -s x 'cmd'

Launches a long job detached, in one line. Ideal for kicking off remote work over ssh.

ssh prod "tmux new -d -s import './import.sh > import.log 2>&1'"

panes & windows

Ctrl-b % splits vertically, " horizontally, c makes a window, o cycles panes, z zooms one pane full-screen.

# Ctrl-b %   split right
# Ctrl-b "   split down
# Ctrl-b z   zoom / unzoom

copy mode

Ctrl-b [ enters scrollback, Ctrl-b ] pastes. The selection keys follow mode-keys, which defaults to emacs unless $EDITOR or $VISUAL contains vi: C-Space then M-w under emacs, Space then Enter under vi.

# Ctrl-b [ then arrows or /search

tmux kill-session

Ends a session and everything in it.

tmux kill-session -t flighttest
tmux kill-server

tmux capture-pane

Dumps a pane's scrollback to a file, which is handy for feeding output to an agent.

tmux capture-pane -pS -2000 -t build > build-scrollback.txt

Git essentials

The daily surface. add -p for commits that say one thing, diff main...HEAD for reviewing a branch against its merge base, and log -S when you need to find the commit that introduced a string — the best archaeology tool git has.

git status -sb

Short status with branch and tracking info.

git status -sb

git add -p

Stages hunk by hunk, so one commit says one thing.

git add -p src/api.ts

git diff variants

Working tree vs index, index vs HEAD, or against another branch.

git diff                 # unstaged
git diff --staged        # what you are about to commit
git diff main...HEAD     # your branch's changes only

git log --oneline --graph

Readable history with branch topology.

git log --oneline --graph --decorate -20
git log --oneline main..HEAD

git log -S

Finds the commit that introduced or removed a string. The single best archaeology tool in git.

git log -S 'MAX_RETRIES' --oneline
git log -p -S 'joystick_calibrate' -- firmware/

git blame

Line-by-line authorship. -w ignores whitespace-only changes.

git blame -w -L 40,60 src/parser.ts

git switch / restore

The modern, unambiguous replacements for the many meanings of checkout.

git switch -c feature/graph-nodes
git restore src/app.ts            # discard working changes
git restore --staged src/app.ts   # unstage

git stash

Parks uncommitted work so you can switch context.

git stash push -m 'wip node layout' -- src/
git stash list
git stash pop

git rebase -i

Rewrites your local commits: squash, reorder, reword, drop.

git rebase -i main
git rebase -i HEAD~5

git push --force-with-lease

Force-pushes only if nobody else has pushed since you last fetched.

git push --force-with-lease origin feature/graph-nodes

git commit --amend

Folds staged changes into the previous commit, or just fixes the message.

git commit --amend --no-edit

Git recovery

Almost nothing in git is truly lost for about thirty days — the expiry for unreachable entries, which is the case a bad reset or rebase creates. reflog records every position HEAD has held, including the ones no branch points at, so a bad reset or rebase is recoverable if you look before you panic.

git reflog

Every position HEAD has held, including ones no branch points at. This is how you undo almost anything.

git reflog -20
git reset --hard HEAD@{3}

git reset --soft/--mixed/--hard

Moves the branch pointer, keeping changes staged, unstaged, or discarding them entirely.

git reset --soft HEAD~1    # undo commit, keep changes staged
git reset HEAD~1           # undo commit, keep changes in working tree
git reset --hard HEAD~1    # undo commit, discard changes

git revert

Creates a new commit that undoes an old one. The safe choice on a shared branch.

git revert a1b2c3d
git revert -m1 <merge-sha>

git cherry-pick

Applies one commit from another branch onto this one.

git cherry-pick a1b2c3d
git cherry-pick a1b2c3d^..e4f5g6h

git bisect

Binary-searches history for the commit that broke something.

git bisect start
git bisect bad
git bisect good v1.4.0
# test, then: git bisect good | git bisect bad
git bisect reset

git bisect run

Automates the search with a test command. Walk away and come back to the answer.

git bisect start HEAD v1.4.0
git bisect run npm test -- parser.test.ts

git clean -nd

Lists untracked files that -fd would delete. Preview first.

git clean -nd
git clean -fd
git clean -fdx    # also removes gitignored files

git fsck --lost-found

Finds dangling commits when even the reflog cannot help.

git fsck --lost-found --no-reflogs

Worktrees for parallel agents

One repository, several working directories, one branch each. This is what makes parallel agent work practical: no shared index, no branch switching underneath a running process, and a separate dev server port per task.

git worktree add

Checks out another branch into a second directory sharing one .git. Two agents can then work without fighting over the index.

git worktree add ../app-graph feature/graph-nodes
git worktree add -b fix/auth ../app-auth main

git worktree list / remove

Shows and cleans up worktrees.

git worktree list
git worktree remove ../app-auth
git worktree prune

parallel agent pattern

One worktree per task means each agent gets isolated files, its own branch, and its own dev server port.

for t in graph auth export; do
  git worktree add -b "feat/$t" "../app-$t" main
done
# then run one agent per directory

Claude Code from the shell

Print mode turns the model into a pipeline stage. Scope tool permissions to specific commands rather than granting Bash wholesale, keep the work on a branch, and read the diff rather than the summary of the diff.

claude

Starts an interactive session in the current directory.

cd ~/projects/app && claude

claude -p "query"

Print mode: runs non-interactively, prints the result, exits. This is what makes it scriptable.

claude -p 'summarise the failing tests and propose a fix'

piping into claude -p

Anything on stdin becomes context.

git diff main...HEAD | claude -p 'review this diff for auth and injection risks'
journalctl -u app -p err --since '1 hour ago' | claude -p 'group these errors by root cause'

--output-format json

Emits structured output you can parse with jq. Options are text, json and stream-json.

claude -p 'list the exported functions in src/api.ts as JSON' --output-format json | jq -r '.result'

--allowedTools

Pre-approves specific tools so an unattended run does not stop at a permission prompt.

claude -p 'run the test suite and fix failures' \
  --allowedTools 'Bash(npm test:*)' 'Read' 'Edit'

claude -c / -r

Continues the most recent conversation, or resumes a specific session by ID.

claude -c
claude -c -p 'now add tests for that'
claude -r abc123 'finish the PR'

--add-dir

Grants access to directories outside the working tree, such as a sibling repo.

claude --add-dir ../shared-types ../firmware

claude mcp

Adds and manages MCP servers for the current project or globally.

claude mcp list
claude mcp add --help

CLAUDE.md

Project memory. Conventions, commands and constraints that get loaded every session.

cat > CLAUDE.md <<'EOF'
## Commands
- test: `pytest -q`
- lint: `ruff check .`

## Conventions
- Pydantic v2 models only; no dict returns from routes.
EOF

--dangerously-skip-permissions

Skips every permission prompt.

# only inside a throwaway container or VM
docker run --rm -it -v "$PWD:/w" -w /w my-agent-image \
  claude -p 'refactor the parser' --dangerously-skip-permissions

headless in a loop

Print mode composes with normal shell tools, so agents become pipeline stages.

for f in src/**/*.py; do
  claude -p "add type hints to $f, edit in place" \
    --allowedTools 'Read' 'Edit' >/dev/null
done

docs

The CLI reference is the source of truth and changes often.

# https://docs.claude.com/en/docs/claude-code/cli-reference
claude --help

Anthropic API from bash

When you want the model without the agent, curl and jq are enough. Build the request body with jq -n --arg rather than string concatenation — prompts contain quotes and newlines, and hand-built JSON breaks on both.

curl the Messages API

A direct request, no SDK required.

curl -fsS https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H 'anthropic-version: 2023-06-01' \
  -H 'content-type: application/json' \
  -d '{"model":"claude-sonnet-4-6","max_tokens":512,"messages":[{"role":"user","content":"hi"}]}' \
  | jq -r '.content[0].text'

building the body with jq

Constructs JSON safely instead of concatenating strings that break on quotes and newlines.

body=$(jq -n --arg m 'claude-sonnet-4-6' --arg p "$(cat prompt.txt)" \
  '{model:$m, max_tokens:1024, messages:[{role:"user", content:$p}]}')
curl -fsS https://api.anthropic.com/v1/messages -d "$body" ...

keys out of your shell history

Read the key from a file or a keychain rather than exporting it in .bashrc.

export ANTHROPIC_API_KEY="$(cat ~/.config/anthropic/key)"
chmod 600 ~/.config/anthropic/key

Shell speed

Small savings, applied hundreds of times a day. Ctrl-R and !$ are the highest-value pair; fzf is the highest-value single install. Set HISTCONTROL=ignoreboth so secrets typed with a leading space stay out of your history file.

Ctrl-R

Reverse-searches your history. Press again to cycle further back.

# Ctrl-R then type part of the command

!! and !$

The previous command, and its last argument.

mkdir -p /srv/app/releases
cd !$
sudo !!

Ctrl-A / E / U / K / W

Line start, line end, delete to start, delete to end, delete previous word.

# Ctrl-A jump to start, Ctrl-U wipe the line

cd - and pushd/popd

Jump back to the previous directory, or keep a stack of them.

cd /var/log && cd -
pushd /srv/app; popd

alias & functions

Aliases for renaming, functions for anything taking arguments.

alias ll='ls -lah'
mkcd() { mkdir -p -- "$1" && cd -- "$1"; }

port killer

Frees a port something forgot to release.

kill -9 "$(lsof -ti :3000)"
fuser -k 3000/tcp

fzf

Fuzzy finder that turns any list into an interactive picker. Its Ctrl-R replacement alone is worth it.

vim "$(fzf)"
git switch "$(git branch --format='%(refname:short)' | fzf)"

clipboard from the shell

Pipe output straight to the clipboard.

cat id_ed25519.pub | xclip -sel clip     # X11
cat id_ed25519.pub | wl-copy               # Wayland
cat id_ed25519.pub | pbcopy                # macOS

history hygiene

Keeps secrets and duplicates out of your history file.

# ~/.bashrc
HISTCONTROL=ignoreboth
HISTSIZE=50000
shopt -s histappend

Tooling & runners

A project that anyone can start needs one discoverable entry point. A Makefile or justfile with dev, test and deploy targets is documentation that cannot go stale, because it is the thing people actually run.

modern replacements

Drop-in upgrades worth installing on any box you use daily.

rg      # grep
fd      # find
bat     # cat with syntax highlighting
eza     # ls
delta   # git diff pager
btop    # top
dust    # du
jless   # jq output pager

direnv

Loads and unloads environment variables as you cd in and out of a project.

echo 'export DATABASE_URL=postgres://localhost/app' > .envrc
direnv allow

uv

Fast Python package and environment manager.

uv venv && source .venv/bin/activate
uv pip install -r requirements.txt
uv run pytest -q

pipx

Installs Python CLI tools into isolated environments so they never fight your project deps.

pipx install ruff
pipx list

nvm / pnpm

Node version switching and a faster package manager.

nvm use 22
pnpm install --frozen-lockfile
pnpm dlx create-next-app@latest

make as a task runner

A Makefile is a discoverable, dependency-free front door to your project's commands.

.PHONY: dev test
dev:
	uvicorn main:app --reload
test:
	pytest -q

just

A task runner with make's interface and none of its build semantics.

just --list
just deploy staging

entr / watchexec

Re-runs a command whenever files change.

rg --files -t py | entr -c pytest -q
watchexec -e ts,tsx -- npm run build

ss + curl smoke test

A three-line health check to paste after any deploy.

systemctl is-active app && \
  ss -tlpn | grep -q ':8000' && \
  curl -fsS localhost:8000/health | jq -e '.status=="ok"'