Linux, from the server up

[    0.000000] Linux version 6.8.0-generic (buildd@lcy02)
[    0.612441] Loading initial ramdisk ...
[  OK  ] Mounted /boot/efi.
[  OK  ] Started Journal Service.
[  OK  ] Reached target Network is Online.
[  OK  ] Started OpenSSH server daemon.
[  OK  ] Started nginx.service.
[  OK  ] Started api.service — Your app, finally.
[  OK  ] Reached target Multi-User System.
web-01 login:

Everything above happened before your app served a single byte. This is a tour of that machinery — what each piece is for, then the commands that drive it.

What Linux actually is

The kernel does five things: it hands out CPU time to processes, hands out memory, talks to disks and network cards, enforces who is allowed to touch what, and provides a filesystem. It does not include bash, ls, systemd, nginx, or apt. Those are userland — regular programs, replaceable, that a distribution bundles together.

So a "distro" is a curated bundle: kernel + GNU userland + a package manager + a set of defaults. Debian is the community-run base; Ubuntu is built from Debian with a predictable release cadence and long-term support.

They share apt, dpkg, the same file layout, and almost all of the same commands. Anything in this guide works on both.

The syscall boundary

There is exactly one door between your code and the kernel, and it is called a system call. When your Node server accepts a connection, it is really calling accept(). When it reads a file, openat() then read().

When it spawns a child process, clone() then execve(). Every language runtime is ultimately a wrapper around a few dozen syscalls.

This is worth internalising because it explains the shape of every problem you will hit. "Too many open files" is the kernel refusing an open(). "Permission denied" is the kernel refusing based on the credentials of the calling process.

"Address already in use" is the kernel refusing a bind(). The error is never mysterious once you know which syscall failed.

# trace a command and count syscalls by name
$ strace -c -f node server.js

# trace only file + network activity of a running process
$ sudo strace -p 4821 -e trace=openat,connect,accept4

# which kernel are we on, and what CPU/memory does it see?
$ uname -a
$ nproc
$ free -h

# which distro and release?
$ cat /etc/os-release
$ lsb_release -a

The filesystem

The layout is a convention called the Filesystem Hierarchy Standard, and it is genuinely useful, not arbitrary. Directories split along two axes: who owns it (the distro vs. you) and does it survive a reinstall.

Path	What lives there, and why you'll go there
/etc	All system configuration. Plain text, editable, version-controllable. Your nginx config, systemd units, cron files, SSH settings. If you're configuring something, it's here.
/var	Data that grows: logs (/var/log), databases (/var/lib), caches, mail queues. This is the directory that fills your disk.
/usr	The distro's own programs and libraries. /usr/bin holds nearly every command you type. Package-manager territory — don't hand-edit.
/usr/local	Software you installed outside the package manager. Deliberately separate so apt never clobbers it.
/opt	Self-contained third-party bundles that ship their own directory tree.
/srv	Data served by this machine. A conventional (and tidy) home for your web app's files.
/home	Human user directories. Service accounts usually live elsewhere or nowhere.
/tmp	Wiped on reboot, often a RAM disk. Anyone can write; use mktemp, never fixed filenames.
/proc	Not a disk. A live view of kernel and process state, rendered as files.
/sys	Also not a disk. Devices, drivers, cgroups — the kernel's object model as files.
/dev	Device nodes. /dev/null, /dev/urandom, /dev/sda.

Everything is a file

This phrase gets repeated without explanation. Concretely: the kernel exposes an enormous amount of state through the ordinary file API, so cat and echo become general-purpose inspection tools. You do not need a special utility to read a process's environment or memory usage — you read a file.

# everything the kernel knows about PID 4821
$ ls /proc/4821/
$ cat /proc/4821/cmdline | tr '\0' ' '   # exact command line
$ sudo cat /proc/4821/environ | tr '\0' '\n'  # its env vars
$ ls -l /proc/4821/cwd /proc/4821/exe     # where it runs, what binary
$ ls -l /proc/4821/fd/                    # every open file + socket

# system-wide
$ cat /proc/loadavg
$ cat /proc/meminfo | head
$ cat /proc/mounts

Disk and mounts

$ df -h                       # free space per mounted filesystem
$ df -i                       # free INODES — can run out with space left
$ du -xh --max-depth=1 /var | sort -h   # biggest dirs under /var
$ sudo lsof +L1                # deleted-but-still-open files eating space
$ lsblk                        # block devices and their mount points
$ findmnt                      # the mount tree, readably

Processes and signals

New processes are created by fork — a process clones itself — followed by exec, which replaces the clone's memory with a different program. That two-step is why a child inherits its parent's environment variables, working directory, open file descriptors, and user. It is also why a subprocess cannot change your shell's directory: it got a copy, not the original.

When a process exits it returns an exit code: 0 means success, anything else means failure. The parent is expected to collect it. If the parent doesn't, the dead child lingers as a zombie (harmless in small numbers, a bug in large ones).

If the parent dies first, the orphaned child is re-parented to PID 1, which always cleans up.

Signals

Signals are the kernel's way of interrupting a process. For web work, four matter:

Signal	Meaning
SIGTERM (15)	"Please shut down." The default for kill and what systemd sends first. Catchable — this is where you finish in-flight requests, close DB pools, then exit. Handle this one.
SIGINT (2)	What Ctrl-C sends. Usually treat it like SIGTERM.
SIGKILL (9)	"Die now." Cannot be caught, blocked, or handled — the kernel removes the process. No cleanup, no flushed buffers. A last resort, not a habit.
SIGHUP (1)	Historically "terminal hung up." Daemons repurposed it to mean "reload your config without dropping connections." nginx -s reload is a SIGHUP.

# the process tree, with PIDs
$ ps -ef --forest
$ pstree -p

# find by name, with full command line
$ pgrep -af node
$ ps aux | grep '[n]ode'       # brackets stop grep matching itself

# who is holding port 3000?
$ sudo ss -ltnp 'sport = :3000'
$ sudo lsof -i :3000

# graceful, then forceful
$ kill 4821            # SIGTERM
$ kill -HUP 4821       # reload config
$ kill -9 4821         # SIGKILL, last resort
$ pkill -f 'node server.js'

# live view, sorted by CPU then memory
$ top -o %MEM
$ htop

const server = app.listen(3000);

const shutdown = async (signal) => {
  console.log(`${signal} received, draining`);
  server.close(async () => {       // stop accepting, finish in-flight
    await db.end();
    process.exit(0);
  });
  setTimeout(() => process.exit(1), 10_000).unref();
};

process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT',  () => shutdown('SIGINT'));

Shell fluency

1. File descriptors and redirection

Every process starts with three open file descriptors: 0 = stdin, 1 = stdout, 2 = stderr. Redirection just points those numbers somewhere else before the program starts. That's the whole model.

cmd >  out.log        # stdout to file, truncating
cmd >> out.log        # stdout to file, appending
cmd 2> err.log        # stderr only
cmd > all.log 2>&1    # stdout to file, THEN stderr to same place
cmd &> all.log        # bash shorthand for the line above
cmd < input.txt       # feed a file to stdin
cmd > /dev/null 2>&1  # discard everything

# order matters: this sends stderr to the TERMINAL, not the file
cmd 2>&1 > all.log    # stderr copied the old stdout first

# see it AND save it
cmd 2>&1 | tee -a all.log

2. Exit codes and chaining

$ ls /nope; echo "exit was $?"    # $? holds the last exit code

build && deploy          # deploy only if build succeeded (exit 0)
build || notify-failure  # notify only if build failed
build; deploy            # run both regardless — rarely what you want

# pipelines report the LAST command's status by default
false | true; echo $?    # prints 0. Surprising and dangerous.

3. Environment, PATH, and shell types

Environment variables are inherited at fork. PATH is a colon-separated list of directories the shell searches, in order, to turn node into /usr/bin/node. When you type a command, the shell walks PATH and runs the first match.

Crucially, which startup files run depends on how the shell was started:

Shell kind	Reads	You get it when
login	/etc/profile, then ~/.bash_profile or ~/.profile	SSH-ing in, or su -
interactive non-login	~/.bashrc	Opening a new terminal tab locally
non-interactive	Nothing. Only what the parent passed in	Cron jobs, systemd services, scripts

$ which node              # first match on PATH
$ command -v node         # portable version; also finds aliases
$ readlink -f "$(command -v node)"   # resolve symlinks to the real binary
$ echo "$PATH" | tr ':' '\n'

# see the environment a non-interactive shell really gets
$ env -i /bin/bash -c 'echo $PATH'

4. Quoting and globbing

The shell expands your line before the program sees it. *.log is turned into a list of filenames by the shell, not by ls.

Unquoted $VAR is split on whitespace and glob-expanded; quoted "$VAR" is passed as one argument. Single quotes suppress all expansion.

file="my report.txt"
rm $file        # tries to delete TWO files: "my" and "report.txt"
rm "$file"      # correct

# safe iteration over filenames, even with spaces/newlines
find /srv/app/logs -name '*.log' -mtime +14 -print0 \
  | xargs -0 -r rm --

# heredoc: multi-line text, expanded
cat > /tmp/note.txt <<EOF
Deployed at $(date -Is)
EOF

# quoted delimiter = NO expansion (use for config files)
cat > /tmp/raw.txt <<'EOF'
Literal $HOME, literal $(date)
EOF

The boot sequence

Stage 1 · Firmware (UEFI or BIOS)

The motherboard initialises hardware and looks for a bootloader. On modern systems this means reading the EFI System Partition — a small FAT filesystem mounted at /boot/efi. On a cloud VM this stage is emulated and takes milliseconds.

Stage 2 · Bootloader (GRUB)

GRUB reads /boot/grub/grub.cfg, shows the menu you usually skip, and loads two files into memory: the kernel (/boot/vmlinuz-*) and the initramfs (/boot/initrd.img-*). It also passes the kernel command line, which includes the crucial argument telling the kernel which device is the root filesystem.

Stage 3 · Kernel + initramfs

The kernel decompresses, detects hardware, and mounts the initramfs — a small temporary root filesystem held in RAM. Its only job is to contain enough drivers to reach the real root filesystem, which might be on LVM, encrypted, or on a RAID array the kernel can't read unaided. Once the real root is mounted, the kernel pivots to it and executes /sbin/init.

Stage 4 · PID 1 (systemd)

On Debian and Ubuntu, /sbin/init is a symlink to systemd. It becomes PID 1, reads its unit files, works out the dependency graph, and starts everything it can in parallel until it reaches the default target — normally multi-user.target for a server. Your app starts here, if you told it to.

$ uptime -p                     # how long since the last boot
$ who -b                        # exact boot timestamp

$ systemd-analyze               # total boot time, split by stage
$ systemd-analyze blame         # slowest units, descending
$ systemd-analyze critical-chain # the dependency path that gated boot

$ journalctl -b                 # the whole current boot
$ journalctl -b -1 -p err       # errors from the PREVIOUS boot
$ journalctl --list-boots       # boots the journal still has

$ dmesg -T --level=err,warn     # kernel ring buffer, human timestamps
$ cat /proc/cmdline             # args GRUB handed the kernel

systemd

Everything systemd manages is a unit, described by a small INI file. The unit's suffix is its type:

Type	What it represents
.service	A process to run and supervise. This is 95% of what you'll write.
.socket	A listening socket systemd holds open, starting the service on first connection.
.timer	A schedule that activates a service. The modern alternative to cron.
.target	A named grouping of units — a synchronisation point, like multi-user.target.
.mount	A filesystem mount point, generated from /etc/fstab or written by hand.
.path	Watches a file or directory and activates a service when it changes.

Where unit files live

Three directories, in increasing order of priority. A file in a higher-priority directory completely replaces one with the same name below it.

Directory	Owner
/lib/systemd/system	Shipped by packages. Never edit — apt upgrade will overwrite you.
/run/systemd/system	Runtime-generated, gone on reboot.
/etc/systemd/system	Yours. Your own units and your overrides go here — and they outrank both of the above.

Anatomy of a service unit

[Unit]
Description=Orders API (Node)
Documentation=https://git.example.com/orders
# Ordering only — does NOT pull these in
After=network-online.target postgresql.service
# Soft dependency: try to start it, don't fail if it can't
Wants=network-online.target
# Hard dependency: if postgres stops, stop this too
Requires=postgresql.service
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=simple                     # process stays in foreground (see note)
User=api
Group=api
WorkingDirectory=/srv/orders-api
EnvironmentFile=/etc/orders-api/env      # chmod 600, owned by root
Environment=NODE_ENV=production
ExecStart=/usr/bin/node /srv/orders-api/server.js
ExecReload=/bin/kill -HUP $MAINPID

Restart=always                  # restart on ANY exit, including exit 0
RestartSec=3                    # wait 3s between attempts
TimeoutStopSec=30               # then SIGKILL after graceful window
KillSignal=SIGTERM

# --- cheap, high-value hardening ---
NoNewPrivileges=true            # cannot gain privileges via setuid
PrivateTmp=true                 # its own /tmp, invisible to others
ProtectSystem=strict            # entire filesystem read-only...
ReadWritePaths=/srv/orders-api/uploads   # ...except this
ProtectHome=true
ProtectKernelTunables=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# --- resource ceilings ---
MemoryMax=1G                    # OOM-kill this unit, not the database
LimitNOFILE=65535               # file descriptors, for many connections

[Install]
WantedBy=multi-user.target      # what `enable` hooks it into

Choosing a Type=

Type	Use when
simple	Your process runs in the foreground and never forks. Default. Correct for node, gunicorn, uvicorn, most modern servers.
exec	Like simple, but systemd waits until the binary has actually been exec'd before considering it started. Slightly better failure reporting.
notify	The process tells systemd "I'm ready" via sd_notify. Best for zero-downtime ordering, since dependents wait for real readiness.
forking	Classic daemons that background themselves. Needs PIDFile=. Avoid unless the software demands it.
oneshot	Runs, does a job, exits. Use with RemainAfterExit=yes for setup tasks, and for timer-driven jobs.

Driving it

# after ANY edit to a unit file
$ sudo systemctl daemon-reload

$ sudo systemctl start api
$ sudo systemctl stop api
$ sudo systemctl restart api      # full stop + start
$ sudo systemctl reload api       # runs ExecReload, keeps connections
$ sudo systemctl enable --now api # start now AND at every boot
$ sudo systemctl disable api

# inspection
$ systemctl status api            # state, PID, memory, last log lines
$ systemctl is-active api
$ systemctl is-enabled api
$ systemctl cat api               # the effective unit file + overrides
$ systemctl show api -p MemoryMax -p Restart
$ systemctl list-units --failed   # everything currently broken
$ systemctl list-dependencies api

# override a PACKAGED unit without editing it
$ sudo systemctl edit nginx       # writes a drop-in .conf snippet

Scheduled work

cron: the five fields

┌─────────── minute        (0-59)
│ ┌───────── hour          (0-23)
│ │ ┌─────── day of month  (1-31)
│ │ │ ┌───── month         (1-12)
│ │ │ │ ┌─── day of week   (0-7, both 0 and 7 = Sunday)
│ │ │ │ │
* * * * *  command to run

# operators
*/15 * * * *      # every 15 minutes
0 */4 * * *       # every 4 hours, on the hour
30 2 * * 1-5      # 02:30, Monday to Friday
0 0 1,15 * *      # midnight on the 1st and 15th
@reboot           # once at boot (use a systemd unit instead)
@daily            # same as 0 0 * * *

Where cron jobs live

Location	Notes
crontab -e	Per-user. Runs as that user. Five fields, no user column. Not in your git repo — easy to lose.
/etc/cron.d/name	Drop-in files. Six fields — a username column sits between the schedule and the command. Deployable with config management. Preferred.
/etc/crontab	System-wide, also six fields. Leave it to the distro.
/etc/cron.{hourly,daily,weekly}	Drop an executable script in; no schedule syntax at all. Run by run-parts. Filenames must have no dots — backup.sh is silently ignored, backup works.

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=""

# m  h  dom mon dow  USER  COMMAND
*/10 *  *   *   *    api   flock -n /tmp/sync.lock /srv/orders-api/bin/sync >> /var/log/orders/sync.log 2>&1
15   3  *   *   *    api   /usr/bin/node /srv/orders-api/scripts/nightly.js >> /var/log/orders/nightly.log 2>&1

# file MUST end with a newline or cron ignores the last line

The four things that make cron jobs fail

1. PATH. Cron gives you a minimal PATH and sources no dotfiles. node, python from a venv, aws, and anything from nvm or asdf will not be found. Use absolute paths, or set PATH= at the top of the file.
2. Environment variables. DATABASE_URL from your .bashrc or .env does not exist. Source it explicitly inside the script.
3. Silent output. Anything printed to stdout/stderr gets mailed to the user — and on a server with no mail transport, discarded. Always redirect to a log file with >> file 2>&1, or log to the journal with logger.
4. Overlap. A job that takes 12 minutes on a 10-minute schedule will pile up until the box falls over. Wrap it in flock -n, which takes a lock and exits immediately if another copy holds it.
5. Percent signs. In a crontab, an unescaped % is turned into a newline. date +%F silently breaks; write date +\%F.

#!/usr/bin/env bash
set -euo pipefail

# everything explicit: no inherited environment to rely on
export PATH=/usr/local/bin:/usr/bin:/bin
cd /srv/orders-api

# load secrets from a root-owned, 0600 file
set -a; source /etc/orders-api/env; set +a

exec /usr/bin/node scripts/nightly.js

# did cron even fire? (Ubuntu/Debian log cron to the journal)
$ journalctl -u cron --since "1 hour ago"
$ journalctl -t CRON --since today

# reproduce cron's environment exactly, then run your command in it
$ env -i /bin/bash --noprofile --norc -c '/srv/orders-api/bin/sync'

# list what's scheduled
$ crontab -l                    # yours
$ sudo crontab -l -u api        # someone else's
$ ls -la /etc/cron.d/ /etc/cron.daily/

systemd timers: the better default

A timer is two files — a .service saying what, and a .timer saying when. It's more typing than a crontab line, and you get: real logs in the journal, exit-status tracking, Persistent= to catch up on missed runs after downtime, dependency ordering, resource limits, the same sandboxing directives as any service, and randomised delays so a fleet of servers doesn't stampede your API at exactly 03:00.

[Unit]
Description=Nightly order reconciliation
After=network-online.target postgresql.service

[Service]
Type=oneshot
User=api
WorkingDirectory=/srv/orders-api
EnvironmentFile=/etc/orders-api/env
ExecStart=/usr/bin/node scripts/nightly.js
# no redirection needed — stdout/stderr go to the journal

[Unit]
Description=Run order reconciliation at 03:15

[Timer]
OnCalendar=*-*-* 03:15:00
RandomizedDelaySec=600    # spread up to 10 min across the fleet
Persistent=true           # if the box was off at 03:15, run on boot
Unit=nightly.service

[Install]
WantedBy=timers.target

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now nightly.timer

$ systemctl list-timers --all        # next run, last run, for everything
$ systemctl start nightly.service    # run it NOW, on demand
$ journalctl -u nightly.service -n 50

# validate a schedule expression before trusting it
$ systemd-analyze calendar "Mon *-*-* 02:30:00"
$ systemd-analyze calendar "*:0/15"   # every 15 minutes

Logs and the journal

System	Holds	Read with
journald	Everything systemd-managed writes to stdout/stderr, plus syslog messages and kernel messages. Structured, indexed, binary format.	journalctl
kernel ring buffer	Kernel messages only — drivers, OOM kills, network resets, disk errors. Fixed-size, overwrites itself.	dmesg
plain files	Apps that write their own files: /var/log/nginx/access.log, /var/log/auth.log, your app's own logs.	tail, grep, less

The big shift with systemd is that your service should not manage log files at all. Write to stdout and stderr; journald captures both, tags them with the unit name, PID, user and timestamp, and handles rotation and disk limits for you. This is the same discipline containers enforce, so it travels well.

# one unit, follow live (the equivalent of tail -f)
$ journalctl -u api -f
$ journalctl -u api -n 200 --no-pager

# time windows — accepts plain English
$ journalctl -u api --since "2 hours ago"
$ journalctl -u api --since "09:00" --until "09:30"
$ journalctl --since yesterday --until today

# severity: 0=emerg 3=err 4=warning 6=info 7=debug
$ journalctl -p err -b            # errors and worse, this boot
$ journalctl -p warning..err

# several units at once, interleaved chronologically
$ journalctl -u api -u nginx -u postgresql --since "30 min ago"

# machine-readable, for piping into jq
$ journalctl -u api -o json-pretty -n 5
$ journalctl -u api -o cat         # message text only, no metadata

# by executable, by user, by PID
$ journalctl /usr/sbin/sshd
$ journalctl _UID=1001
$ journalctl _PID=4821

# how much disk is the journal using, and prune it
$ journalctl --disk-usage
$ sudo journalctl --vacuum-time=14d
$ sudo journalctl --vacuum-size=500M

$ sudo mkdir -p /var/log/journal
$ sudo systemd-tmpfiles --create --prefix /var/log/journal
$ sudo systemctl restart systemd-journald

# then cap it in /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=1G
MaxRetentionSec=30day

The kernel ring buffer

dmesg shows what the kernel itself said. It is where you find the truth about hardware, disk I/O errors, network interface resets, and — most relevant for web work — the OOM killer, which terminates processes when the machine runs out of memory. An app that vanishes with no application-level error and no systemd stop request was almost certainly OOM-killed, and only dmesg will tell you.

$ sudo dmesg -T                        # -T = human-readable timestamps
$ sudo dmesg -T --level=err,crit,alert
$ sudo dmesg -w                        # follow, like tail -f

# the smoking gun for a vanished process
$ sudo dmesg -T | grep -iE 'out of memory|oom-kill|killed process'
$ journalctl -k -b --grep='Out of memory'

Plain log files and rotation

nginx, PostgreSQL and older software still write their own files. logrotate — itself run by a daily timer — renames, compresses and eventually deletes them according to per-package rules in /etc/logrotate.d/.

/var/log/orders/*.log {
    daily
    rotate 14              # keep two weeks
    compress
    delaycompress          # don't gzip the most recent rotation
    missingok              # don't error if the file isn't there yet
    notifempty
    create 0640 api adm    # mode/owner for the fresh file
    sharedscripts
    postrotate
        systemctl reload orders-api >/dev/null 2>&1 || true
    endscript
}

$ sudo tail -f /var/log/nginx/error.log
$ sudo tail -f /var/log/nginx/access.log | grep -v ' 200 '

# top 20 IPs, then top paths, from an access log
$ awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
$ awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# search compressed rotations too
$ zgrep -h '500 ' /var/log/nginx/access.log.*.gz | wc -l

# dry-run a rotation config
$ sudo logrotate -d /etc/logrotate.d/orders-api
$ sudo logrotate -f /etc/logrotate.d/orders-api   # force it now

Resources and limits

Load average — not a percentage

uptime reports three numbers: the average number of processes running or waiting over the last 1, 5 and 15 minutes. On Linux this includes processes blocked on disk I/O, not just CPU. So the number only means something relative to your core count: a load of 4.0 on a 4-core box is fully busy; on a 16-core box it's idle.

The 1/5/15 spread tells you direction. 8.2 / 3.1 / 1.4 means something started going wrong minutes ago and is escalating. 1.4 / 3.1 / 8.2 means the incident is already resolving.

Memory — and why free looks alarming

Linux deliberately uses all spare RAM as page cache for disk data. That memory is instantly reclaimable, so "almost no free memory" is normal and healthy. The number that matters is available, not free.

For a process, RSS (resident set size) is physical RAM actually in use; VSZ (virtual size) is address space reserved and is often wildly larger — a JVM or a Go binary can show gigabytes of VSZ while using a fraction of it. Judge by RSS.

$ uptime                          # load average, vs. `nproc` cores
$ free -h                         # look at "available", not "free"
$ vmstat 1 5                      # 5 one-second samples; watch si/so (swap)

# top 10 processes by real memory use
$ ps aux --sort=-rss | head -11
$ ps -eo pid,user,rss,%cpu,etime,cmd --sort=-rss | head

# per-unit accounting, straight from cgroups
$ systemd-cgtop
$ systemctl status api | grep Memory

# is anything swapping? swap usage is fine; swap THRASHING is not
$ swapon --show

File descriptors

Every open file, every socket, every pipe is a descriptor, and both the kernel and each process have a ceiling. Hit it and you get EMFILE: too many open files — which on a web server usually means a leak (unclosed HTTP agents, DB connections, or file handles), not genuine demand.

$ ulimit -n                       # soft limit for THIS shell
$ ulimit -Hn                      # hard ceiling

# the limits a RUNNING process actually has
$ cat /proc/4821/limits | grep -i files
$ ls /proc/4821/fd | wc -l        # how many it's using right now

# system-wide
$ cat /proc/sys/fs/file-nr        # allocated / free / max

Disk

A full disk breaks things in confusing ways: PostgreSQL refuses writes, nginx can't log, sessions fail, and deploys half-complete. The three usual culprits are logs, Docker images and build caches, and old package files.

$ df -h /                                     # confirm which fs is full
$ sudo du -xh --max-depth=1 / | sort -h | tail -15

$ sudo journalctl --vacuum-size=200M          # journal
$ sudo apt-get clean                          # cached .deb files
$ sudo apt-get autoremove --purge             # orphaned packages + old kernels
$ docker system df                            # what docker is holding
$ docker system prune -a --volumes            # DESTRUCTIVE — read df output first

$ sudo lsof +L1                               # deleted files still held open

Users and permissions

UID 0 is root and bypasses all permission checks. Regular humans get UIDs from 1000 up. Between roughly 100 and 999 sit system users: accounts created for daemons, with no password, no home directory, and /usr/sbin/nologin as their shell so nobody can log in as them.

$ id                              # your uid, gid, and all groups
$ id api
$ getent passwd api               # better than grepping /etc/passwd
$ groups api
$ who; w                          # who is logged in right now
$ last -n 20                      # recent login history

# a proper service account: no login, no home, no password
$ sudo adduser --system --group --no-create-home \
      --shell /usr/sbin/nologin api

# add a human to a group (takes effect at next login)
$ sudo usermod -aG docker deploy   # -a is vital: without it you REPLACE groups

Reading a permission string

-rwxr-x---  1 api  api   4096 Aug 20 09:12 deploy.sh
│└┬┘└┬┘└┬┘    └┬┘  └┬┘
│ │  │  │      │    └── group: members of "api"
│ │  │  │      └─────── owner: user "api"
│ │  │  └────────────── other: --- (nothing)
│ │  └───────────────── group: r-x (read, execute)
│ └──────────────────── owner: rwx (read, write, execute)
└────────────────────── type: - file, d dir, l symlink, s socket

Octal:  r=4  w=2  x=1   →  rwxr-x--- = 750

On a directory the bits mean something different, and this trips everyone up: r lets you list the names inside, x lets you traverse into it and access files by name, w lets you create and delete entries. Deleting a file therefore depends on the directory's permissions, not the file's. And you need x on every directory in a path — if nginx can read /srv/app/public/index.html but gets 403, check x on /srv, /srv/app and /srv/app/public.

Mode	Use for
644	Ordinary files: source, static assets, public config
755	Directories, and executable scripts
600	Secrets: .env, API keys, SSH private keys, TLS keys
640	Files the owner writes and one group reads — logs, typically
700	Private directories: ~/.ssh

The three extra bits

• setuid (4000) — the program runs as its owner, not as you. This is how passwd (owned by root) can edit /etc/shadow. Shows as s in the owner's execute slot. Never set this on your own scripts.
• setgid (2000) — on a directory, new files inherit the directory's group. Genuinely useful for a shared upload directory.
• sticky (1000) — in a world-writable directory, only the file's owner may delete it. This is why /tmp is 1777 and not a free-for-all.

$ sudo chown -R api:api /srv/orders-api
$ sudo chmod 600 /etc/orders-api/env
$ sudo chmod 750 /srv/orders-api

# dirs 755, files 644 — in one pass, without a loop
$ sudo find /srv/orders-api -type d -exec chmod 755 {} +
$ sudo find /srv/orders-api -type f -exec chmod 644 {} +

# shared upload dir: group-writable, group inherited
$ sudo chgrp www-data /srv/uploads
$ sudo chmod 2775 /srv/uploads

# default mode for NEW files (022 → 644 files, 755 dirs)
$ umask

sudo

sudo runs one command as another user, logging who did what — to the journal on any systemd box (journalctl -t sudo), and additionally to /var/log/auth.log on Ubuntu, which still installs rsyslog. Debian 12 no longer does, so that file may not exist. That audit trail is the point — it's why you disable direct root login and give humans sudo instead.

$ sudo -u api node scripts/one-off.js   # run AS the service user
$ sudo -i                               # full root login shell
$ sudo -l                               # what am I allowed to run?

# this FAILS: redirection is done by YOUR shell, before sudo runs
$ sudo echo "x" > /etc/protected     # permission denied
# these work:
$ echo "x" | sudo tee /etc/protected >/dev/null
$ sudo sh -c 'echo "x" > /etc/protected'

# NEVER edit /etc/sudoers directly — visudo validates before saving
$ sudo visudo
$ sudo visudo -f /etc/sudoers.d/deploy   # drop-in, preferred

SSH

Key authentication is asymmetric: your private key never leaves your laptop; the matching public key is appended to ~/.ssh/authorized_keys on the server. At login the server sends a challenge only the private key can answer. Nothing secret crosses the wire, so there is no password to brute-force, phish or leak.

# ed25519: shorter, faster, and the current default recommendation
$ ssh-keygen -t ed25519 -C "reuben@laptop"

# copy the PUBLIC key to a server you can already reach
$ ssh-copy-id -i ~/.ssh/id_ed25519.pub deploy@web-01

# load into the agent so you type the passphrase once per session
$ ssh-add ~/.ssh/id_ed25519
$ ssh-add -l

# diagnose a refused login — -vvv shows every key it offers
$ ssh -vvv deploy@web-01

~/.ssh/config

Worth setting up properly once. It turns ssh -i ~/.ssh/prod_ed25519 -p 2222 [email protected] into ssh web-01, and it applies to scp, rsync and git too.

Host web-01
    HostName 203.0.113.10
    User deploy
    Port 2222
    IdentityFile ~/.ssh/prod_ed25519
    IdentitiesOnly yes        # offer ONLY this key, not all of them

# reach a private box through a bastion, without landing on the bastion
Host db-01
    HostName 10.0.3.14
    User deploy
    ProxyJump web-01

Host *
    ServerAliveInterval 30    # stop idle sessions being dropped
    ServerAliveCountMax 3
    AddKeysToAgent yes

Hardening sshd

PermitRootLogin no             # humans use sudo, not root
PasswordAuthentication no      # keys only — the big one
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowUsers deploy              # explicit allow-list
MaxAuthTries 3
LoginGraceTime 20
X11Forwarding no
AllowAgentForwarding no        # see the gotcha below
ClientAliveInterval 300

# 1. validate the syntax BEFORE restarting
$ sudo sshd -t

# 2. confirm your settings actually won. Drop-ins load in LEXICAL order
#    and the FIRST value found wins, so a lower-numbered file beats you.
#    Ubuntu cloud images ship 50-cloud-init.conf with password auth ON.
$ sudo sshd -T | grep -iE 'passwordauthentication|permitrootlogin'

# 3. keep your current session open; restart in it
$ sudo systemctl restart ssh

# 4. from a SECOND terminal, prove you can still get in
$ ssh deploy@web-01

# who has been trying? Debian 12 dropped rsyslog, so there may be
# no auth.log at all — the journal is the reliable source on both.
$ journalctl -u ssh --since today | grep -E 'Failed|Accepted'
$ sudo grep -E 'Failed|Accepted' /var/log/auth.log | tail -30   # Ubuntu only

The other things SSH carries

# sync a build, deleting removed files, showing progress
$ rsync -avz --delete --exclude '.git' ./dist/ web-01:/srv/app/dist/

# pull a file down
$ scp web-01:/var/log/nginx/error.log ./

# LOCAL forward: reach the server's private Postgres from your laptop
# localhost:15432 → (through web-01) → 10.0.3.14:5432
$ ssh -L 15432:10.0.3.14:5432 web-01
$ psql -h 127.0.0.1 -p 15432 -U app orders

# run one command and exit
$ ssh web-01 'systemctl is-active api'

The firewall

The model is: deny all incoming, allow all outgoing, then open individual ports. Note that a cloud provider's security groups are a separate layer sitting in front of the host — traffic must pass both, so a rule that looks right can still be blocked one level up.

# ALWAYS allow SSH before enabling, or you're locked out
$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing
$ sudo ufw allow 22/tcp comment 'ssh'
$ sudo ufw allow 80,443/tcp comment 'web'
$ sudo ufw enable

$ sudo ufw status verbose
$ sudo ufw status numbered
$ sudo ufw delete 3

# restrict a port to one source — e.g. Postgres from the app subnet only
$ sudo ufw allow from 10.0.3.0/24 to any port 5432 proto tcp

# rate-limit ssh: blocks an IP after 6 attempts in 30s
$ sudo ufw limit 22/tcp

# what the kernel actually ended up with
$ sudo nft list ruleset | head -40

Networking and sockets

Binding: 127.0.0.1 vs 0.0.0.0

When your app calls listen(), it binds to a specific interface. 127.0.0.1 (loopback) accepts connections only from the same machine. 0.0.0.0 accepts on every interface, including the public one.

The correct architecture for a web app is: your app binds to 127.0.0.1 (or better, a Unix socket), and nginx — which does bind publicly on 80 and 443 — proxies to it. Your application server never faces the internet directly, so a misconfigured firewall can't expose it.

# l=listening t=tcp u=udp n=numeric p=process
$ sudo ss -ltnp

State  Local Address:Port   Process
LISTEN 127.0.0.1:3000       node      ← private, correct for an app
LISTEN 0.0.0.0:80           nginx     ← public, correct for a proxy
LISTEN 0.0.0.0:5432         postgres  ← PUBLIC DATABASE. Fix this.

$ sudo ss -ltnp 'sport = :3000'     # one port
$ ss -tn state established           # current connections
$ ss -s                              # socket summary by state

Interfaces and routes

$ ip -brief addr                # interfaces and their IPs, compactly
$ ip route                      # routing table; "default via" is your gateway
$ ip route get 1.1.1.1          # which interface/source IP would be used
$ ip -s link show eth0          # packet counters, errors, drops

# Ubuntu configures interfaces via netplan (YAML → systemd-networkd)
$ ls /etc/netplan/
$ sudo netplan try              # applies, auto-reverts if you don't confirm

Name resolution

Turning api.example.com into an address goes through several steps, and skipping over them is why "DNS is fine, I checked with dig" is so often wrong. dig queries a DNS server directly; your application goes through the C library resolver, which consults /etc/nsswitch.conf, then /etc/hosts, and only then DNS via /etc/resolv.conf.

$ getent hosts api.example.com    # uses the SAME path your app uses
$ dig +short api.example.com      # asks DNS directly, ignores /etc/hosts
$ dig api.example.com @1.1.1.1    # ask a specific resolver

$ cat /etc/hosts                  # static overrides win
$ cat /etc/resolv.conf            # often a symlink to systemd-resolved
$ resolvectl status               # the real upstream servers in use
$ resolvectl flush-caches

Testing from the server itself

# full request/response headers, following redirects
$ curl -sSvL https://api.example.com/health

# status code and timing breakdown only
$ curl -s -o /dev/null \
    -w '%{http_code} dns=%{time_namelookup} tls=%{time_appconnect} total=%{time_total}\n' \
    https://api.example.com

# bypass DNS to test a specific origin before cutting over
$ curl -sv --resolve api.example.com:443:10.0.3.20 https://api.example.com

# is the app itself up, behind the proxy?
$ curl -si http://127.0.0.1:3000/health

# can we even reach a port?
$ nc -zv db.internal 5432
$ mtr -rwc 20 api.example.com     # traceroute + ping, combined

Packages

$ sudo apt update                 # refresh the package INDEX only
$ sudo apt upgrade                # install newer versions of what's installed
$ sudo apt full-upgrade           # will remove packages if needed to upgrade

$ sudo apt install nginx
$ sudo apt remove nginx           # binaries go, config stays
$ sudo apt purge nginx            # config goes too
$ sudo apt autoremove --purge     # orphaned deps and old kernels

$ apt policy nginx                # installed vs candidate vs available
$ apt show nginx
$ apt-cache depends nginx

# which package owns this file? which files did this package install?
$ dpkg -S /usr/sbin/nginx
$ dpkg -L nginx

Why the distro's Node is ancient

Debian and Ubuntu freeze package versions at release and then only backport security fixes — that stability is the entire product. So apt install nodejs on a two-year-old release gives you a runtime several majors behind. For language runtimes you almost always want a separate source:

• A vendor apt repo (NodeSource, PostgreSQL's PGDG, Docker's own) — pinned to a major version, and unattended-upgrades keeps patching it. Best default for servers.
• nvm / fnm / asdf — per-user version managers. Great on a laptop, awkward on a server because they live in a shell profile that systemd and cron never read.
• Containers — sidestep the question by baking the runtime into the image.

# keys go in /etc/apt/keyrings, NOT the deprecated apt-key
$ sudo install -m 0755 -d /etc/apt/keyrings
$ curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key \
    | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg

$ echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_22.x nodistro main" \
    | sudo tee /etc/apt/sources.list.d/nodesource.list

$ sudo apt update && sudo apt install nodejs

# pin a package so upgrades never move it
$ sudo apt-mark hold nodejs
$ apt-mark showhold

Unattended security updates

$ sudo apt install unattended-upgrades
$ sudo dpkg-reconfigure -plow unattended-upgrades
$ sudo unattended-upgrade --dry-run --debug

# does anything need a reboot to take effect? (new kernel, glibc)
$ ls /var/run/reboot-required && cat /var/run/reboot-required.pkgs

Deploying an app

Why a reverse proxy at all?

Your Node or Python server can listen on 443 directly. You put nginx in front because it terminates TLS, serves static files far faster, buffers slow clients so a bad connection can't tie up an application worker, compresses responses, sets security headers, rate-limits, and — most usefully — lets you restart the app without dropping the listening socket on the public port.

Step 1 · A service user and a home for the code

$ sudo adduser --system --group --no-create-home --shell /usr/sbin/nologin api
$ sudo mkdir -p /srv/orders-api /etc/orders-api
$ sudo chown -R api:api /srv/orders-api

# secrets: root-owned, readable by nobody else
$ sudo install -o root -g root -m 600 /dev/null /etc/orders-api/env
$ sudo nano /etc/orders-api/env

NODE_ENV=production
PORT=3000
DATABASE_URL=postgres://app:[email protected]:5432/orders
SESSION_SECRET=...

Step 2 · The systemd unit

Use the api.service from the systemd section. The essentials: User=api, EnvironmentFile=, Restart=always, and a MemoryMax= ceiling so a leak can't take the database down with it.

Step 3 · nginx in front

upstream orders_api {
    server 127.0.0.1:3000;
    keepalive 32;                  # reuse upstream connections
}

# Connection must be EMPTY for upstream keepalive to work. Setting it to
# "upgrade" on every request pools nothing — this map sends it only when
# the client actually asked to upgrade.
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      '';
}

server {
    listen 80;
    server_name api.example.com;
    # certbot will add the 443 block and a redirect here

    # static assets straight from disk — never touch Node
    location /static/ {
        alias /srv/orders-api/public/;
        expires 30d;
        access_log off;
        try_files $uri =404;
    }

    location / {
        proxy_pass http://orders_api;
        proxy_http_version 1.1;

        # preserve the client's identity for your app's logs
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # websockets when asked for; empty otherwise, so keepalive works
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_connect_timeout 5s;
        proxy_read_timeout    60s;
        proxy_buffering       on;   # shields your app from slow clients
    }

    client_max_body_size 20m;      # default is 1m — cause of most 413s
    gzip on;
    gzip_types application/json text/css application/javascript;
}

$ sudo ln -s /etc/nginx/sites-available/orders-api /etc/nginx/sites-enabled/
$ sudo rm /etc/nginx/sites-enabled/default
$ sudo nginx -t                  # ALWAYS test before reloading
$ sudo systemctl reload nginx    # reload ≠ restart: no dropped connections

Step 4 · TLS with certbot

$ sudo apt install certbot python3-certbot-nginx
$ sudo certbot --nginx -d api.example.com

# renewal is a systemd timer installed for you — verify it
$ systemctl list-timers | grep certbot
$ sudo certbot renew --dry-run
$ sudo certbot certificates       # expiry dates

Step 5 · A deploy script that can fail honestly

#!/usr/bin/env bash
set -euo pipefail

cd /srv/orders-api

git fetch --all
git reset --hard origin/main
npm ci --omit=dev
npm run build

sudo systemctl restart orders-api

# wait for the health endpoint before declaring success
for i in {1..30}; do
  if curl -fsS http://127.0.0.1:3000/health >/dev/null; then
    echo "healthy after ${i}s"; exit 0
  fi
  sleep 1
done

echo "health check FAILED — recent logs:" >&2
journalctl -u orders-api -n 40 --no-pager >&2
exit 1

Unix sockets instead of a port

Slightly faster than TCP and — more importantly — impossible to reach from the network, because a socket is a filesystem object governed by ordinary permissions. Have your app listen on /run/orders-api/app.sock, add RuntimeDirectory=orders-api to the unit so systemd creates and cleans up the directory, and point nginx at proxy_pass http://unix:/run/orders-api/app.sock;.

nginx runs as www-data, so add it to the api group and make the socket group read/write — mode 0660, via UMask=0007 in the unit. Read alone is not enough: connecting to a Unix stream socket requires write permission on it, so a 0640 socket gives nginx connect() failed (13: Permission denied) and a 502.

Containers

Namespaces — what the process can see

A namespace gives a process a private version of a global resource. Each one maps onto something you already know from Docker:

Namespace	Gives the process its own…
pid	Process table. Your app is PID 1 inside; ps sees nothing else. Why a container dies when its main process exits.
net	Interfaces, routes, ports, firewall rules. Why localhost in a container isn't the host, and why two containers can both bind 3000.
mnt	Filesystem tree. The image's root becomes /. Bind mounts and volumes are entries grafted into this tree.
uts	Hostname.
ipc	Shared memory and semaphores.
user	UID/GID mapping — root inside can map to an unprivileged UID outside. The basis of rootless containers.

cgroups — what the process can use

Control groups meter and cap CPU, memory, I/O and process counts. docker run --memory=512m writes a value into a cgroup file. So does MemoryMax=512M in a systemd unit — they are the same mechanism, so systemd's resource controls and Docker's flags feel alike.

Every systemd service already runs in its own cgroup, which is how systemctl status can report a unit's memory use.

OverlayFS — where the image comes from

A union filesystem stacks read-only image layers under a single thin writable layer. Reads fall through the stack; writes copy the file up into the writable layer. This is why layers are shared and cached between images, why docker build can skip unchanged steps, and why writing large files inside a container is slower than writing to a mounted volume.

# a container process is visible on the HOST, with a different PID
$ docker run -d --name web nginx
$ docker inspect -f '{{.State.Pid}}' web
$ ps -p 48213 -o pid,user,cgroup,cmd

# its namespaces, as files
$ sudo ls -l /proc/48213/ns/

# its resource limits, as files
$ cat /sys/fs/cgroup/system.slice/docker-*.scope/memory.max

# join a running container's namespaces without docker exec
$ sudo nsenter -t 48213 -n ss -ltnp   # its network view
$ sudo nsenter -t 48213 -m ls /        # its filesystem view

Containers vs. systemd services

They solve overlapping problems. A single app on a single VM is genuinely simpler as a systemd service — no registry, no image builds, no daemon, and journald handles logs.

Containers earn their complexity when you need identical environments across machines, multiple conflicting runtime versions, or an orchestrator. Both sit on the same kernel machinery, so the understanding transfers either way.

Troubleshooting

502 Bad Gateway

nginx reached your app's address and got nothing usable back. Almost always the app, not nginx.

$ systemctl status orders-api            # is it even running?
$ journalctl -u orders-api -n 80         # why did it stop / crash-loop?
$ curl -si http://127.0.0.1:3000/health  # does it answer locally?
$ sudo ss -ltnp 'sport = :3000'          # bound to the right address?
$ sudo tail -50 /var/log/nginx/error.log # nginx's side of the story

If nginx logs connection refused, the app isn't listening. If upstream timed out, it's listening but too slow — raise proxy_read_timeout only after finding out why.

EADDRINUSE / "Address already in use"

Something already holds the port — usually a previous copy of your app that didn't exit.

$ sudo ss -ltnp 'sport = :3000'
$ sudo lsof -i :3000
$ sudo systemctl stop orders-api     # stop it PROPERLY first
$ kill 4821                          # only if it's an orphan

If systemd keeps resurrecting it, that's Restart=always doing its job — stop the unit instead of killing the PID.

Works in my shell, fails in cron

Different environment. Cron sources no dotfiles, so PATH is minimal and your app's env vars don't exist.

# reproduce cron's environment exactly
$ env -i /bin/bash --noprofile --norc -c '/srv/orders-api/bin/sync'

$ journalctl -u cron --since "1 hour ago"   # did it even fire?
$ readlink -f "$(command -v node)"             # use this absolute path

Fix by using absolute paths, setting PATH= at the top of the cron file, sourcing your env file inside the script, and redirecting output to a log so the next failure isn't silent.

The disk is full

Logs, Docker, or a deleted-but-open file.

$ df -h /
$ df -i /                                      # inodes, not bytes?
$ sudo du -xh --max-depth=1 / | sort -h | tail -12
$ sudo lsof +L1                                # deleted, still held open
$ sudo journalctl --vacuum-size=200M
$ docker system df

The process vanished with no error

No application log, no systemd stop request — this is the OOM killer's signature.

$ sudo dmesg -T | grep -iE 'out of memory|oom-kill|killed process'
$ journalctl -u orders-api --since "1 hour ago" | tail -30
$ free -h; ps aux --sort=-rss | head

Cap the offender with MemoryMax= so it dies inside its own cgroup instead of taking a neighbour with it, then go find the leak.

Permission denied

Check the service's user, then walk the whole path — you need x on every parent directory, not just r on the file.

$ systemctl show orders-api -p User -p Group
$ namei -l /srv/orders-api/public/index.html   # perms at EVERY level
$ sudo -u api ls -l /srv/orders-api            # test AS the service user
$ sudo -u api cat /etc/orders-api/env

namei -l is the underused tool here — it prints the ownership and mode of each component of a path in one pass.

Config change had no effect

Something between you and the running process is still holding the old version.

$ sudo systemctl daemon-reload      # after editing ANY unit file
$ systemctl cat orders-api          # the EFFECTIVE config incl. drop-ins
$ sudo nginx -t                     # syntax error = old config still live
$ sudo nginx -T | grep -A5 server_name   # full resolved nginx config

Everything is slow

Work out which of the four resources is the constraint before changing anything.

$ uptime; nproc                   # load vs cores
$ vmstat 1 5                      # r=runnable, si/so=swapping, wa=io wait
$ free -h                         # "available", not "free"
$ iostat -xz 1 3                  # %util near 100 = disk-bound
$ ss -s                           # connection counts by state
$ systemd-cgtop                   # which unit is eating it

High wa in vmstat means you're waiting on disk. High si/so means you're swapping and need more RAM. Neither is fixed by adding CPU.

I can't SSH in any more

Before assuming lockout, check the firewall, sshd config, key permissions, and whether the disk is full.

# from your machine — -vvv shows exactly which key was offered and refused
$ ssh -vvv deploy@web-01

# from the cloud provider's serial/recovery console:
$ sudo sshd -t                              # config valid?
$ systemctl status ssh
$ sudo ufw status
$ ls -ld ~/.ssh; ls -l ~/.ssh/authorized_keys  # must be 700 and 600
$ sudo journalctl -u ssh -n 40   # auth.log may not exist on Debian 12

sshd silently refuses keys if ~/.ssh or authorized_keys are group- or world-writable. The reason appears in the server's ssh journal, never in the client's error.

The general method

1. Reproduce the boundary. Test it locally on the box (curl 127.0.0.1), then from outside. That one test halves the search space.
2. Read the log closest to the failure. journalctl -u <unit> -n 100 before anything else. Then nginx, then dmesg.
3. Check what changed. journalctl --since around the incident, plus your deploy history and /var/log/apt/history.log.
4. Verify the assumption, don't trust it. systemctl cat for the real config, ss -ltnp for the real binding, sudo -u api for the real permissions.
5. Change one thing. Then re-test at the same boundary.