Systems design

Data structures taught what to keep things in — the array, the hash table, the B-tree — and priced each by hand. Algorithms taught what to do with them, and why the design that wins on paper can lose on the machine. This guide is about what happens when one process, one machine, one database is no longer enough: how a real system stores a row, keeps a copy of it, stays consistent, caches, and fails — and how to walk into a design problem, read it for the one force it turns on, and drive it. It is the third of three and assumes the first two. Big-O and the growth shapes, the B-tree are used here without rebuilding them, with a pointer back when it matters. Everything is measured against a real PostgreSQL 16 and Redis 7 on one four-vCPU machine, and every number names the machine it was taken on, because the number you should trust is the one you measured, not the one you memorised.

The latency numbers you memorised are already wrong

There is a table you have seen. "Latency Numbers Every Programmer Should Know" — an L1 cache reference at half a nanosecond, main memory at 100, a disk seek at ten milliseconds, a round trip across the country at 150, printed in a neat column that every systems-design article reprints. It is a genuinely useful table. It is also mostly from a 2009 talk, and time has moved several of its rows by more than a factor of ten.

Here is one, in writing, in a file that ships today. LevelDB's design document — Google's, in the source tree, describing a database people run right now — states a constant out loud:

Assuming a disk IO rate of 100MB/s (ballpark range for modern drives) ...

Sequential disk, 100 MB/s. Measure it on the machine these words were written on — a four-vCPU Intel Xeon at 2.10 GHz, kernel 6.18.44 — reading a gigabyte back through O_DIRECT so the page cache cannot cheat:

sequential write 1 GiB:                          165 MB/s
sequential read  1 GiB (O_DIRECT, 1 MiB blocks): 2324 MB/s
random read 8 KiB x 20000 (O_DIRECT):            25241 IOPS, 39.6 us/read

2,324 MB/s. Twenty-three times the number written into the design document, and the document is current. This is not a knock on LevelDB — it is the whole lesson of the chapter. An absolute latency number is a fact about a machine on a date. It rots. What does not rot is the shape: sequential is much faster than random, memory is much faster than disk, disk is much faster than network, and each of those gaps is a couple of orders of magnitude. Memorise the ratios and re-measure the constants, because the constants are already wrong and yours will be too.

So the rest of this chapter is not a table to memorise. It is a ladder you build once, on your own hardware, and then carry in your head as a habit — the habit of pricing a request before you build it.

The ladder, one rung at a time

Here is the cost of a dozen operations on this box, each run twenty thousand times after a warm-up, sorted, and reported at the median and the tail. Read it from the top down as a climb from "free" to "expensive":

operation                                           p50        p99      p99.9
python function call                             164 ns     307 ns     543 ns
dict lookup (in-process cache hit)               110 ns     165 ns     414 ns
hash 1 KiB (sha256)                             1.21 us    2.54 us   14.76 us
compress 1 KiB (zlib-1)                        20.67 us   36.67 us   52.31 us
json.dumps small object                         3.18 us    7.93 us   23.75 us
json.loads small object                         2.52 us    6.59 us   23.88 us
TCP round trip, loopback, 4 bytes              31.95 us   63.80 us   96.67 us
redis GET (loopback)                           79.52 us  134.92 us  312.44 us
redis SET (loopback)                           90.60 us  227.33 us  408.38 us
postgres SELECT 1 (loopback)                   67.40 us  126.50 us  459.44 us
postgres pkey lookup, cached                   69.53 us  119.37 us  278.98 us
postgres single-row INSERT+COMMIT (fsync)     274.89 us  893.57 us    2.32 ms

The units change three times on the way down. A function call and a dict lookup are nanoseconds — this is the in-process cache hit, the thing you do freely. Hashing and JSON are microseconds, a thousand times more, but still all inside one process. Cross a socket — even to 127.0.0.1, even to a server on the same machine — and you are paying tens of microseconds for the round trip before the other end does any work. A cached Postgres primary-key lookup is 70 microseconds; the network is most of that, not the query. And the last rung is the one to sit with: a single row inserted and committed durably costs 275 microseconds at the median and 2.3 milliseconds at the 99.9th — four hundred times a JSON parse, for one small row.

That last jump has a name and a cause, and it is worth its own rung.

The most expensive thing on the ladder is telling the truth

Watch what happens when you ask the disk to promise the write is safe versus letting it lie. This is pg_test_fsync, which does exactly one 8 KB write over and over, each time flushing it all the way to durable storage, and then does the same write without the flush:

        fdatasync                          9524.986 ops/sec     105 usecs/op
        fsync                              9375.587 ops/sec     107 usecs/op
Non-sync'ed 8kB writes:
        write                           2622127.046 ops/sec       0 usecs/op

105 microseconds to make 8 KB durable; 0 microseconds to write it and not wait. A factor of a thousand, and it is the difference between "the operating system has the bytes" and "the bytes will survive the power going out". A write that returns in zero time has only handed your data to a buffer in RAM; the fdatasync is the part where the disk physically commits, and physics charges for it. Every durable commit in this guide — the write-ahead log, synchronous replication, the transactional outbox — is paying this 105 microseconds, and it is why the durable insert sat at the bottom of the ladder.

There is a second lesson hiding in the same tool. Ask it to write 16 kilobytes as one flush, then as sixteen separate flushed kilobytes:

         1 * 16kB open_sync write          9417.506 ops/sec     106 usecs/op
        16 *  1kB open_sync writes          634.534 ops/sec    1576 usecs/op

Same sixteen kilobytes, fifteen times the cost when you flush them one at a time, because the fixed price of a durable flush is paid per flush, not per byte. This is why databases batch commits, why a log is written sequentially, and why "one fsync of the WAL file may suffice to commit many transactions" — a sentence you will meet again in chapter 4, now with a price attached.

The same DRAM, thirty-five times apart

One rung of the ladder deserves opening up, because it decides more than it looks like it should. "Main memory" is a single number on the famous table, but memory does not have one speed. Chase a pointer randomly through a working set of a given size, touching one 64-byte line at a time, and watch the per-access cost climb as the set outgrows each cache:

working set        ns/access
16 KiB (L1)            2.21
256 KiB (L2)           6.24
4 MiB (L3)            38.35
16 MiB               137.70
256 MiB (DRAM)       158.90

sequential 256 MiB scan: 18.9 ms = 13.3 GiB/s (4.50 ns per 64B line)

The random walk through 256 MiB of DRAM costs 158.90 nanoseconds per access. A sequential scan of the same 256 MiB — same chip, same bytes — costs 4.50 nanoseconds per line. A factor of thirty-five, from nothing but the order you touch the addresses in. The hardware fetches a whole 64-byte line and guesses at the next one; walk forwards and every guess is right and the next line is already arriving, jump around and every guess is wrong and you wait the full latency each time. This single ratio is the argument behind half of this guide — why a B-tree beats a binary tree on disk, why a database counts pages and not comparisons, why laying a row out in the right column order makes the table smaller. Data structures measured this cliff in its array chapter; here it is again, on a different machine, saying the same thing. Big-O counts your accesses and is silent about how far apart they landed, and thirty-five times is a large silence.

The connection is a resource with a wall

The ladder priced one request. A system serves many at once, and there the surprise is a hard limit you do not choose. Ask this Postgres how many clients it will admit, then send it more:

=== max_connections === 100
=== 128 clients: what actually happens ===
FATAL:  sorry, too many clients already

One hundred, by default, and the hundred-and-first is refused outright — not queued, not slowed, rejected. And here is the trap: a hundred permitted connections is not a hundred useful ones. This box has four vCPUs, so it can do about four things at once; the other ninety-six permitted connections, if they all arrive with work, do not run in parallel — they pile up and wait, and the next chapter measures exactly what that waiting costs. The gap between how many connections you are allowed and how many are actually doing work is where connection pools, thread pools and bulkheads all live. Hold the gap in mind; it is the reason PgBouncer exists and the reason "just raise max_connections" makes things worse.

Estimation is a habit, not a lookup

You now have the constants for a back-of-the-envelope estimate, and the point of the ladder is that you use it without looking anything up. A service doing 1,000 requests a second, each making three cached Postgres lookups and one durable write: the reads are 3 × 70 µs = 210 µs of database time per request, the write is 275 µs, so about half a millisecond of database work per request and 1,000 of them a second — half a second of database time demanded per wall-clock second, on one core's worth of the four you have. That is a comfortable estimate you did in your head, and it already tells you the write is your expensive step and the reads are cheap. Every later chapter is an exercise in this: name the operation, price it off a ladder you measured, multiply by the rate, and see which number is large before you build anything.

Reach for this the next time someone says a number should be "fast enough" or quotes you a latency figure from a blog. Ask which machine it was measured on and when, and if the answer is a table from a conference talk, measure it yourself — the ladder takes an afternoon to build and it will disagree with the folklore by a factor of ten in at least one place. The moment you catch yourself trusting a memorised constant over a measured one is the moment this chapter was written for.

A system at 80% load is not 80% busy

Your dashboard says the database is at 80% CPU. The obvious reading is that you have 20% left, so you can take maybe another quarter of the traffic before you need a bigger machine. That reading is wrong, and it is wrong in a way that has taken down more systems than any bug, because the system does not degrade when it runs out of headroom — it degrades long before, and the last chapter's connection wall was the crude version of a curve this chapter draws exactly. At 80% utilisation a request is already waiting five times its own service time. The 20% that looks like headroom is sitting in a denominator, and denominators do not subtract.

To see why, you need one identity first — the closest thing queueing theory has to a law of conservation, and it falls out of your own database.

The identity you cannot escape

Take three quantities. λ (lambda) is the arrival rate: requests per second coming in. W is the time each request spends in the system, from arrival to departure — the response time. L is the number of requests in the system at any instant, the ones being served plus the ones waiting. Little's Law says these three are locked together:

L = λ × W

That is it. The average number of requests inside your system equals how fast they arrive times how long each one stays. It assumes nothing about the distribution of arrivals or service times, nothing about how many servers you have, nothing about scheduling — it needs only that the system is stable and nothing leaks. It is not a model of your system; it is an accounting identity your system cannot violate, the way a bathtub's water level cannot violate "inflow times dwell time".

You can watch it hold in your own database. pgbench runs a fixed number of clients, each firing one transaction, waiting for the reply, and firing the next — so the number of clients is L, the number of requests in the system. Run it at seven client counts against this box's Postgres and multiply the throughput it reports by the latency it reports:

 clients     tps   mean lat ms   L = tps×W    error
       1    1307        0.765        1.00       +0.0%
       2    2246        0.890        2.00       -0.1%
       4    3499        1.142        4.00       -0.1%
       8    4785        1.661        7.95       -0.6%
      16    4748        3.339       15.85       -0.9%
      32    5243        6.007       31.50       -1.6%
      64    5015       12.538       62.88       -1.8%

Throughput times latency comes back as the client count, to within two percent, at every load. You did not model anything. You measured two numbers the benchmark happened to print and their product was the third, because the identity leaves no room for it to be otherwise. This is the cheapest profound thing in the guide: L, λ and W are one equation, so you can never move one without moving another, and every later chapter is a trick for choosing which one moves.

Where the throughput went, and where the latency came from

Now read the same table the other way. From 1 client to 64, throughput rose from 1,307 to about 5,000 and then stopped — call it a factor of four, and it saturated by 8 clients. Over that same stretch latency climbed from 0.77 ms to 12.5 ms, a factor of sixteen. Past four clients, on a four-vCPU box, adding load bought almost no throughput and paid for it entirely in latency.

The read-only path makes it brutal, because there is no disk in the way to blur it:

 4 clients: 119,281 tps   0.034 ms
64 clients:  74,983 tps   0.854 ms

Sixteen times the clients bought twenty-five times the latency and less throughput — 75,000 transactions a second against 119,000. The machine has four cores; four clients keep them busy; the other sixty clients do not add parallelism, they add a line, and the line is pure waiting that also thrashes the cache and costs throughput. "More concurrency" past the core count is not more work done. It is a longer queue.

The curve, and its knee

Here is the shape behind the numbers. Call ρ (rho) the utilisation — the fraction of the time the server is busy, arrival rate over service rate. A standard model of a single queue (Poisson arrivals — arrivals that are nicely random, spread out independently — one server, called M/M/1) gives the average time in the system as service time divided by one minus utilisation:

W = S / (1 − ρ)

Read 1 − ρ as the idle fraction, and the formula says your response time is your service time divided by how idle you are. Divide by a shrinking number and you get an exploding one. Priced out on a 10 ms service time:

   rho   arrivals/s     W (ms)     W/S
 0.500         50.0      20.00      2.0
 0.700         70.0      33.33      3.3
 0.800         80.0      50.00      5.0
 0.900         90.0     100.00     10.0
 0.950         95.0     200.00     20.0
 0.990         99.0    1000.00    100.0

Read the W/S column — response time as a multiple of the underlying work. At 80% a request takes five times its service time. At 90%, ten times. At 99%, a hundred times. The work each request needs never changed; it is 10 ms the whole way down. What changed is how long it waits behind others, and that wait is governed by 1 − ρ, so the curve is flat and boring until about 70% and then turns into a wall. This is the "knee". The distance from 80% to 90% utilisation is ten percentage points on your dashboard and a doubling of latency underneath it. Headroom is not the 20% you have left; it is the 1 − ρ in the denominator, and it runs out non-linearly.

Two honesty notes, because a formula presented as gospel is worse than none. First, this curve is optimistic. It assumes nicely random arrivals and service times; real traffic is burstier and real service times have the fat tail the next chapter is about, and both make the true curve climb sooner and steeper than this one. Second, simulating the same queue and comparing to the formula, the two agree within one percent up to ρ = 0.9 and then the simulation comes in lower at ρ = 0.99 — 732 ms against the predicted 1,000 — not because the formula is wrong but because a queue this close to saturation takes far longer than any finite run to reach its steady state. The wall is real; you just cannot benchmark your way to the top of it, which is itself worth knowing.

One fast server beats sixteen slow ones

The identity has a counter-intuitive corollary that settles a recurring argument. Suppose you have a fixed total capacity — 800 requests a second — and an offered load of 640 (so ρ = 0.8 either way), and you can arrange it as one fast server or several slow ones that add up to the same throughput. Which is faster?

     1 server  @  1.25 ms  ->  mean response   6.25 ms
     2 servers @  2.50 ms  ->  mean response   6.94 ms
     4 servers @  5.00 ms  ->  mean response   8.73 ms
     8 servers @ 10.00 ms  ->  mean response  12.86 ms
    16 servers @ 20.00 ms  ->  mean response  21.91 ms

One fast server beats sixteen slow ones by three and a half times, at identical total capacity and identical utilisation — and this is with a shared queue feeding them, which is the friendly case. The reason is that a single fast server can put its whole speed behind whichever request is at the front, while sixteen slow ones each crawl through their own work; a request unlucky enough to land behind a long one on a slow server cannot borrow the idle capacity of the other fifteen. Splitting one big machine into many small ones costs latency for free. Only adding capacity — raising the total, lowering ρ — buys it back. "Just add more small instances" and "consolidate onto fewer bigger ones" are not equivalent, and the arithmetic says which way to lean.

Reach for this chapter whenever a latency graph turns from flat to vertical with no obvious cause, or whenever someone proposes to run a service "hot" at 85–90% to save money. The lever is always the same equation: to cut W you cut ρ (shed load, add capacity), or you cut S (make each request cheaper — a cache, an index, a smaller query), or you accept a longer L (a queue, and the honesty to bound it). Add a cache, add a replica, shard it, shed load — these are not four unrelated tricks. They are four ways of moving one variable in W = S/(1 − ρ), and the rest of the guide is that sentence, elaborated.

One slow response in a hundred delays most of your requests

Your mean response time is 2 milliseconds. Your users are complaining that the app is slow. Both of these are true at once, and the reason is that the mean is the one statistic that cannot see the problem. An average is a single number standing in for a whole distribution, and it hides its worst behaviour by definition — a thousand fast requests and ten catastrophic ones average out to something that looks healthy and satisfies no one. The number your users feel is not the middle of the distribution. It is the edge, and the edge behaves nothing like the middle.

Start with the plainest possible version of the claim: a database, one client, no concurrency, no contention of any kind, running the same tiny transaction ten thousand times. If latency were a stable thing, every run would cost about the same. Measure the percentiles — the p50 is the value half the runs beat, the p99 the value 99% beat, the max the single worst:

clients   p50     p90     p99     p999     max      p99/p50
1         0.807   1.017   1.412   3.416    22.226   1.7

Half the transactions finished in 0.8 ms. The worst one took 22.2 ms — twenty-seven times the median — with nobody else on the system. Nothing was competing for the database. There was no queue. And still one request in ten thousand was twenty-seven times slower than typical, because underneath a single query sit a checkpoint that periodically flushes dirty pages, a page the kernel evicted and has to fetch back, the garbage collector, the scheduler handing the CPU to something else for a slice. Those events are rare and they are large, and they do not care that you only sent one request. The tail is not something concurrency creates. It is always there; concurrency only widens it. That is the first thing to unlearn — the tail is a property of the system, not of the load.

The tail outruns the median

Now add load and watch the two ends of the distribution move at different speeds:

clients    tps     p50      p99      p999      max
1          1185    0.807    1.412    3.416     22.226
16         2856    4.705   18.381   73.028    120.586
64         3684   12.130   79.866  138.337    238.759

From 1 client to 64, throughput roughly tripled, the median grew fifteenfold — and the p99 grew fifty-six-fold, from 1.4 ms to 79.9 ms. The tail does not track the median; it pulls away from it. Under load the p99/p50 ratio climbs from 1.7 to 6.6, which is another way of saying that as a system fills, the gap between a typical request and a bad one widens faster than either number alone suggests. If you are watching averages, you see a gentle rise. Your worst-served users are on a curve several times steeper, and they are the ones who tell other people your product is slow.

Why a rare slowness becomes a common one

Here is where the tail stops being a monitoring nuisance and becomes an architectural force. Modern requests fan out. A single page load calls a service that calls twenty others; a search query hits a hundred shards and waits for all of them; a feed assembles itself from dozens of backends. The user waits for the slowest of everything their request touched, and slowness that is rare per-backend stops being rare once you take the max over many.

Suppose each backend is slow — above its own p99 — just one percent of the time, and a request has to wait for all of them. The chance that at least one is slow is one minus the chance they are all fast, and if the backends are independent the all-fast probability is 0.99 multiplied by itself once per backend. (Independent probabilities multiply — the two-sentence version of a rule the maths guide Numbers and algebra builds properly in its chapter on probability; here it is just 0.99 to the power of the fan-out.) Compute it:

    fan-out   1:  1.0%
    fan-out  10:  9.6%
    fan-out 100: 63.4%
    fan-out 200: 86.6%

At a fan-out of 100, a per-backend slowness of one-in-a-hundred becomes an end-to-end slowness of 63.4% — nearly two requests in three are slow, built entirely out of backends that are each fast 99% of the time. This is the central result of Dean and Barroso's "The Tail at Scale" (Communications of the ACM, 2013), and it reports exactly this: a one-in-a-hundred slow response delays 63% of user requests once a request fans out to a hundred servers. The arithmetic above reproduces that figure on its own, which is the point — the mechanism is not a claim to take on faith, it is 1 − 0.99¹⁰⁰. Push the per-backend rarity to one-in-a-thousand and you buy back headroom, but only until the fan-out reaches a thousand, where you are back to 63%. At scale, the p99 of a component is the median experience of the whole. You cannot fix this by making the common case faster; you have to attack the tail directly.

Attacking the tail: send a second copy

The cheapest attack is almost impudent. If a request has not come back by the time it is already in unusual territory — say its own p95 — send a second copy to a different server and take whichever answers first. Most requests finish before the p95 and never trigger a duplicate, so the extra load is small; the few that would have been slow get a second roll of the dice. Simulate it on a heavy-tailed service (median 10 ms, a long right tail) and measure the cost:

  no hedge : p50=  9.98   p99=  81.13   p99.9= 162.35   1.00x work
  hedge@p95: p50= 10.00   p99=  57.06   p99.9=  79.51   1.05x work

The p99.9 was halved for five percent more work. The median did not move, because the median never triggered a hedge; only the tail did, and only the tail paid. This is the result Dean and Barroso report from Google's own systems — hedging after a short delay cut the p99.9 of retrieving a thousand values from 1,800 ms to 74 ms for about 2% more requests — and the shape is the same one the simulation shows: a small, bounded increase in total work buys a large cut in the worst case, because you only spend the extra work where the first copy was already losing.

A timeout is a percentile you chose

Once you believe the tail exists, one piece of everyday advice stops being arbitrary. "Set your timeout to the p99.9 of the dependency" sounds like a magic number until you see the distribution: a timeout is a horizontal line drawn across it, and everything to the right of the line you have chosen to abandon. Draw it at the p99.9 and you give up on the worst one in a thousand calls — the ones already so slow that waiting for them helps no one and ties up a connection your other requests need. Draw it too tight and you abandon requests that would have succeeded; too loose and a hung dependency holds your resources until you fall over. The right value is not a constant you look up. It is a point you pick on a curve you measured, and it is the first of the failure-handling levers chapter 12 builds on — every one of which is, underneath, a decision about which part of the tail to stop waiting for.

Reach for this chapter the moment "the average looks fine" collides with "users say it's slow", and any time a request in your system fans out to more than a handful of backends. Put a p99 and a max on your dashboards next to the mean, and when the max is twenty times the median with no load on the box, believe it — that is the tail showing you it was there all along, waiting for a fan-out to multiply it into everyone's problem.

A row lives on a page, and the log is written first

You run UPDATE accounts SET balance = balance + 1 WHERE id = 42. One column, one row, a few bytes changed. Postgres reads eight kilobytes off disk to do it and writes a durable record before it touches the table at all. To understand why — and to understand every layer built on top of storage in this guide — you have to stop thinking of a row as a thing and start thinking of it as a resident of a page, because the page, not the row, is the unit the database actually moves.

Data structures ended on the B-tree and the rule that on disk you count pages, not comparisons, because a page is what a disk hands you in one go. This chapter is where that page becomes concrete. Postgres stores every table as an array of fixed 8 KB pages, and you can open one and read it.

What is actually inside 8,192 bytes

Install pageinspect, put a hundred thousand rows in a table, and look at the header of its first page:

 lsn        | lower | upper | special | pagesize | version
 0/2BAEC418 |   548 |   600 |    8192  |    8192  |    4

heap_page_items(...) limit 3:
 lp | lp_off | lp_len | t_xmin | t_xmax | t_ctid | t_hoff
  1 |   8136 |     55 | 334816 |      0 | (0,1)  |     24
  2 |   8080 |     53 | 334816 |      0 | (0,2)  |     24

tuples on page 0: 131

A page has a fixed structure, and every number above follows from it. The first 24 bytes are a header. Then two things grow toward each other from opposite ends: an array of line pointers grows down from the top, and the row data — the tuples — is filled in from the bottom up. lower (548) is where the line-pointer array currently ends; upper (600) is where the tuple data currently begins; the gap between them, 52 bytes here, is the free space left on the page.

Now the arithmetic closes on itself. Each line pointer is exactly 4 bytes — an (offset, length) pair saying where on the page its row sits and how long it is. The array starts right after the 24-byte header, so the number of line pointers is (lower − 24) / 4 = (548 − 24) / 4 = 131. And the page holds exactly 131 tuples. The count is not stored anywhere as a number; it is implied by where lower sits, and it comes out to the tuple count on the nose. That is what "a row lives on a page" means literally: a row is a stretch of bytes near the bottom of an 8 KB block, found through a 4-byte pointer near the top, and t_hoff = 24 tells you its own data begins 24 bytes in, after its per-row header. A point lookup does not fetch a row. It fetches this whole page and reads the row out of it.

Column order is worth 27% of your table

Here is a consequence that costs real disk and that almost nobody orders their columns around. Take six columns — three smallint (2 bytes) and three bigint (8 bytes) — and build the same table twice with the same data, changing only the order the columns are declared in:

    t     |  bytes   | pages | lp_len
 pad_bad  | 15319040 |  1870 |   72
 pad_good | 12050432 |  1471 |   54

Seventy-two bytes per row against fifty-four; 1,870 pages against 1,471. The "bad" table is 27% larger, holding the same values, because of a rule the hardware imposes: a value has to begin at an address that is a multiple of its own size. An 8-byte bigint must start on an 8-byte boundary. So in the order smallint, bigint, smallint, bigint, …, after each 2-byte smallint the database inserts 6 bytes of padding to push the next bigint onto its boundary — dead space, repeated per row, per page, forever. Declare the three bigint columns first and the three smallint columns last, and the padding vanishes: the wide fixed-width types pack tight, and the narrow ones fill the tail. Same schema, same data, a quarter less storage and a quarter fewer pages to read, from column order alone. This is the same concern for memory layout that Data structures raised in its array chapter — there it was cache-line contiguity, here it is field alignment — one level down, charging rent on disk.

The value too big for the page

A page is 8 KB, so what happens to a value bigger than a page? It cannot live inline, so Postgres moves it out of line into a side table, a mechanism it calls TOAST (The Oversized-Attribute Storage Technique). The documentation names the trigger:

The TOAST management code is triggered only when a row value to be stored in a table is wider than TOAST_TUPLE_THRESHOLD bytes (normally 2 kB).

Store a 3,200-byte value and a 128,000-byte value of genuinely random (incompressible) text and watch where the bytes go:

 id | length | pg_column_size
  1 |   3200 |           3200
  2 | 128000 |         128000
 main_heap | toast_heap
      8192  |    139264

The main table stayed one page; the 139 KB of value data went to the TOAST heap, chunked so four chunks fit a page. This matters because a query that never selects the big column never pays to read it — the main row stays small and dense. But there is a trap that turns a TOAST demonstration into a compression demonstration, and it catches people writing exactly this kind of test:

 note                        | length | pg_column_size
 compressible repeat(x,3000) |   3000 |            44

Three thousand bytes of the letter x compresses to 44, so it never crosses the threshold and never TOASTs at all. Postgres compresses before it decides to go out of line, so a value that looks oversized can vanish. Test TOAST with repeat('x', 3000) and you will conclude it does not fire; test it with random bytes and you see the real behaviour. The lesson beyond the trap: the size of a row on disk is the compressed size, and how compressible your data is decides how much of it fits in a page.

Why the log is written first

Now back to the opening puzzle — why an update writes a durable record before it changes the table. When Postgres modifies a page, it does two writes: it changes the page in memory (to be flushed to the table file whenever convenient), and it appends a description of the change to the write-ahead log (WAL) and flushes that to disk immediately. The rule, from the documentation, is absolute:

changes to data files (where tables and indexes reside) must be written only after those changes have been logged, that is, after WAL records describing the changes have been flushed to permanent storage.

The log is written first, and this is the trick that makes durability affordable. Recall chapter 1: a durable flush costs about 105 microseconds, and it costs that per flush, not per byte. If a transaction dirtied ten scattered pages and had to flush all ten to their scattered homes on disk, it would pay for ten random writes. Instead it flushes one thing — the log — and the log is a single file written sequentially, appended to, so the flush is one cheap sequential sync instead of ten expensive random ones. The documentation says exactly this:

only the WAL file needs to be flushed to disk to guarantee that a transaction is committed ... The WAL file is written sequentially, and so the cost of syncing the WAL is much less than the cost of flushing the data pages. Furthermore, when the server is processing many small concurrent transactions, one fsync of the WAL file may suffice to commit many transactions.

That last sentence is chapter 1's batching result, restated by the database: because the fixed cost is per flush, many transactions can share one. The table pages catch up lazily; if the server crashes first, the log is replayed on restart to redo anything that had not yet reached the table. Durability is a promise kept by the log, not by the data files, and the log is cheap because it is sequential and shared. Hold onto this, because it is the whole foundation of the chapters that follow: a replica is a machine replaying your WAL, synchronous commit is a choice about how far down this flush to wait, and the transactional outbox is one more writer piggybacking on the same single commit.

Reach for this chapter the next time a table is mysteriously twice the size you expected, or a bulk load is bottlenecked on disk while barely touching the data. The size question is usually column order or an un-TOASTed wide value sitting inline; the write question is usually the WAL, and the next chapter shows what an index adds to every one of those writes.

An index is a second copy, and layout pays the bill

Data structures built a B-tree and read a small one straight out of a database file, ending on the promise that the Systems design guide would show what the database does with it. Here is where that promise comes due, and it starts with a measurement that should settle any doubt about why B-trees won. Put five million rows in a table, index the primary key, and read the index's own structure out of Postgres with pageinspect:

  heap  |  pkey  | k_idx | heap_pages
 249 MB | 107 MB | 34 MB |     31848

bt_metap('big_pkey'):  root=412  level=2

 btpo_level | pages |  items  | avg_items_per_page
          2 |     1 |      48 |                 48     <- root
          1 |    48 |   13709 |                286
          0 | 13662 | 5013661 |                367     <- leaves

Five million keys, and the tree is three levels deep. One root page holds 48 downward pointers; each of those points to a level-1 page holding about 286; each of those points to a leaf holding about 367 keys. The number of keys the tree can hold is the fanout raised to the height, and 48 × 367 × 367 = 6.46 million, comfortably above five million — which is why three levels suffice. To find any one of five million rows, Postgres reads the root, one level-1 page, one leaf, and then one heap page for the row itself. Four page reads to locate a row among five million. That is the B-tree's entire pitch, measured rather than asserted, and it is the same shape Data structures found in miniature.

The read side is the side that lies to you

An index makes that lookup fast, so the instinct is to add one to every column you filter on. The instinct is wrong, and it is wrong because an index is not a lens you lay over the table — it is a second copy of the indexed columns, a whole separate B-tree that has to be updated on every insert, update and delete. You measure the read benefit; you do not measure the write cost, so you undercount it. Put it on a scale. Insert 200,000 rows into the same table with three different index sets, measuring the write-ahead log each insert generates and the wall-clock time:

                      WAL total   bytes/row   wall time   heap / index size
pkey only                 30 MB         157       0.49 s   11 MB /  4408 kB
+1 btree                  44 MB         232       0.82 s   11 MB /  9440 kB
+3 btrees                 77 MB         405       1.63 s   11 MB /    25 MB

Three secondary indexes cost 2.5 times the WAL, three times the wall-clock time, and produce more index than table — 25 MB of index against 11 MB of heap. Every row inserted now writes itself into four B-trees, each of which may split a page, and each of those changes is logged. The read you wanted got faster; four writes you did not think about got slower, and on a write-heavy table that trade can invert entirely. There is a second thing hiding in the top row: pkey-only WAL is 157 bytes per row for a row of about 50 bytes. The excess is full-page writes — the first time a page is touched after a checkpoint, the whole 8 KB page goes into the log so a crash mid-write can be repaired, and 157 bytes a row is that 8 KB amortised across the rows that shared each page.

The planner is doing arithmetic you can redo

So an index exists, and a query could use it or could scan the whole table. Which one runs is decided by the planner, and its decision is not a mystery — it is a cost, in arbitrary units, computed from constants you can read and multiply yourself. Ask Postgres to explain a full scan of the five-million-row table:

Seq Scan on big  (cost=0.00..81847.92 rows=4999992 width=23)

That 81,847.92 is not magic. A sequential scan reads every page once and processes every row, and the planner prices each: seq_page_cost is 1.0 per page, cpu_tuple_cost is 0.01 per row. The table is 31,848 pages holding 5,000,000 rows, so:

31,848 pages × 1.0    =  31,848
5,000,000 rows × 0.01 =  50,000
                total =  81,848   (planner: 81,847.92)

The two numbers match to the rounding of the row estimate. You can do the planner's job on the back of an envelope, and that is the point — its choices are legible, not oracular. (One constant is worth distrusting: random_page_cost defaults to 4, meaning a random page read is priced as four times a sequential one. That was true for spinning disks and is wrong for the SSD under this box, where chapter 1 measured random 8 KB reads at 39.6 µs against sequential reads streaming at 2.3 GB/s. Most SSD deployments retune it to somewhere between 1.1 and 2.0, and doing so changes which plans win.)

The crossover is nowhere near where the folklore puts it

Common wisdom says an index scan stops paying once a query returns more than about 5% of a table. Measure where Postgres actually switches from using the index to scanning the whole table, by asking for a rising fraction of the rows:

selectivity   plan chosen         cost
       0.1%   Bitmap Heap Scan   13,853
       1.0%   Bitmap Heap Scan   34,301
       8.0%   Bitmap Heap Scan   41,434
      15.0%   Bitmap Heap Scan   49,807
      40.0%   Bitmap Heap Scan   79,454
      70.0%   Seq Scan           94,347

The switch happens between 40% and 70%, not at 5%. Postgres keeps using the index far past where folklore says it should, because a bitmap scan — which collects all the matching row locations from the index first, sorts them into page order, then reads each page once — stays cheaper than reading everything until the query wants most of the table. The 5% rule is a memory of a different era and a different random_page_cost. The real crossover is a number your own planner will tell you, and it is much higher than you think.

The fact that beats the index

Now the measurement that reorders your whole mental model. Take the same table and ask for the same number of rows — about 250,000, five percent — two ways. Once through a column whose values are scattered randomly across the table (k), once through the primary key, whose values are stored in physical order (id). Watch the pages each one touches:

uncorrelated k < 5000:    Bitmap Heap Scan, Heap Blocks: exact=31,844,  Buffers read 25,250
correlated  id < 250000:  Index Scan,                                   Buffers read  2,277

Same row count. Thirty-one thousand, eight hundred heap pages against two thousand, two hundred seventy-seven — a factor of fourteen, and the uncorrelated query touched essentially every page in the table to return a twentieth of it. Nothing changed but where the rows physically sit. The 250,000 correlated rows live in a contiguous run of pages, so reading them is a short sequential sweep; the 250,000 uncorrelated rows are sprinkled one every few rows across all 31,848 pages, so fetching them means visiting nearly every page for a handful of rows each. The index found the row locations in both cases, cheaply — a couple of hundred index pages. The heap paid the bill, and the size of the bill was set by layout, not by the index. This is why "add an index" is not the whole answer to a slow query: the index finds the pointers, but physical clustering decides how many pages those pointers land on, and that is what you actually wait for.

The write-optimised mirror

Everything so far is a B-tree updated in place, which is read-optimised: lookups are three page reads, but every write may have to find and modify a page in the middle of the tree. Flip the priority — make writes cheap and let reads work harder — and you get the other family, the log-structured merge tree (LSM) under RocksDB, Cassandra and most write-heavy stores. Instead of updating a tree in place, an LSM appends every write to an in-memory buffer, flushes it as a sorted file, and later merges those files in the background into larger sorted files arranged in levels, each level ten times the size of the one above. LevelDB's own design document prices the merging:

we will pick one 2MB file from level L. In the worst case, this will overlap ~ 12 files from level L+1 ... The compaction will therefore read 26MB and write 26MB.

Twenty-six megabytes read and written to fold in two — and RocksDB states the general figure plainly: "the usual write amplification of leveled compaction ... is often larger than 10." So an LSM does not escape the cost of maintaining sorted structure on disk; it moves it, off the write path and into background compaction, at the price of writing each piece of data roughly ten times over its life. The read side is the mirror image: a lookup may have to check the memory buffer and one file per level before it is sure a key is absent. Neither family is free. A B-tree pays on every write to keep reads at three page reads; an LSM pays 10× write amplification in the background to keep the write path an append. Which you want is the read/write ratio, which is the same question this whole guide keeps returning to.

Reach for this chapter whenever a query that "has an index" is still slow, or a table's write throughput falls off as you add features. The slow indexed read is almost always the heap paying for poor clustering, not the index failing; the slowing writes are almost always the indexes you added for reads, each one a second copy maintained on every change. Count your indexes the way you count durable flushes — each is a write-amplified structure with a bill that comes due on the write path, which is exactly where the next chapter's replica is standing.

A replica is always behind, so read-your-writes fails by default

You add a read replica to take load off the primary, point your reads at it, and a user posts a comment and immediately does not see their own comment. The bug report says "sometimes my post disappears and then comes back". It is not a bug. It is the defining property of a replica, and chapter 4 already told you what a replica is: a machine replaying your write-ahead log. It cannot replay a change until the change has been logged, shipped over the network, and applied — so at the instant the primary says "committed", the replica has not seen it yet. The question is not whether the replica is behind. It is always behind. The question is how far, and whether you built something that assumes it is not.

The anomaly is near-total, not occasional

Measure it on the tightest possible setup: a primary and a streaming replica on the same machine, loopback network, no cross-datacenter distance at all. Write a row on the primary, then immediately read it back from the replica, two hundred times:

writes: 200   reads on replica that did NOT see the write: 195 (97.5%)
time until the replica caught up, us: p50=385 p95=732 max=5924

The replica is 385 microseconds behind at the median, and that tiny lag is enough that 97.5% of immediate reads miss the write. This is the intuition to install and it is the opposite of what "eventual consistency" suggests to most people. Read-your-writes does not fail rarely, as an unlucky edge case. It fails almost always, because the write and the read are microseconds apart and the replication lag, however small, is longer than that gap. Across a real network — a different availability zone, a millisecond of round trip — the window is wider and the miss rate is a flat 100%. The loopback case is the charitable one, and it already fails.

There is a second, subtler anomaly. Point a client at a pool of replicas with slightly different lag, and two consecutive reads can go backwards in time. Force it by pausing replay on the replica while the primary keeps writing:

client reads leader  -> 205 rows
client reads replica -> 200 rows   (replay paused; 5 behind)
pg_stat_replication: replay behind by 920 bytes

A user who read 205 rows a moment ago now reads 200 — the count went down, with no deletion anywhere. This is monotonic reads violated, and a read-replica pool that load-balances across replicas of differing lag produces it naturally: request one lands on a caught-up replica, request two on a lagging one, and time appears to run backwards. Neither of these is exotic; both are what a naive replica pool does on a Tuesday.

The dial, with a price on every detent

You fix read-your-writes by making the primary wait — but wait for what, exactly? Postgres exposes the choice as synchronous_commit, five levels of "how far down the pipe must a commit reach before I tell you it succeeded", and each level is a point on the latency-versus-durability trade. Priced with a thousand single-row commits at each level on this box:

  synchronous_commit=off           p50=0.179 ms   5065 commits/s
  synchronous_commit=local         p50=0.437 ms   2083 commits/s
  synchronous_commit=remote_write  p50=0.619 ms   1565 commits/s
  synchronous_commit=on            p50=0.891 ms   1070 commits/s
  synchronous_commit=remote_apply  p50=0.905 ms   1072 commits/s

Read it top to bottom as a ladder of promises. off — do not even wait for the local disk flush; fast, and a crash can lose committed data. local — the local WAL is durably flushed (chapter 1's fsync, and the jump from off is 0.26 ms, about two of those 105 µs flushes). remote_write — the replica has the bytes in its OS buffers. on — the replica has flushed them to its own disk. remote_apply — the replica has applied them, so a read on the replica now sees the write. That last level is the exact fix for the anomaly this chapter opened with, and its price is printed: 0.9 ms a commit against 0.18 ms, roughly five times the latency to guarantee a read on the replica sees your write. This is a real dial you turn per transaction, not a property of the system, and the important thing is that the choice has a number attached — you are buying consistency in units of milliseconds. (Add the cross-datacenter round trip, typically about a millisecond per availability-zone hop, on top of each remote level to translate this to a real deployment.)

The same trap wears a Kafka logo

This is not a Postgres quirk; it is the shape of every replicated system, and the most common way people get it wrong has a name. Kafka's acks=all sounds like it means "all replicas have it". It does not, and Kafka's own documentation is blunt about the trap:

"acknowledgement by all replicas" does not guarantee that the full set of assigned replicas have received the message. By default, when acks=all, acknowledgement happens as soon as all the current in-sync replicas have received the message. For example, if a topic is configured with only two replicas and one fails (i.e., only one in sync replica remains), then writes that specify acks=all will succeed. However, these writes could be lost if the remaining replica also fails.

"All" means all currently in-sync replicas, and that set can shrink to one. A write acknowledged by "all" replicas can be acknowledged by a single replica that then dies, and the write is gone. The setting that means what people think acks=all means is min.insync.replicas — a floor on how many replicas must be in sync for a write to be accepted at all. acks=all without min.insync.replicas is remote_apply without checking anyone is actually there.

Reach for this chapter the moment "it works on my machine but sometimes not in production" involves a read after a write, or any time you put a read replica behind a load balancer. The rule is small and absolute: a replica is behind, so a read that must see a specific recent write goes to the primary or waits for remote_apply, and a read that must not go backwards is pinned to one replica. Everything else you route to a replica freely, and the next two chapters are about what "consistent" even means once there is more than one copy that can disagree.

Snapshot isolation has a hole, and its name is write skew

Two doctors are on call. Each is tired and wants to go home, and the system's one rule is that at least one doctor must stay on call. Each opens the app, which checks the rule — are there other doctors on call? — sees the other one, and, satisfied, takes themselves off. Both checks ran against a database that was correct at the moment each was read. Both commits succeed. Now zero doctors are on call, and no single transaction did anything wrong. This is write skew, and it is the specific, named hole in the isolation level most databases give you by default — a hole you can fall through even when you thought you had turned concurrency safety on.

The letters in ACID that this chapter is about are the I, isolation: the promise that concurrent transactions do not step on each other. The trouble is that "isolation" is not one guarantee but a ladder of them, and where you stand on the ladder decides which anomalies can happen to you. Postgres documents the ladder as a table, and it is worth reading as the contract it is:

Isolation level	Dirty read	Non-repeatable read	Phantom read	Serialization anomaly
Read uncommitted	not in PG	possible	possible	possible
Read committed	not possible	possible	possible	possible
Repeatable read	not possible	not possible	not in PG	possible
Serializable	not possible	not possible	not possible	not possible

Two things in that table are already surprising, and Postgres says both in words. First, its weakest level is not as weak as the name: "PostgreSQL's Read Uncommitted mode behaves like Read Committed" — you cannot get dirty reads here at all. Second, its Repeatable Read is stronger than the standard requires: it "does not allow phantom reads ... higher guarantees are acceptable". Postgres's Repeatable Read is in fact snapshot isolation — every statement in the transaction sees a single frozen snapshot of the database as of the transaction's start. That is a strong, clean guarantee. It is also the one with the hole.

Every cell of the table, on two real connections

You do not have to trust the table. Open two concurrent connections and reproduce each row. Start with a non-repeatable read — read a value, let another transaction commit a change, read again:

  READ_COMMITTED   first read=100  second read after concurrent commit=150  -> NON-REPEATABLE READ
  REPEATABLE_READ  first read=100  second read after concurrent commit=100  -> stable

Under Read Committed the second read sees the new value; under Repeatable Read the snapshot holds and the value does not move. Exactly as the table promised. Now the one that bites real code — a lost update, the read-modify-write every application writes: read a balance, add ten, write it back, from two transactions at once:

  READ_COMMITTED   two +10 on 100 -> final=110    LOST UPDATE
  REPEATABLE_READ  two +10 on 100 -> ERROR: 40001 could not serialize access due to concurrent update

Under Read Committed the answer is 110, not 120 — one of the two updates vanished, silently, because both read 100 before either wrote. Under Repeatable Read the second commit cannot proceed and Postgres raises error 40001. Snapshot isolation caught the lost update by refusing to let two transactions that read the same row both write it. So far it looks like Repeatable Read is the safe choice.

The cell snapshot isolation cannot fill

Then run the doctors:

  READ_COMMITTED   doctors still on call = 0    WRITE SKEW
  REPEATABLE_READ  doctors still on call = 0    WRITE SKEW
  SERIALIZABLE     doctors still on call = 1    ERROR: 40001 could not serialize access
                   due to read/write dependencies among transactions

Repeatable Read does not stop it. Both doctors go off call under snapshot isolation, and the invariant breaks. The reason is precise and worth holding: the lost update was two transactions writing the same row, which snapshot isolation detects. Write skew is two transactions writing different rows — each doctor updates their own record — while reading a shared condition. There is no write conflict on any single row, so first-committer-wins has nothing to fire on. Each transaction is individually correct against its own snapshot; it is only together that they violate the rule, and snapshot isolation reasons one transaction at a time. Only Serializable, which tracks the read/write dependencies between transactions, sees that these two cannot both be true and aborts one with 40001.

Notice the two failures carry the same error code. The lost update raised 40001; the write skew raised 40001. Different anomalies, different mechanisms, one code — which is the whole application-level lesson: you retry on 40001. Your code does not need to know whether it hit a concurrent-update conflict or a read/write-dependency conflict; it needs to catch 40001, roll back, and try again, because the database is telling you it refused to let a race corrupt your data and you should replay the transaction on fresh state. A serializable transaction without a retry loop is a bug waiting for load.

Why the names lie, and the fix

The deeper trap is believing the table's four columns are the whole story — that if you forbid dirty reads, non-repeatable reads and phantoms, you are serializable. You are not, and this was the central point of Berenson and colleagues' 1995 critique of exactly these definitions:

The prominence of the table compared to this extra proviso leads to a common misconception that disallowing the three phenomena implies serializability.

Write skew is the proof. Snapshot isolation forbids all three classic phenomena and is still not serializable, because "serializable" means the outcome is equivalent to running the transactions one at a time in some order, and no serial order of the two doctors leaves zero on call — one of them would have seen the other already gone. The three ANSI phenomena are a checklist that misses this, and a database can pass the checklist and still corrupt an invariant. Snapshot isolation is a specific, named guarantee with a specific, named gap, and knowing the gap's name is what lets you find it before production does.

The fixes are concrete. Promote the transaction to Serializable and add a retry loop, paying the abort-and-retry cost. Or make the write conflict visible to snapshot isolation by forcing the transactions to touch the same row — SELECT ... FOR UPDATE on the rule's rows, or a materialised counter of doctors-on-call that both transactions update, turning the write skew into a lost update that first-committer-wins already catches. Either way the move is the same: give the database a conflict on a row it can see.

Reach for this chapter whenever an invariant that "can't happen" happens under concurrency — a double-booked slot, an overdrawn balance, a uniqueness rule two requests both slipped past. The symptom is that each transaction looks correct in isolation and the corruption only appears when two run at once. That is write skew, your isolation level is snapshot isolation, and the fix is to make the conflict a row the database can lock — or to go serializable and retry on 40001. The next chapter is about what "consistent" means once the copies are on different machines and no single database can see the conflict at all.

CAP was never "pick two of three"

Every slide about distributed systems has the same triangle: Consistency, Availability, Partition-tolerance, pick two. It is memorable, it is on the whiteboard in every interview, and it is wrong in a way that matters, because it frames a theorem about behaviour during a network failure as a menu you order from. You do not pick P. Partitions are not a design choice you decline; they are packets that do not arrive, and they happen to you. To see what CAP actually says you have to start one level down, with the arithmetic the triangle skips — quorums — because a quorum is where "consistent" and "available" stop being adjectives and become counts.

A quorum is set intersection, and that is the whole idea

Spread your data across N copies. A write goes to W of them and returns; a read consults R of them and takes the newest. The one question that decides whether a read can see the latest write is: does every read set share at least one node with every write set? If they always overlap, some node in your read has the write. If they can miss, you can read stale. Compute it exhaustively over every possible choice of nodes:

  N   W   R  R+W   >N?   always overlaps?   tolerates loss (writes/reads)
  3   1   1    2 False              False        2/2
  3   2   2    4  True               True        1/1
  5   2   2    4 False              False        3/3
  5   3   3    6  True               True        2/2
  5   4   1    5 False              False        1/4
  5   5   1    6  True               True        0/4

The "overlaps" column is identical to the "R + W > N" column, in every row, for a plain arithmetic reason: two subsets of an N-set are forced to share an element exactly when their sizes add to more than N (if R + W > N, they cannot fit into disjoint halves). So R + W > N is not a rule of thumb; it is the pigeonhole principle, and it is the entire content of "strong consistency" in a quorum store. The table also shows the cost: to guarantee overlap at N = 3 you need W = 2, R = 2, which tolerates losing exactly one node on each side. Want to survive two failures? You need N = 5. Consistency and fault-tolerance are both bought in units of nodes, and the quorum condition prices them.

What CAP actually claims

Now the theorem, stated with the definitions the triangle hides. Gilbert and Lynch proved CAP in 2002 in the asynchronous model — a network with no clock and no bound on how long a message can take — and their words for the three letters are not the everyday ones; paraphrased, they are:

• Consistency means linearizability: every operation appears to take effect instantaneously at a single point in time, and everyone sees the same order. This is not the C in ACID. ACID's C is "my transaction respects my constraints"; CAP's C is "there is one authoritative history of a single object". Different word, different guarantee.
• Availability means every request to a non-failing node returns a non-error response — eventually, with no time bound. It is not "99.99% uptime". A system that answers every request in an hour is available by this definition and useless by yours.
• Partition is the network splitting so that some nodes cannot reach others.

Read that way, the theorem is nearly obvious: if the network partitions and a write lands on one side, a node on the other side either answers with stale data (sacrificing consistency) or refuses to answer until the partition heals (sacrificing availability). It cannot do both, because it cannot tell the difference between "the other side is slow" and "the other side is gone". That is all CAP says, and Brewer himself — who coined the conjecture — argued in 2012 that the "two of three" formulation is misleading: partitions are rare, so when there is no partition there is no reason to give up either C or A; the choice is not global but per-operation; and the useful question is what a system does during a partition and how it recovers, not which two letters it "keeps".

PACELC: the branch that runs all the time

CAP only says anything when the network is partitioned, which on a healthy network is almost never. So what governs your system the other 99.9% of the time? PACELC names it: if there is a Partition, choose Availability or Consistency; Else, choose Latency or Consistency. The else branch is the one running right now, on every request, and its trade is the one you actually live with — every synchronous replica you wait for buys consistency and costs latency. You already priced this branch in chapter 6: the five synchronous_commit levels, from 5,065 commits a second at off to 1,072 at remote_apply, are a PACELC "else" dial with a number on each detent. CAP is the dramatic branch that fires during an outage; PACELC's latency-versus-consistency is the mundane branch that fires on every commit, and it is the one that has shaped more systems.

Consensus, and why nobody puts it on the request path

When you truly need one authoritative answer across machines — who is the leader, which shard owns this key — you need consensus, and the two classic mechanisms show why it is expensive. Two-phase commit (2PC) has a coordinator ask everyone to prepare, then tell everyone to commit. Its flaw is in the gap: if the coordinator dies after prepare and before commit, every participant sits holding locks, unable to proceed, forever. Postgres ships 2PC turned off for exactly this reason:

show max_prepared_transactions;  ->  0
begin; insert ...; prepare transaction 'demo';
ERROR:  prepared transactions are disabled
HINT:  Set max_prepared_transactions to a nonzero value.

The documentation's own warning is that "it is unwise to leave transactions in the prepared state for a long time" — they hold locks and block vacuum — and that you should keep the feature disabled unless an external manager is watching. That is the blocking property of 2PC stated by a vendor rather than a textbook.

Raft fixes the blocking by replacing the single coordinator with a majority vote: a leader is elected, the log is replicated, and progress needs only a majority alive — Raft's paper states it plainly: "fully functional (available) as long as any majority of the servers are operational ... a typical cluster of five servers can tolerate the failure of any two servers." That majority is the same R + W > N quorum from the top of this chapter, now voting on a log instead of a value. But consensus is not free, and the paper's timing requirement is why it never sits on the data path: broadcastTime ≪ electionTimeout ≪ MTBF (mean time between failures), with the broadcast time "0.5ms to 20ms" because each RPC must reach stable storage, and "when the leader crashes, the system will be unavailable for roughly the election timeout" of 10 to 500 ms. So consensus runs the control plane — leader election, shard placement, cluster membership, distributed locks — the small amount of metadata that must be unambiguous. The data plane runs at async-replication speed. Kafka's controller, etcd behind Kubernetes, and Postgres HA managers like Patroni are all this shape: consensus for the few facts that must be singular, cheap replication for the flood of data that need not be.

Reach for this chapter whenever someone says a system "chose AP" or "chose CP" as if it were a fixed trait, or proposes to run application transactions through a consensus protocol for safety. The honest questions are narrower and answerable: what does this do during a partition, what does it cost in latency when there is no partition, and is this datum a control-plane fact that needs consensus or a data-plane fact that does not. Partitioning the data — as opposed to partitioning the network — is the next chapter, and it is a choice you do get to make.

You will shard later than the design articles say

Every "how we scaled" article starts a company with a billion users and shards the database on page one, and the reader walks away believing sharding is an early move. It is one of the last. Splitting one database into many buys you capacity you almost certainly already had, and it charges for it in cross-shard joins, distributed transactions, rebalancing, and hot keys — the very problems the rest of this chapter is about. Before you take on any of that, it is worth measuring how much one machine can actually do, because the number is larger than the folklore admits and it grows every year.

The ceiling of one machine keeps rising

This guide's entire cluster runs on four virtual CPUs, and on that hardware one Postgres does over 5,200 write transactions a second (each a multi-row TPC-B transaction) and about 119,000 indexed reads a second. Now look at what a single machine can be in 2026, from AWS's published specifications (dated, because these numbers move upward every year): a U7i instance offers up to 1,920 vCPUs and 32 TiB of memory; a single io2 Block Express volume delivers up to 256,000 IOPS. And Postgres's own documented limit on one table is 32 TB. Put those together: this four-vCPU box already does thousands of writes a second, a single available instance has 480 times its cores and two thousand times its memory, and one table can hold thirty-two terabytes before the engine itself objects.

The consequence is blunt: almost nobody running a web application is anywhere near the ceiling of one machine. A service doing a few thousand writes a second and holding a few hundred gigabytes — which describes most products that will ever exist — fits on one box with room to grow into a bigger box for years. Every shard added before that ceiling is reached is complexity bought to solve a problem you do not have.

So the honest order of operations, when a single database starts to strain, puts sharding dead last:

1. Fix the query. One missing index or one accidental sequential scan (chapter 5) is usually the whole problem.
2. Add the index the access pattern needs — and no more, since each one taxes writes.
3. Add memory, so the working set stays in cache and the disk stops being the bottleneck.
4. Add a read replica (chapter 6) to move reads off the primary.
5. Move cold data out — archive or tier the rows nobody queries.
6. Partition within one node — Postgres declarative partitioning splits a table into pieces on one machine, keeping each index small and vacuum cheap, with no distributed transactions.
7. Then, and only then, shard across machines.

Six cheaper moves come first, and most systems never exhaust them.

When you do shard, the naive scheme moves everything

Suppose you have earned the right to shard. The obvious method is to hash the key and take it modulo the number of nodes — the same key-to-slot trick Data structures used inside a hash table, now choosing a machine instead of an array slot. It works perfectly until you add a node, and then watch what the modulus does:

    4 -> 5 nodes: 79.95% of keys move   (ideal: 20.00%)
    8 -> 9 nodes: 88.89% of keys move   (ideal: 11.11%)
  100 -> 101 nodes: 99.00% of keys move (ideal:  0.99%)

Going from 100 nodes to 101, 99% of your keys move to a different machine — nearly every key in the system, all at once, because changing the divisor from 100 to 101 changes almost every remainder. The ideal is that you should only have to move the fraction that belongs on the new node, about 1%. Modulo hashing moves a hundred times that. During the move, cache is cold everywhere and the network saturates, which is why adding a shard to a modulo-sharded system is an outage.

The ring, and why one point per node is not enough

The fix is consistent hashing: hash the nodes onto a ring as well as the keys, and each key belongs to the next node clockwise. Add a node and it only steals the keys in the arc between it and the previous node — nobody else moves. Karger's paper, which introduced this, proves the property directly:

Monotonicity is immediate. When a new bucket is added, the only items that move are those that are now closest to one of the new bucket's associated points. No items move between old buckets.

Measure it, and the move-fraction drops from 99% to where it should be. But naively placing each node at a single point on the ring has its own failure, and it is severe:

  vnodes=   1: load spread 0.00x..3.08x of even   adding 11th node moves 15.65%
  vnodes= 100: load spread 0.78x..1.14x of even   adding 11th node moves  8.82%
  vnodes= 500: load spread 0.97x..1.02x of even   adding 11th node moves  9.36%

With one point per node the ring is wildly unbalanced: one node got zero keys and another got 3.08 times its fair share, because random points do not divide a circle evenly. This is not a tuning nuisance; it makes the scheme unusable, and it is exactly what Dynamo fixed by giving each node many virtual nodes — a hundred points on the ring instead of one — so the arcs average out. At 100 virtual nodes the spread tightens to 0.78×–1.14× and the move-fraction lands near the ideal 9%. Virtual nodes are not an optimisation on consistent hashing; they are the part that makes it work.

The key no hash can save

Consistent hashing with virtual nodes solves rebalancing and balance across nodes. It does nothing for the problem that actually takes shards down, because that problem is not about how keys are distributed — it is about one key. Real traffic is not uniform; it is Zipf-distributed, a few keys enormously more popular than the rest. Measure the hottest single key as the skew rises:

  zipf s=1.0: shard load 0.61x..2.09x of even; hottest single key ~=  8.32% of all traffic
  zipf s=1.2: shard load 0.33x..3.66x of even; hottest single key ~= 19.67% of all traffic

At realistic skew, one key is 20% of all traffic and the busiest shard carries eleven times the load of the quietest. No hash function helps, because the key is atomic — every hash sends all of that key's traffic to exactly one shard, and that shard melts. The celebrity's post, the viral link, the one product everyone is buying: sharding cannot split it, because splitting requires a place to put the halves and there is only one key. The real fixes live elsewhere — split the key artificially with a random suffix and merge on read, or (the next chapter) put a cache in front of the hot key so most of its reads never reach the shard at all.

Reach for this chapter when someone proposes sharding, and ask the question the design articles skip: what does one machine actually do, and are we near it? Usually you are not, and the answer is an index, more memory, or a replica. When you genuinely must shard, use consistent hashing with many virtual nodes so a resize is not an outage — and know that it buys you balance and rebalancing, not relief from the hot key, which is a caching problem wearing a partitioning costume.

Cache size sets the hit rate; the policy barely moves it

You already run a cache, and it is not Redis. Every response your server sends carries a Cache-Control header, and browsers and CDNs obey it, so the first cache to understand is the one in the HTTP spec — starting with the two directives that everyone swaps. no-cache sounds like "do not cache this". It does not mean that. RFC 9111 is exact:

The no-cache response directive, in its unqualified form (without an argument), indicates that the response MUST NOT be used to satisfy any other request without forwarding it for validation and receiving a successful response.

The no-store response directive indicates that a cache MUST NOT store any part of either the immediate request or the response and MUST NOT use the response to satisfy any other request.

So no-cache means "store it, but revalidate before you use it" — the cache keeps the copy and sends a conditional request each time to ask "still good?", saving the download when the answer is yes. no-store is the one that means "do not write it down". If you want a response kept out of caches entirely, no-store is the directive; no-cache still stores it. The whole HTTP caching model is these directives plus the distinction between a shared cache that "stores responses for reuse by more than one user" (a CDN) and a private one "dedicated to a single user" (the browser). Get the directive right and a CDN absorbs your traffic; get it wrong and you either leak private data into a shared cache or force a revalidation on every hit.

The policy question is the wrong question

When you do reach for a Redis in front of your database, the first argument people have is which eviction policy to use — LRU (evict the least-recently-used key), LFU (the least-frequently-used), or random — and it is a waste of breath. Run the same Zipf-distributed workload against Redis under every policy, capped at 8 MB:

  allkeys-lru      hit rate=80.94%
  allkeys-lfu      hit rate=81.56%
  allkeys-random   hit rate=78.91%
  noeviction       hit rate=81.05%   (with 24,934 OOM errors)

The best policy beats random eviction by 2.65 percentage points. Random. The sophisticated choice you were about to agonise over is worth less than three points against throwing darts. What actually moves the number is not the policy but the size:

    maxmemory=  2 mb   keys held=  1894   hit rate=58.91%
    maxmemory=  4 mb   keys held=  5929   hit rate=72.94%
    maxmemory=  8 mb   keys held= 14042   hit rate=81.61%
    maxmemory= 16 mb   keys held= 28598   hit rate=85.70%
    maxmemory= 32 mb   keys held= 28598   hit rate=85.70%
    maxmemory= 64 mb   keys held= 28598   hit rate=85.70%

Doubling from 2 to 4 MB buys 14 points — five times what the best policy bought. And then the curve does something you should exploit: it flattens dead. From 16 MB on, the hit rate is stuck at 85.70% and the key count at 28,598, because at that point the entire working set — the keys actually being requested — fits in the cache, and buying more memory caches keys nobody asks for. This is a curve you can generate for your own traffic, and it answers the only cache-sizing question that matters: how much does the next gigabyte buy? Here the honest answer past 16 MB is "nothing", and knowing where your own curve flattens is worth more than any amount of policy tuning. (Redis's LRU is itself deliberately approximate — it samples a few keys rather than tracking exact recency — and raising the sample count from 3 to 50 bought half a point for sixteen times the work, so even Redis agrees the policy is not where the value is.)

The stampede is a different problem entirely

Hit rate is the steady-state story. The failure that actually pages you is what happens the instant a hot key expires or the cache restarts cold. Two hundred requests arrive for the same key, the cache misses on all of them, and all two hundred go to the database at once:

  cache-aside, no protection    origin calls=199   p50= 469.7 ms   p99= 538.4 ms
  single-flight lock (SET NX)   origin calls=  1   p50=   1.0 ms   p99= 227.3 ms

Without protection, 199 of the 200 requests hit the database — a thundering herd that can take the origin down exactly when a popular key expires, which is the worst possible moment. The fix is single-flight: the first request to miss takes a lock (SET key:lock NX), computes the value, and fills the cache; everyone else waits for it instead of piling onto the origin. It collapses 199 origin calls to 1, and the median request from 470 ms to 1 ms. One SET NX turns a stampede into a single computation everyone shares.

Here is where three distinct fixes get muddled into "cache stampede handling", and they solve three different failures:

• Single-flight solves many requests missing the same key at once — one recompute per key, the result above.
• TTL jitter solves a different stampede: a batch of keys all written together expire together. Set 200 keys with the same 60-second TTL and all 200 expire in the same second; add ±10% jitter and the expiries spread over 13 seconds. Single-flight does nothing for this, because each key is a separate stampede; jitter is what desynchronises them.
• Serve-stale-while-revalidate solves making the user wait at all — return the expired value immediately and refresh it in the background, so a slow recompute never blocks a reader.

They are not interchangeable, and reaching for the wrong one leaves the failure in place. Single-flight will not save you from synchronised expiries; jitter will not save you from one viral key; neither hides recompute latency from the reader.

Why the invalidation bugs live in one place

There is a taxonomy of caching patterns — cache-aside, read-through, write-through, write-behind — but the distinction worth carrying is what each does about the two ways a cache can be wrong: a stale read (the cache has an old value) and a lost write (an update that never reached the store). In cache-aside, the pattern almost everyone actually uses, the application reads the cache, falls back to the database on a miss, and writes the value back — which means the application owns invalidation. When the underlying data changes, your code must remember to delete or update the cached copy, and there is no invariant enforcing it. That is why cache-invalidation bugs are legendary: the cache and the database are two stores kept in sync by hand, and the moment a code path updates the database and forgets the cache, they disagree, silently, until a user sees a stale price. Write-through (write to cache and store together) trades write latency for fewer stale reads; the choice is which failure you would rather have. But cache-aside's popularity is exactly why "the cache is wrong" is a bug you will debug more than once — the correctness was never automatic, it was a delete you had to remember.

Reach for this chapter when a cache is "not helping" or when the database falls over at a suspiciously round moment — the top of the hour, a deploy, a key going viral. A low hit rate is almost always a size problem, so plot the curve before you touch the policy; a database that dies in bursts is almost always a stampede, so add single-flight to the hot keys and jitter to synchronised TTLs; and a value that is wrong rather than slow is almost always cache-aside invalidation your code forgot. The next chapter moves from caching a read to reliably performing a write exactly once, which is a harder promise than it sounds.

Exactly-once delivery is impossible; exactly-once processing is a transaction

A service sends a charge request to a payment provider, and the network times out. Did the charge go through? You cannot know. The request might have been lost on the way, or it might have succeeded and the reply lost on the way back — and from where you sit, a lost request and a lost response are identical. So you have two choices, and only two. Retry, and risk charging the card twice. Do not retry, and risk never charging it. That is the whole problem of delivery in one example, and Kafka's own documentation states the impossibility plainly:

If a producer attempts to publish a message and experiences a network error, it cannot be sure if this error happened before or after the message was committed.

Because the sender cannot tell those apart, exactly-once delivery is not achievable, and no product sells it honestly. What you get is a choice of which error to prefer:

At most once — Messages may be lost but are never redelivered. At least once — Messages are never lost but may be redelivered. Exactly once — Each message is processed once and only once.

Read those as a menu and you will pick "exactly once" every time, which is why teaching the menu first is a mistake. The trick is to make the choice not matter, and the tool for that is idempotency: design the operation so that doing it twice has the same effect as doing it once. Charge by a client-supplied idempotency key, and a retried charge is recognised and ignored. Once processing is idempotent, you want at-least-once delivery — never lose a message, freely redeliver — because redelivery is now free. Exactly-once processing is at-least-once delivery plus an idempotent consumer. It is not a third delivery mode; it is the second mode with the duplicates made harmless. Kafka's warning is aimed exactly here: many systems claim exactly-once, and "it is important to read the fine print, because sometimes these claims are misleading."

The idempotency key returns the first answer

Make it concrete. Store the idempotency key with a primary-key constraint, in the same transaction as the effect, and let the database's uniqueness enforce once-only:

   charged
   replayed {'amount': 500, 'attempt': 1}   <- client retried the same request
   replayed {'amount': 500, 'attempt': 1}
  orders created by 3 delivery attempts of one request: 1

Three attempts, one order. The second and third attempts hit the unique constraint on the key, the handler catches the violation, and — the subtle part — it returns the first attempt's stored response, not a fresh one. A retry must replay what the original returned; it must not re-execute and it must not report a different answer, or the client sees {'attempt': 3} and concludes something changed when nothing did. The stored response is the idempotency, and returning it is the whole discipline.

The dual-write problem, and one commit that fixes it

Here is where most async systems have a bug they have never noticed. The handler does two things: it writes to its database, and it publishes an event to a broker. Two systems, two writes, no atomicity between them. Crash in the gap and you either committed the database change but never published the event (a lost notification), or published an event for a transaction that then rolled back (a phantom notification). This is the dual-write problem, and you cannot solve it by reordering the two writes — there is no order that closes the gap.

The fix is to make it one write. Instead of publishing to the broker, insert the event into an outbox table in the same transaction as the business change:

  after commit:            orders: 1   outbox unpublished: 1
  after a rolled-back txn: orders: 1   outbox: 1   (neither wrote)

One commit now covers both the order and the intent to publish it — they land together or not at all, because they are the same transaction. A separate relay process then reads the outbox and publishes to the broker at-least-once, marking rows published as it goes. If the relay crashes mid-publish it republishes, which is fine, because the consumer is idempotent. The outbox converts an impossible cross-system atomic write into an ordinary single-database commit plus an idempotent replay — the two tools you already have.

Postgres is a queue until it isn't

Reaching for Kafka the moment you need a queue is a common over-build, because the database you already run is a capable queue. The mechanism is one clause: SELECT ... FOR UPDATE SKIP LOCKED, which lets each worker claim rows while skipping the rows another worker has locked, instead of blocking behind them. Eight workers draining a thousand jobs:

  8 concurrent workers drained 1000 jobs in 73 ms, no job taken twice
  per-worker counts: [110, 110, 120, 120, 130, 130, 140, 140]  total=1000
  double-claimed rows: 0

That is about 13,700 jobs a second, with zero double-claims, from the database you already operate, transactionally, with no new system to run. For most web applications that is the right answer, and you should say so before you propose a broker. SKIP LOCKED is what makes it correct under concurrency: without it, workers serialise behind each other's locks; with it, they fan out across the ready rows.

When you do outgrow it, the reason is usually one of the things a log gives that a queue does not — and the queue-versus-log distinction is worth drawing. A queue deletes a message once it is acknowledged; a log (Kafka) keeps it, and each consumer holds its own offset, so "events are not deleted after consumption". That is why a log supports replay and many independent consumers of the same stream, and a queue more naturally supports competing consumers and per-message acknowledgement. Neither is an upgrade of the other.

The three properties you will have to reason about

Three concerns come with async work, and each is a trade you name rather than a setting you flip. Backpressure: an unbounded queue turns a throughput problem into an unbounded-latency problem, which is worse, because the work is stale by the time it runs — chapter 2's utilisation curve wearing different clothes. A bounded queue that rejects when full is a feature, not a limitation. Dead-letter queues: a message that fails forever will block its partition forever, so you move it aside to a DLQ after N attempts — and the cost, which you must accept consciously, is that ordering is now broken for that key, because you let later messages pass the one you set aside. Ordering: Kafka guarantees order only within a partition, and partition is chosen by key — events "with the same event key ... are written to the same partition ... in exactly the same order as they were written." Every "we need ordering" requirement is really "we need ordering per something" — per user, per account, per device — and naming that something is the actual design decision, because it is also your partition key and therefore your unit of parallelism.

Reach for this chapter whenever a retry might double an effect, or a notification is sometimes sent for something that did not happen, or a job is occasionally processed twice. The cure is almost never a better broker; it is an idempotency key stored with the effect, an outbox so the write and the publish are one commit, and the honest recognition that you are buying at-least-once and making the duplicates harmless. What happens when the retries themselves become the problem — when everyone retries at once during an outage — is the next chapter.

When a system is failing, doing less beats trying harder

A dependency slows down. Every service calling it, sensibly, retries. Every service calling those services sees them slow down too, and retries. The load on the struggling dependency does not stay flat while it recovers — it multiplies, precisely when it has the least capacity to absorb it. This is the counter-intuitive heart of failure handling: the instinctive responses — retry harder, wait for the slow call, keep trying — are the ones that turn a blip into an outage. The whole discipline is learning to do less when things go wrong.

Start with the multiplication, because it is worse than it looks. A client request passes through three layers of services, and each layer retries up to three times on failure:

  1 attempt  per layer ->   1 call
  2 attempts per layer ->   8 calls
  3 attempts per layer ->  27 calls
  4 attempts per layer ->  64 calls

Three retries at each of three layers is 27 calls at the bottom for one request at the top, because the retries compound multiplicatively down the stack — 3 × 3 × 3. During an outage, when the bottom layer is already failing, your own retry logic multiplies its load 27-fold. No individual service is misbehaving; each is following reasonable advice. Together they build a retry storm that keeps a struggling service down long after it would have recovered.

Jitter is the mechanism, not a refinement

"Add exponential backoff" is the standard fix and it is not enough, because unjittered backoff keeps all the clients synchronised — they all failed at the same instant, so they all wait the same doubling delay and all retry together, colliding in lockstep at every step. Measure a hundred clients contending for one slow resource under five strategies:

strategy                total calls   time to drain (s)
no backoff                    10095                2.01
exponential, no jitter         5149               94.29
full jitter                     919                3.70
equal jitter                    871                4.51

Unjittered exponential backoff took 25 times longer to drain than full jitter, did 5.6 times the work — and was slower than no backoff at all. Doing the sophisticated thing wrong lost to doing nothing. Full jitter — wait a random time between zero and the exponential ceiling — desynchronises the clients so they stop colliding, and it is what the AWS SDKs actually ship. Botocore's retry code spells the formula out in its docstring: t_i = rand(0, 1) * min(2 ** attempt, MAX_BACKOFF) — that random multiplier is full jitter, with a default of three attempts. Jitter is not a tweak on backoff; it is the part that makes backoff work.

Budgets beat counts, breakers come in two kinds

A per-request retry count cannot see the storm, because each request retrying three times looks reasonable in isolation — it is the fleet retrying that amplifies. The fix is a retry budget: a shared pool of retry capacity for the whole client, so the total retry volume is capped no matter how many requests want to retry. Botocore ships exactly this — a RetryQuota with INITIAL_CAPACITY = 500, charging each retry from the pool — and Envoy recommends budgets over static counts for the same reason: "the overall retry volume cannot explode and cause large scale cascading failure."

The word "circuit breaker" then hides two different mechanisms, and Envoy usefully splits them. What Envoy calls circuit breaking is a concurrency limit — a cap on how many requests can be in flight to a dependency at once — and its rationale is this chapter's thesis: "It's nearly always better to fail quickly and apply back pressure downstream as soon as possible." A concurrency limit protects you from a slow dependency, by refusing to pile up unbounded work waiting on it. The trip-on-consecutive-failures-and-eject mechanism that folklore calls a circuit breaker, Envoy calls outlier detection: "dynamically determining whether some number of hosts ... are performing unlike the others and removing them from the healthy load balancing set ... a form of passive health checking." That protects the pool from one bad host. Two mechanisms, one name — a concurrency limit bounds your exposure to a slow dependency; outlier ejection removes a single sick instance. You often want both, and knowing they are different is what lets you reason about which failure each one addresses. Bulkheads are the same idea applied to isolation: give each dependency its own limited pool of connections or threads, so one saturated dependency cannot consume every worker and take the rest of your service down with it.

Admission control: the limiter, and where it lies

When the load is more than you can serve, the honest move is to shed it — reject some requests fast so the rest succeed, rather than accept everything and fail everything. The tool is a rate limiter, and the choice of limiter is a real trade. The simplest, a fixed-window counter, has a flaw that disqualifies it:

  allowed just before boundary: 100, just after: 100
  -> 200 requests in a 0.2 s span, against a '100 per minute' limit (2x the intended rate)

A client sends 100 at the end of one window and 100 at the start of the next — 200 requests in a fraction of a second, twice the limit it is supposed to enforce, because the counter resets on a clock boundary the client can straddle. A token bucket fixes the burstiness cleanly: tokens refill at the sustained rate and the bucket depth caps the burst, so a burst of 60 against a depth-20 bucket admits exactly 20 and denies 40 — two knobs, rate and burst, with two distinct meanings. A sliding-window log is exact — it stores a timestamp per request and counts the true window — but exactness costs memory: about 26 bytes per request in the window, which at high rates per key is real.

And the limiter has a cost of its own that decides where it can live:

  5000 decisions in 580 ms = 8,619 decisions/s from one Python client, single Redis

A centralised limiter adds a network round trip to every request — and at 8,619 decisions a second from one client against a Redis that itself does 33,000 GETs a second, the bottleneck is the round trip, not Redis. That is why real distributed rate limiting pushes local token buckets to each node with periodic reconciliation, or shards the limiter by key: you cannot afford a synchronous hop on the hot path just to decide whether to allow the hot path.

The error budget is a permission slip

All of this is in service of a reliability target, and the last tool makes the target a number people can argue about honestly. Perfect reliability is the wrong goal — Google's SRE book makes the case that 100% is almost never the right target, because it is more reliability than users notice and costs far more to reach — and the error budget operationalises that. Turn an availability target — a service-level objective, or SLO — into an allowance:

  SLO 99.90%  ->  30,000 failed requests  =  43.2 min of full outage
  SLO 99.95%  ->  15,000 failed requests  =  21.6 min
  SLO 99.99%  ->   3,000 failed requests  =   4.3 min

Notice how little "one more nine" costs you: going from 99.9% to 99.99% cuts your monthly failure allowance from 30,000 requests to 3,000, from 43 minutes of outage to four. That budget is a permission slip: while you are under it, ship features fast and spend the budget on risk; when you blow through it, stop shipping and fix reliability. It converts an unwinnable argument between "move faster" and "be more stable" into a single shared number that decides which one wins this week. And the availability arithmetic behind it is worth carrying — 99.9% is nearly nine hours of downtime a year, 99.99% is under an hour — because every extra nine is roughly ten times the engineering cost, and most products do not need the nine they claim to want.

Reach for this chapter whenever a small failure becomes a large one — a slow dependency that takes the whole service down, an outage that will not end after the cause is fixed, load that spikes hardest exactly when you are weakest. The moves are all the same shape: retry with jitter and a budget so you do not amplify, cap concurrency and shed load so you fail fast instead of failing everything, and hold a reliability target you can actually spend. Doing less, on purpose, when the system is failing. The next chapter turns from operating a system to designing one, where these levers become things you plan for rather than reach for.

A design is a chain, and each link fixes the next

The "system design interview" has a folklore around it — a rubric of things you are supposedly graded on, published nowhere and repeated everywhere by prep vendors. It is worth being honest about what is actually known. The few primary facts come from companies describing their own process: Meta says its design interview runs about 45 minutes and almost never involves coding, that you spend it talking and drawing on a whiteboard, and that a full loop is up to six such conversations. That is about all that is established. The claims that cost is now graded explicitly, that "Design Kafka" appears at senior levels, that estimation is a hire signal — plausible, unverified, and not worth memorising.

What is worth learning is underneath the interview and does not depend on it: a good design is built as a dependency chain, where each decision constrains the next, and the order is not a ritual but a consequence of what needs what. Skip a link and the ones after it are guesses. Here is the chain.

Requirements → estimate. You cannot size anything until you know what it does, so first pin the functional requirements (what it must do) and the non-functional ones (how much, how fast, how available). Then turn them into numbers with the back-of-envelope habit from chapter 1: requests per second, data per year, read-to-write ratio. This is the load-bearing step, because the estimate decides the entire shape of the system.

Estimate → one node or many. With numbers in hand, the first real question is whether one machine suffices — and chapter 9 measured why it usually does. A service doing thousands of writes a second and hundreds of gigabytes fits on one box with a replica, and saying so with numbers is a stronger answer than reflexively sharding. Most designs that reach for a distributed architecture on slide one were sized wrong or not sized at all.

API → access patterns. Before the schema, define the operations: the handful of calls the system exposes. The API is not decoration; it is the set of access patterns, and access patterns are what a data model has to serve. "Look up a link by its short code" and "list a user's links newest-first" are different access patterns, and they demand different structures.

Access patterns → data model → indexes. The access patterns fix the schema, and the schema plus the patterns fix the indexes — because an index (chapter 5) exists to serve a specific query, and every index you cannot tie to an access pattern is a write tax with no reader. You derive indexes from the API, not from staring at the tables.

Indexes → write cost → what breaks first. Each index costs on every write (chapter 5 measured 2.5× the WAL for three of them), so the indexes fix the write cost, and the write cost against the estimate tells you the pressure point. Now you can answer the question the whole chain was built to answer: what breaks first, and how would you know? — the primary running out of write throughput, a hot key melting one shard, a queue growing without bound. That answer is the design.

Notice that each arrow is a genuine constraint, not a convention. The estimate really does decide whether one node works; the API really does fix the access patterns; the access patterns really do fix the schema; the schema really does fix the indexes; the indexes really do set the write cost. Follow the chain and every later decision falls out of an earlier one with a reason attached. Jump to "let's use Cassandra" before the estimate and you are choosing a data model before you know the access patterns before you know the load — three links skipped, three guesses stacked.

This chain also explains what seniority sounds like in a design discussion, which is a claim about engineering rather than about any company's scorecard. A junior answer names components: "we'll use a load balancer, a cache, and a database." True, and empty, because it is an inventory with no forces. A senior answer names trade-offs with numbers: "one Postgres handles this at 3% of its measured write capacity, so we start there and add a read replica when reads cross the primary's headroom" — a decision with a quantity behind it. A staff answer names what will break first and how you would know: "the write path is fine for years, but a single viral key is 20% of traffic by the Zipf measurement, so the failure mode is a hot partition, and I'd watch per-key request rates to catch it." Each level up is less about knowing more components and more about attaching numbers and failure modes to the ones you name. That is the same thing every prior chapter has done — measured the cost and named what breaks — which is why the honest way to prepare for a design conversation is not to memorise architectures but to have built and priced the pieces.

Reach for this chain whenever you are handed a blank design — in a review, an interview, or a real project starting from nothing. Start at requirements, force yourself to the estimate before you name a single technology, and let each link constrain the next, out loud, with the number that justifies it. The next two chapters walk the chain end to end on two real prompts, and the point of both is the same: the arithmetic does most of the work, and the honest answer is usually smaller than it first looks.

A URL shortener is one Postgres and a cache

"Design a URL shortener" is the classic opening prompt, and it is a trap for the eager, who reach immediately for sharding, a distributed ID service, and a global cache mesh. Walk the chain from the last chapter instead, starting with the estimate, and the architecture shrinks to almost nothing — which is the whole lesson.

The estimate. Say a hundred thousand new links a day, a generous number for most products, with a 100-to-1 read-to-write ratio because links are created once and clicked many times:

  100,000 new links/day   = 1.2 writes/s,  3.5/s at peak
  100:1 reads             = 116 reads/s,  347/s at peak
  measured row            = 134 bytes  -> 4.9 GB/year
  5 years                 = 182,500,000 rows, 24 GB   (PostgreSQL relation limit: 32 TB)
  peak write vs one node  = 3.5/s of a measured 4,672 write txn/s = 0.07%
  peak read  vs one node  = 347/s of a measured 91,561 reads/s   = 0.38%

Read the last two lines. Peak writes are 0.07% of one node's measured capacity; peak reads, 0.38%. (Those capacities are the shortener's own single-row insert-with-unique-check and point lookup, measured directly — a little different from chapter 2's heavier TPC-B and read-only pgbench on the same box, which is why this guide always names what a number measured.) Five years of links is 24 GB against a 32 TB table limit. The honest answer to "design a URL shortener" at this scale is one Postgres and a cache, and being able to say that with numbers — rather than apologising for not sharding — is the senior answer. Everything interesting is in the details the arithmetic leaves, not in the scale.

The key. The one genuine design decision is how to generate the short code, and there are three options with real trade-offs. A counter (a bigserial) encoded in base62 is dense, sequential, and gives the shortest codes — but sequential codes are guessable and leak how many links exist, letting anyone enumerate every link. A hash of the URL is deterministic (the same URL yields the same code, which may or may not be what you want) but needs collision handling. A random code avoids enumeration and is what most shorteners use — and it walks straight into the birthday problem from chapter 5's WAL benchmark: collisions among random draws become likely near the square root of the space, far sooner than "the space is full". Watch it bite:

  random 7-hex-char codes, space = 268,435,456
  199,928 codes drawn -> 72 collisions   (expected n²/2N = 75)

Two hundred thousand codes from a space of 268 million produced 72 collisions, matching the birthday estimate of 75 almost exactly — collisions at 0.07% of the space filled, not 50%. So a random-key shortener must handle collisions (insert, catch the unique-violation, retry with a new code — chapter 11's idempotency machinery in miniature), and it must size the space for the birthday bound, not the population. Base62 gives room to spare: six characters is 56.8 billion codes — 1,556 years at this rate — and seven is 3.5 trillion, so the codes stay short even with generous headroom. The unique index on the code (a B-tree from chapter 5) is what enforces no-collisions and what every read looks the code up through.

The read path and the cache. Reads outnumber writes 100 to 1, and a redirect is the same tiny lookup every time, so it caches beautifully — put the hot codes in Redis in front of the code-to-URL lookup and the database barely sees the read traffic. The redirect status code is itself a caching decision, the exact one chapter 10 was about. A 301 (permanent) tells the browser to cache the redirect, so subsequent clicks never reach your server at all — fast and cheap, but you lose click analytics and you can never change or disable that link, because the browser has stopped asking. A 302 (temporary) is not cached, so every click hits you — you keep analytics and control, and you pay for every redirect. The choice is the no-cache/no-store trade wearing a status code: cache aggressively and lose visibility, or stay in the loop and pay per click. Most shorteners choose 302 precisely because the analytics are the product.

What breaks first. Not the write path — 0.07% of a node has years of runway. It is the hot key. When one link goes viral it becomes, by chapter 9's Zipf measurement, a single key carrying a large fraction of all traffic, hammering one cache entry and one database row. The fix is chapter 10's, not chapter 9's: no hash reshuffle helps an atomic key, so you front the hot code with a cache (which a 301 or a CDN does for free) and, if it is truly extreme, serve it from the edge. The staff-level sentence is naming this before it happens: "the write path is trivial for years; the failure mode is a viral link becoming a hot key, and I'd watch per-code request rates to see it coming."

Reach for this walk whenever a prompt or a real project sounds big and you suspect it is not. Size it first: writes per second, bytes per year, against one node's measured capacity. Nine times in ten the answer is one primary, a replica, and a cache, and the engineering is in the key generation, the collision handling, and the one hot key — not in a distributed architecture you were about to build for load you do not have. The next prompt is the one where the machinery genuinely earns its place.

A webhook system is where every failure lesson comes due

Design a system that delivers webhooks: when something happens in your product, notify each customer's HTTP endpoint. It is the prompt closest to what you actually run, and it is the one where every lesson from the async and failure chapters comes due at once — because you are calling someone else's server, which is slow, flaky, and occasionally a black hole, and you are on the hook for delivering to it reliably anyway. Walk the chain.

The estimate. Five million events a day is a busy but ordinary product:

  5,000,000 events/day = 58/s average, 231/s at 4x peak
  measured: Postgres SKIP LOCKED drained 1,000 jobs, 8 workers, 90 ms = 11,111 jobs/s
  peak need = 231/s = 2.08% of one node's measured queue capacity
  at a 2% permanent-failure rate: 100,000 DLQ entries/day

Peak delivery is 2% of what one Postgres queue does (chapter 11's SKIP LOCKED, re-measured here at a slightly slower 90 ms — queue throughput wobbles run to run, but the fraction is tiny either way). So, again, no broker required to start — the database you already run is the queue. The engineering is entirely in the delivery guarantees, and that is where the failure chapters cash in.

Never lose an event: the outbox. The event is produced inside the transaction that changed your data, and it must not be lost if that transaction commits or emitted if it rolls back — the dual-write problem from chapter 11. So you write the event to an outbox table in the same commit as the business change, and a relay drains the outbox into the delivery queue. One commit, no lost or phantom events.

Deliver at-least-once, safely: idempotency keys. You cannot know whether a delivery that timed out was received (chapter 11: a lost request and a lost response are indistinguishable), so you retry, which means the customer may receive the same webhook twice. The contract, then, is at-least-once delivery plus a delivery id in every payload, so the customer can dedupe on their side — you make their processing idempotent by giving them the key to do it with. Promising exactly-once would be a lie; promising at-least-once with a dedupe key is the honest, buildable guarantee.

Retry without a storm: jitter, a schedule, a budget. A customer's endpoint will be down sometimes, so you retry on a backoff — and chapter 12 already proved it must be jittered or the retries synchronise into a storm. The schedule is a design knob with a plain meaning — how long does a customer get to be down before you give up:

   8 attempts, base 1 s, cap 1 h -> p50 window 0.04 h,  p99 0.06 h
  12 attempts, base 1 s, cap 1 h -> p50 window 0.57 h,  p99 0.96 h

Twelve jittered attempts give a customer about an hour of downtime to recover before you stop trying. And because you are retrying across many customers at once, the retries need a budget (chapter 12), not just a per-delivery count, so a fleet of failing endpoints cannot amplify into a self-inflicted flood.

The endpoint that fails forever: the DLQ, and the ordering it breaks. After the last attempt, a delivery goes to a dead-letter queue rather than blocking everything behind it — at a 2% permanent-failure rate that is 100,000 DLQ entries a day, a real operational surface you must build tooling for. And the DLQ costs you something chapter 11 named: ordering. Once you set one failed delivery aside and let later ones through, that customer's events are no longer in order. If a customer needs ordered delivery, you cannot DLQ past a stuck event — you must stop that customer's stream — so "ordered" and "never blocked" are in tension, and which you promise is a design decision, made per-customer by their partition key.

The slow customer, and why the bulkhead is the fix. Here is the failure that separates a system that survives from one that does not, and it comes straight from chapter 2's identity. A customer's endpoint degrades from 50 milliseconds to 30 seconds — not down, just slow. At your peak rate, Little's Law says how many deliveries are stuck in flight:

  endpoint  0.05 s -> L =    12 in flight
  endpoint  1.00 s -> L =   231 in flight
  endpoint  5.00 s -> L = 1,157 in flight
  endpoint 30.00 s -> L = 6,944 in flight

One slow customer, and 6,944 deliveries are in flight where twelve were before, each holding a worker, a connection, a slot. This is head-of-line blocking: one slow item at the front holds up everything queued behind it. Retrying makes it worse — more attempts, more workers held. The retry is not the fix and neither is a bigger pool, because the number grows with their latency without bound. The fix is a bulkhead (chapter 12): cap the concurrent deliveries per customer at, say, 20. Now one slow customer ties up 20 slots instead of 6,944, and the other customers are untouched. The bulkhead even tells you the truth about that customer — at 30-second responses, 20 slots deliver only 0.67 a second, so their own backlog grows and you will eventually shed or DLQ them — but it does so without letting them take down everyone else. That last sentence is the best single line in the worked set: at 231 a second and 30-second responses, the bulkhead, not the retry, is the fix, and it falls straight out of L = λW.

Reach for this walk when a system's job is to call things it does not control — webhooks, payment providers, third-party APIs, any integration. The shape is always the same: an outbox so you never lose the intent, at-least-once with a dedupe key so retries are safe, jittered retries on a budget so you do not storm, a DLQ for the permanently broken with eyes open about the ordering it breaks, and a per-dependency bulkhead so one slow partner cannot consume the whole system. Every one of those is a chapter you already built, standing up together on one design. The final chapter collects them into the tool you re-skim before the next one.

Which force is this, and what will it cost you

Every chapter of this guide taught a force — a thing that pushes on a system until it bends: utilisation, the tail, durability, layout, replication lag, contention, skew, the cost of a durable write. The skill the guide was actually building is the one this chapter names: looking at a system you have never seen and knowing, from its symptoms, which force is acting on it and what relieving that force will cost you. That recognition is not memory. It is reading a symptom back to its force, and you have now felt each force break something real and measured the price of the fix, so the table below is a set of arguments you have already had, not a list to memorise.

The decision table

Each row is a force, the symptom that reveals it, what you reach for, and — the column that keeps you honest — what that costs. Every entry is a chapter you built and measured.

Symptom you see	The force	Reach for	What you give up	Ch
Latency exploding, throughput flat	Utilisation past the knee	Cut ρ (shed, add capacity) or cut S (cache, index)	Money, or a longer queue you must bound	2
p99 is fine per-service, awful end-to-end	Tail × fan-out	Hedge, timeout at a percentile, cut fan-out	~5% more work, or abandoned slow requests	3
Must survive a crash	Durability	WAL, fsync, synchronous replication	~105 µs per flush; latency per sync level	1, 4, 6
Indexed query still slow	Physical layout	Cluster the data; the index only finds pointers	Re-clustering cost; one sort order	5
Writes slowing as you add features	Index write-amplification	Drop unused indexes, or an LSM store	Read speed, or read amplification	5
Read after write sometimes missing	Replication lag	Route to primary, or remote_apply	~5× commit latency	6
An "impossible" invariant breaks under load	Write skew	Serializable + retry on 40001, or lock the row	Aborts and retries under contention	7
Need one answer across machines	Consensus	Quorum R+W>N; consensus on the control plane only	Latency, availability during a partition	8
One node filling up	Capacity ceiling	The six escape hatches, then shard	Cross-shard joins, rebalancing	9
One key melting one shard	Skew (hot key)	Cache or split the key — no hash helps	A special-case path	9, 10
Cache barely helping	Working set > cache	More size (plot the curve), not a better policy	Memory	10
Origin dying in bursts	Stampede	Single-flight, TTL jitter, stale-while-revalidate	A little staleness or complexity	10
Retries double an effect	At-least-once delivery	Idempotency key + outbox (one commit)	A dedupe store and its bookkeeping	11
Small failure becoming large	Retry amplification	Jitter, retry budget, circuit breaker, shed	Some rejected requests, fast	12
One slow dependency taking you down	Head-of-line blocking	A per-dependency bulkhead (concurrency limit)	That dependency's throughput	12, 15

Read down the "what you give up" column and the shape of the subject appears: there is no free choice. Every fix moves a cost somewhere else — consistency for latency, read speed for write speed, memory for hit rate, throughput for isolation. A design is not a search for the option with no downside. It is choosing which downside you can live with, and being able to say which one, with a number, is the entire craft.

When the table runs out

The table stops where a system gets unusual, so carry the procedure that built it — four questions that place almost anything, the same four the design chain drilled.

What is the load, and does one node survive it? Estimate first, always. Requests per second, bytes per year, read-to-write ratio, against one machine's measured capacity. Chapter 9's lesson is that the answer is usually yes, and reaching for distribution before you have done this arithmetic is the most common and most expensive mistake in the subject.

Which resource saturates first? Every system has one bottleneck that gives out before the others — a CPU, a disk's fsync rate, a single hot row, the network. Find it, because chapter 2's curve says the whole system's latency explodes when that resource passes its knee, and no amount of headroom elsewhere helps. Optimising anything but the bottleneck is motion without progress.

What must be consistent or durable, and what may be stale? This is the dial with a price tag, from chapters 6 through 8. Name the data that must survive a crash and the reads that must see the latest write — and, just as important, the far larger set of data that may lag, because that is where all your latency headroom and availability live. Systems fail when everything is treated as needing the strongest guarantee, which is neither affordable nor necessary.

What breaks first as it grows, and how would you know? The staff-level question. Trace the load forward and name the first thing to give — the write path, a hot key, an unbounded queue — and then name the metric that would warn you. A design you cannot describe the failure of is a design you do not yet understand.

The honest close

It is worth ending where the guide's siblings did, with candour about the exam this material is nominally for. There is no single "system design round", the published rubrics are thin, and most of what circulates as interview lore is prep-vendor content marketing. What is real, and what every honest description of a good engineer agrees on, is exactly the four questions above turned into behaviour: establish the load before naming a technology, say why you chose what you chose with a number attached, name the trade-off you are accepting, and know what will break first. That is not a trick for a whiteboard. It is what designing a system is, and it survives whatever the interview format does next — and, more to the point, it makes you better at the systems you already run, which was the reason to read this at all.

So the thing to carry out of all three guides is one habit, not a catalogue. Read the problem for its force, size it before you build, choose which cost you will pay and say so out loud, and price everything you reach for. You have now built and measured every force in the table underneath it — the page, the log, the queue, the replica, the cache, the retry — rather than being handed the table cold. That is the difference between memorising the patterns and understanding them, and it is the difference this guide, and the two before it, were written to make.

Further reading

Primary sources, one line each on what it is good for. Every number in this guide traces to one of these or to a script re-run on the four-vCPU machine it was written on.

• PostgreSQL 16 documentation source and backend READMEs (postgres/postgres, REL_16_STABLE) — read as primary rather than as a website: doc/src/sgml/storage.sgml for the page and TOAST, wal.sgml for the write-ahead log, mvcc.sgml for the isolation table and snapshot isolation, high-availability.sgml for replication, ref/prepare_transaction.sgml for why 2PC ships off, limits.sgml for the 32 TB relation limit.
• pageinspect, pg_test_fsync, pg_test_timing, pgbench — the tools that made the storage, durability and queueing chapters measurable rather than asserted; every table in chapters 1–9 came from one of them on the live cluster.
• Kafka 4.1.0 documentation (apache/kafka, 4.1.0/docs/) — design.html for the delivery-semantics taxonomy, the exactly-once fine print, and the acks=all/ISR trap; introduction.html for per-partition ordering. Kafka 4.x is KRaft-only, so any ZooKeeper material is stale.
• Redis 7.0 (redis/redis, 7.0/redis.conf and src/evict.c) — the eight maxmemory-policy values and the approximate-LRU maxmemory-samples trade the caching chapter measured. Note the 2024 licence change and the Valkey fork if you are choosing it today.
• LevelDB design document (google/leveldb, doc/impl.md) and the RocksDB wiki (facebook/rocksdb, Leveled-Compaction.md) — the LSM levels, compaction, and the vendor's own "larger than 10" write-amplification figure; also the object lesson of the "100MB/s" constant now off by 20×.
• RFC 9111 (HTTP Caching), from its editing source (httpwg/http-core) — the exact meaning of no-cache, no-store, s-maxage, Vary and heuristic freshness.
• Envoy architecture docs (envoyproxy/envoy, docs/root/intro/arch_overview/upstream/) — circuit_breaking.rst and outlier.rst, which split the one word "circuit breaker" into a concurrency limit and outlier ejection.
• The AWS SDK retry source — boto/botocore (botocore/retries/) for the shipped Full Jitter formula, the RetryQuota, and the client-side token bucket; aws/aws-sdk-go-v2 for the 2026 throttle-aware backoff. Read the source, not the 2015 blog post whose formula it no longer matches.
• Berenson, Bernstein, Gray, Melton, O'Neil & O'Neil, "A Critique of ANSI SQL Isolation Levels" (SIGMOD 1995 / MSR-TR-95-51) — why forbidding the three ANSI phenomena is not serializability, and where write skew lives.
• Ongaro & Ousterhout, "In Search of an Understandable Consensus Algorithm" (Raft) — the majority-quorum availability property and the broadcastTime ≪ electionTimeout ≪ MTBF timing that keeps consensus off the data path.
• Karger et al., "Consistent Hashing and Random Trees" (STOC 1997) — monotonicity, and the balance properties that virtual nodes exist to fix.
• Dean & Barroso, "The Tail at Scale" (CACM 2013) — fan-out tail amplification and hedging; the mechanism (1 − 0.99ⁿ) is reproduced here, the headline figures reported.
• Gilbert & Lynch (2002), Brewer's "CAP Twelve Years Later" (2012), and Abadi's PACELC (2012) — CAP stated precisely, retracted as "two of three", and extended to the else-branch that runs all the time.
• Google SRE book, "Embracing Risk" and "Service Level Objectives" — the error budget as a permission slip, and why 100% is the wrong target.
• Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — the book-length version of chapters 5–11; excellent, and predates Kafka 4.x, PostgreSQL 12–16 and every current instance type, so re-check version-dependent claims against the vendor docs above.