The File That Wouldn't Die

What we built

Four laptops on home Wi-Fi, one shared folder. Press play.

It worked, and it survived crashed laptops. It also had every classic distributed-systems bug in miniature. Let's fix them one at a time.

TCP sends bytes, not messages

We ended every message with the word aosppdfs and read 1,024 bytes at a time. Watch what can go wrong:

import struct

async def send(writer, payload: bytes):
    writer.write(struct.pack("!I", len(payload)) + payload)   # 4-byte length, then the bytes
    await writer.drain()                                       # backpressure: wait if the socket is full

async def recv(reader) -> bytes:
    (size,) = struct.unpack("!I", await reader.readexactly(4))
    if size > MAX_FRAME:
        raise ValueError("frame too large")                    # never trust a length blindly
    return await reader.readexactly(size)                      # exactly that many, however TCP splits them

One lock for everything

Every thread shared one global directory behind one lock. Four reads and two writes arrive:

Snapshots: readers grab the current immutable map; a writer builds a new one and swaps the pointer. At thousands of connections, swap threads for asyncio streams.

Where does a file live?

We picked a random peer for the copy, then broadcast the whole directory so everyone could find it. Hashing makes the location computable instead:

Real 32-bit hashes of 120 file names, 16 virtual points per server. Consistent hashing moves about 1/n of the files; mod n moves most of them.

How many copies must say yes?

Keep N = 3 copies. A write waits for W of them; a read asks R of them. Break some servers and try it:

async def put(name, data, if_match, W=2):
    chunks = [data[i:i + CHUNK] for i in range(0, len(data), CHUNK)]    # 4 MiB pieces
    ids = [sha256(c) for c in chunks]                                     # the checksum is the chunk's name
    replicas = ring.owners(name, n=3, distinct_zones=True)

    acks = await first(W, [r.store(ids, chunks) for r in replicas], timeout=0.5)
    if len(acks) < W:
        raise Unavailable("not enough copies, try again")                 # never claim it was saved

    return await metadata.compare_and_set(                                # Raft: one truth per name
        name, expect=if_match, new={"chunks": ids, "version": if_match + 1})

Two people, one file

On the web this is just ETag + If-Match. For offline edits, vector clocks spot concurrent versions.

Deleting is a write

Who's still alive?

Our peer list was typed by hand. Real clusters gossip: every second, each server tells a couple of random others what it knows.

Missed heartbeats make a server suspect; gossip spreads the verdict in about log(n) rounds. See gossip protocols.

Is my copy still right?

Disks rot quietly. Hash every chunk, then hash the hashes into a tree, and two replicas can find a difference without shipping the data:

Copies, or math?

Erasure codes rebuild lost pieces from any k survivors (Reed–Solomon). Hot files: copies. Cold files: math.

Agreeing on names

Bytes can live anywhere. But "which version of notes.txt is current?" needs one answer. That's consensus:

Raft picks consistency over availability: a minority can't write. That's the C in CAP, chosen on purpose.

Fast downloads

From one laptop to the internet

Same ideas at every step. Press play, and it grows from the bottom up.

PUT    /files/notes.txt   If-Match: "v3"         → 200 ETag: "v4"  |  412 someone beat you
GET    /files/notes.txt   Range: bytes=0-4194303 → 206 the first chunk
GET    /files/notes.txt   If-None-Match: "v4"    → 304 your copy is current
DELETE /files/notes.txt   If-Match: "v4"         → 204 and a tombstone
GET    /health                                   → 200 for the load balancer

Adding a server

Watching it heal

The number that matters most isn't latency. It's how many chunks are missing a copy right now.

2020 vs today, one page

1. Messages on a TCP stream
   end marker (2020)length prefix

   Any bytes, no hangs. Cost: a 4-byte header.

2. Many connections
   thread each + one lock (2020)asyncio + snapshots

   Readers never wait. Cost: async code everywhere.

3. Where a file lives
   random peer (2020)hash ring, zone-aware

   Computable, and growth moves 1/n. Cost: virtual nodes to balance.

4. How many copies
   2 (2020)3, W = 2, R = 2

   Survives one loss during repair. Cost: 3× storage.

5. Two writers
   last writer wins (2020)compare-and-set

   No silent loss. Cost: clients retry.

6. Deletes
   rename to "name *" (2020)versioned tombstones + GC

   We were right; add expiry. Cost: tombstones take space.

7. Membership
   typed IP list (2020)seeds + gossip

   Servers come and go. Cost: tuning suspicion timeouts.

8. Metadata
   broadcast to all (2020)a Raft group

   One truth per name. Cost: a majority must be up.

9. Bit rot
   trust the disk (2020)checksums + Merkle repair

   Corruption is found, not served. Cost: hashing CPU.

10. Getting in
    raw sockets on Wi-Fi (2020)HTTPS, tokens, a load balancer

    Safe to expose. Cost: one more hop.

The well-earned drink

Every Brew Miles trip ends with one:

1. Frame your bytes, and version everything, deletes included.
2. Hash to place, copy to survive, and spread copies across failure zones.
3. Make writes conditional and pick quorums that overlap.
4. Gossip to notice, Raft to agree, Merkle trees to repair.
5. Watch under-replication, and kill a server on purpose before one dies on its own.