Distributed consensus
Quorum
Keep a replicated store correct while the network lies to it.
You play a cluster operator running five nodes through partitions, crashes and clock skew. You watch nodes, messages, elections, logs diverging and truncating, split brain.
What you'll come away knowing
- Why a majority quorum works: any two majorities of the same cluster share at least one member, so two leaders of the same term cannot both be elected.
- Why the minority side of a partition cannot commit anything, however healthy it looks from inside.
- What actually causes split brain — and that a faster failure detector makes it worse, not better.
- Why an election timeout that is tight relative to network latency produces an election storm rather than a fast failover.
- That commit latency is set by the majority-th fastest replica — so two slow disks in a five-node cluster cost you nothing, and three cost you everything.
- What a leader lease assumes about clock skew, and what happens the moment that assumption breaks.
How the model works
The cluster runs Raft as specified in the Ongaro and Ousterhout paper: terms, randomized election timeouts, RequestVote with the log up-to-date check, AppendEntries with the prevLogIndex/prevLogTerm consistency check, followers truncating conflicting suffixes, and a commit index that only advances for entries from the current term replicated on a majority.
The network model gives every link a latency distribution, a loss rate, duplication and reordering. Partitions are arbitrary groupings you draw yourself. Nodes have clock skew and drift, crash and restart with either persistent or volatile state, and their disks have an fsync latency you can turn up.
A set of clients issues reads and writes with timeouts and retries. A history checker watches the whole run from outside and flags lost writes, stale reads, duplicate application, and two nodes accepting writes at once. That checker is the scoring oracle — availability is worth points, but a single lost acknowledged write fails the run.
There is also a deliberately naive mode: primary-backup with a heartbeat failure detector and no quorum or fencing at all. It exists so you can watch split brain happen on purpose, and then fix it.
Every constant used — latencies, timeouts, sizes — is listed with its source in docs/NOTES-quorum.md, along with what the model deliberately simplifies. The numbers are illustrative, not measurements.
Scenarios
- TutorialElect a leader
Five nodes, no faults. Watch a term start and a leader emerge.
- TutorialMinority partition
Cut the cluster 2/3 and watch why the old leader cannot commit.
- CoreSplit brain
Naive mode: two primaries, both taking writes. Then fix it.
- CoreElection storm
Timeouts too tight for the latency. Tune them until it settles.
- CoreDivergence and truncation
A leader dies holding uncommitted entries. Watch the followers realign.
- CoreSlow disk, slow cluster
The network is fine. The commit latency is not.
- CoreThree nodes or five?
How many simultaneous failures can each survive, and what does the fifth node cost you?
- HardClock skew breaks the lease
Lease reads assume bounded skew. Break the assumption.
Questions
Is this a real Raft implementation?
It implements the algorithm from the paper, including the five safety properties, and those properties are asserted in tests against randomized chaos runs. It is not a production consensus library: there is no snapshotting or log compaction in v1, and it runs against a simulated network rather than a real one.
Are the timeouts realistic?
They start from the paper’s suggested ranges — a 150 to 300 ms randomized election timeout with a heartbeat well under that — and every constant is documented in the notes with its source. They are illustrative, not measurements of any deployed cluster.
What does the history checker actually check?
It has the sim’s global view, so it can be stricter than a real black-box checker. It flags a write that was acknowledged to a client but is not present in the committed log, a read that returns a value older than a previously acknowledged write to the same key, an entry applied twice, and any interval where two nodes both accepted writes.
Why is correctness scored harder than availability?
Because that is the trade the algorithm is making. A system that goes unavailable during a partition has done its job; a system that stayed up and lost an acknowledged write has not. The scoring reflects that: a lost write is a failing grade regardless of uptime.
Can I break it on purpose?
That is the point. Drag nodes apart to draw a partition, kill and restart them, slow a link or a disk, skew a clock, tighten the election timeout until the cluster thrashes, or switch to naive mode and watch two primaries take writes.
Does the share link really replay my run exactly?
Yes. A run is a seed plus a log of your inputs at the exact ticks you made them, and the model has no access to the clock or to any randomness it does not own. Replaying that log reproduces the run tick for tick.