What it is
A Python implementation of the Raft consensus algorithm, forming a distributed key-value store over HTTP. Five nodes give a quorum of three, so the cluster keeps serving with two of them down.
Nothing is forwarded between nodes. A follower refuses a read or write with a 421 naming the current leader, and the client retries against it, which is the approach §8 of the Raft paper describes. Built from In Search of an Understandable Consensus Algorithm by Ongaro and Ousterhout.
Code: github.com/oliverstubbs/raft-kv-store
What I built
Leader election on a randomised 150 to 300ms timer, with pre-vote: a candidate first asks whether a vote would be granted, without anyone changing term, so a node returning from a partition can rejoin without deposing a healthy leader. That part is an extension to standard Raft, from §9.6 of Ongaro's thesis.
Log replication on a 50ms heartbeat, each follower checking an entry lines up with its own log before appending.
A NOOP on election. Every new leader appends one, which is what lets entries inherited from an earlier term commit without waiting for the next client write to come along.
Reads from the leader, served only after it confirms a majority is still following it, so a leader that has been deposed but not yet noticed cannot answer with stale data.
Crash-safe persistence. last_applied is committed in the same transaction as the data it applies, and term and vote are written together in a single commit, so a restart can never vote twice in one term.
Tests. Six files covering the consensus logic directly, plus a failover script that kills the leader mid-run and checks the write survives.
Two Compose modes. A development shape with hot reload and debug endpoints, and a deployment shape with neither, because a reloading node triggers an election and makes any measurement meaningless.
What I learned
The bug that took longest to find was a liveness one. Heartbeats were sent from the same loop that applied committed entries, and applying needed a lock that every in-flight write was also queuing for. Under load the heartbeats waited behind the writes: 25 concurrent writes stretched the gap to around 200ms, 100 stretched it to 750ms, well past the election timeout, and a perfectly healthy leader kept losing its leadership to a follower that had simply stopped hearing from it. Moving heartbeats into their own task, on a path that never takes the lock, fixed it. No single piece was wrong. The problem only existed under load.
Pre-vote had a quieter bug. Failover took about seven seconds instead of under one, and nothing errored. I had used one timer to answer two questions: should I start an election, and have I heard from a leader recently. A candidate resets that timer just before asking for pre-votes, so it would turn down its peer's request, and with one node already down that peer was the only vote left to get. Giving the second question its own timestamp brought failover back under a second.
A node can also stop being the leader without knowing it. It gets partitioned, nobody tells it, and it carries on answering reads with data that is already stale. I thought a node could just check whether it was still the leader, but it can't, it has to ask the others. That is why a read is only served once a majority has confirmed it.
The log is kept in the database and in a list in memory, so reads are fast. When a database write failed after the list had already been changed, the two disagreed, lookups quietly returned the wrong entry's term, and a follower started accepting entries it should have rejected. Nothing crashed. The same goes for truncation: cutting a follower's log whenever new entries overlap it looks harmless, but the leader resends overlapping entries every round, so a delayed duplicate would chop off entries that were already committed.
A passing test told me very little until I had broken the code it was meant to protect and watched it fail. Several of mine passed either way.
What it does not do yet
Documented in full in the repo, but the main gaps: there is no snapshotting, so the log grows without bound and is read into memory in full on every start; membership is static, so the cluster cannot change size without a restart; and log backtracking drops a follower's next index by one per round trip, so a node starting from an empty log catches up slowly.