Database field guide · 22

Raft Consensus

Terms, quorums, replicated logs, and failure recovery

Build an operational model of Raft leader election, log replication, quorum commit, membership, and consistent reads.

Franck Pachot7 chaptersRaft · consensus · quorum · replication

Build the mental model before choosing the mechanism.

This minibook was AI-generated from Franck Pachot's archived blog posts. Links to the original articles are included for source context and verification.

A term gives leadership a generation

Raft divides time into monotonically increasing terms. A server is a follower, candidate, or leader, and election timeouts trigger candidacy when heartbeats disappear.

A candidate needs votes from a majority.
At most one leader can win a term when voting rules hold.
Randomized election timeouts reduce repeated split votes.

The leader orders the replicated log

Clients send writes to the leader, which appends log entries and replicates them to followers. The log index provides order while the term identifies the leadership generation that created an entry.

Followers reject entries that do not match their preceding log.
Conflict repair truncates an uncommitted divergent suffix.
State machines apply committed entries in log order.

Commit requires a quorum

An entry from the leader's current term becomes committed after replication to a majority. Majority intersection ensures that a later elected leader contains committed history.

A three-replica group tolerates one unavailable replica.
A five-replica group tolerates two but increases replication work.
Acknowledgement policy determines client-visible durability.

Election safety depends on log freshness

A voter grants its vote only to a candidate whose log is at least as up to date as its own. This prevents a candidate missing committed entries from becoming leader.

Term is compared before log index.
Committed entries survive leadership changes.
Uncommitted entries may be overwritten after failover.

Reads need a leadership proof

A leader can serve a linearizable read only after confirming it still leads a quorum or using an equivalent lease with bounded clock assumptions. Follower reads require an explicit staleness or safe-time contract.

A heartbeat alone does not make arbitrary follower reads current.
ReadIndex avoids appending a log entry for every read.
Local follower reads exchange freshness for latency.

Operations happen per consensus group

Distributed databases run many Raft groups, commonly one per tablet. Leaders, replicas, log retention, snapshots, and membership changes must be balanced across nodes.

A healthy cluster can still have one unhealthy group.
Snapshots catch up followers whose logs are no longer retained.
Joint membership changes preserve quorum intersection during reconfiguration.

Field manual

Concrete mechanics, diagnostic evidence, and executable patterns to carry into a real system.

Calculate failure tolerance

A Raft group with N voting replicas commits through a majority and tolerates floor((N-1)/2) unavailable voters, assuming the remaining voters can communicate.

replicas = 3
quorum = replicas // 2 + 1      # 2
failure_tolerance = (replicas - 1) // 2  # 1

Separate leader and follower health

For each tablet or consensus group, retain leader identity, current term, committed index, applied index, follower match indexes, and heartbeat age. Cluster-level node health is too coarse.

group_health = {
  'term': current_term,
  'leader': leader_uuid,
  'commit_index': commit_index,
  'apply_lag': commit_index - applied_index,
  'replication_lag': commit_index - follower_match_index,
}

Model quorum latency

A write waits for the leader's local path and the fastest remote acknowledgements needed for a majority. The slowest replica is irrelevant until it is needed for quorum or blocks log retention.

commit_latency ≈ leader_append
               + kth_fastest_follower_round_trip
               + durability_and_apply_boundary
# For RF=3, k=1 remote follower when the leader counts in the majority.

Test partitions, not only crashes

A useful fault test isolates leaders from different subsets, delays messages, restarts stale followers, and verifies that only a majority side accepts writes while committed data survives re-election.

1. Record term, leader, commit index, and client request ID.
2. Partition the leader from both followers.
3. Verify a majority elects a new leader and the minority rejects writes.
4. Heal, verify one log, then retry ambiguous requests idempotently.

Source articles

Optional deep dives with the complete experiments and product-version context behind this guide.

Fault Tolerance with Raft and no Single Point of Failuredev.to · 2024-09-05 Write Buffering to Reduce Raft Consensus Latency in YugabyteDBdev.to · 2024-09-14 Local reads from 🚀YugabyteDB raft followersdev.to · 2022-06-27