Independent computer or process.
Own memory + own clock.
Nodes do not share instantaneous state.
Side 58
Distributed
Systems
A study of computation spread across unreliable machines. Distributed systems replace shared memory with messages, introducing delay, partial failure, uncertain ordering and the need for explicit coordination.
node→message→failure→coordination→consistency
06system lenses
05failure modes
05coordination problems
58Side
Distribution changes the failure model.
One process can no longer directly know the state of another; it only receives messages that may be delayed, duplicated or lost.
Communication crosses a network.
Delay is variable.
A missing reply does not reveal whether the peer failed or the network is slow.
Copies can diverge.
Who knows the latest value?
Replication creates availability and coordination problems simultaneously.
Some parts fail while others continue.
Partial failure.
Distributed systems must operate without assuming all components share fate.
Rejoin without corrupting state.
Replay, reconcile, elect?
Recovery logic is part of the protocol, not an afterthought.
There is no perfectly shared clock.
Distributed systems often need to reason about order without relying on synchronized wall time.
Physical clocks diverge.
Synchronization reduces but does not eliminate uncertainty.
Causal order without global time.
If one event can influence another, the system can order them causally.
Represent ordering abstractly.
Lamport clocks track a partial temporal relation among events.
Track causal histories.
Vector clocks can distinguish concurrent updates from causally ordered ones.
Bound waiting.
Timeouts turn uncertainty into a decision point without proving failure.
Grant authority for bounded time.
Leases depend on carefully managed clock assumptions.
Replication trades coordination for availability and performance.
Copies increase resilience and read capacity, but writes must be propagated and reconciled.
Leader
One node orders writes.
Followers replicate the leader’s log or state.
Multi-leader
Several nodes accept writes.
Useful across regions but creates conflict-resolution work.
Leaderless
Clients write to several replicas.
Quorums and repair mechanisms reconcile divergent copies.
Lag
Replicas are temporarily behind.
Read behavior depends on whether stale results are acceptable.
Conflict
Concurrent writes disagree.
Systems need deterministic merge or application-level resolution rules.
Consensus creates one agreed history despite failures.
Protocols such as Raft and Paxos coordinate nodes on an ordered sequence of decisions.
Choose a coordinator.
Nodes must avoid accepting conflicting leaders for the same logical term.
Require overlapping majorities.
Overlap prevents two independent decisions from both appearing committed.
Replicate an ordered history.
State machines can replay the same committed commands deterministically.
Decide when an entry is durable enough.
Consensus separates proposed work from work known to be committed.
Never decide conflicting values.
Safety may be preserved even when progress temporarily stops.
Eventually make progress.
Progress depends on enough functioning nodes and communication.
Consistency defines what clients are allowed to observe.
Different guarantees make different promises about ordering and freshness.
| Model | Guarantee | Cost |
|---|---|---|
| Linearizable | Operations appear to occur atomically in real-time order | Coordination and latency |
| Sequential | All clients see one order consistent with program order | Still coordinated, weaker than real-time |
| Read-your-writes | A client sees its own completed updates | Session routing/state |
| Eventual | Replicas converge if updates stop | Temporary divergence allowed |
| Causal | Causally related operations preserve order | Metadata and dependency tracking |
Reliability comes from designing around failure, not assuming it away.
Distributed systems need idempotence, retries, redundancy and clear ownership of uncertain work.
Retry
Repeat transiently failed operations, but make repeated execution safe where possible.
Idempotence
Ensure duplicate requests do not create duplicate effects.
Backoff
Slow retries to avoid amplifying an overloaded dependency.
Circuit breaker
Stop sending work to a dependency that is repeatedly failing.
Dead letter
Preserve work that cannot be processed automatically for later inspection.
Observability
Correlate logs, metrics and traces across nodes to reconstruct failure paths.
Designing Data-Intensive ApplicationsKleppmann · distributed data systems
Distributed SystemsTanenbaum & van Steen · broad foundation
In Search of an Understandable Consensus AlgorithmOngaro & Ousterhout · Raft
Distributed AlgorithmsNancy Lynch · formal foundations