Durability and Consistency
Durability
An append is acknowledged once a majority of voters has replicated it. A single node failure cannot lose an acknowledged write. With the default raft.wal.fsync = "never", a replica holds the newest appends in the page cache, so a host crash of a majority of voters at once can lose the writes acknowledged after their last fsync. Set always to acknowledge only after fsync (see Configuration). Acknowledged data lives in replicated hot state across the cluster, and a background worker flushes older segments to S3 on a configurable interval, after which the data inherits S3-grade durability.
In a cross-region deployment with a 5-second flush interval, per-message durability is approximately 9-10 nines. The window where acknowledged-but-unflushed data is at risk from a simultaneous multi-region failure is measured in seconds.
Consistency
Writes are linearizable. Each Raft group serializes appends through its current leader, which assigns a total order.
Catch-up reads may be served by any replica that has applied the relevant stream state. Followers can return already replicated historical bytes locally because stream positions are immutable. Tail-sensitive reads still preserve protocol-visible semantics: HEAD, Stream-Up-To-Date: true, Stream-Closed: true, and offset=now are generated by the leader path unless the follower has applied a terminal closed state. Requests that need a write-side access transition, such as expiry or TTL touch, are also routed to the leader.
A follower can be behind an acknowledged append. If a catch-up cursor exceeds
its local tail, it checks that cursor through a quorum-confirmed leader read
instead of returning 416 from its stale prefix. A genuinely invalid cursor
still returns 416; without a confirmed leader, the request redirects or
returns a retryable 503.
A catch-up client recovering from an acknowledged append can request
consistency=leader. That opt-in routes the ordinary GET through the owning
Raft leader instead of treating a follower-local tail as authoritative. The
leader first confirms with a majority of voters that it still leads (Raft
ReadIndex) and waits until it has applied everything committed up to that
point, so the read is linearizable: it reflects every append acknowledged
before the read started, including appends acknowledged by a newer leader. A
leader cut off from the majority redirects to the new leader or answers 503
with Retry-After instead of serving its stale state. The confirmation costs
one round trip to the voters; concurrent leader reads to the same Raft group
share rounds, and a read that arrives while a round is in flight waits for the
next one. On the leader, a read waits for its confirmation before it reaches
the group, so appends and other reads to the group do not queue behind it. A
leader that misses a confirmation under load retries it for up to one election
timeout before it answers 503; a read that arrived during a round can wait
for that round and its own, so up to two. A client should retry a 503 after
Retry-After. It is not a live-read mode; long-poll and SSE already have
leader-owned watcher semantics. The Read path
section of the extensions specification states this contract normatively.
HEAD, GET …/bootstrap, snapshot reads and the registration of a long-poll
or SSE reader take the same leadership confirmation on the leader, as do
offset=now on a consistency=leader read.
A long-poll or SSE reader takes everything else it needs from that
registration: its offset=now position, whether the stream exists, its
content type and incarnation all come from the leader's state at or after the
confirmed point. A long-poll also answers 416 there for an offset beyond the
tail; an SSE session reports it as an error event from its first read. Its
first read is served only by that leader while it still leads; a node that
lost leadership in between redirects or answers 503 instead of asking a new
leader that may not have applied everything yet. A long-poll that times out
answers 204 at the requested offset.
offset=now without consistency=leader skips the confirmation and uses the
leader's applied state, so right after a leader change it may resolve to a
tail that lags the committed one.
With consistency=leader, Stream-Up-To-Date: true on a read means it reached the committed tail. On a local read it means the tail as the serving node knew it: a leader that has lost leadership without knowing it yet can mark a stale tail. A response without it is partial; keep reading from Stream-Next-Offset.
Availability and durability (standard layout)
Three voting replicas across availability zones, plus two non-voting replicas in a second region.
| Property | What it means |
|---|---|
| Write availability | Writes continue as long as a majority of voters is healthy. The layout tolerates any single voting-AZ failure. Non-voting replicas hold extra copies but do not vote. |
| Read availability | Any reachable replica can serve replicated historical catch-up data locally. Reads that need fresh tail metadata, leader-owned state changes, or live watcher ownership still track write availability per group, not per cluster. |
| Per-message durability | ~9-10 nines with a 5-second flush interval. |
| Annual zero-loss probability | ~3-4 nines. Probability of no data-loss events in a year. |