ursuladurable streams
v0.7.6 · apache-2.0

Exactly-Once Writes

Network retries can produce duplicate appends. Ursula provides server-side deduplication so your application doesn't need its own bookkeeping. Three headers form the identity:

  • Producer-Id: a stable client identifier (UUID, hostname, etc.).
  • Producer-Epoch: bumped on producer restart. Must be ≥ the last epoch the server saw from this producer.
  • Producer-Seq: a per-epoch sequence number. Starts at 0 for a new epoch and must increase by exactly 1 per append.

Both epoch and seq are capped at 2^53 − 1 so they survive a round-trip through JSON.

How dedup works

The server records an exact receipt for every accepted sequence in the current epoch from each Producer-Id per stream, within a per-stream receipt window of 1,024 items, and always keeps each producer's newest receipt. On an append:

  • Delayed duplicate (same epoch and any seq already accepted): silently deduplicated. The response carries that sequence's original byte and record ranges, even when newer sequences have committed, as long as its receipt is in the window. The newest sequence always gets its ranges. An older duplicate whose receipt left the window is still deduplicated (never written twice) and answered 204 with Producer-Seq but without range headers.
  • Next in sequence (seq = last_seq + 1): accepted.
  • Out of order (seq skips ahead): rejected with 409. The response carries Producer-Expected-Seq, the sequence the server wanted, and Producer-Received-Seq.
  • Epoch regression (epoch less than the last accepted one): rejected with 403 Forbidden. The response's Producer-Epoch is the current epoch. A producer that gets 403 has been replaced by a newer instance and should stop writing.

Restart and epoch hygiene

When a producer crashes and restarts, bump Producer-Epoch. Otherwise the server still expects the next contiguous seq within the old epoch, and the producer (which has lost its in-memory seq counter) will collide. A new epoch starts the seq counter back at 0.

Per-stream state

Dedup state is scoped per stream. The same Producer-Id writing to two different streams maintains two independent epoch/seq counters, so you do not have to elect a single global writer per producer.

The server retains exact receipts for the current epoch per (producer_id, stream) pair, bounded by the stream's receipt window. Older epochs and their receipts are dropped once a higher epoch is observed. A producer that has not written to a stream for 7 days is forgotten: its next append must start at sequence 0 (a later sequence gets 409), so bump the epoch or restart the sequence after long pauses. A stream holds at most 4,096 producers: a new producer beyond that replaces the least recently seen producer idle for at least an hour, and when none has been idle that long its append is rejected with 429 and a producer_limit body.

When you don't need this

For append-only workloads where each write is independent (event logging, audit trails), exactly-once headers are optional. Just POST without them. Add them when retries are real and double-applies would be visible.