ursuladurable streams
v0.7.6 · apache-2.0

Offsets

An offset is a position inside a stream. Clients use offsets to read from a specific point or resume where they left off.

Format

Offsets are opaque tokens. Ursula returns them as 20-character strings, for example "00000000000000000042", but their content carries no meaning for clients. Take each offset from a server response and pass it back unchanged. To tell which of two offsets of the same stream comes first, compare them lexicographically. Don't parse offsets or compute new ones from them.

Two special values are accepted on read requests:

  • offset=-1: start from the beginning of the stream. After a retention advance has trimmed the beginning, it answers 410 Gone like any trimmed offset; continue from that response's Stream-Next-Offset, which is the earliest offset still readable.
  • offset=now: start from the stream's current tail. Useful for "only new data" subscriptions.

Response headers

After every read or append, the server returns two related headers:

  • Stream-Next-Offset: the offset to pass back on the next request. Every successful read and append response carries it (on SSE, in each control event's streamNextOffset instead of a header), except an append answered as a duplicate whose receipt has left the receipt window (exactly-once writes). Error responses carry it only where the read and append status tables say so.
  • Stream-Cursor: set on live (long-poll, SSE) responses. A counter for caches, not a position: echo it as ?cursor= on the next live request. Each response's cursor is past the one the request sent, so successive long-poll URLs differ and a cache can't serve the same empty response in a loop.

Resume from Stream-Next-Offset (in SSE, the latest control event's streamNextOffset) in every read mode. Stream-Cursor carries no stream identity, so it doesn't reveal that a stream was recreated; see Recreated streams.

Stability across snapshots

Offsets are stable across snapshot publishes. Publishing a snapshot at an offset doesn't renumber later offsets or delete earlier data. A separate retention advance trims the data before its offset. A read at a trimmed offset returns 410 Gone, and its Stream-Next-Offset header is the earliest offset still readable, which HEAD also reports as Stream-Retained-Offset.

Recreated streams

An offset doesn't identify which stream it came from. If a stream is deleted and recreated at the same URL, an offset saved from the old stream may be accepted by the new one and return the new stream's data. HEAD returns a Stream-Incarnation token that changes when a stream is deleted and recreated. A client that keeps offsets across a possible recreate should store the incarnation with them and compare it for equality before resuming. A restore from backup keeps the incarnation but can rewind the stream to the backup's tail, so offsets read after that backup was taken are not valid on the restored stream.