ursuladurable streams
v0.7.6 · apache-2.0

Durable Streams Extensions

Document: Durable Streams Protocol Extensions Status: Draft Date: 2026-07-18

Abstract

This document defines extensions to the Durable Streams Protocol. It adds bucket namespacing, snapshot and bootstrap semantics, multipart bootstrap delivery, a JSON message text rule, and read path parameters.

The snapshot and bootstrap design in sections 2 and 3 is based on a proposal by Loro. Ursula implements part of it: there is no latest-snapshot redirect, no snapshot deletion and no conditional publish.

All base protocol semantics remain in effect. Where this document is silent, the base protocol governs.

Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174].

Table of Contents

JSON Message Text

This section defines the "normalized message" that the base protocol's JSON mode refers to. It is a storage rule, not an optional extension, and has no extension token. A server MUST apply it to every stream whose content type is application/json (with or without parameters).

It applies to every JSON write: the body of a creating PUT and the body of POST. It applies to every read representation: the default NDJSON view, SSE, and bootstrap parts (Section 3).

  1. The server MUST validate each request body as JSON text per [RFC8259]. \uXXXX escapes that encode unpaired surrogates are valid.
  2. After top-level array flattening, the server MUST store each message as the original text with insignificant whitespace removed, followed by one LF (U+000A). Insignificant whitespace is U+0020, U+0009, U+000A and U+000D outside string literals. The server MUST NOT reorder object members, drop or merge duplicate member names, or rewrite number or string literal text.
  3. The nesting depth of a scalar is 0. The depth of an array or object is 1 plus the maximum depth of its elements or member values; an empty array or object has depth 1. A message whose depth exceeds 127 MUST be rejected with 400 Bad Request. Depth is measured per message, after flattening, so a top-level array body may be one level deeper than its deepest message.
  4. A body with invalid JSON or invalid UTF-8 MUST be rejected with 400 Bad Request, and no message from that request is committed.
  5. Every read representation MUST carry the stored message text unchanged. Where a representation embeds the message in a larger JSON document, the embedded text MUST be the stored text.
  6. Offsets and ETags refer to the stored text.

Messages stored before a server adopted this rule are not rewritten. Clients MUST NOT assume that numbers survive storage as any particular binary floating-point value: a literal such as 1.50e3 or 1e400 is stored and returned as written.

Compatibility note (informative). Earlier Ursula releases re-serialized JSON messages: they sorted object members, kept only one of several duplicate members, rewrote numbers through a binary double, and rejected lone-surrogate escapes and numbers outside the double range. This rule removes these behaviors for new writes. The depth limit is unchanged for a bare message; a message inside an array body gains one level.

A JSON message boundary is an offset whose preceding stored byte is LF, plus the stream's first offset. Stored messages contain no other LF, so the boundaries are exactly the ends of the stored messages.

Client Event Time

The server does not assign a timestamp to messages. Applications that need event time SHOULD include one client-supplied timestamp in each JSON value, for example a captured_at field in a telemetry event.

Client event times MAY be out of order and MUST NOT change committed message order, stored bytes, or offsets. Their HTTP metadata representation and any secondary index over them are outside this document; Ursula's experimental event-time indexer is one such index.

1. Buckets

Every stream belongs to a bucket. The stream URL takes the form {base_url}/{bucket_id}/{stream_id}.

1.1. Identifier Constraints

  • bucket_id: MUST match ^[a-z0-9_-]{4,64}$. Globally unique within the service.
  • stream_id: Any UTF-8 string. MUST NOT exceed 122 bytes. MUST NOT contain /, \0, or ... On the bucketed / surface, the combined {bucket_id}/{stream_id} key MUST also not exceed 122 bytes.
  • The literal local stream ID streams is reserved, and a stream_id MUST NOT start with $, which is reserved for bucket-level subresources.

1.2. Bucket Lifecycle

In Ursula, buckets are implicit namespaces. A bucket exists once a stream has been created in it; PUT /{bucket_id}/{stream_id} needs no earlier bucket create. PUT /{bucket_id} is kept for clients that create the bucket first: it validates bucket_id, changes no state, and answers 201 Created for a valid ID or 400 Bad Request for an invalid one. The client surface has no bucket metadata read, stream listing or bucket delete. Removing a bucket with everything in it is an operator action, the admin-plane bucket purge, which permanently fences the bucket ID: later stream creates in it answer 404. It is not a protocol operation.

This section describes Ursula's server profile. Earlier drafts of this document required explicit bucket creation and defined bucket metadata, stream listing and bucket deletion in Sections 1.3 to 1.5. Those section numbers stay unused so that later section numbers do not change.

1.6. Stream Operations

All stream operations defined in the base protocol apply under {base_url}/{bucket_id}/{stream_id}. A path segment after {stream_id} names a stream subresource (Sections 2 and 3), never another stream. Ursula answers 404 Not Found for any other path with three or more segments under a bucket (paths under /__ursula/ are operational endpoints, not buckets).

The names $transaction, append-batch, attrs, bootstrap, retention and snapshot are reserved for any future co-location form and stream subresource; apart from the $ rule in Section 1.1 they remain valid stream IDs. Earlier drafts defined a three-segment path-affinity form and a group-local $transaction endpoint in Sections 1.7 and 1.8; those features are removed and the section numbers stay unused.

2. Snapshots

A snapshot is a materialized representation of a stream's content from offset -1 (inclusive) to snapshot_offset (exclusive). Snapshots enable clients to skip full replay and resume from a compacted state.

2.1. Offset Conventions

  • Offsets are opaque tokens as defined in the base protocol. Clients MUST use server-returned Stream-Next-Offset values.

The messages of a JSON stream are its LF-terminated messages (JSON Message Text); servers MUST use the same message boundaries for snapshot and retention offset validation and for bootstrap parts. A non-JSON stream has no message boundaries that this extension exposes: any committed offset is a valid snapshot or retention offset, and a bootstrap part MAY contain several consecutive messages. This extension does not require any particular binary framing format.

2.2. Publish Snapshot

PUT {stream_url}/snapshot/{snapshot_offset}

Creates a snapshot at the specified offset. The snapshot represents the materialized result of folding all updates in the range [-1, snapshot_offset).

Request Headers:

  • Content-Type: The content type of the snapshot blob. Servers MUST store this value and return it on subsequent reads. Defaults to application/octet-stream.

Response Codes:

  • 204 No Content: Snapshot published successfully.
  • 400 Bad Request: snapshot_offset is invalid or, for a JSON stream, not a message boundary.
  • 404 Not Found: Stream does not exist.
  • 409 Conflict: snapshot_offset exceeds the current tail, or the server cannot produce a consistent view at that offset.
  • 410 Gone: snapshot_offset is older than the current earliest retained offset.
  • 413 Payload Too Large: Snapshot exceeds the server's size limit.
  • 503 Service Unavailable: For a JSON stream, the server could not verify that snapshot_offset is a message boundary. It MUST NOT publish an unverified offset; the client MAY retry.

Note (informative). Ursula keeps snapshot bodies of up to 32 MiB in replicated state. With a cold backend, it stores bodies of at least its external-payload threshold (1 MiB by default) as cold-tier objects instead and accepts up to 1 GiB; only the reference is replicated, and the digest is the same as for an inline body.

Idempotency:

The server MUST return Stream-Snapshot-Digest for a successful publish. Repeating a publish at the same offset with the same content type and body MUST succeed and return the same digest. Publishing different content at an already-published offset MUST return 409 Conflict.

Retention Effect:

Publishing a snapshot MUST NOT change the earliest retained offset. Retention is a separate, explicit operation:

PUT {stream_url}/retention/{retained_offset}

Like every request that addresses a stream, a retention request MAY carry the Stream-Incarnation precondition (Section 2.4); its success response carries Stream-Incarnation. The server MUST reject a retention boundary that is not aligned to committed data (for a JSON stream, not a message boundary: 400 Bad Request), moves backward, or is newer than the latest published snapshot. As for a snapshot, a JSON retention boundary the server cannot verify MUST NOT be applied (503 Service Unavailable). After success, offsets less than retained_offset MUST return 410 Gone. A server MAY retain less than requested, at a committed boundary at or below retained_offset; the success response's Stream-Retained-Offset then reports the effective boundary, and offsets below it MUST return 410 Gone. Ursula retains exactly the requested offset.

Note (informative). Ursula checks a JSON boundary at apply time when the byte before the offset is still in memory. Otherwise it reads that byte from the leader (as a consistency=leader read), and applies the operation only if the stream was not deleted and recreated in between; a failed read answers 503.

Publishing a new snapshot replaces the previously visible snapshot immediately. The superseded snapshot MAY be garbage-collected asynchronously after the new snapshot becomes visible. Clients MUST NOT depend on older snapshots remaining readable after overwrite.

Concurrency:

Snapshot creation MUST NOT block concurrent appends. The snapshot's consistency boundary is snapshot_offset. Updates at or beyond that offset MUST NOT be folded into the snapshot.

2.4. Snapshot Metadata on HEAD and Stream Incarnation

HEAD {stream_url}

Servers MUST include Stream-Snapshot-Offset and Stream-Snapshot-Digest when a latest visible snapshot exists for the stream. Apart from bootstrap (Section 3), HEAD is the only way to learn which snapshot is the latest.

Response Headers:

  • Stream-Snapshot-Offset: The latest visible snapshot offset.
  • Stream-Snapshot-Digest: A digest of the latest visible snapshot's content type and body, equal to the digest its publish returned.
  • Stream-Retained-Offset: The earliest retained offset (Section 2.2, Retention Effect).
  • Stream-Incarnation: An opaque token that identifies this incarnation of the stream.

Semantics:

  • Absence of Stream-Snapshot-Offset means the stream has no visible snapshot.
  • To read the latest snapshot, a client takes Stream-Snapshot-Offset from HEAD and fetches it with Read Snapshot (Section 2.5).
  • A server implementing this extension MUST send Stream-Retained-Offset on every successful HEAD. Reads at offsets below it MUST return 410 Gone. Before any retention advance it is the offset of the stream's first byte. Like every offset, it is opaque: clients pass it back as offset or compare it lexicographically with other offsets of the same stream.
  • A server implementing this extension MUST send Stream-Incarnation on every successful HEAD. It MUST change when the stream is deleted and recreated at the same URL, and MUST NOT change otherwise. Clients MUST compare it only for equality. Offsets do not identify an incarnation, so a client that stores offsets and snapshot offsets across a possible recreate SHOULD store the incarnation with them and send it as a request precondition (below). An operator restore from backup keeps each stream's incarnation although it can rewind the stream to the backup's tail; offsets read after the backup was taken are not valid on the restored stream.

Stream-Incarnation on responses:

A server implementing this extension MUST send Stream-Incarnation on every successful response of a request that addresses a stream: create (201 and 200), append and close, every read (catch-up, the long-poll 204, and the SSE response), HEAD, bootstrap, snapshot publish and read, and retention. A client learns the token from its first request, without a separate HEAD.

Stream-Incarnation as a request precondition:

Requests address streams by URL only, so a client holding state from a deleted incarnation (offsets, producer state, a local cache) would otherwise act on a stream recreated at the same URL, including one recreated by TTL expiry and a later create, without any delete. Every request that addresses a stream MAY carry Stream-Incarnation: create (PUT), append (POST, including close), every read (catch-up, long-poll, SSE), HEAD, DELETE, bootstrap, snapshot publish and read, and retention.

  • Absent: the request behaves as the base protocol and this extension define it, unchanged.
  • Present: the server compares the value for equality with the stream's current incarnation, atomically with the operation. A write is compared when it is applied: a write that was accepted before a delete and recreate but applied after it is refused. A read is compared against the same stream state that serves it. A live read (long-poll or SSE) ends when the stream it was opened against is deleted or recreated: a long-poll answers 404 or 412, and an SSE stream ends with an error event.
  • Mismatch: 412 Precondition Failed. A refused write applies nothing; a refused read returns no data, though, like any request that reaches a stream, it may still renew that stream's sliding TTL. The response carries the current Stream-Incarnation when the stream exists. 412, not 409, so it never mixes with Stream-Seq and producer conflicts.
  • Evaluation follows RFC 9110 §13.2: a request that would answer 404 without the header still answers 404, except that a PUT with the header for a stream that does not exist answers 412 and creates nothing. Likewise a read that would answer 410 or 416 without the header still does; only a request that would otherwise succeed (or a PUT that would create) is answered 412.
  • Malformed: a value that is not a token the server could have issued, or more than one value, answers 400 Bad Request.

2.5. Read Snapshot

GET {stream_url}/snapshot/{snapshot_offset}

Returns the snapshot blob at the specified offset.

Response Headers:

  • Content-Type: The content type stored at publish time.
  • Stream-Snapshot-Offset: The snapshot offset.
  • Stream-Snapshot-Digest: The snapshot digest returned at publish time.
  • Stream-Next-Offset: The next offset after the snapshot.
  • Stream-Up-To-Date: true: Present when no update follows the snapshot (the snapshot offset is the stream's tail); absent otherwise. Clients then read on from Stream-Next-Offset.

Response Codes:

  • 200 OK: Snapshot exists.
  • 404 Not Found: Snapshot does not exist or has been garbage-collected.

3. Bootstrap

Bootstrap provides single-request initialization: a snapshot (if any) plus retained updates after the snapshot point, returned as a single ordered response that preserves per-message content types. A server returns either every retained update after the snapshot point, or a contiguous prefix of them and an honest continuation offset from which the client reads the rest through ordinary reads (Section 3.3).

3.1. Request

GET {stream_url}/bootstrap

Query Parameters:

  • /bootstrap is a one-shot initialization endpoint. It does not define any query parameters of its own.
  • Servers SHOULD reject any live query parameter on /bootstrap with 400 Bad Request.

3.2. Response

Response Headers:

  • Content-Type: multipart/mixed; boundary=<token>
  • Stream-Snapshot-Offset: The snapshot offset, or -1 if no snapshot exists.
  • Stream-Next-Offset: The offset immediately after the last returned update byte. When the response contains no update parts, it is the snapshot point (defined below).
  • Stream-Up-To-Date: true: Present when Stream-Next-Offset equals the stream's tail offset at the time the response was produced. Otherwise the server MUST either omit the header, as for ordinary reads in the base protocol, or send false. Clients MUST treat an absent header as false.

The snapshot point S is the visible snapshot's offset when a snapshot exists, and otherwise the earliest retained offset.

Completeness:

A server MUST return one of the following:

  • Complete: update parts holding every retained byte in [S, tail), with Stream-Next-Offset equal to the tail and Stream-Up-To-Date: true.
  • Partial: update parts holding the retained bytes in [S, P) for some offset P with S ≤ P < tail, with Stream-Next-Offset: P and no Stream-Up-To-Date: true. P MAY equal S, in which case the response contains the snapshot part and no update parts.

In both cases the update parts MUST hold the retained bytes starting exactly at S, contiguous and in stream order: a server MUST NOT skip or duplicate any byte between S and Stream-Next-Offset. For a JSON stream each part MUST be exactly one message. For a non-JSON stream a part MAY contain several consecutive messages, and a part boundary carries no meaning. Stream-Next-Offset MUST be an offset that ordinary reads accept, and MUST NOT be below S.

A server MAY choose the partial form for any reason, for example because some messages after S are held only in a storage tier that does not keep per-message boundaries in memory. Serving /bootstrap never requires a server to read such a tier. A client MUST NOT treat a partial response as an error.

Note (informative). Ursula never reads cold storage for update parts while serving /bootstrap (a snapshot body stored in the cold tier is streamed from it as the first part), and it omits Stream-Up-To-Date from a partial response. For a non-JSON stream it returns [S, tail) as one part when S is at or above the stream's cold frontier, every byte of that range is still hot and the range is at most 8 MiB; otherwise it returns the partial form with P = S (the snapshot part only). For a JSON stream, under the same hot-range conditions, it splits [S, tail) on LF into one part per message, and otherwise returns the partial form with P = S; it caps the update parts of one response at 8 MiB, returning the partial form with P at the last message boundary that keeps the parts within the cap, and always includes at least one message, even when that message alone exceeds the cap. Each response uses a fresh 128-bit random multipart boundary.

Response Body:

The body is an RFC 2046 multipart/mixed entity. The multipart boundary, not any outer binary framing, delimits the parts. Servers SHOULD choose an unpredictable boundary for every response, so that no stream content can contain it.

  1. First part: Snapshot message. If a snapshot exists, the part body MUST be the raw snapshot bytes and the part Content-Type MUST equal the snapshot blob's stored content type.
  2. Subsequent parts: Zero or more parts of retained updates starting at the snapshot point, as required by the completeness rules above: one message per part for a JSON stream; one or more consecutive messages per part for any other stream. The part Content-Type MUST be the content type of the update. For streams with a fixed stream-level content type, all update parts will normally share that value.

If no snapshot exists, the server MUST still emit an empty first part, Stream-Snapshot-Offset MUST be -1, and that empty part's Content-Type MUST be application/octet-stream. Update parts, if any, then begin at the earliest retained offset.

Servers MUST NOT wrap the entire bootstrap response in an outer binary framing container, and they MUST NOT reinterpret or reframe binary payloads when constructing bootstrap parts.

Example:

HTTP/1.1 200 OK
Content-Type: multipart/mixed; boundary=rr-bootstrap-9f1c2e7a
Stream-Snapshot-Offset: 0000000000000012
Stream-Next-Offset: 0000000000000026
Stream-Up-To-Date: true
Cache-Control: no-store

--rr-bootstrap-9f1c2e7a
Content-Type: application/octet-stream

<snapshot-bytes>
--rr-bootstrap-9f1c2e7a
Content-Type: application/json

{"op":"set","path":["title"],"value":"hello"}
--rr-bootstrap-9f1c2e7a
Content-Type: application/json

{"op":"insert","path":["body",0],"value":"world"}
--rr-bootstrap-9f1c2e7a--

A partial response that returns only the snapshot. The client continues with GET {stream_url}?offset=0000000000000012:

HTTP/1.1 200 OK
Content-Type: multipart/mixed; boundary=rr-bootstrap-4b7d01c3
Stream-Snapshot-Offset: 0000000000000012
Stream-Next-Offset: 0000000000000012
Cache-Control: no-store

--rr-bootstrap-4b7d01c3
Content-Type: application/octet-stream

<snapshot-bytes>
--rr-bootstrap-4b7d01c3--

Response Codes:

  • 200 OK: Bootstrap data returned.
  • 404 Not Found: Stream does not exist.

3.3. Follow-Up Live Reads

Bootstrap is one-shot only:

  • /bootstrap returns an ordered initialization payload once.
  • /bootstrap does not support live=sse or live=long-poll.
  • After applying the bootstrap response, clients MUST continue tailing through the ordinary stream read APIs using the returned Stream-Next-Offset.
  • When the response does not carry Stream-Up-To-Date: true, the ordinary catch-up reads from Stream-Next-Offset return the remaining updates; clients MUST apply them before treating their state as current.
  • Clients SHOULD prefer GET {stream_url}?offset=<offset>&live=sse and fall back to GET {stream_url}?offset=<offset>&live=long-poll when SSE is unavailable.

3.4. Compatibility

  • Regular GET {stream_url}?offset=... MUST NOT return snapshot bytes. Snapshots are only delivered through /bootstrap.
  • When a server returns 410 Gone for a read (because retention has advanced), clients SHOULD call /bootstrap to rebuild state, then continue tailing from the returned Stream-Next-Offset through the ordinary read APIs.
  • If a server has begun returning 410 Gone for a stream (i.e., earliest retained offset > -1), the server MUST ensure /bootstrap is available and can provide a snapshot covering that earliest retained offset.
  • Clients MUST parse /bootstrap as an ordered sequence of parts. For a JSON stream each update part is one message; for any other stream, clients MUST NOT treat a part boundary as a message boundary.

7. Ordinary SSE Compatibility

Ordinary SSE behavior is defined by the base protocol. Binary SSE: see Section 9.5.

  • Extension endpoints MUST NOT redefine the payload shape of ordinary binary SSE event: data frames unless they document an explicit endpoint-local contract.

8. URL Encoding

snapshot_offset values originate from Stream-Next-Offset and MAY contain characters requiring URL encoding. Clients and servers MUST apply standard URL encoding/decoding when using offset values in path segments.

9. Read Path

This section collects read parameters and read representations that apply to every stream and that the base protocol leaves to the server.

9.1. Long-Poll Timeout

timeout_ms={t} bounds how long a live=long-poll read waits for new data before the server answers 204 No Content. The default is 1000. An unparseable value means the default, and a parsed value is clamped to 1..60000. It applies only to live=long-poll reads; every other read, including live=sse, ignores it.

9.2. Leader Consistency

consistency={c} selects how fresh a catch-up read must be.

  • local, the default: any replica MAY serve bytes it has applied. Offsets are immutable, so those bytes are correct, but a replica that lags MAY omit appends that were already acknowledged.
  • leader: the read MUST be linearizable. It MUST reflect every append acknowledged before the request was received, including appends acknowledged by a newer leader.
  • Any other value MUST be rejected with 400 Bad Request. consistency=leader combined with live MUST be rejected with 400 Bad Request; live reads follow the leader's watchers already.

A server that cannot prove the read is linearizable MUST NOT serve it. It answers 307 Temporary Redirect to the current leader when it knows one, and otherwise 503 Service Unavailable with Retry-After. Clients SHOULD follow the redirect and retry the 503 after the advertised delay.

Note (informative). Ursula forwards a consistency=leader read from a follower to the group's leader. The leader confirms with a majority of voters that it still leads (Raft ReadIndex) and waits until it has applied everything committed up to that point. Concurrent reads to the same Raft group share confirmation rounds. A leader cut off from the majority, or one that cannot confirm in time, answers 307 or 503 rather than its own state. Ursula applies the same confirmation to HEAD, /bootstrap, snapshot reads and the registration of a long-poll or SSE reader.

9.3. Response Size

A server MAY cap how many stream bytes one response returns. Ursula caps every read response, and every SSE data event, at 8 MiB (8,388,608 bytes).

max_bytes={n} on an offset read asks for at most n bytes. A value above the server's cap is clamped to it. Ursula ignores a value that is not a decimal integer and applies the cap alone; clients SHOULD send a positive integer. On a live read, max_bytes bounds each response or SSE data event; Ursula raises a smaller value to 1 byte, and to 4 bytes on a text SSE read so an event can hold a whole UTF-8 character.

A response that max_bytes or the cap shortened is partial: it MUST NOT carry Stream-Up-To-Date: true, and its Stream-Next-Offset is the offset just past the returned bytes. Clients continue from Stream-Next-Offset until a response carries Stream-Up-To-Date: true.

9.4. JSON Stream Reads

Reads of an application/json stream return the stored messages (see JSON Message Text) in their stored order, each followed by one LF, as Content-Type: application/x-ndjson. HEAD reports the configured content type, application/json. SSE responses for such a stream carry Stream-Data-Content-Type: application/x-ndjson.

max_bytes and the server's cap (Section 9.3) count bytes of this representation, so a response or SSE data event MAY end inside a message. Clients MUST resume from Stream-Next-Offset and buffer an incomplete trailing line until the rest arrives. SSE data: lines are transport framing, not message boundaries.

9.5. Binary SSE

A live=sse read of a stream whose content type is neither text/* nor application/json carries binary data:

  • The response MUST include Stream-Sse-Data-Encoding: base64, and Stream-Data-Content-Type MUST be the stream's content type.
  • Each event: data payload MUST be the standard base64 encoding [RFC4648] of a contiguous byte range of the stream, with no JSON envelope. The server MAY split the text across several data: lines; the client joins them, removes the line breaks and decodes.
  • Each data event is followed by an event: control event whose streamNextOffset is the offset just past the decoded bytes.

There is no client opt-in or opt-out: the stream's content type decides the encoding.