Durable Streams Extensions

Document: Durable Streams Protocol Extensions Status: Draft Date: 2026-07-18

Abstract

This document defines extensions to the Durable Streams Protocol. It adds bucket namespacing, snapshot and bootstrap semantics, multipart bootstrap delivery, mutable stream attributes, Ursula-specific append batching for high-throughput small-event writes, and stable record coordinates for JSON streams.

The snapshot and bootstrap design in sections 2 and 3 was proposed by Loro; Ursula adopts that design.

All base protocol semantics remain in effect. Where this document is silent, the base protocol governs.

Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174].

Extension Capability Advertisement

Buckets
Snapshots
Bootstrap
Append Batch
Stream Attributes
JSON Record Coordinates
Ordinary SSE Compatibility
URL Encoding

Extension Capability Advertisement

Stream-Extensions is a response header for advertising protocol extensions that are active for the target stream. Its value is a comma-separated list of extension tokens registered by the specifications that define them. A server MUST advertise only tokens whose extension semantics apply to the target stream; server-wide support for an extension is not sufficient. An extension specification defines its token and the responses on which an active implementation MUST advertise it.

Clients MUST ignore unrecognized extension tokens. A client that depends on an extension MUST verify its token before relying on extension-specific request parameters, response headers, or body semantics.

1. Buckets

Every stream belongs to a bucket. The stream URL takes the form {base_url}/{bucket_id}/{stream_id}.

1.1. Identifier Constraints

bucket_id: MUST match ^[a-z0-9_-]{4,64}$. Globally unique within the service.
stream_id: Any UTF-8 string. MUST NOT exceed 122 bytes. MUST NOT contain /, \0, or ... On the bucketed / surface, the combined {bucket_id}/{stream_id} key MUST also not exceed 122 bytes.
The literal local stream ID streams is reserved for the bucket listing endpoint /{bucket_id}/streams.

1.2. Create Bucket

PUT /{bucket_id}

Buckets MUST be created explicitly. PUT /{bucket_id}/{stream_id} MUST NOT implicitly create a bucket.

Response Codes:

201 Created: Bucket created successfully.
400 Bad Request: bucket_id is invalid.
409 Conflict: Bucket already exists.

1.3. Get Bucket Metadata

GET /{bucket_id}

Response Codes:

200 OK: Bucket exists. Returns a JSON object with at least bucket_id (string) and streams (integer, count of streams in the bucket).
400 Bad Request: bucket_id is invalid.
404 Not Found: Bucket does not exist.

1.4. List Bucket Streams

GET /{bucket_id}/streams?prefix={prefix}&after={cursor}&limit={n}

prefix filters bucket-local stream IDs by prefix. When omitted, all bucket-local stream IDs are eligible.
after is an exclusive cursor over bucket-local stream IDs.
limit defaults to 1000 and must be in 1..=1000.

The response body is:

{
  "bucket_id": "agents",
  "prefix": "user-",
  "stream_count": 2,
  "streams": [
    {
      "stream_id": "user-1",
      "status": "Open",
      "content_type": "text/plain",
      "tail_offset": 42,
      "created_at_ms": 1735689600000,
      "last_write_at_ms": 1735689601000
    }
  ],
  "next_cursor": "user-2",
  "has_more": true
}

streams is sorted lexicographically by stream_id.

Response Codes:

200 OK: Matching streams returned.
400 Bad Request: bucket_id or query parameters are invalid.
404 Not Found: Bucket does not exist.

1.5. Delete Bucket

DELETE /{bucket_id}

A bucket MUST be empty (zero streams) before deletion.

Response Codes:

204 No Content: Bucket deleted.
400 Bad Request: bucket_id is invalid.
404 Not Found: Bucket does not exist.
409 Conflict: Bucket is not empty.

1.6. Stream Operations

All stream operations defined in the base protocol apply under {base_url}/{bucket_id}/{stream_id}. When the bucket does not exist, the server MUST return 404 Not Found.

2. Snapshots

A snapshot is a materialized representation of a stream's content from offset -1 (inclusive) to snapshot_offset (exclusive). Snapshots enable clients to skip full replay and resume from a compacted state.

2.1. Offset Conventions

Offsets are opaque tokens as defined in the base protocol. Clients MUST use server-returned Stream-Next-Offset values.

Servers that expose snapshots and /bootstrap as message sequences MUST use one consistent retained-message boundary model across ordinary reads, snapshot-offset validation, and bootstrap responses. This extension does not require any particular binary framing format.

2.2. Publish Snapshot

PUT {stream_url}/snapshot/{snapshot_offset}

Creates a snapshot at the specified offset. The snapshot represents the materialized result of folding all updates in the range [-1, snapshot_offset).

Request Headers:

Content-Type: The content type of the snapshot blob. Servers MUST store this value and return it on subsequent reads. Defaults to application/octet-stream.
Stream-Snapshot-Match: Optional digest of the currently visible snapshot. If supplied and it does not match, the server MUST reject the publish with 409 Conflict.

Response Codes:

204 No Content: Snapshot published successfully.
400 Bad Request: snapshot_offset is invalid or not aligned to a committed message boundary.
404 Not Found: Stream does not exist.
409 Conflict: snapshot_offset exceeds the current tail, or the server cannot produce a consistent view at that offset.
410 Gone: snapshot_offset is older than the current earliest retained offset.
413 Payload Too Large: Snapshot exceeds the server's size limit.

Idempotency:

The server MUST return Stream-Snapshot-Digest for a successful publish. Repeating a publish at the same offset with the same content type and body MUST succeed and return the same digest. Publishing different content at an already-published offset MUST return 409 Conflict.

Retention Effect:

Publishing a snapshot MUST NOT change the earliest retained offset. Retention is a separate, explicit operation:

PUT {stream_url}/retention/{retained_offset}

The server MUST reject a retention boundary that is not aligned to committed data, moves backward, or is newer than the latest published snapshot. After success, offsets less than retained_offset MUST return 410 Gone.

For record-aware JSON streams, servers also expose:

PUT {stream_url}/snapshot?record={record}
PUT {stream_url}/retention?record={record}

The server resolves the stable record ordinal to the canonical byte boundary before applying the operation.

Publishing a new snapshot replaces the previously visible snapshot immediately. The superseded snapshot MAY be garbage-collected asynchronously after the new snapshot becomes visible. Clients MUST NOT depend on older snapshots remaining readable after overwrite.

Concurrency:

Snapshot creation MUST NOT block concurrent appends. The snapshot's consistency boundary is snapshot_offset. Updates at or beyond that offset MUST NOT be folded into the snapshot.

2.3. Read Latest Snapshot

GET {stream_url}/snapshot

Returns a redirect to the latest visible snapshot resource, if one exists.

Response:

307 Temporary Redirect when a latest snapshot exists. The Location header points to {stream_url}/snapshot/{snapshot_offset}.
404 Not Found when no snapshot exists.

2.4. Snapshot Metadata on HEAD

HEAD {stream_url}

Servers SHOULD include Stream-Snapshot-Offset when a latest visible snapshot exists for the stream.

Response Headers:

Stream-Snapshot-Offset: The latest visible snapshot offset.
Stream-Snapshot-Digest: A digest of the latest visible snapshot's content type and body.

Semantics:

Absence of Stream-Snapshot-Offset means the stream has no visible snapshot.
When present, the value MUST refer to the same latest visible snapshot that GET {stream_url}/snapshot would resolve to.

2.5. Read Snapshot

GET {stream_url}/snapshot/{snapshot_offset}

Returns the snapshot blob at the specified offset.

Response Headers:

Content-Type: The content type stored at publish time.
Stream-Snapshot-Offset: The snapshot offset.
Stream-Snapshot-Digest: The snapshot digest returned at publish time.
Stream-Next-Offset: The next offset after the snapshot.
Stream-Up-To-Date: Boolean indicating whether the stream has updates beyond the snapshot.

Response Codes:

200 OK: Snapshot exists.
404 Not Found: Snapshot does not exist or has been garbage-collected.

2.6. Delete Snapshot

DELETE {stream_url}/snapshot/{snapshot_offset}

Deletes the snapshot identified by snapshot_offset, subject to bootstrap safety rules.

Response Codes:

404 Not Found: Snapshot does not exist or has been superseded.
409 Conflict: snapshot_offset refers to the latest visible snapshot and cannot be deleted because /bootstrap would become incomplete.

Implementations that expose only one visible snapshot MAY make superseded snapshots unreachable immediately after overwrite. In that case, deleting an older offset will return 404 Not Found.

3. Bootstrap

Bootstrap provides single-request initialization: a snapshot (if any) plus all retained updates after the snapshot point, returned as a single ordered response that preserves per-message content types.

3.1. Request

GET {stream_url}/bootstrap

Query Parameters:

/bootstrap is a one-shot initialization endpoint. It does not define any query parameters of its own.
Servers SHOULD reject any live query parameter on /bootstrap with 400 Bad Request.

3.2. Response

Response Headers:

Content-Type: multipart/mixed; boundary=<token>
Stream-Snapshot-Offset: The snapshot offset, or -1 if no snapshot exists.
Stream-Next-Offset: The next offset after all returned data.
Stream-Up-To-Date: Boolean.

Response Body:

The body is an RFC 2046 multipart/mixed entity. Each MIME part is one logical bootstrap message. The multipart boundary, not any outer binary framing, defines the bootstrap message boundaries.

First part: Snapshot message. If a snapshot exists, the part body MUST be the raw snapshot bytes and the part Content-Type MUST equal the snapshot blob's stored content type.
Subsequent parts: Retained updates after the snapshot point, one update message per part. Each part body MUST be exactly one retained update message, and the part Content-Type MUST be the content type of that update. For streams with a fixed stream-level content type, all update parts will normally share that value.

If no snapshot exists, the server MUST still emit an empty first part, Stream-Snapshot-Offset MUST be -1, and that empty part's Content-Type MUST be application/octet-stream. Update parts then begin at the earliest retained offset.

Servers MUST NOT wrap the entire bootstrap response in an outer binary framing container. Each update part contains exactly one retained update message from the stream. Implementations MUST preserve the same retained-message boundaries they use for ordinary reads and snapshot-offset validation, and they MUST NOT reinterpret or reframe binary payloads when constructing bootstrap parts.

Example:

HTTP/1.1 200 OK
Content-Type: multipart/mixed; boundary=rr-bootstrap-9f1c2e7a
Stream-Snapshot-Offset: 0000000000000012
Stream-Next-Offset: 0000000000000026
Stream-Up-To-Date: true
Cache-Control: no-store

--rr-bootstrap-9f1c2e7a
Content-Type: application/octet-stream

<snapshot-bytes>
--rr-bootstrap-9f1c2e7a
Content-Type: application/json

{"op":"set","path":["title"],"value":"hello"}
--rr-bootstrap-9f1c2e7a
Content-Type: application/json

{"op":"insert","path":["body",0],"value":"world"}
--rr-bootstrap-9f1c2e7a--

Response Codes:

200 OK: Bootstrap data returned.
404 Not Found: Stream does not exist.

3.3. Follow-Up Live Reads

Bootstrap is one-shot only:

/bootstrap returns an ordered initialization payload once.
/bootstrap does not support live=sse or live=long-poll.
After applying the bootstrap response, clients MUST continue tailing through the ordinary stream read APIs using the returned Stream-Next-Offset.
Clients SHOULD prefer GET {stream_url}?offset=<offset>&live=sse and fall back to GET {stream_url}?offset=<offset>&live=long-poll when SSE is unavailable.

3.4. Compatibility

Regular GET {stream_url}?offset=... MUST NOT return snapshot bytes. Snapshots are only delivered through /bootstrap.
When a server returns 410 Gone for a read (because retention has advanced), clients SHOULD call /bootstrap to rebuild state, then continue tailing from the returned Stream-Next-Offset through the ordinary read APIs.
If a server has begun returning 410 Gone for a stream (i.e., earliest retained offset > -1), the server MUST ensure /bootstrap is available and can provide a snapshot covering that earliest retained offset.
Clients MUST parse /bootstrap as an ordered message sequence. Message boundaries come from MIME parts.

4. Append Batch

Append Batch is an Ursula-specific implementation extension. It is not part of the base Durable Streams Protocol and clients MUST NOT assume it exists on other Durable Streams servers.

The extension reduces HTTP request overhead for workloads that append many small independent records. It is a transport batching mechanism only: each frame in the request is committed as an independent logical append with its own acknowledgement. Servers MUST NOT concatenate the frames into one stream record.

4.1. Request

Bucketed Ursula route:

POST /{bucket_id}/{stream_id}/append-batch

The request Content-Type applies to every frame and MUST match the stream's configured content type.

The request body is a sequence of binary frames:

uint32_be payload_len
payload bytes
uint32_be payload_len
payload bytes
...

payload_len is a 32-bit unsigned integer encoded in network byte order. It is the number of payload bytes that immediately follow. Zero-length frames are invalid.

Ursula currently accepts at most 512 frames per public append-batch request and a 32 MiB total body. Servers MAY reject larger batches with 413 Payload Too Large.

4.2. Unsupported Per-Append Controls

The current Ursula append-batch endpoint does not support controls that require per-frame headers:

Stream-Closed
Producer-Id, Producer-Epoch, Producer-Seq
Stream-Record-Match
Stream-Seq

Clients that need producer fencing, idempotent producer sequence checks, stream preconditions, or atomic stream close MUST use the standard POST {stream_url} append operation.

4.3. Response

On a syntactically valid batch request, the response is 200 OK with a JSON array. The array order matches the request frame order. Each element acknowledges one logical append:

[
  { "status": 204, "stream_next_offset": 3 },
  { "status": 204, "stream_next_offset": 6 }
]

Acknowledgement fields:

status: HTTP status code for the frame-level logical append.
stream_next_offset: Present when the frame append succeeds or when an error can report the stream tail.
stream_closed: Present and true if the append observes a closed stream.
error_code: Present for failed frame-level appends.
error: Human-readable failure detail for failed frame-level appends.

Malformed framing, missing Content-Type, unsupported headers, or an oversized request MAY fail the entire HTTP request with 400 Bad Request or 413 Payload Too Large.

Frame-level failures are reported inside the JSON array. A successful outer 200 OK does not mean every frame committed. Clients MUST inspect every acknowledgement.

4.4. Ordering and Compatibility

Frames are submitted in request order and response acknowledgements preserve that order. For a single stream, successful frames advance the same stream offsets that standard POST {stream_url} would have produced.

All standard stream read APIs remain unchanged. Data appended through Append Batch is read through ordinary GET, long-poll, SSE, snapshot, and bootstrap paths using the same offsets and content type rules as standard appends.

5. Stream Attributes

Stream Attributes are an Ursula-specific metadata extension for mutable, application-owned stream attributes. They are separate from the base protocol's stream metadata returned by HEAD {stream_url}.

Attributes are not part of the append-only byte stream. Updating attributes MUST NOT change stream offsets, ETag, content type, closed state, snapshots, bootstrap payloads, or ordinary read results.

5.1. Attribute Object

The attribute document is a JSON object with the following optional fields:

{
  "title": "Support session",
  "metadata": {
    "purpose": "customer-support",
    "environment_id": "env_019e2590d33f711fabf42f2857cecd8a",
    "agent": {
      "id": "agent_019e390add9f7bac9b6cc806db46fcbd",
      "version": 2
    }
  }
}

Fields:

title: Optional string. Interpreted by the application.
metadata: Optional JSON object for arbitrary application metadata.

Ursula does not define first-class semantics for application concepts such as agents or environments. If an application wants to associate such information with a stream, it SHOULD store that information inside metadata.

Unknown top-level fields are ignored: servers MUST accept attribute objects containing unrecognized top-level fields and MUST NOT store them. Clients that need additional fields MUST place them inside metadata.

The encoded attribute object MUST NOT exceed 16 KiB. Servers MUST reject larger attribute documents with 400 Bad Request. Attributes are replicated with stream state, so the limit keeps log entries and snapshots small.

An empty attribute object ({}) is equivalent to no attributes.

5.2. Create Stream With Attributes

Attributes MAY be supplied when creating a stream:

PUT {stream_url}
Stream-Attrs: {"title":"Support session","metadata":{"purpose":"customer-support"}}

Stream-Attrs is an optional request header whose value is the JSON attribute object.

When a stream already exists, the create request is idempotent only if the existing stream attributes match the supplied attributes after empty-object normalization. A mismatch MUST return 409 Conflict, the same as other create-configuration mismatches.

5.3. Read Attributes

GET {stream_url}/attrs

Response:

200 OK: Returns the current attribute object as application/json.
404 Not Found: The stream does not exist or has expired.

When no attributes are set, the response body is {}.

5.4. Replace Attributes

PUT {stream_url}/attrs
Content-Type: application/json

{"title":"Renamed session","metadata":{"purpose":"debugging"}}

The request body replaces the complete attribute object. Partial merge semantics are not defined by this extension.

Response Codes:

204 No Content: Attributes were replaced, or the submitted object already matched the stored attributes.
400 Bad Request: The request is missing Content-Type, the content type is not application/json, the body is not a valid attribute JSON object, or the attribute object exceeds the size limit.
404 Not Found: The stream does not exist or has expired.

Submitting {} clears the stream attributes. Attributes MAY be updated after a stream is closed because stream closure only forbids further appends.

6. JSON Record Coordinates

The JSON Record Coordinates extension adds a stable logical message coordinate to streams configured with Content-Type: application/json. It is a pure superset of the base protocol: offsets remain the canonical exact-resume cursor, ordinary offset reads remain byte-oriented, and clients that do not understand record coordinates continue to operate unchanged.

This extension registers the token json-record-coordinates-v1. It is active only for an application/json stream for which the server maintains the record model defined below. Responses to HEAD and stream operations targeting such a stream MUST advertise this token in Stream-Extensions. A server MUST NOT advertise this token for a stream on which the extension is inactive, including a stream with a non-JSON content type.

Record-aware clients MUST verify this token before relying on record query parameters or response headers and MUST reject a record-aware response that omits it.

6.1. Record Model

Each logical JSON message has a zero-based unsigned integer record ordinal. Ordinals are assigned consecutively in committed stream order and are scoped to one stream.

For one stream:

F is the first retained record ordinal.
N is the next record ordinal and record tail.
Retained records occupy [F, N).
O(r) is the canonical Durable Streams offset at the first byte of record r.

A new empty stream has F = N = 0. An ordinal MUST NOT be reused or changed after it becomes externally visible. Retention MAY advance F, but MUST NOT renumber surviving records.

Record ordinals are independent from offsets, Stream-Seq, idempotent-producer sequence numbers, and sequence fields inside application JSON values. Clients MAY perform arithmetic on ordinals, but MUST continue to treat offsets as opaque tokens.

The JSON-mode normalization rules in the base protocol define the records:

A non-array JSON value creates one record.
A top-level JSON array creates one record per array element after exactly one level of flattening.
An empty array creates no records on PUT and remains invalid on POST.

An implementation MUST preserve each normalized message boundary and its ordinal across hot storage, WAL, cold flush, compaction, snapshot, restart, and retention. Coalescing physical storage objects MUST NOT coalesce the logical record coordinate space.

6.2. Client Event Time

This extension does not assign a server timestamp to records. Applications that need event time SHOULD include one client-supplied timestamp in each JSON value, for example a captured_at field in a telemetry event.

Client event times MAY be out of order and MUST NOT change committed record order, ordinals, canonical bytes, or offsets. Their HTTP metadata representation and any secondary index over them are outside this extension.

6.3. Create and Append Acknowledgements

When PUT creates a JSON stream with initial messages, or POST appends one or more JSON messages, successful responses MUST include:

Stream-Record-Start: Inclusive ordinal of the first record created by this operation.
Stream-Record-Next: Exclusive ordinal after the last record created by this operation.

For an operation that creates no records, Stream-Record-Start and Stream-Record-Next MUST both equal the current record tail.

All messages from one JSON request body MUST commit atomically and receive one contiguous ordinal range, or the operation MUST fail without making any ordinal visible. Concurrent appenders MUST receive disjoint ranges in committed stream order.

When an idempotent-producer retry is deduplicated, the server MUST return the original Stream-Record-Start and Stream-Record-Next values.

For example, if the current record tail is 42:

POST {stream_url}
Content-Type: application/json

[
  {"type":"navigation_started"},
  {"type":"network_response","status":500}
]

The successful response includes:

Stream-Extensions: json-record-coordinates-v1
Stream-Record-Start: 42
Stream-Record-Next: 44
Stream-Next-Offset: <opaque-offset>

6.4. Conditional Append by Record Tail

A writer MAY require the current record tail to match an expected value:

Stream-Record-Match: 42

The server MUST accept the append only when the current N equals the supplied unsigned integer. A mismatch MUST return 412 Precondition Failed with the current Stream-Record-Next and Stream-Next-Offset. This precondition composes with, but does not replace, Stream-Seq or idempotent-producer headers.

6.5. Record Metadata on HEAD

For a JSON stream implementing this extension, HEAD {stream_url} MUST include:

Stream-Record-First: First retained record ordinal F.
Stream-Record-Next: Next record ordinal N.

6.6. Record-Aware Read Parameters

A record-aware read selects its starting position with exactly one of:

record=<ordinal>: Start at the specified record boundary.
record=now: Start at the current record tail.
tail_records=<count>: Start at max(F, N - count).

These parameters are mutually exclusive with each other and with offset. tail_records MUST be a non-negative integer.

The following rules apply:

record < F MUST return 410 Gone with Stream-Record-First and Stream-Record-Next.
record = N is a valid up-to-date empty read.
record > N MUST return 400 Bad Request with Stream-Record-Next.
tail_records=0 is equivalent to record=now.

Record-aware reads MAY also specify max_records=<positive-integer>. max_records limits the number of complete records returned. max_records without a record-aware start parameter MUST return 400 Bad Request.

max_bytes and max_records MUST NOT appear in the same request. A record-aware read that includes max_bytes MUST return 400 Bad Request. This keeps the base protocol's partial-byte behavior separate from the extension's complete-record guarantee.

6.7. Record-Aligned Read Responses

A successful record-aware catch-up or long-poll response MUST:

begin at a complete record boundary;
contain only complete normalized JSON messages;
return Content-Type: application/x-ndjson by default;
return at most max_records messages when supplied;
include Stream-Record-First, Stream-Record-Start, Stream-Record-Next, and Stream-Next-Offset.

Stream-Record-Start is the resolved starting ordinal. Stream-Record-Next is the continuation ordinal after the last complete returned record. Stream-Next-Offset MUST identify the same logical boundary as Stream-Record-Next.

An empty successful response MUST set Stream-Record-Start and Stream-Record-Next to the resolved starting ordinal.

Example:

HTTP/1.1 200 OK
Content-Type: application/x-ndjson
Stream-Extensions: json-record-coordinates-v1
Stream-Record-First: 0
Stream-Record-Start: 42
Stream-Record-Next: 44
Stream-Next-Offset: <opaque-offset>

{"type":"navigation_started"}
{"type":"network_response","status":500}

Ordinary GET {stream_url}?offset=... behavior is unchanged and MAY still end in the middle of a JSON record as defined by the base protocol.

6.8. Optional Record Envelope View

A record-aware client MAY request record_view=envelope. This parameter MUST be combined with a record-aware start parameter.

The response MUST use:

Content-Type: application/vnd.durable-stream-records+ndjson

Each response line MUST be a JSON object containing:

record: Record ordinal.
value: The original normalized JSON value.

Example:

{"record":42,"value":{"captured_at":"2026-07-18T10:29:59.900Z","type":"navigation_started"}}
{"record":43,"value":{"captured_at":"2026-07-18T10:30:00.100Z","type":"network_response","status":500}}

The envelope is an alternate read representation only. It MUST NOT change canonical stream bytes, offsets, record ordinals, ETags for the canonical representation, snapshots, bootstrap data, or ordinary reads.

6.9. Live Reads and SSE

Record start parameters compose with live=long-poll and live=sse.

SSE control events for record-aware reads MUST add:

{
  "streamNextOffset": "<opaque-offset>",
  "streamFirstRecord": 0,
  "streamNextRecord": 44,
  "upToDate": true
}

For the default JSON view, SSE data events retain the base protocol's JSON-mode representation and MAY contain multiple newline-delimited messages. Control-event record coordinates MUST refer to the boundary after every complete message represented before that control event.

With record_view=envelope, each SSE event: data MUST contain exactly one envelope object and the response MUST include Stream-Data-Content-Type: application/vnd.durable-stream-record+json.

A reconnecting client MAY use either the last streamNextRecord or streamNextOffset. Offset-based reconnection remains the base-protocol path.

6.10. Retention, Snapshots, and Bootstrap

Retention MUST advance F only at a record boundary. A trimmed ordinal MUST return 410 Gone; surviving records keep their original ordinals.

When a snapshot advances the earliest retained offset, that offset MUST map to one record boundary. Snapshot publication at an intra-record offset MUST return 400 Bad Request. Snapshot and bootstrap responses SHOULD include Stream-Record-First and Stream-Record-Next when the underlying stream implements this extension. Bootstrap update parts MUST preserve one logical JSON message per update part.

Implementations MAY reclaim ordinal index entries only after the corresponding records fall before F and are no longer required by a visible snapshot or bootstrap response.

6.11. Append Batch Integration

For JSON streams implementing this extension, each successful Append Batch frame receives its own contiguous record range. Frame acknowledgements defined in Section 4.3 MUST additionally include:

stream_record_start;
stream_record_next.

One outer Append Batch request remains transport batching, not an atomic multi-frame transaction. A failed frame assigns no visible record ordinal; successful later frames continue from the last committed record tail.

6.12. Compatibility and Non-Goals

A base Durable Streams client continues to append JSON values or arrays, read by offset, and receive canonical NDJSON. It MAY ignore all Stream-Record-* and Stream-Extensions headers.

This extension does not define:

schema validation or schema evolution;
indexes over arbitrary JSON fields;
SQL, Arrow, Parquet, or DataFusion integration;
consumer groups;
per-record arbitrary byte headers;
authentication or authorization;
a bidirectional append-session transport;
record framing for arbitrary binary content types.

6.13. Correctness Requirements

Implementations MUST preserve these invariants:

Every successfully appended normalized JSON message receives exactly one ordinal.
Ordinals are contiguous in committed stream order and are never reused.
One atomic append receives one contiguous ordinal range or no range.
A deduplicated append returns its original ordinal range.
O(r) always identifies the first canonical byte of record r.
Offset reads and record reads expose the same canonical JSON sequence.
Record-aware reads never return partial records.
Flush, compaction, snapshot, restart, and retention never change a surviving record's ordinal.
N - F equals the number of retained logical messages.
Base-protocol clients observe no semantic change.

Conformance tests SHOULD cover JSON array flattening, concurrent append ordering, idempotent retry, record-tail preconditions, partial offset reads, record-aligned reads, retention, snapshot/bootstrap boundaries, cold flush, restart, and live-read reconnection.

7. Ordinary SSE Compatibility

Ordinary SSE behavior, including binary stream handling, is defined by docs/specs/durable-stream.md.

For binary streams, ordinary event: data payloads are raw base64 text and the response MUST include stream-sse-data-encoding: base64.
Extension endpoints MUST NOT redefine the payload shape of ordinary binary SSE event: data frames unless they document an explicit endpoint-local contract.

8. URL Encoding

snapshot_offset values originate from Stream-Next-Offset and MAY contain characters requiring URL encoding. Clients and servers MUST apply standard URL encoding/decoding when using offset values in path segments.

Abstract

Terminology

Table of Contents

Extension Capability Advertisement

1. Buckets

1.1. Identifier Constraints

1.2. Create Bucket

1.3. Get Bucket Metadata

1.4. List Bucket Streams

1.5. Delete Bucket

1.6. Stream Operations

2. Snapshots

2.1. Offset Conventions

2.2. Publish Snapshot

2.3. Read Latest Snapshot

2.4. Snapshot Metadata on HEAD

2.5. Read Snapshot

2.6. Delete Snapshot

3. Bootstrap

3.1. Request

3.2. Response

3.3. Follow-Up Live Reads

3.4. Compatibility

4. Append Batch

4.1. Request

4.2. Unsupported Per-Append Controls

4.3. Response

4.4. Ordering and Compatibility

5. Stream Attributes

5.1. Attribute Object

5.2. Create Stream With Attributes

5.3. Read Attributes

5.4. Replace Attributes

6. JSON Record Coordinates

6.1. Record Model

6.2. Client Event Time

6.3. Create and Append Acknowledgements

6.4. Conditional Append by Record Tail

6.5. Record Metadata on HEAD

6.6. Record-Aware Read Parameters

6.7. Record-Aligned Read Responses

6.8. Optional Record Envelope View

6.9. Live Reads and SSE

6.10. Retention, Snapshots, and Bootstrap

6.11. Append Batch Integration

6.12. Compatibility and Non-Goals

6.13. Correctness Requirements

7. Ordinary SSE Compatibility

8. URL Encoding