# Ursula - full documentation

> Open-source Distributed Durable Streams over HTTP, backed by S3.

Source: https://ursula.tonbo.io

---

# Ursula

> [!NOTE]
> Ursula is built by <a href="https://tonbo.io/" target="_blank" rel="noopener noreferrer"><img src="/assets/tonbo.png" alt="" width="16" height="16" style={{verticalAlign: "-3px", borderRadius: "3px", marginRight: "4px"}} />Tonbo</a>.

Ursula is a self-hosted, distributed server for the replayable, append-only event timelines behind document edits, agent runs, workflows, and chat. It speaks the [Durable Streams Protocol](/docs/specs/durable-stream) over plain HTTP and SSE.

Quorum-replicated in-memory front-ends give you single-digit-millisecond appends and live tail on top of S3 durability, with no separate broker, no batched 250 ms commits, and no S3 Express bill.

- **HTTP-native.** `PUT` creates, `POST` appends, `GET` replays or tails with long-poll/SSE. Any HTTP client is a valid client.
- **One timeline per resource.** A stream per document, session, workflow, room, or agent run, instead of a few high-throughput pipelines.
- **Thread-per-core × multi-Raft.** Each stream hashes to one Raft group and one owner core. Hot-path requests touch that core only, with no cross-core synchronization. Hundreds to thousands of small groups co-exist per node, so a slow follower for one group never stalls another.
- **Hot ring + S3 cold tier.** Writes commit at Raft quorum in an in-memory ring. Older segments flush to S3 in the background. A single `GET` transparently spans both tiers.

## Try it

Start a single in-memory node with Docker (or build from source, see [Install](/docs/install)):

```bash
docker run --rm -p 4437:4437 ghcr.io/tonbo-io/ursula:0.3.6
```

Create a stream, append bytes, and read them back:

```bash
curl -X PUT http://127.0.0.1:4437/demo
curl -X PUT http://127.0.0.1:4437/demo/hello

curl -X POST http://127.0.0.1:4437/demo/hello \
  -H 'Content-Type: application/octet-stream' \
  --data-binary 'hello world'

curl 'http://127.0.0.1:4437/demo/hello?offset=-1'
```

Continue with the [Quick Start](/docs/quick-start) to replay and tail streams live, and [Deploy a Cluster](/docs/deploy-cluster) when you're ready for the production three-voter shape.

## Learn more

- **[Why Ursula](/docs/why-ursula):** the four properties Ursula keeps that other servers force you to trade
- **[Architecture](/docs/architecture/overview):** thread-per-core, multi-Raft, hot/cold tiers
- **[Protocol Spec](/docs/specs/durable-stream):** Durable Streams plus [Ursula's extensions](/docs/specs/extensions)
- **[ursulactl](/docs/cli):** the operator CLI for a running cluster (leadership drain, status, readiness gates)

## Credits

- **[ElectricSQL](https://electric-sql.com/)** for the original Durable Streams Protocol that Ursula implements.
- **[Loro](https://loro.dev/)** for the snapshot and replay extension design that Ursula adopted on top of the base protocol.

---

# Why Ursula

## What Ursula keeps

A new generation of event streams lives outside the broker network. Document editors, agents, and durable workflows need timelines that browsers, mobile apps, and serverless functions can read, write, and tail over the public internet. That asks for HTTP-native, distributed, S3-backed infrastructure, not the SDK-locked, single-network shape Kafka-style brokers were built for.

The [Durable Streams Protocol](/docs/specs/durable-stream) nails that wire format, but its reference server is a single process: a node loss is data loss. The other servers we evaluated each force you to give up one of four things this primitive deserves to keep:

- **Open-source self-hosting.**
- **Low write latency** (sub-50 ms appends, no batching window required).
- **Plain S3 economics** (cold tier on standard S3, no S3 Express tier, no per-GB SaaS markup).
- **Quorum-replicated durability** (acknowledged writes survive a single-node failure).

Ursula keeps all four. For the head-to-head numbers, see [How Ursula compares](/docs/competitive-comparison).

## One timeline per entity

Most event systems multiplex many entities into shared channels and rebuild per-entity state downstream. Ursula inverts that: each document, session, task, room, or agent run gets its own durable timeline. Writers append directly to it and readers resume from its offsets.

Treat it as a per-entity durable log runtime, purpose-built for that pattern rather than a general event backbone.

## What you get from one stream per entity

- **Replayable recovery.** When a worker, sandbox, or agent restarts, replay its log and continue.
- **Live tails with simple clients.** Catch-up reads, long-poll, or SSE, all over plain HTTP.
- **Lifecycle in the same primitive.** Snapshots, bootstrap, and TTL ride the timeline instead of being separate infrastructure.

## The tradeoff

Ursula makes one timeline per entity cheap. In exchange, it isn't the most general primitive for cross-system event distribution or arbitrary stream processing. Keeping the per-entity model simple is what makes recovery, inspection, and operation tractable for long-running application state.

For latency and durability numbers, see [Architecture](/docs/architecture/overview) and [Competitive comparison](/docs/competitive-comparison).

---

# Install

This page gets a **single Ursula node** running on your machine so you can try the API. There are two ways to do it:

| Pick | When |
| ---- | ---- |
| **Docker** | You just want to try Ursula. One command, no toolchain. |
| **Build from source** | You want local binaries, plan to hack on Ursula, or can't run Docker. |

Both start the same in-memory, non-replicated node on port `4437`. Nothing is persisted across restarts. For a production cluster, see [Deploy a Cluster](/docs/deploy-cluster).

## Docker

Every release publishes a multi-arch image to GHCR:

```bash
docker run --rm -p 4437:4437 ghcr.io/tonbo-io/ursula:0.3.6
```

The image runs as a non-root user and contains both `ursula` (the server) and `ursulactl` (the operator CLI). Images are published only from release tags, so pin the version you want to run.

## Build from source

You need rustup, a C compiler, and `pkg-config`. The repository pins the toolchain in `rust-toolchain.toml`, so rustup selects the right nightly automatically.

### Installing the prerequisites (macOS, Debian/Ubuntu)

On macOS:

```bash
brew install rustup pkg-config
rustup-init -y
```

On Debian or Ubuntu:

```bash
sudo apt-get update
sudo apt-get install -y build-essential pkg-config curl
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

Clone, build, and start the server:

```bash
git clone https://github.com/tonbo-io/ursula.git
cd ursula
cargo build --release -p ursula -p ursula-ctl
./target/release/ursula server
```

The server is now listening on `127.0.0.1:4437`. The build also produces `target/release/ursulactl`, the operator CLI you will use once a cluster is up.

## Verify

With the node running, from another terminal:

```bash
curl http://127.0.0.1:4437/__ursula/metrics
```

A JSON snapshot of runtime state means the node is up.

## Next

- [Quick Start](/docs/quick-start): create streams, append, replay, and tail live over SSE
- [Deploy a Cluster](/docs/deploy-cluster): the production three-voter shape with OpenTofu + Helm
- [Configuration](/docs/configuration): persistence, presets, and S3 cold storage

---

# Quick Start

This page assumes a single node is already running on `127.0.0.1:4437`. Starting one is a single Docker or Cargo command, see [Install](/docs/install). The default node is in-memory and nothing survives a restart. For disk-backed persistence, see [Configuration](/docs/configuration).

Drive the HTTP API with `curl` from another terminal.

## Acknowledge the bucket

Bucket creation in the current build is an idempotent acknowledgement - Ursula returns `201` whether or not the name has been seen before. It's still worth issuing because clients and docs assume the call:

```bash
curl -X PUT http://127.0.0.1:4437/demo
```

## Create a stream

```bash
curl -X PUT http://127.0.0.1:4437/demo/hello
```

## Append data

```bash
curl -X POST http://127.0.0.1:4437/demo/hello \
  -H 'Content-Type: application/octet-stream' \
  --data-binary 'first message'

curl -X POST http://127.0.0.1:4437/demo/hello \
  -H 'Content-Type: application/octet-stream' \
  --data-binary 'second message'
```

Each successful append returns `204 No Content` with a `Stream-Next-Offset` header. Add `Producer-Id` / `Producer-Epoch` / `Producer-Seq` if you need [exactly-once retries](/docs/concepts/exactly-once-writes).

## Read everything from the beginning

```bash
curl -i 'http://127.0.0.1:4437/demo/hello?offset=-1'
```

The body contains the appended bytes. Response headers include `Stream-Next-Offset`, `Stream-Up-To-Date`, and an `ETag`.

## Subscribe for live updates

Open a second terminal:

```bash
curl -N 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse'
```

`-N` keeps `curl` from line-buffering the SSE stream. Append more data from the first terminal and you'll see it arrive immediately as `event: data` lines. Binary streams are delivered as raw base64 text in `event: data` payloads (`Stream-Sse-Data-Encoding: base64`). See [binary SSE](/docs/concepts/binary-sse) for details.

## Inspect runtime state

For a multi-node cluster the canonical day-2 tool is [`ursulactl`](/docs/cli). It speaks to every node, summarises leadership, and wraps restarts in safe drain and catch-up steps. For a single local node you can either point it at a one-line manifest:

```bash
cat > /tmp/local.json <<'JSON'
{"nodes": [{"id": 1, "http_url": "http://127.0.0.1:4437", "host": "127.0.0.1"}]}
JSON

ursulactl status --config /tmp/local.json
```

Or hit the underlying JSON endpoint directly:

```bash
curl http://127.0.0.1:4437/__ursula/metrics
```

The raw endpoint is also what `ursulactl` consumes. Use it directly when you want the full snapshot or are building custom tooling.

## Next steps

- [Deploy a Cluster](/docs/deploy-cluster): the production three-voter shape with OpenTofu + Helm
- [Configuration](/docs/configuration): persistence, presets, and S3 cold storage in one place
- [ursulactl](/docs/cli): the operator CLI you'll use once a cluster is up
- [API overview](/docs/api/overview): the HTTP surface Ursula currently exposes
- [Streams](/docs/concepts/streams): the core stream abstraction and lifecycle
- [Architecture](/docs/architecture/overview): thread-per-core, multi-Raft internals

---

# Clients

Ursula speaks plain HTTP and Server-Sent Events. There is no required client library. Any HTTP client in any language works.

The examples elsewhere in these docs use `curl` because it's universal. The same routes, headers, and query parameters apply to every other client.

## Minimal examples

<Tabs>
<Tab title="curl">

```bash
# create a bucket and stream
curl -X PUT http://127.0.0.1:4437/demo
curl -X PUT http://127.0.0.1:4437/demo/hello

# append
curl -X POST http://127.0.0.1:4437/demo/hello \
  -H 'Content-Type: application/octet-stream' \
  --data-binary 'hello world'

# catch-up read
curl 'http://127.0.0.1:4437/demo/hello?offset=-1'

# live tail
curl 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse'
```

</Tab>
<Tab title="Python">

```python
import requests

base = "http://127.0.0.1:4437"

requests.put(f"{base}/demo")
requests.put(f"{base}/demo/hello")

requests.post(
    f"{base}/demo/hello",
    headers={"Content-Type": "application/octet-stream"},
    data=b"hello world",
)

# catch-up read
resp = requests.get(f"{base}/demo/hello", params={"offset": -1})
print(resp.content)

# live tail with SSE
with requests.get(
    f"{base}/demo/hello",
    params={"offset": -1, "live": "sse"},
    stream=True,
) as r:
    for line in r.iter_lines():
        if line:
            print(line.decode())
```

</Tab>
<Tab title="TypeScript / fetch">

```ts
const base = "http://127.0.0.1:4437";

await fetch(`${base}/demo`, { method: "PUT" });
await fetch(`${base}/demo/hello`, { method: "PUT" });

await fetch(`${base}/demo/hello`, {
  method: "POST",
  headers: { "Content-Type": "application/octet-stream" },
  body: "hello world",
});

// catch-up read
const data = await (await fetch(`${base}/demo/hello?offset=-1`)).text();

// live tail with native EventSource
const es = new EventSource(`${base}/demo/hello?offset=-1&live=sse`);
es.addEventListener("data", (e) => console.log(e.data));
```

</Tab>
</Tabs>

## From a browser

The examples above assume same-origin access. A page served from a different origin than the gateway needs the gateway to allow that origin — see [cross-origin reads](/docs/security#cross-origin-reads-opt-in):

```bash
ursula gateway ... --cors-allowed-origin https://app.example.com
```

Two things to know once it is on:

- **Read your continuation headers.** The gateway sends `Access-Control-Expose-Headers: *`, so `Stream-Next-Offset` and friends are readable. Without exposure a browser can read one page and never advance, which looks like the stream ending. If a `fetch` succeeds but `response.headers.get("stream-next-offset")` is `null`, the origin is not allowing your origin.
- **`EventSource` cannot send `Authorization`.** It has no header API, so it only works against `public_read` streams. For a private live tail, use `fetch` and read the body:

  ```ts
  const response = await fetch(`${base}/${bucket}/hello?offset=-1&live=sse`, {
    headers: { Authorization: `Bearer ${token}` },
  });
  const reader = response.body!.pipeThrough(new TextDecoderStream()).getReader();
  ```

  A subscription ends when its credential expires, and says so: the final frame is `event: credential-expired`. Treat it as a reconnect signal rather than end of stream — refresh the token and re-read from the last `Stream-Next-Offset` you saw. Any other termination deserves the same handling.

## Notes for client implementers

- After every read and append, the server returns `Stream-Next-Offset`. Track it. Use it as the `offset` query parameter on the next read. Don't construct offsets manually. They're opaque.
- For binary streams over SSE, `data` events carry raw base64 text and the response includes `Stream-Sse-Data-Encoding: base64`. Decode the data event first, then interpret the bytes using `Stream-Data-Content-Type`. See [Binary SSE](/docs/concepts/binary-sse).
- For exactly-once writes, send `Producer-Id`, `Producer-Epoch`, `Producer-Seq` headers and retry on network errors. The server deduplicates. See [Exactly-once writes](/docs/concepts/exactly-once-writes).
- For conditional writes, use `Stream-Seq` to enforce ordering from one logical writer. For JSON streams that need a compare-current-tail guard across writers, use Ursula's `Stream-Record-Match` extension. See [Conditional writes](/docs/concepts/conditional-writes).

---

# Deploy a Cluster

Production Ursula is a static-membership Raft cluster: three voting nodes across availability zones, a durable Raft log per node, and a shared S3 bucket for the cold tier.

The recommended way to run it is **Kubernetes via the Helm chart**, with **OpenTofu** provisioning the cloud prerequisites:

- **Starting from scratch on AWS**: use the OpenTofu stack in `deploy/eks`. It provisions the VPC, EKS cluster, storage, S3 bucket, and IAM identities, and generates the Helm values to install with.
- **Already have a Kubernetes cluster**: `helm install` directly from GHCR and supply your own S3 bucket and storage class.

If you are not running Kubernetes, see [bare metal](#without-kubernetes-bare-metal--vms) at the end.

## Recommended: OpenTofu + Helm on EKS

The repository includes an OpenTofu reference stack under [`deploy/eks`](https://github.com/tonbo-io/ursula/tree/main/deploy/eks). It provisions a three-AZ VPC and EKS cluster, one managed node group per zone, the EBS CSI and Pod Identity add-ons, an encrypted `gp3` StorageClass, a versioned S3 bucket, and least-privilege identities for Ursula and the event-time indexer. It writes the complete Helm input to `generated-values.yaml` and a dedicated kubeconfig without touching `~/.kube/config`.

One-time setup: create a versioned, encrypted S3 bucket for OpenTofu state, copy `backend.tf.example` to the ignored `backend.tf` with a unique state key, and copy `terraform.tfvars.example` to the ignored `terraform.tfvars`. In the tfvars, pick an explicit image tag and restrict the EKS public API to your operator or CI CIDRs (the stack rejects `0.0.0.0/0` and `::/0`).

After that, the complete deployment path is:

```bash
cd deploy/eks
tofu init
tofu apply
KUBECONFIG=./kubeconfig helm install ursula ../../charts/ursula --namespace ursula --create-namespace -f generated-values.yaml
KUBECONFIG=./kubeconfig helm test ursula --namespace ursula
```

See [`deploy/eks/README.md`](https://github.com/tonbo-io/ursula/tree/main/deploy/eks) for inputs, cost, state, and teardown guidance. OpenTofu owns the AWS prerequisites and Helm owns the namespace-scoped Ursula workloads.

## Existing Kubernetes cluster: Helm

The chart and images are published to GHCR on every release. One command starts a three-voter cluster:

```bash
helm install ursula oci://ghcr.io/tonbo-io/charts/ursula --version 0.3.6
```

The chart defaults to the `ghcr.io/tonbo-io/ursula` image pinned to the chart's `appVersion`. The default install runs three voters and 64 Raft groups, with durable per-pod Raft logs on PVCs, a headless peer Service, a client Service, and a quorum-protecting PodDisruptionBudget.

For production, add shared S3 cold storage and workload identity:

```yaml
s3:
  bucket: my-ursula-bucket
  region: us-east-1
  prefix: ursula-prod

coldStorage:
  enabled: true

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ursula-s3
```

For MinIO or another S3-compatible backend, set `s3.endpoint`. Prefer workload identity over static S3 credentials. To use a registry mirror, set `global.image.repository`, `global.image.tag`, and optionally `global.imagePullSecrets`.

`server.replicaCount` controls the voter set for a **fresh** cluster and supports `1`, `3`, and `5`. Production should use `3`, or `5` only when tolerating two simultaneous voter failures justifies the larger write quorum. Changing it on an initialized cluster is not safe Raft voter reconfiguration. Safe scaling needs the future operator workflow.

## Verify

```bash
helm test ursula
```

The test mounts the chart-generated cluster manifest and runs `ursulactl wait-ready`. It succeeds only when every node reports the expected Raft group count and every group has a leader. To query a node directly:

```bash
kubectl port-forward svc/ursula 4437:4437
curl http://127.0.0.1:4437/__ursula/metrics
```

## Production notes

- Run three voters across three availability zones, each with its own zonal persistent volume for the Raft log. Never use memory Raft storage or ephemeral voter data in production.
- Shared S3 is required for any multi-node cluster: replicas must be able to read chunks flushed by any leader.
- Put stateless gateway replicas behind authenticated TLS ingress and keep voter and peer Services private. Ursula itself has no TLS or auth (see [Security](/docs/security)).
- Kubernetes rolling updates do not transfer Raft leaders on their own. For an operationally safe restart, wrap the platform's restart in [ursulactl's maintenance verbs](/docs/cli): `ursulactl drain` the node, let Kubernetes restart the pod, then `ursulactl wait --node N` and `undrain` before moving to the next one. See [Operations](/docs/operations) for day-2 work.
- There are no dedicated health probes yet. Cluster readiness comes from `helm test` or `ursulactl wait-ready`. The mutating admin surface is bound to pod loopback (`127.0.0.1:4438`) and is reached with `kubectl port-forward`.

## Optional: event-time indexer

`ursula-indexer` is an optional worker pool that builds queryable event-time indexes from streams, kept outside the voter processes so query and compaction work never touches commit latency:

```yaml
indexer:
  enabled: true
  replicaCount: 2
  s3:
    prefix: indexes
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ursula-index
```

Register streams dynamically with `PUT /v1/indexes/{id}` on the internal indexer Service. No Helm upgrade is needed. The Service is an internal ClusterIP. If remote applications need read access, front it with an authenticated, path-aware proxy and never expose the registration or administration routes.

## Without Kubernetes (bare metal / VMs)

The same static-membership model works by hand. Every node runs the same config file, and only the node ID differs:

```toml
[server]
listen = "0.0.0.0:4437"

[raft]
group_count = 256
init_membership_per_group = true

[raft.wal]
backend = "disk"
path = "/var/lib/ursula"

[storage.cold]
backend = "s3"
root = "ursula-prod"

[storage.cold.s3]
bucket = "my-ursula-bucket"
region = "us-east-1"

[[raft.peers]]
node_id = 1
url = "http://10.0.0.1:4437"

[[raft.peers]]
node_id = 2
url = "http://10.0.0.2:4437"

[[raft.peers]]
node_id = 3
url = "http://10.0.0.3:4437"
```

Start each node with its own ID:

```bash
ursula --config /etc/ursula/ursula.toml --node-id 1   # node 2 and 3 likewise
```

`init_membership_per_group = true` is only needed on the very first start of a fresh cluster. Flip it to `false` afterwards. Then verify with a one-file manifest:

```bash
cat > cluster-manifest.json <<'JSON'
{
  "nodes": [
    {"id": 1, "http_url": "http://10.0.0.1:4437", "host": "10.0.0.1"},
    {"id": 2, "http_url": "http://10.0.0.2:4437", "host": "10.0.0.2"},
    {"id": 3, "http_url": "http://10.0.0.3:4437", "host": "10.0.0.3"}
  ]
}
JSON

ursulactl wait-ready --config cluster-manifest.json --expected-groups 256
ursulactl status      --config cluster-manifest.json
```

Every config key (peers, listeners, WAL, cold-tier tuning) is documented in [Configuration](/docs/configuration).

---

# Configuration

Everything configurable in Ursula lives on this page. Ursula is configured by a TOML config file, an optional resource preset, and a small set of CLI overrides, merged in this order:

```text
built-in defaults < preset defaults < config file < CLI overrides
```

The normal startup command is:

```bash
ursula --config /etc/ursula/ursula.toml --node-id 1
```

With no config file at all, `ursula` starts a single-node in-memory runtime. That's what [Install](/docs/install) uses.

## CLI flags

| Flag | Default | Notes |
| ---- | ------- | ----- |
| `--config FILE` | searches `./ursula.toml`, `/etc/ursula/ursula.toml`, then user config paths | TOML config file |
| `--preset PRESET` | `default` when no config file is found, otherwise none | Resource preset: `default`, `dev`, `tiny`, `small`, `standard`, or `large` |
| `--node-id ID` | config file or preset | Overrides `raft.node_id` |

CLI `--node-id` intentionally overrides any `raft.node_id` value in the config file so the same file can be mounted on every node. Helm derives it from the StatefulSet ordinal.

## Resource presets

`--preset` supplies a base configuration before the config file is merged, and config-file values win. Pick by node size:

| Preset | Intended use | Main defaults |
| ------ | ------------ | ------------- |
| `default`, `dev` | Local development | Single-node in-memory runtime |
| `tiny` | Memory-bound tests or very small nodes | 64 groups, 64 MiB cold cache, 4 MiB cold flushes, 2 cold writes, 8 MiB hot/admission caps, 64 MiB HTTP in-flight body budget |
| `small` | Small nodes or cost-sensitive tests | 128 groups, 64 MiB cold cache, 4 MiB cold flushes, 2 cold writes, 16 MiB hot/admission caps, 64 MiB HTTP in-flight body budget |
| `standard` | Production baseline | 256 groups, 256 MiB cold cache, 8 MiB cold flushes, 4 cold writes, 64 MiB hot/admission caps, 256 MiB HTTP in-flight body budget |
| `large` | Larger nodes with more cache and write headroom | 512 groups, 512 MiB cold cache, 16 MiB cold flushes, 8 cold writes, 128 MiB hot/admission caps, 512 MiB HTTP in-flight body budget |

## Server and runtime

| Config key | Default | Purpose |
| ---------- | ------- | ------- |
| `server.listen` | `127.0.0.1:4437` | Client/API listener |
| `server.cluster_listen` | none | Optional separate Raft/cluster listener |
| `runtime.core_count` | `available_parallelism` | Worker threads, each pinned to its mailbox event loop |
| `runtime.live_read_max_waiters_per_core` | `65536` | SSE waiter cap per core. `0` disables the limit |
| `observability.tokio_console` | `false` | Enables Tokio console when the binary is built with the feature |

Without `server.cluster_listen`, `server.listen` serves both client/API and Raft/cluster routes, and each `[[raft.peers]].url` should point at that address. With `server.cluster_listen` set, `server.listen` is client/API-only and peer URLs must point at the cluster listener instead. Note that the same peer URL is currently also used for HTTP leader redirects, so it must be reachable by clients or the gateway as well as by peers.

## Raft

| Config key | Default | Purpose |
| ---------- | ------- | ------- |
| `raft.node_id` | none | This node's stable ID, usually overridden per host with `--node-id` |
| `raft.group_count` | `core_count × 16` | Total Raft groups. Higher values improve stream-to-group hash spread |
| `raft.wal.backend` | `memory` | `memory` (volatile, survives no restart) or `disk` (durable, required for clusters) |
| `raft.wal.path` | none | Required for `backend = "disk"`. Raft logs live under `PATH/raft-log`, and removing the directory wipes state cleanly |
| `raft.peers` | empty | Static gRPC peer list, with one `[[raft.peers]]` entry per voter, including this node |
| `raft.init_membership_per_group` | `false` | One-time per-group membership bootstrap. Set `true` on the very first start of a fresh cluster, then flip back to `false` |

Pick one WAL backend per cluster and use it on every peer.

## Cold storage

Ursula keeps recent data in an in-memory hot ring on every replica and flushes older segments and snapshot blobs to a cold backend. Multi-node clusters should use S3 or an S3-compatible object store so every replica can read chunks flushed by any leader. All replicas must point at the same shared bucket.

```toml
[storage.cold]
backend = "s3"
root = "ursula-prod-20260518"

[storage.cold.s3]
bucket = "my-ursula-bucket"
region = "us-east-1"
# endpoint = "http://127.0.0.1:9000"
# access_key_id = "AKIA..."
# secret_access_key = "..."
# session_token = "..."
# server_side_encryption = "aes256"   # default; "aws-kms" or "none"
# kms_key_id = "arn:aws:kms:..."      # customer managed key for "aws-kms"
```

When `access_key_id` and `secret_access_key` are omitted, Ursula uses the standard AWS SDK credential chain (instance profile, environment, profile, and so on). Prefer that over static keys. Set `endpoint` for S3-compatible stores such as MinIO, R2, or TOS.

Every cold-tier object write requests server-side encryption by default (`server_side_encryption = "aes256"`, i.e. SSE-S3) — free on AWS S3, and Raft snapshot uploads inherit the same setting. `aws-kms` switches to SSE-KMS, using the AWS managed key unless `kms_key_id` names a customer managed key. MinIO honors SSE headers only when a KMS/KES is configured; for a MinIO deployment without one, set `server_side_encryption = "none"` explicitly or writes fail with an SSE error.

| Config key | Default | Purpose |
| ---------- | ------- | ------- |
| `storage.cold.backend` | `none` | `none`, `memory`, or `s3` |
| `storage.cold.root` | none | Prefix prepended to every cold key. Use a date-stamped value for benchmark runs so cleanup can't touch production data |
| `storage.cold.s3.bucket` | none | Required when `backend = "s3"` |
| `storage.cold.s3.server_side_encryption` | `aes256` | SSE mode for every object write: `aes256` (SSE-S3), `aws-kms`, or `none` |
| `storage.cold.s3.kms_key_id` | none | Customer managed KMS key; only valid with `aws-kms` |
| `storage.cold.flush_interval` | `1s` | Background flush worker tick interval |
| `storage.cold.flush_size` | `8MiB` | Target bytes flushed per group per tick |
| `storage.cold.flush_max_concurrency` | `4` | Parallel cold writes in flight |
| `storage.cold.max_hot_size_per_group` | `64MiB` | Backpressure ceiling: when a group's hot bytes exceed this, new writes get HTTP `503` until the flush catches up. Explicit `0` disables the cap |

Flush defaults are conservative, so tune them under load. A typical benchmark profile drops the interval to `200ms`, raises concurrency to `32`, and bumps the per-group ceiling. Advanced deployments can split the flush threshold and batch size with `storage.cold.flush_min_hot_size` and `storage.cold.flush_max_size`, but the single `storage.cold.flush_size` knob is the normal path. Once a group's accumulated hot payload reaches the threshold, Ursula packs eligible slices from that group into one immutable object up to the configured maximum; large single-stream slices keep the same direct-object layout.

Raft snapshotting does not force a partial cold flush. Payload below the cold threshold stays in the bounded hot tail and is included in the next group snapshot, so a short snapshot interval does not turn small tails into small cold objects.

Helm and the EC2 scripts map their deployment inputs onto these same keys, so this table applies to every deployment method.

## Complete examples

Disk-backed single node (streams survive a restart):

```toml
[server]
listen = "127.0.0.1:4437"

[raft]
node_id = 1
group_count = 16

[raft.wal]
backend = "disk"
path = "./data"
```

```bash
ursula --config ./ursula.toml
```

Three-node durable cluster (same file on each host, override the node ID per host):

```toml
[server]
listen = "0.0.0.0:4437"

[runtime]
core_count = 16

[raft]
node_id = 1
group_count = 256
init_membership_per_group = true

[raft.wal]
backend = "disk"
path = "/var/lib/ursula"

[storage.cold]
backend = "s3"
root = "ursula-prod-20260518"

[storage.cold.s3]
bucket = "my-ursula-bucket"
region = "us-east-1"

[[raft.peers]]
node_id = 1
url = "http://10.0.0.1:4437"

[[raft.peers]]
node_id = 2
url = "http://10.0.0.2:4437"

[[raft.peers]]
node_id = 3
url = "http://10.0.0.3:4437"
```

```bash
ursula --config /etc/ursula/ursula.toml --node-id 1
```

See [Deploy a Cluster](/docs/deploy-cluster) for the full deployment walk-through.

---

# Security

> [!WARNING]
> Ursula does not terminate TLS, authenticate clients, or restrict admin endpoints. Treat the listening port as fully trusted. Run it on a private network behind a reverse proxy that owns TLS termination and request authentication.

The current `v0.x` security model is deliberately narrow. Ursula is built to slot behind your existing edge layer, not to be one.

## What Ursula does

- **Quorum-acknowledged writes.** An append is acknowledged only after a majority of voters has replicated it.
- **Per-group backpressure.** When a group's hot ring exceeds `storage.cold.max_hot_size_per_group`, appends to that group return `503` with `Retry-After` until cold flush catches up. Per-group, not global or per-client.
- **Stream-level isolation.** Streams hash to disjoint Raft groups and disjoint owner cores. A hot stream on one group cannot starve writes on a different group on a different core.

## What Ursula does not do

Handle the following outside Ursula:

- **TLS / HTTPS.** The public listener serves plain HTTP. No built-in `rustls`.
- **Inter-node encryption.** Peer gRPC (Raft heartbeats, append-entries, snapshots, and leader-read checks) runs over h2c. Peers must share a private network. Non-leader HTTP writes return a `307` redirect to the current group leader rather than being forwarded over gRPC.
- **API authentication on nodes.** Ursula nodes themselves accept any caller with network reach. Bearer-token validation is available as an opt-in feature of the gateway (see below); node listeners must stay on a private network either way.
- **Authorization / multi-tenancy on nodes.** Nodes enforce no per-user, per-bucket, or per-scope ACLs. The gateway's opt-in access control provides a bucket-level tenant boundary; anything finer stays upstream.
- **Admin endpoint isolation.** `/__ursula/metrics`, `/__ursula/flush-cold/*`, `/__ursula/raft/*`, and the public stream endpoints share the same listener with no auth gate.
- **Per-client rate limiting.** A single noisy client can saturate a core's mailbox or a group's hot ring.
- **Health/readiness endpoints.** No `/healthz` or `/readyz`. Use `/__ursula/metrics` as a process-alive probe (it serves only after the runtime initializes).
- **At-rest encryption beyond the cold tier.** Cold-tier S3 writes (including Raft snapshots) request SSE-S3 by default (`storage.cold.s3.server_side_encryption`, switchable to `aws-kms` or `none`). The hot ring is in memory; WAL and Raft log directories live on disk in plaintext — use full-disk encryption at the host level. Per-tenant KMS keys and client-side encryption are out of scope.

CORS is permissive (`Access-Control-Allow-Origin: *`). Restrict at the proxy for browser traffic.

Tenant offboarding has a first-class erasure path: the admin-plane [bucket purge endpoint](/docs/operations#tenant-offboarding-bucket-purge) removes a tenant's streams, bucket, quota, and cold objects idempotently, leaving other tenants untouched. It deliberately retains aggregate monotonic usage counters so asynchronous accounting cannot miss committed work; see the operations note for the remaining identifier-erasure limitation.

## Gateway access control (opt-in)

A shared or internet-facing deployment can enable OAuth resource-server checks on `ursula gateway`. The feature is off by default; without the flags the gateway keeps its original trusted pass-through behavior.

```bash
ursula gateway \
  --upstream http://ursula-0:4437 \
  --auth-issuer https://issuer.example \
  --auth-audience https://streams.example \
  --auth-policy /etc/ursula/policy.toml
```

- **Authentication.** `Bearer` credentials are validated as RFC 9068 JWT access tokens: the header must declare `typ: at+jwt` (OIDC ID tokens are rejected), the signature must verify against the issuer's JWKS, and `iss`, `aud`, `sub`, `client_id`, `iat`, `exp`, and `jti` must all be present and valid. The JWKS location comes from `--auth-jwks-url` or RFC 8414 metadata discovery; keys are cached by `kid` and refetched on rotation.
- **Tenant boundary.** The bucket is the top-level namespace and logical tenant boundary. The policy file declares each bucket's owners (issuer-qualified subjects) and whether anonymous reads are allowed:

```toml
[[bucket]]
id = "tenant-a"
public_read = true
owners = [{ issuer = "https://issuer.example", subject = "user-1" }]
```

- **Concealment.** Unknown buckets, private buckets probed by strangers, and write attempts without ownership all answer the same `404` a missing resource would, so a private stream's existence is not observable.
- **Credential termination.** The gateway strips `Authorization` before forwarding; upstream nodes never see end-user credentials and must remain on a private network.
- **Anonymous public reads.** `public_read` grants exactly the read-only actions (read, head, tail, snapshot read) to unauthenticated callers — never writes, deletes, or bucket administration.
- **Subscriptions do not outlive their credential.** A live tail is one long-lived request, so a single admission check at connection time would turn a short-lived token into an unbounded read. The gateway ends an SSE subscription at the credential's `exp` with a final `event: credential-expired` frame, so a client can tell expiry from end-of-stream and re-authenticate. Anonymous reads on `public_read` buckets have no credential and so no deadline. Note that this bounds duration, not revocation: a token revoked before its `exp` keeps an open subscription until then.

An access-controlled gateway can additionally enable per-tenant admission limits and usage accounting:

```bash
ursula gateway ... \
  --quota-policy /etc/ursula/quotas.toml \
  --usage-log /var/log/ursula/usage.jsonl
```

- **Quotas** (`--quota-policy`): per-bucket request rate (429 with `Retry-After`), concurrent live-read connections, and request body size. Limits are gateway-process-local; a horizontally scaled deployment multiplies effective limits by replica count. Ursula's own `503` backpressure semantics are unchanged. Data-plane quotas (stream count, retained bytes) are enforced inside Ursula as per-group backstops: `PUT /__ursula/quota/{bucket}` replicates `max_streams` / `max_retained_bytes` records to every Raft group, and each group rejects creates/appends that would exceed the limits against its local counters with `429` (no `Retry-After`: these are capacity caps, not rate limits). Because a bucket's streams hash across groups, the cluster-wide bound is `limit x group_count` - an abuse backstop; exact tenant-level enforcement belongs to the gateway, which reads aggregated `/__ursula/usage`.
- **Usage** (`--usage-log`): per-tenant request, ingress, and egress byte counters aggregated by `(bucket, principal, action class)` and appended as sequence-numbered JSONL batches on `--usage-flush-secs` intervals. `--usage-chunk-bytes` adds a `chunks` counter alongside them: each append is counted as `ceil(bytes / unit)`, never below one, summed per request. It exists because that sum cannot be recovered afterwards — two appends of 5 KiB and 25 KiB and two of 10 KiB and 20 KiB agree on both request count and byte total while owing four units and three — so a deployment that charges per write unit has to be handed the sum rather than the ingredients. The unit size is a pricing choice and Ursula does not pick one; 10 KB and 25 KB are both in use. A batched append costs proportionally less than the same records sent individually, which is the intended incentive: batching is cheaper to serve. A failing sink delays reporting (batches queue and merge) but never blocks requests or drops counts. Egress is counted from actually streamed bytes, including SSE bodies. Committed-truth counters (append bytes surviving retries, retained bytes) come from Ursula's replicated state and are a separate, complementary ledger.

### Cross-origin reads (opt-in)

`public_read` is only nominally public until the origin answers CORS: without it, browser JavaScript cannot read a public stream cross-origin. Allowed origins are deployment policy, so nothing is sent unless you list them.

```bash
ursula gateway ... \
  --cors-allowed-origin https://app.example.com \
  --cors-allowed-origin https://studio.example.com
```

Pass `*` instead to allow any origin.

Three properties are deliberate:

- **Credentials are never allowed.** Ursula authenticates from an `Authorization` header the caller sets explicitly, which CORS does not treat as credentials, and no cookies are involved. So `*` grants no ambient access — a cross-origin page must still present its own bearer token.
- **`Access-Control-Expose-Headers: *`.** A read carries its continuation in response headers (`stream-next-offset`, `stream-record-next`, `stream-cursor`), and a browser cannot see those without exposure — an unexposed client can read one page and never advance. The wildcard is only honoured while credentials stay disallowed, which is the second reason they are.
- **Preflight never consults the resource.** `OPTIONS` arrives without `Authorization`, so a per-bucket answer would tell an unauthenticated caller whether a private bucket exists. The preflight reply is identical for every path and is produced before authorization runs, which keeps concealment intact.

> [!NOTE]
> `EventSource` cannot set request headers, so a browser cannot open an SSE tail on a **private** stream with it. Use `fetch` with a `ReadableStream` and an `Authorization` header. `EventSource` is fine for `public_read` streams.

## Recommended deployment

```
                    Untrusted internet
                            │
                            v
                  ┌─────────────────────┐
                  │   Reverse proxy     │  TLS, authn, per-client
                  │ (nginx / Envoy / …) │  rate limiting, CORS
                  └──────────┬──────────┘
                             │ plain HTTP, private network
                ┌────────────┼────────────┐
                v            v            v
           ┌────────┐   ┌────────┐   ┌────────┐
           │ Ursula │   │ Ursula │   │ Ursula │
           │  node  │   │  node  │   │  node  │
           └────────┘   └────────┘   └────────┘
                             ↕ gRPC h2c on private network
                          (Raft replication)
```

### Checklist

- **Bind to the private interface.** Set `server.listen = "10.0.0.X:4437"` or use a security group / firewall so the listener is unreachable from public networks.
- **Terminate TLS at the proxy.** Ursula stays plain HTTP on the internal side.
- **Authenticate at the proxy.** Validate the caller (OAuth2, mTLS, signed requests) and reject unauthenticated traffic before it reaches Ursula.
- **Block admin paths from public traffic.** Deny `/__ursula/*` on the public listener and allow it only on an internal or ops network.
- **Use IAM roles for S3.** Omit static `storage.cold.s3.access_key_id` / `storage.cold.s3.secret_access_key` values and let the AWS SDK credential chain discover credentials.
- **Encrypt data volumes.** Apply full-disk encryption to `raft.wal.path`.
- **Keep peer traffic private.** Never route gRPC peer traffic across the public internet.

## Reporting vulnerabilities

Open a GitHub Security Advisory on [tonbo-io/ursula](https://github.com/tonbo-io/ursula/security/advisories) rather than a public issue.

---

# Resumable AI Stream

> Stream LLM tokens through a durable timeline so clients can refresh, reconnect, and resume without losing a token.

Instead of piping a model's output straight into one HTTP response, the generating worker appends tokens to a stream, and clients replay and tail that stream. A client that reconnects mid-generation replays the answer so far and continues from the same position, without losing or duplicating tokens. The upstream [`chat-aisdk` example](https://github.com/durable-streams/durable-streams/tree/main/examples) uses the same pattern.

## One stream per run

Create a text stream per generation, with a TTL so finished runs clean themselves up:

```bash
curl -X PUT http://127.0.0.1:4437/agent/run-42 \
  -H 'Content-Type: text/plain' \
  -H 'Stream-TTL: 86400'
```

## Producer: append tokens, close at the end

The worker that talks to the model appends each chunk. Producer headers make retries safe: a chunk re-sent after a network error is deduplicated instead of appended twice:

```ts
const base = "http://127.0.0.1:4437";
let seq = 0;

for await (const chunk of modelResponse) {
  await fetch(`${base}/agent/run-42`, {
    method: "POST",
    headers: {
      "Content-Type": "text/plain",
      "Producer-Id": "worker-1",
      "Producer-Epoch": "1",
      "Producer-Seq": String(seq++),
    },
    body: chunk.text,
  });
}

// Seal the stream: readers get an EOF signal, further appends return 409.
await fetch(`${base}/agent/run-42`, {
  method: "POST",
  headers: { "Stream-Closed": "true" },
});
```

If the worker crashes and restarts, bump `Producer-Epoch` and start `Producer-Seq` back at `0` (see [Exactly-once writes](/docs/concepts/exactly-once-writes)).

## Consumer: replay, then tail

`offset=-1` replays everything appended so far, then stays live for new tokens. Reopening the page mid-generation therefore shows the partial answer immediately and continues streaming:

```ts
const es = new EventSource(`${base}/agent/run-42?offset=-1&live=sse`);

es.addEventListener("data", (e) => {
  answerEl.textContent += e.data;
});

es.addEventListener("control", (e) => {
  if (JSON.parse(e.data).streamClosed) es.close(); // generation finished
});
```

Or watch it from a terminal:

```bash
curl -N 'http://127.0.0.1:4437/agent/run-42?offset=-1&live=sse'
```

The stream is durable and readable by any number of clients, so the same transcript can be tailed from a second device, read after the fact, or consumed by another process. The producer does not need to know about its readers.

## Next

- [Chat Room](/docs/examples/chat-room): the same pattern with JSON records and multiple writers
- [Read stream](/docs/api/read): catch-up, long-poll, and SSE read modes
- [Exactly-once writes](/docs/concepts/exactly-once-writes): producer epoch and sequence rules

---

# Chat Room

> One JSON stream per room: append messages, replay history by record, and tail new messages live.

A chat room is a JSON timeline: every message is one record, Ursula assigns record ordinals in commit order, and every participant replays history and tails the same stream. This mirrors the chat examples in the [upstream Durable Streams repo](https://github.com/durable-streams/durable-streams/tree/main/examples).

## One stream per room

```bash
curl -X PUT http://127.0.0.1:4437/chat/room-7 \
  -H 'Content-Type: application/json'
```

`application/json` streams advertise `json-record-coordinates-v1`: each appended JSON object becomes one complete record with a stable zero-based ordinal, so clients paginate and resume by message, not by byte offset.

## Send a message

```bash
curl -X POST http://127.0.0.1:4437/chat/room-7 \
  -H 'Content-Type: application/json' \
  --data '{"user":"ada","text":"hello","sent_at":"2026-07-23T09:00:00Z"}'
```

Each append returns the record range it created in `Stream-Record-Start` / `Stream-Record-Next`. Appending a top-level JSON array creates one record per element, which is useful for flushing an offline outbox in one request. To make send retries safe against double-posting, add the [exactly-once producer headers](/docs/concepts/exactly-once-writes) with one `Producer-Id` per device.

## Load recent history

Fetch the last 50 messages as self-describing `{record, value}` envelopes, newest ordinals last:

```bash
curl 'http://127.0.0.1:4437/chat/room-7?tail_records=50&record_view=envelope'
```

```json
{"record":118,"value":{"sent_at":"2026-07-23T09:00:00Z","text":"hello","user":"ada"}}
{"record":119,"value":{"sent_at":"2026-07-23T09:00:07Z","text":"hey ada","user":"lin"}}
```

JSON records are validated and normalized on append, so key order may differ from what the sender wrote.

## Tail new messages live

`record=now` skips history and streams only messages committed from now on. In envelope mode, every SSE data event carries exactly one `{record, value}` object, so the client parses each event directly:

```ts
const base = "http://127.0.0.1:4437";
const es = new EventSource(
  `${base}/chat/room-7?record=now&record_view=envelope&live=sse`,
);

es.addEventListener("data", (e) => {
  const { record, value } = JSON.parse(e.data);
  renderMessage(record, value);
});
```

Control events publish `streamNextRecord`. Store it and reconnect with `record=<streamNextRecord>` to resume without missing or repeating a message. Deduplicate by `record` if you combine a history fetch with a live tail.

## Next

- [Resumable AI Stream](/docs/examples/resumable-ai-stream): the single-writer variant for streaming model output
- [Record Coordinates](/docs/concepts/record-coordinates): the complete record replay model
- [Browser Telemetry](/docs/examples/browser-telemetry): an advanced example with event-time indexing on top of records

---

# Browser Telemetry

> Collect replayable browser events with JSON record coordinates and one client event timestamp.

Browser telemetry is a natural fit for an `application/json` stream: the browser appends ordinary JSON over HTTP, Ursula assigns stable record ordinals in commit order, and consumers can replay or tail complete events without interpreting opaque offsets.

The runnable example is in [`examples/browser-telemetry`](https://github.com/tonbo-io/ursula/tree/main/examples/browser-telemetry).

## Data model

Keep one timestamp: the time captured by the browser. It remains application data and may arrive out of order after offline buffering or retries.

```json
{
  "captured_at": "2026-07-18T10:30:00.100Z",
  "type": "network_response",
  "status": 500
}
```

Ursula does not reorder the stream by `captured_at` and does not invent a second protocol timestamp. Record `42` means the 42nd committed JSON event. Its opaque offset identifies the same byte boundary for base Durable Streams clients.

## Browser collector

Point plain `fetch` at a same-origin route that proxies to the Ursula stream. The gateway owns browser authentication and CORS policy.

```js
const events = [{
  captured_at: new Date().toISOString(),
  type: "navigation",
  duration_ms: performance.getEntriesByType("navigation")[0]?.duration,
}];

const response = await fetch("/telemetry/events", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(events),
  keepalive: true,
});

console.log(
  response.headers.get("Stream-Record-Start"),
  response.headers.get("Stream-Record-Next"),
);
```

One array append is normalized into complete NDJSON records. Each successful response returns the contiguous record range assigned to that operation.

## Replay and live tail

Read complete events from a known ordinal:

```http
GET /telemetry/events?record=42&max_records=100
```

Check that the response advertises `Stream-Extensions: json-record-coordinates-v1`, then continue from its `Stream-Record-Next` header. For a self-describing representation, add `record_view=envelope`:

```bash
curl 'http://127.0.0.1:4437/telemetry/events?record=42&max_records=100&record_view=envelope'
```

```json
{"record":42,"value":{"captured_at":"2026-07-18T10:30:00.100Z","type":"network_response","status":500}}
```

For a live native `EventSource`, add `live=sse`. Every envelope-mode SSE data event contains exactly one `{record, value}` object, and control events publish `streamNextRecord` for reconnection. See [Record Coordinates](/docs/concepts/record-coordinates) for the complete replay model.

## Event-time queries

Event time is an ordinary secondary index, not another Durable Streams coordinate. The optional `ursula-indexer` crate:

1. consumes the envelope view in record order
2. extracts `captured_at` and buffers `(captured_at, record)` in memory
3. flushes immutable sorted Parquet parts to S3 and advances `CURRENT` with an ETag compare-and-swap
4. publishes `indexed_through_record` and `durable_through_record` so queries can distinguish the live prefix from the crash-recoverable prefix
5. returns record ordinals for time queries, which callers resolve back through Ursula
6. reports a retention gap instead of presenting an incomplete result.

The source stream remains canonical and the derived index can be rebuilt from retained records. Equal timestamps are ordered by record ordinal.

This flow uses only HTTP/SSE. It does not depend on Append Session.

```bash
cargo run -p ursula --bin ursula -- indexer \
  --stream-url http://127.0.0.1:4437/telemetry/browser-telemetry \
  --s3-bucket my-telemetry-index \
  --s3-prefix production/browser-telemetry \
  --cache-dir ./target/browser-telemetry-cache
```

For Kubernetes, deploy one shared worker pool with its own S3 prefix and workload identity:

```yaml
s3:
  bucket: my-telemetry-index
  region: us-east-1

indexer:
  enabled: true
  replicaCount: 2
  s3:
    prefix: production/telemetry-indexes
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ursula-telemetry-index
```

When an application creates a browser stream, register it dynamically without changing the Helm release:

```bash
curl -X PUT http://ursula-indexer:4493/v1/indexes/browser-session-42 \
  -H 'Content-Type: application/json' \
  -d '{"stream_url":"http://ursula-gateway:4437/sessions/browser-session-42","timestamp_field":"captured_at"}'
```

The registration preflight rejects sources that are not JSON or do not advertise record coordinates. The fixed worker pool then balances `(stream, record range)` tasks: many small browser streams share pods, while a hot stream can occupy several workers. Per-stream S3 namespaces isolate checkpoints and queries without becoming compute-assignment boundaries.

S3 stores immutable content-addressed Parquet parts and versioned manifests. Flushes are split into UTC-day event-time partitions, and bounded size-tiered compaction merges only a fixed number of new parts within one day. Historical compacted parts are never rewritten when later telemetry arrives. Manifest-aware garbage collection preserves a configurable recent-generation and in-flight grace window, then removes superseded and failed-CAS objects. Compaction and GC use a separate bounded cache and index instance, so their Parquet and S3 work does not hold the HTTP query mutex. The serving cache is bounded and disposable, so a replacement instance can recover from `CURRENT` without a persistent volume. For local development, use `--object-dir ./target/browser-telemetry-objects` in place of the S3 options.

The query API is ordinary HTTP:

```http
GET http://ursula-indexer:4493/v1/indexes/browser-session-42/events?from=2026-07-18T10:00:00Z&until=2026-07-18T11:00:00Z&limit=100
GET http://ursula-indexer:4493/v1/indexes/browser-session-42/status
```

A query returns records sorted by `(captured_at_ms, record)`, plus a cursor and a fixed source-record watermark:

```json
{
  "indexed_from_record": 40,
  "indexed_through_record": 1200,
  "durable_through_record": 1152,
  "through_record": 1200,
  "records": [
    {"captured_at_ms": 1784370600100, "record": 42},
    {"captured_at_ms": 1784370600100, "record": 57}
  ],
  "next": {"captured_at_ms": 1784370600100, "record": 57}
}
```

Fetch the next page with all three cursor fields from the first response:

```http
GET /v1/indexes/browser-session-42/events?from=2026-07-18T10:00:00Z&until=2026-07-18T11:00:00Z&limit=100&through_record=1200&after_captured_at_ms=1784370600100&after_record=57
```

`through_record` prevents newly indexed source records from appearing halfway through one paginated query. In worker-pool mode, the visible and durable watermarks are the same contiguous S3-published prefix, and completed ranges beyond a gap remain invisible until the missing range commits. Resolve returned record ordinals through Ursula to fetch the canonical event values.

Registration captures the source's current retained boundary as `indexed_from_record`, so browser telemetry can be enabled for an existing stream after record 0 has been trimmed. The registration response, `/status`, query JSON, and the `indexed-from-record` query header make that lower bound explicit. An invalid query returns `400`. A deterministically invalid source record or conflicting event time blocks further advancement for that registration. Its existing durable prefix remains queryable and `/status` reports the blocked record until an operator repairs the source and calls `POST /v1/indexes/{id}/status/resume`. If retention later advances past an unfinished registered range, the resulting `409` cannot be resumed because those records are no longer recoverable.

---

# Streams

A stream is an append-only byte sequence addressed by a URL: `/{bucket}/{stream}`. Once data is written it cannot be modified or reordered. Only new data can be appended.

## Naming

Stream IDs may contain any byte that is not `/`, `\0`, or `..`. Maximum stream ID length is 122 bytes, and the combined bucket-plus-stream path is also capped at 122 bytes. The literal name `streams` is reserved (it's the bucket-level listing endpoint).

A workspace app might use `/workspace-a/doc-123`. An agent system might use `/agents/run-2026-05-13-abc`. Anything within the rules above is fair.

## Content type

A stream's content type is set on creation (PUT) or on first append (POST) and defaults to `application/octet-stream` if no `Content-Type` header is supplied. Subsequent appends must declare the same content type, or the server rejects them with `400`. The content type rides along on reads so clients can dispatch on it without inspecting payloads.

## Lifecycle

Every stream is in one of these states:

- **Open.** Accepts appends. The default on creation.
- **Closed.** Sealed by `Stream-Closed: true` on a POST. Readers receive an EOF signal. Further appends return `409`. Close is irreversible.
- **Expired.** Past its `Stream-TTL` or `Stream-Expires-At` deadline. Reads and writes return `404`. Expired streams are eventually garbage-collected.
- **Deleted.** Removed by `DELETE`. Reads return `404`.

`Stream-TTL` (seconds, relative) and `Stream-Expires-At` (RFC 3339 absolute) are mutually exclusive. Sending both yields `400`.

## Related

- [Buckets](/docs/concepts/buckets): how streams are grouped
- [Snapshots](/docs/concepts/snapshots): compacting a long stream
- [Bootstrap](/docs/concepts/bootstrap): efficient first-load for new clients

---

# Buckets

Streams are organized into buckets. A bucket is a namespace, like a folder that holds related streams under one URL prefix:

```
/{bucket}/{stream}
```

A collaborative editing app might use one bucket per workspace, with one stream per document: `/workspace-a/doc-1`, `/workspace-a/doc-2`. An agent platform might use one bucket per tenant: `/tenant-acme/run-xyz`.

## Naming

Bucket IDs match the regex `[a-z0-9_-]{4,64}`: 4–64 bytes, lowercase ASCII letters and digits plus `_` and `-`. Uppercase letters, `/`, `.`, and most punctuation are rejected with `400`.

The combined bucket-plus-stream URL path also has a 122-byte ceiling. With a 64-byte bucket name, you have 58 bytes left for the stream ID.

## Lifecycle

- **`PUT /{bucket}`** creates the bucket. Idempotent if it already exists.
- **`GET /{bucket}`** returns metadata.
- **`DELETE /{bucket}`** removes the bucket but only when empty. If any stream still exists, `DELETE` returns `409 Conflict` with `bucket_not_empty`. There is no cascading delete. Remove streams first.
- **`GET /{bucket}/streams`** lists streams in the bucket with optional prefix filtering and cursor-based pagination (page size up to 1000, default 1000).

## Related

- [Streams](/docs/concepts/streams)
- [API: list streams](/docs/api/list-streams)

---

# Offsets

An offset is a position inside a stream. Clients use offsets to read from a specific point or resume where they left off.

## Format

Offsets are numeric (`u64` byte positions internally) but are returned to clients as 20-character zero-padded decimal strings, for example `"00000000000000000042"`, so they sort lexicographically. Treat them as opaque tokens: read the value from the server's response and pass it back unchanged.

Two special values are accepted on read requests:

- `offset=-1`: start from the very beginning of retained data (the earliest still-available offset).
- `offset=now`: start from the stream's current tail. Useful for "only new data" subscriptions.

## Response headers

After every read or append, the server returns two related headers:

- **`Stream-Next-Offset`**: always present. The numeric position to use for the next request.
- **`Stream-Cursor`**: set on live (long-poll, SSE) responses. An opaque token that bundles the stream identity and epoch with the offset, so a reconnecting client lands on the same stream version it was reading before.

For pure catch-up reads, `Stream-Next-Offset` is enough. For live tailing across reconnects, prefer `Stream-Cursor` (passed as `?cursor=`). It surfaces stream re-creation as a clean error rather than silently re-reading new data under the same name.

## Stability across snapshots

Offsets are byte positions, not sequence numbers, and they're stable across snapshot publishes. Publishing a snapshot at offset 100 does not renumber later offsets or delete earlier data. A separate retention advance makes data before offset 100 eligible for garbage collection. Reads to trimmed offsets return `410 Gone` with a `stream-earliest-offset` header pointing at the first still-available position.

## Related

- [Read modes](/docs/concepts/read-modes)
- [Record Coordinates](/docs/concepts/record-coordinates): complete JSON values with stable, arithmetic-friendly ordinals
- [Snapshots](/docs/concepts/snapshots)
- [Bootstrap](/docs/concepts/bootstrap): recovery when offsets you remember have been trimmed

---

# Record Coordinates

JSON streams can expose a stable record number for every committed JSON value. Use records when your application thinks in events or messages and needs to replay complete values, count progress, or resume without parsing an opaque byte offset.

Record Coordinates are available only on `application/json` streams that advertise `json-record-coordinates-v1` in the `Stream-Extensions` response header.

## Three different values

| Value | Assigned by | Meaning | Client arithmetic |
| --- | --- | --- | --- |
| Offset | Ursula | Exact position in the canonical byte stream | No, pass it back unchanged |
| Record | Ursula | Zero-based JSON value number in commit order | Yes, records occupy ranges such as `[42, 44)` |
| `captured_at` | Your application | When the event happened at the source | Query through an external event-time index |

Records and offsets identify the same committed stream order at different levels. A record points to a complete JSON value, and an offset points to a byte boundary. `captured_at` is ordinary JSON data and may be out of order after retries, offline buffering, or backfill. It never changes the committed record order.

```json
{
  "captured_at": "2026-07-18T10:30:00.100Z",
  "type": "network_response",
  "status": 500
}
```

Ursula does not create a second server timestamp or reorder this event by `captured_at`.

## Append and receive a record range

A top-level JSON array is normalized into one record per array element. If the current record tail is `42`, this append creates records `42` and `43`:

```bash
curl -i -X POST http://127.0.0.1:4437/demo/events \
  -H 'Content-Type: application/json' \
  --data-binary '[
    {"captured_at":"2026-07-18T10:30:00.100Z","type":"navigation"},
    {"captured_at":"2026-07-18T10:29:59.900Z","type":"network_response"}
  ]'
```

The successful response includes:

```http
Stream-Extensions: json-record-coordinates-v1
Stream-Record-Start: 42
Stream-Record-Next: 44
Stream-Next-Offset: <opaque-offset>
```

The half-open range `[42, 44)` is safe to store, compare, and count. A deduplicated producer retry returns the original range.

## Replay complete records

Read up to 100 complete records beginning at record `42`:

```bash
curl -i 'http://127.0.0.1:4437/demo/events?record=42&max_records=100'
```

The body is NDJSON. Continue with the returned `Stream-Record-Next`:

```http
Stream-Record-First: 0
Stream-Record-Start: 42
Stream-Record-Next: 44
Stream-Next-Offset: <opaque-offset>
```

Use `record=now` to wait only for future records, or `tail_records=100` to read the most recent retained records. Add `live=long-poll` or `live=sse` for live delivery.

When a consumer needs the ordinal beside each value, request the envelope view:

```bash
curl 'http://127.0.0.1:4437/demo/events?record=42&max_records=100&record_view=envelope'
```

```json
{"record":42,"value":{"captured_at":"2026-07-18T10:30:00.100Z","type":"navigation"}}
{"record":43,"value":{"captured_at":"2026-07-18T10:29:59.900Z","type":"network_response"}}
```

If retention has removed the requested record, Ursula returns `410 Gone` and reports the new first retained ordinal in `Stream-Record-First`. Surviving records are never renumbered.

## Conditional append by record tail

Use `Stream-Record-Match` when a writer should append only if no other record has committed since it last read the stream:

```bash
curl -i -X POST http://127.0.0.1:4437/demo/events \
  -H 'Content-Type: application/json' \
  -H 'Stream-Record-Match: 44' \
  --data-binary '{"captured_at":"2026-07-18T10:31:00Z","type":"checkpoint"}'
```

A mismatch returns `412 Precondition Failed` with the current `Stream-Record-Next`.

## Query by captured time

`captured_at` is not a Durable Streams coordinate, so `GET ?captured_at=...` is intentionally not part of the stream protocol. The optional `ursula-indexer` tails the envelope view, builds a rebuildable S3-backed event-time index, and returns record ordinals for a time range. Callers can then resolve those records through Ursula.

The index reports `indexed_through_record` and `durable_through_record`. Pagination pins one `through_record` watermark so newly indexed events do not move a query between pages. See the [Browser Telemetry example](/docs/examples/browser-telemetry) for the complete setup and query flow.

## Related

- [Offsets](/docs/concepts/offsets)
- [Append API](/docs/api/append)
- [Read API](/docs/api/read)
- [Browser Telemetry](/docs/examples/browser-telemetry)

---

# Read Modes

Streams support three read modes, all via `GET`:

- **Catch-up**: `GET /b/s?offset=-1` returns all data from the given offset immediately. Use this to sync historical data.
- **Long-poll**: `GET /b/s?offset=...&live=long-poll` returns immediately if data is available, otherwise holds the connection until new data arrives or a timeout. Good for simple polling loops.
- **SSE**: `GET /b/s?offset=...&live=sse` opens a persistent Server-Sent Events connection. The server pushes new data as it arrives. This is the recommended mode for real-time frontends (`EventSource` in the browser works out of the box).

---

# Exactly-Once Writes

Network retries can produce duplicate appends. Ursula provides server-side deduplication so your application doesn't need its own bookkeeping. Three headers form the identity:

- **`Producer-Id`**: a stable client identifier (UUID, hostname, etc.).
- **`Producer-Epoch`**: bumped on producer restart. Must be ≥ the last epoch the server saw from this producer.
- **`Producer-Seq`**: a per-epoch sequence number. Starts at `0` for a new epoch and must increase by exactly `1` per append.

Both epoch and seq are capped at `2^53 − 1` so they survive a round-trip through JSON.

## How dedup works

The server records the exact receipt for every accepted sequence in the current epoch from each `Producer-Id` per stream. On an append:

- **Delayed duplicate** (same epoch and any seq already accepted): silently deduplicated. The response carries that sequence's original byte and record ranges, even when newer sequences have committed.
- **Next in sequence** (`seq = last_seq + 1`): accepted.
- **Out of order** (seq skips ahead or goes backward): rejected with `409 producer_seq_conflict`. The response body includes the `expected_seq` the server wanted.
- **Epoch regression** (new epoch less than last accepted): rejected with `409`.

## Restart and epoch hygiene

When a producer crashes and restarts, **bump `Producer-Epoch`**. Otherwise the server still expects the next contiguous seq within the old epoch, and the producer (which has lost its in-memory seq counter) will collide. A new epoch starts the seq counter back at `0`.

## Per-stream state

Dedup state is scoped per stream. The same `Producer-Id` writing to two different streams maintains two independent epoch/seq counters, so you do not have to elect a single global writer per producer.

The server retains exact receipts for the current epoch per `(producer_id, stream)` pair. Older epochs and their receipts are dropped once a higher epoch is observed.

## When you don't need this

For append-only workloads where each write is independent (event logging, audit trails), exactly-once headers are optional. Just POST without them. Add them when retries are real and double-applies would be visible.

## Related

- [Conditional writes](/docs/concepts/conditional-writes): coordinate multiple writers, different problem
- [API: append](/docs/api/append)

---

# Conditional Writes

Writers appending to the same stream sometimes need coordination. Ursula supports two complementary request-header guards without introducing locks or cross-stream transactions.

## Stream-Seq

`Stream-Seq` is a client-supplied monotonic sequence token. The server tracks the last accepted value per stream and rejects any append whose `Stream-Seq` is not lexicographically greater than the previous one.

This is useful when a single logical writer wants to enforce ordering without relying on server-side ETags. For instance, an agent that numbers its steps and wants the server to reject out-of-order delivery.

## Stream-Record-Match

`Stream-Record-Match` is an Ursula extension for JSON streams using [Record Coordinates](/docs/concepts/record-coordinates). The server accepts the append only when the current record tail equals the supplied ordinal.

Use it when multiple writers may race after reading the same stream state:

```http
POST /demo/events
Content-Type: application/json
Stream-Record-Match: 44

{"type":"checkout"}
```

A mismatch returns `412 Precondition Failed` with the current `Stream-Record-Next`, allowing the client to re-read and retry.

## When to use

- **Stream-Seq**: single-writer ordering. "Reject this if my writes arrive out of order."
- **Stream-Record-Match**: multi-writer optimistic concurrency on a JSON stream. "Append this only if nobody has committed another record since I read."
- **Producer-Id / Producer-Epoch / Producer-Seq** ([exactly-once writes](/docs/concepts/exactly-once-writes)): deduplicate retries from a producer that may resend the same logical append after a network hiccup or restart.
- **Neither**: append-only workloads where every write is independent (e.g. event logging). Just POST.

---

# Snapshots

As a stream grows, replaying it from `offset=-1` becomes expensive. Snapshots solve this by storing a compacted representation of all data up to a given offset.

```
PUT    /{bucket}/{stream}/snapshot/{offset}    publish a snapshot
PUT    /{bucket}/{stream}/retention/{offset}   advance retained history
GET    /{bucket}/{stream}/snapshot             read the latest snapshot
GET    /{bucket}/{stream}/snapshot/{offset}    read a specific snapshot
```

## Who publishes

Snapshots are an **application-level** operation. Ursula stores them but does not produce them. The writer that knows how to compute the merged state is responsible for publishing. A CRDT editor, for example, periodically computes the merged document and publishes it as the latest snapshot. An agent system might snapshot accumulated tool-call state.

Application snapshots are separate from the Raft-internal snapshots Ursula takes for replication and recovery. The two never collide.

## Size and content type

Snapshot bodies are capped at **128 MiB**. The snapshot has its own content type (separate from the stream's), stored as `snapshot_content_type` and defaulting to `application/octet-stream`. A binary CRDT document can be snapshotted under `application/octet-stream` even when the stream itself carries `application/json` deltas.

## Checkpoint and retention

Publishing a snapshot is safe on its own: it creates a replay checkpoint but leaves all stream history readable. After the application has verified the checkpoint, it may explicitly advance retention to the same or an older checkpoint boundary with `PUT /{bucket}/{stream}/retention/{offset}`. Only that operation makes earlier reads return `410 Gone` and makes old cold-tier data eligible for GC.

JSON streams also accept record coordinates: `PUT /{bucket}/{stream}/snapshot?record=N` and `PUT /{bucket}/{stream}/retention?record=N`.

## Immutability

Snapshots are immutable at an offset. Retrying the same body and content type is idempotent and returns the same `Stream-Snapshot-Digest`; different content at the same offset returns `409`. `Stream-Snapshot-Match` can conditionally publish against the current digest. The `DELETE /{bucket}/{stream}/snapshot/{offset}` endpoint exists for protocol-level completeness, but Ursula refuses every delete: targeting the latest snapshot returns `409`, anything else returns `404`. To replace a snapshot, publish a new one at a higher offset.

## Related

- [Bootstrap](/docs/concepts/bootstrap): fetch latest snapshot plus post-snapshot updates in one request
- [API: publish snapshot](/docs/api/publish-snapshot)
- [API: read snapshot](/docs/api/read-snapshot)

---

# Bootstrap

Bootstrap is the new-client initialization endpoint. A single request returns everything a client needs to catch up:

1. The latest snapshot (if any)
2. All updates after the snapshot, up to the stream's current tail

```
GET /{bucket}/{stream}/bootstrap
```

The response is `multipart/mixed`:

- **First part.** The snapshot. If no snapshot exists, this part is present but empty.
- **Subsequent parts.** Incremental updates from the snapshot offset (or from the earliest retained offset, if no snapshot exists).

Each part has its own `Content-Type`: the snapshot part uses `snapshot_content_type`. Update parts use the stream's content type. The client does not need to know in advance whether a snapshot exists. The same parsing path handles both cases.

## When to use bootstrap vs catch-up

| You want… | Use |
| --- | --- |
| Latest state, fastest first-load | `/bootstrap` |
| Replay every byte from the beginning | `GET /{b}/{s}?offset=-1` |
| Resume from a known offset that's still retained | `GET /{b}/{s}?offset=N` |
| Resume from an offset the server has since trimmed (`410 Gone`) | `/bootstrap`, then continue from the snapshot offset |

## Live continuation

Bootstrap does not accept `?live=sse`. Combining the multipart body with an SSE event stream is rejected with `400`. To go live after bootstrap, finish reading the bootstrap response, then open a separate `GET /{b}/{s}?offset=<snapshot_offset>&live=sse` (or pass the `cursor` returned by bootstrap).

## Related

- [Snapshots](/docs/concepts/snapshots)
- [Read modes](/docs/concepts/read-modes)

---

# Durability and Consistency

## Durability

An append is acknowledged once a **majority of voters** has replicated it. A single node failure cannot lose an acknowledged write. Acknowledged data lives in replicated hot state across the cluster, and a background worker flushes older segments to S3 on a configurable interval, after which the data inherits S3-grade durability.

In a cross-region deployment with a 5-second flush interval, per-message durability is approximately 9-10 nines. The window where acknowledged-but-unflushed data is at risk from a simultaneous multi-region failure is measured in seconds.

## Consistency

Writes are **linearizable**. Each Raft group serializes appends through its current leader, which assigns a total order.

Catch-up reads may be served by any replica that has applied the relevant stream state. Followers can return already replicated historical bytes locally because stream positions are immutable. Tail-sensitive reads still preserve protocol-visible semantics: `HEAD`, `Stream-Up-To-Date: true`, `Stream-Closed: true`, and `offset=now` are generated by the leader path unless the follower has applied a terminal closed state. Requests that need a write-side access transition, such as expiry or TTL touch, are also routed to the leader.

`Stream-Up-To-Date` on each read tells the client whether more committed data exists past `next_offset`. `false` means keep paging.

## Availability and durability (standard layout)

Three voting replicas across availability zones, plus two non-voting replicas in a second region.

| Property | What it means |
|---|---|
| **Write availability** | Writes continue as long as a majority of voters is healthy. The layout tolerates any single voting-AZ failure. Non-voting replicas hold extra copies but do not vote. |
| **Read availability** | Any reachable replica can serve replicated historical catch-up data locally. Reads that need fresh tail metadata, leader-owned state changes, or live watcher ownership still track write availability per group, not per cluster. |
| **Per-message durability** | ~9-10 nines with a 5-second flush interval. |
| **Annual zero-loss probability** | ~3-4 nines. Probability of no data-loss events in a year. |

---

# Binary SSE

The SSE wire format is text-only: every `data:` field must be valid UTF-8. Ursula therefore advertises the decoded data payload type with `Stream-Data-Content-Type` on every SSE read.

For `text/*` streams, data events carry UTF-8 text directly.

For `application/json` streams, data events carry newline-delimited JSON and the response includes `Stream-Data-Content-Type: application/x-ndjson`. SSE `data:` lines are transport framing, not JSON record boundaries, so clients should buffer until newline before parsing records.

For other content types (`application/octet-stream`, custom binary), Ursula base64-encodes the data event payload as raw text. It is not wrapped in a JSON envelope:

```text
event: data
data: AQIDBAUG
data: BwgJCg==

event: control
data: {"streamNextOffset":"123456_789","streamCursor":"abc"}
```

The choice is automatic, determined by the stream's content type. There is no client opt-in. What you get is what the stream's content type implied. The response advertises base64 data with `Stream-Sse-Data-Encoding: base64`. When that header is absent, the `data` event payload is already UTF-8 text with the type identified by `Stream-Data-Content-Type`.

## Decoding

For binary streams, concatenate the `data:` lines for an `event: data`, remove line breaks inserted by SSE framing, then base64-decode the resulting text. Interpret the decoded bytes according to `Stream-Data-Content-Type`.

For text and JSON streams, the `data` field is already the payload text. JSON streams use newline-delimited JSON (`application/x-ndjson`), so buffer until newline before parsing.

Browser `EventSource` works for both modes. Only the `data` handler differs.

## Related

- [Read modes](/docs/concepts/read-modes)
- [API: read](/docs/api/read)
- [Length-prefixed framing](/docs/concepts/len-prefixed-framing): when one SSE event contains multiple application records

---

# Length-Prefixed Framing

Streams are raw byte sequences. The protocol does not impose message boundaries. Every append and every read returns whatever bytes are in flight, possibly mid-record. For streams carrying many small messages (CRDT operations, agent events), Ursula recommends a simple framing convention:

```
[4-byte big-endian length][payload bytes]
```

Each application-level append writes one framed record. Each read returns a concatenation of framed records that the client parses incrementally.

## Server is frame-agnostic

The server **does not** validate or enforce frames. It stores and serves raw bytes. That means:

- A read may end mid-frame. The client must handle partial frames and resume parsing on the next read.
- An ill-formed frame (wrong length, truncated payload) is the producer's bug, not a server-rejectable error.
- The framing convention is per-application. Writers and readers of a stream must agree on it.

## When you don't need framing

- Streams using `application/json` typically use newline-delimited JSON or a single self-describing document. JSON has its own delimiters.
- Streams holding one logical blob per stream don't need framing at all.

Use framing when one stream carries many independently meaningful messages and the client wants to dispatch on each one without scanning.

## Related

- [Streams](/docs/concepts/streams)
- [Binary SSE](/docs/concepts/binary-sse): over SSE, each event is one delivery. Framing lets one event carry multiple records.

---

# API overview

Ursula exposes a small public HTTP API for durable append-only streams. Most users only need the `/{bucket}/{stream}` route family, which maps cleanly to buckets and stream IDs.

This API is Ursula's implementation of the [Durable Streams Protocol](/docs/specs/durable-stream), defined by the durable-streams community, with additional route families for compatibility and deployment needs.

## Bucket operations

| Method | Path | Description |
| -------- | ----------------------------- | ------------------------------------------------------------ |
| `PUT` | `/{bucket}` | [Create a bucket](/docs/api/create-bucket) |

Bucket-level `GET` (metadata), `DELETE`, and `GET /{bucket}/streams` (list) are part of the Durable Streams Protocol but are not yet implemented in Ursula. Track stream existence at the application layer for now.

## Stream operations

| Method | Path | Description |
| -------- | ----------------------------- | ------------------------------------------------------------ |
| `PUT` | `/{bucket}/{stream}` | [Create a stream](/docs/api/create-stream) |
| `POST` | `/{bucket}/{stream}` | [Append data or close](/docs/api/append) |
| `POST` | `/{bucket}/{stream}/append-batch` | Batched append. See [extensions spec](/docs/specs/extensions#append-batch) |
| `GET` | `/{bucket}/{stream}` | [Read (catch-up, long-poll, SSE)](/docs/api/read) |
| `HEAD` | `/{bucket}/{stream}` | [Get stream metadata](/docs/api/head-stream) |
| `GET` / `PUT` | `/{bucket}/{stream}/attrs` | [Read or replace stream attributes](/docs/api/stream-attrs) |
| `DELETE` | `/{bucket}/{stream}` | [Delete a stream](/docs/api/delete-stream) |

## Bootstrap and snapshots

| Method | Path | Description |
| -------- | --------------------------------------------- | ---------------------------------------------------- |
| `GET` | `/{bucket}/{stream}/bootstrap` | [Snapshot + tail replay](/docs/api/bootstrap) |
| `GET` | `/{bucket}/{stream}/snapshot` | [Read latest snapshot](/docs/api/read-snapshot) |
| `GET` | `/{bucket}/{stream}/snapshot/{offset}` | [Read snapshot at offset](/docs/api/read-snapshot) |
| `PUT` | `/{bucket}/{stream}/snapshot/{offset}` | [Publish a snapshot](/docs/api/publish-snapshot) |

## Common request patterns

### Create a bucket and stream

```bash
curl -X PUT http://127.0.0.1:4437/demo
curl -X PUT http://127.0.0.1:4437/demo/hello
```

### Append data

```bash
curl -X POST http://127.0.0.1:4437/demo/hello \
  -H 'Content-Type: application/octet-stream' \
  --data-binary 'hello world'
```

### Read from the beginning

```bash
curl 'http://127.0.0.1:4437/demo/hello?offset=-1'
```

### Subscribe with SSE

```bash
curl 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse'
```

## Related concepts

- [Read modes](/docs/concepts/read-modes): catch-up vs long-poll vs SSE
- [Bootstrap](/docs/concepts/bootstrap): snapshot + tail recovery
- [Snapshots](/docs/concepts/snapshots): snapshot lifecycle
- [Exactly-once writes](/docs/concepts/exactly-once-writes): producer deduplication
- [Conditional writes](/docs/concepts/conditional-writes): sequence and JSON record-tail guards

---

# Create bucket

<ParamField path="bucket" type="string" required>
 Bucket ID. The Durable Streams Protocol specifies `[a-z0-9_-]{4,64}`. Client-side conformance is recommended even though Ursula does not currently enforce the regex.
</ParamField>

## Response

| Status | Meaning |
| ------ | -------------------------------------- |
| `201` | Bucket created (idempotent - Ursula returns `201` whether or not the bucket existed before). |

> [!NOTE]
> In the current Ursula implementation `PUT /{bucket}` is a no-op acknowledgement and always returns `201`. Bucket existence is implicit. Streams created under any bucket name will succeed. Validation of the bucket ID (`400`) and conflict detection (`409`) are part of the Durable Streams Protocol but are not yet wired up here.

<RequestExample>
```bash Example
curl -X PUT http://127.0.0.1:4437/demo
```
</RequestExample>

---

# Create stream

<ParamField path="bucket" type="string" required>
 Bucket ID. The Durable Streams Protocol specifies `[a-z0-9_-]{4,64}`. Ursula does not currently enforce this regex but client-side conformance is recommended.
</ParamField>

<ParamField path="stream" type="string" required>
 Stream ID within the bucket. Cannot contain `\0` and segments cannot equal `..`. Note: Ursula does not currently enforce the Durable Streams Protocol's 122-byte stream-ID ceiling - clients should still respect it for conformance.
</ParamField>

<ParamField header="Content-Type" type="string">
 Content type of the initial payload (e.g. `application/json`). Becomes the stream's content type.
</ParamField>

<ParamField header="Stream-Closed" type="string">
 Set to `true` to close the stream immediately after creation.
</ParamField>

<ParamField header="Stream-TTL" type="string">
 Time-to-live in seconds. The stream will expire after this duration.
</ParamField>

<ParamField header="Stream-Expires-At" type="string">
 Absolute expiration timestamp (RFC 3339). Mutually exclusive with `Stream-TTL`.
</ParamField>

<ParamField header="Stream-Seq" type="string">
 Client-supplied monotonic sequence token. Rejects creates whose `Stream-Seq` is not lexicographically greater than the previous value seen for this stream.
</ParamField>

<ParamField header="Stream-Attrs" type="string">
 JSON stream attributes. See [stream attributes](/docs/api/stream-attrs). If the stream already exists, the submitted attributes must match the stored attributes for the create request to be idempotent.
</ParamField>

<ParamField header="Producer-Id" type="string">
 Producer identity for [exactly-once writes](/docs/concepts/exactly-once-writes).
</ParamField>

<ParamField header="Producer-Epoch" type="string">
 Producer epoch (must accompany `Producer-Id`).
</ParamField>

<ParamField header="Producer-Seq" type="string">
 Producer sequence number (must accompany `Producer-Id`).
</ParamField>

<ParamField body="body" type="binary">
 Optional initial payload. If provided, becomes the first entry in the stream.
</ParamField>

## Response

| Status | Meaning |
| ------ | ------------------------------------------------------------------------- |
| `201` | Stream created. |
| `200` | Stream already exists (idempotent). |
| `400` | Invalid stream ID, invalid headers, or bad JSON payload. |
| `409` | Stream already exists with different content type, or sequence conflict. |

Response headers include `Location`, `Content-Type`, `Stream-Next-Offset`, and lifetime headers (`Stream-TTL` / `Stream-Expires-At`) when set. `Stream-Closed: true` is set if the create request also closed the stream. `ETag` is set on reads only.

When the initial payload creates records in an `application/json` stream advertising `json-record-coordinates-v1`, the response also includes `Stream-Record-Start` and `Stream-Record-Next`. A top-level JSON array creates one record per element.

<RequestExample>
```bash Create empty stream
curl -X PUT http://127.0.0.1:4437/demo/hello
```

```bash Create with initial payload
curl -X PUT http://127.0.0.1:4437/demo/hello \
  -H 'Content-Type: application/json' \
  --data-binary '{"msg": "first entry"}'
```

```bash Create with TTL
curl -X PUT http://127.0.0.1:4437/demo/ephemeral \
  -H 'Stream-TTL: 3600'
```

```bash Create with attributes
curl -X PUT http://127.0.0.1:4437/demo/session-1 \
  -H 'Stream-Attrs: {"title":"Support session","metadata":{"purpose":"customer-support"}}'
```

</RequestExample>

> [!NOTE]
> If a stream with the same ID already exists and has identical configuration, the response is `200 OK` (idempotent). If the existing stream differs in content type, closed state, retention, or stream attributes, the response is `409 Conflict`.

> [!NOTE]
> See [Record Coordinates](/docs/concepts/record-coordinates) for JSON record ranges and complete-record replay.

---

# Append

<ParamField path="bucket" type="string" required>
 Bucket ID.
</ParamField>

<ParamField path="stream" type="string" required>
 Stream ID within the bucket.
</ParamField>

<ParamField header="Content-Type" type="string" required>
 Must match the stream's content type (set at PUT or first POST). Required when the body is non-empty. Mismatch returns `400`.
</ParamField>

<ParamField header="Stream-Closed" type="string">
 Set to `true` to close the stream after this append.
</ParamField>

<ParamField header="Stream-Seq" type="string">
 Client-supplied monotonic sequence token. The server tracks the last accepted value per stream and rejects any append whose `Stream-Seq` is not lexicographically greater than the previous one.
</ParamField>

<ParamField header="Producer-Id" type="string">
 Stable producer identity (UUID, hostname, etc.) for [exactly-once writes](/docs/concepts/exactly-once-writes). Dedup state is per-stream.
</ParamField>

<ParamField header="Producer-Epoch" type="string">
 Producer epoch. Bumped on producer restart. Must be ≥ the last epoch the server accepted for this `Producer-Id`. A new epoch resets the seq counter. Max value `2^53 − 1`.
</ParamField>

<ParamField header="Producer-Seq" type="string">
 Producer sequence number. Starts at `0` for a new epoch and must increase by exactly `1` per append. Exact `(epoch, seq)` duplicates are silently deduplicated. Max value `2^53 − 1`.
</ParamField>

<ParamField header="Stream-Record-Match" type="string">
 For JSON streams with Record Coordinates, require the current record tail to equal this ordinal. A mismatch returns `412` with the current `Stream-Record-Next`.
</ParamField>

<ParamField body="body" type="binary" required>
 The bytes to append. Must not be empty unless `Stream-Closed: true` is set (close-only request).
</ParamField>

## Response

| Status | Meaning |
| ------ | ------------------------------------------------------------------ |
| `204` | Append successful (default success response, no body). |
| `200` | Append successful with body - returned when a `Producer-Id` was supplied and the append was not deduplicated, so the response carries producer ack headers. |
| `400` | Empty body without `Stream-Closed: true`, missing content type, or bad JSON. |
| `404` | Stream not found. |
| `409` | Stream is already closed, or sequence/producer conflict. |
| `412` | `Stream-Record-Match` did not match the current JSON record tail. |
| `503` | Cold-write backpressure. Retry after the duration in `Retry-After`. |

Response headers include `Stream-Next-Offset` (always). When a `Producer-Id` was supplied, the server echoes the accepted `Producer-Epoch` and `Producer-Seq` so the producer can confirm what was durably recorded. `Stream-Closed: true` is set if this request closed the stream. `ETag` is set on reads only, not on appends.

For an `application/json` stream advertising `json-record-coordinates-v1`, the response also includes `Stream-Record-Start` and `Stream-Record-Next`. A JSON object creates one record, and a top-level array creates one record per element. The returned half-open range identifies exactly the records created by this request.

<RequestExample>
```bash Append binary
curl -X POST http://127.0.0.1:4437/demo/hello \
  -H 'Content-Type: application/octet-stream' \
  --data-binary 'hello world'
```

```bash Append JSON
curl -i -X POST http://127.0.0.1:4437/demo/hello \
  -H 'Content-Type: application/json' \
  --data-binary '{"captured_at":"2026-07-18T10:30:00.100Z","event":"click"}'
```

```bash Append multiple JSON records
curl -i -X POST http://127.0.0.1:4437/demo/hello \
  -H 'Content-Type: application/json' \
  --data-binary '[{"event":"click"},{"event":"navigation"}]'
```

```bash Close a stream
curl -X POST http://127.0.0.1:4437/demo/hello \
  -H 'Stream-Closed: true'
```
</RequestExample>

> [!NOTE]
> Appends to JSON streams are validated and normalized. The server may coalesce multiple concurrent appends into a single batch for performance.

> [!NOTE]
> See [Record Coordinates](/docs/concepts/record-coordinates) for record ranges, complete-record reads, and the difference between committed record order and client event time.

---

# Read stream

<ParamField path="bucket" type="string" required>
 Bucket ID.
</ParamField>

<ParamField path="stream" type="string" required>
 Stream ID within the bucket.
</ParamField>

<ParamField query="offset" type="string">
 Starting offset. Use `-1` to read from the beginning, or a numeric offset.
</ParamField>

<ParamField query="cursor" type="string">
 Opaque cursor token returned by a previous read. Alternative to `offset`.
</ParamField>

<ParamField query="stream-cursor" type="string">
 Alias for `cursor`.
</ParamField>

<ParamField query="record" type="string">
 For a JSON stream with Record Coordinates, start at a complete record boundary. Supply a zero-based ordinal or `now` for the current record tail. Mutually exclusive with `offset` and `tail_records`.
</ParamField>

<ParamField query="tail_records" type="number">
 Start at the most recent retained records: `max(first_record, next_record - count)`. Mutually exclusive with `record` and `offset`.
</ParamField>

<ParamField query="max_records" type="number">
 Maximum number of complete records to return. Requires `record` or `tail_records` and cannot be combined with `max_bytes`.
</ParamField>

<ParamField query="record_view" type="string">
 Set to `envelope` with a record-aware start to return one `{record, value}` object per NDJSON line.
</ParamField>

<ParamField query="live" type="string">
 Live mode: `sse` for Server-Sent Events, `long-poll` for long-polling. Omit for catch-up read.
</ParamField>

<ParamField query="max_bytes" type="number">
 Maximum bytes to read in one response or SSE data batch. For UTF-8 SSE data, Ursula may shorten the emitted batch so `Stream-Next-Offset` lands on a valid text boundary.
</ParamField>

## Read modes

<Tabs>
 <Tab title="Catch-up">
 No `live` parameter. Returns all available data from the given offset immediately.

    ```bash
    curl 'http://127.0.0.1:4437/demo/hello?offset=-1'
    ```
 </Tab>
 <Tab title="Long-poll">
 `live=long-poll`. Returns immediately if data is available, otherwise holds the connection until new data arrives or a ~3 second timeout.

    ```bash
    curl 'http://127.0.0.1:4437/demo/hello?offset=42&live=long-poll'
    ```
 </Tab>
 <Tab title="SSE">
 `live=sse`. Opens a persistent Server-Sent Events connection. The server pushes data events as new entries are appended. Includes periodic heartbeat comments.

    ```bash
    curl 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse'
    ```
 </Tab>
</Tabs>

## Response

| Status | Meaning |
| ------ | ------------------------------------------------------------- |
| `200` | Data returned (catch-up or long-poll with data). |
| `204` | No new data at the requested offset (catch-up only). |
| `400` | Invalid offset or live mode. |
| `404` | Stream not found or expired. |
| `410` | Requested offset has been trimmed (data no longer available). |

Response headers include `Stream-Next-Offset`, `Stream-Cursor`, `ETag`, `Stream-Up-To-Date`, `Stream-Closed`, and `Content-Type`.

Record-aware responses also advertise `json-record-coordinates-v1` in `Stream-Extensions` and include `Stream-Record-First`, `Stream-Record-Start`, and `Stream-Record-Next`. Continue with `record=<Stream-Record-Next>`. A record below `Stream-Record-First` returns `410`, and a record beyond the current tail returns `400`.

Streams written with `Content-Type: application/json` are returned as newline-delimited JSON with response `Content-Type: application/x-ndjson`. `max_bytes` applies to the encoded byte stream, so a response can end mid-line. Resume from `Stream-Next-Offset` and buffer any incomplete trailing line before parsing. `HEAD` reports the configured stream content type, while read responses report the wire representation.

## SSE event format

In SSE mode, the server sends:

- **Data events** (`event: data`): stream payload in the `data` field. The response `Stream-Data-Content-Type` header identifies the data payload type (`application/x-ndjson` for JSON streams, the original content type for other streams). SSE `data:` lines are transport lines, not guaranteed message boundaries. JSON clients should buffer until newline before parsing records. For binary streams, data is base64-encoded (controlled by the `Stream-Sse-Data-Encoding` header).
- **Control events** (`event: control`): JSON metadata including the current offset and stream state.
- **Heartbeat comments**: periodic `:` lines to keep the connection alive through proxies.

<RequestExample>
```bash Catch-up
curl 'http://127.0.0.1:4437/demo/hello?offset=-1'
```

```bash Long-poll
curl 'http://127.0.0.1:4437/demo/hello?offset=42&live=long-poll'
```

```bash SSE tail
curl 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse'
```

```bash Read complete JSON records
curl -i 'http://127.0.0.1:4437/demo/hello?record=42&max_records=100'
```

```bash Read the latest JSON records with ordinals
curl 'http://127.0.0.1:4437/demo/hello?tail_records=100&record_view=envelope'
```

```bash Tail future JSON records over SSE
curl -N 'http://127.0.0.1:4437/demo/hello?record=now&record_view=envelope&live=sse'
```
</RequestExample>

> [!NOTE]
> See [read modes](/docs/concepts/read-modes), [Record Coordinates](/docs/concepts/record-coordinates), [binary SSE](/docs/concepts/binary-sse), and [offsets](/docs/concepts/offsets) for more details.

---

# Head stream

<ParamField path="bucket" type="string" required>
 Bucket ID.
</ParamField>

<ParamField path="stream" type="string" required>
 Stream ID within the bucket.
</ParamField>

<ParamField header="If-None-Match" type="string">
 ETag for conditional request. Returns `304` if the stream state has not changed.
</ParamField>

## Response

| Status | Meaning |
| ------ | ------------------------------------------------------ |
| `200` | Stream found. |
| `304` | Stream state unchanged (when `If-None-Match` matches). |
| `404` | Stream not found or expired. |

Response headers include:

| Header | Description |
| ------------------------ | --------------------------------------------------------- |
| `Content-Type` | The stream's content type. |
| `Stream-Next-Offset` | The next writable offset (= current length). |
| `ETag` | Stream state ETag (encodes offset and open/closed state). |
| `Stream-Closed` | Present and `true` if the stream is closed. |
| `Stream-Snapshot-Offset` | Present if a snapshot exists, showing its offset. |
| `Stream-TTL` | Remaining TTL in seconds (if a TTL was set). |
| `Stream-Expires-At` | Expiration timestamp (if set). |
| `Cache-Control` | `no-store`. |

For an `application/json` stream implementing Record Coordinates, `HEAD` also includes:

| Header | Description |
| --- | --- |
| `Stream-Extensions` | Contains `json-record-coordinates-v1`. Clients must verify this before relying on record parameters or headers. |
| `Stream-Record-First` | First retained record ordinal. |
| `Stream-Record-Next` | Current record tail and next ordinal to be assigned. |

<RequestExample>
```bash Example
curl -I http://127.0.0.1:4437/demo/hello
```
</RequestExample>

> [!TIP]
> Use `If-None-Match` with a previously received `ETag` to efficiently poll for state changes without transferring data.

> [!NOTE]
> See [Record Coordinates](/docs/concepts/record-coordinates) for complete-record replay and client event-time queries.

---

# Stream attributes

Stream attributes are mutable, application-owned JSON metadata stored beside a stream. They are not part of the append-only byte stream and do not affect offsets, reads, snapshots, bootstrap, content type, or closed state.

The protocol details are defined in the [extensions spec](/docs/specs/extensions#5-stream-attributes).

<ParamField path="bucket" type="string" required>
 Bucket ID.
</ParamField>

<ParamField path="stream" type="string" required>
 Stream ID within the bucket.
</ParamField>

## Attribute object

```json
{
  "title": "Support session",
  "metadata": {
    "purpose": "customer-support",
    "environment_id": "env_019e2590d33f711fabf42f2857cecd8a",
    "agent": {
      "id": "agent_019e390add9f7bac9b6cc806db46fcbd",
      "version": 2
    }
  }
}
```

Top-level fields are optional:

| Field | Type | Description |
| ---------- | ------ | ----------------------------------- |
| `title` | string | Application-defined display title. |
| `metadata` | object | Arbitrary application metadata. |

Ursula does not define top-level fields for application concepts such as agents or environments. Store that data inside `metadata` when needed.

Unknown top-level fields are ignored and not stored. Place any additional fields inside `metadata`.

The encoded attribute object is limited to 16 KiB. Larger documents are rejected with `400 Bad Request`.

## Read attributes

```http
GET /{bucket}/{stream}/attrs
```

| Status | Meaning |
| ------ | ------------------------------------------ |
| `200` | Attributes returned as `application/json`. |
| `404` | Stream not found or expired. |
| `410` | Stream is gone. |

When no attributes are set, the response body is `{}`.

## Replace attributes

```http
PUT /{bucket}/{stream}/attrs
Content-Type: application/json

{"title":"Renamed session","metadata":{"purpose":"debugging"}}
```

`PUT` replaces the complete attribute object. It does not merge with the previous value. Submitting `{}` clears attributes.

| Status | Meaning |
| ------ | ------------------------------------------------------------- |
| `204` | Attributes replaced, cleared, or already equal to the request. |
| `400` | Missing `Content-Type`, non-JSON content type, invalid JSON, or attributes over the 16 KiB limit. |
| `404` | Stream not found or expired. |
| `410` | Stream is gone. |

Attributes can be updated after a stream is closed. Stream closure only prevents further appends.

## Create with attributes

You can set initial attributes on stream creation with the `Stream-Attrs` header:

```bash
curl -X PUT http://127.0.0.1:4437/demo/session-1 \
  -H 'Stream-Attrs: {"title":"Support session","metadata":{"purpose":"customer-support"}}'
```

If the stream already exists, the submitted attributes must match the stored attributes for the create request to return `200 OK`. A mismatch returns `409 Conflict`.

<RequestExample>
```bash Read attributes
curl http://127.0.0.1:4437/demo/session-1/attrs
```

```bash Replace attributes
curl -X PUT http://127.0.0.1:4437/demo/session-1/attrs \
  -H 'Content-Type: application/json' \
  --data-binary '{"title":"Renamed session","metadata":{"purpose":"debugging"}}'
```

```bash Clear attributes
curl -X PUT http://127.0.0.1:4437/demo/session-1/attrs \
  -H 'Content-Type: application/json' \
  --data-binary '{}'
```
</RequestExample>

---

# Publish snapshot

Publishes a new checkpoint blob at the given offset. The snapshot replaces any previously published snapshot, but does not delete stream history. Retention advances only through the separate `/retention` endpoint.

<ParamField path="bucket" type="string" required>
 Bucket ID.
</ParamField>

<ParamField path="stream" type="string" required>
 Stream ID.
</ParamField>

<ParamField path="offset" type="string" required>
 Stream offset this snapshot represents: the 20-character zero-padded decimal token.
</ParamField>

For JSON streams, `PUT /{bucket}/{stream}/snapshot?record={record}` resolves a stable record ordinal to its byte offset and publishes at that boundary.

<ParamField header="Content-Type" type="string">
 Content type of the snapshot blob, stored separately from the stream's own content type. Defaults to `application/octet-stream`.
</ParamField>

<ParamField header="Stream-Snapshot-Match" type="string">
 Optional digest of the currently visible snapshot. The publish succeeds only if it still matches.
</ParamField>

<ParamField body="body" type="binary" required>
 The snapshot blob bytes. Maximum size is **128 MiB**. Larger bodies return `413`.
</ParamField>

## Response

| Status | Meaning |
| ------ | --------------------------------------------------------------- |
| `204` | Snapshot published successfully. |
| `400` | Invalid offset or content type. |
| `404` | Stream not found or expired. |
| `409` | Stale publish (a newer snapshot exists) or offset out of range. |
| `410` | Snapshot offset is older than the retained stream history. |
| `413` | Snapshot body exceeds the maximum allowed size. |

Response headers include `Stream-Next-Offset`, `Stream-Snapshot-Offset`, and `Stream-Snapshot-Digest`. Repeating the same offset, content type, and body is idempotent and returns the same digest. Reusing an offset with different content returns `409`.

<RequestExample>
```bash Example
curl -X PUT 'http://127.0.0.1:4437/demo/hello/snapshot/00000000000000000042' \
  -H 'Content-Type: application/json' \
  --data-binary '{"state": "aggregated snapshot data"}'
```
</RequestExample>

## Advance retention

`PUT /{bucket}/{stream}/retention/{offset}` explicitly discards history before a published checkpoint. For JSON streams, `PUT /{bucket}/{stream}/retention?record={record}` accepts a record ordinal. The boundary must be monotonic, aligned, and no newer than the visible checkpoint.

After a successful `204`, reads before the retained offset return `410 Gone`. This separate step lets clients publish and verify a checkpoint before making history unreachable.

---

## Delete snapshot

`DELETE /{bucket}/{stream}/snapshot/{offset}`

Attempting to delete the current visible snapshot is not allowed.

| Status | Meaning |
| ------ | ----------------------------------- |
| `404` | No snapshot at the given offset. |
| `409` | Cannot delete the current snapshot. |

> [!NOTE]
> Snapshots are immutable at a given offset. A byte-identical retry is idempotent; a different body at the same offset conflicts. Publishing does not advance retention.

---

# Read snapshot

## Latest snapshot

`GET /{bucket}/{stream}/snapshot` redirects to the latest published snapshot's offset-specific URL.

<ParamField path="bucket" type="string" required>
 Bucket ID.
</ParamField>

<ParamField path="stream" type="string" required>
 Stream ID.
</ParamField>

| Status | Meaning |
| ------ | ------------------------------------------------------------- |
| `307` | Redirect to `/{bucket}/{stream}/snapshot/{offset}`. |
| `404` | Stream not found, expired, or no snapshot has been published. |

Response headers include `Location`, `Stream-Next-Offset`, `Stream-Snapshot-Offset`, `Stream-Snapshot-Digest`, and `Stream-Up-To-Date`.

---

## Snapshot at offset

`GET /{bucket}/{stream}/snapshot/{offset}` returns the snapshot blob at a specific offset.

<ParamField path="offset" type="string" required>
 Snapshot offset: the 20-character zero-padded decimal token returned by previous reads or `Stream-Snapshot-Offset` headers.
</ParamField>

| Status | Meaning |
| ------ | -------------------------------------- |
| `200` | Snapshot blob returned. |
| `404` | Stream, snapshot, or offset not found. |

Response headers include `Content-Type`, `Stream-Next-Offset`, `Stream-Snapshot-Offset`, `Stream-Snapshot-Digest`, `Stream-Up-To-Date`, and `Stream-Closed`.

<RequestExample>
```bash Follow redirect to latest
curl -L 'http://127.0.0.1:4437/demo/hello/snapshot'
```

```bash Read specific offset
curl 'http://127.0.0.1:4437/demo/hello/snapshot/00000000000000000042'
```
</RequestExample>

> [!NOTE]
> Snapshot reads go through a linearizable freshness check to ensure you see the latest published snapshot. If the snapshot blob hasn't replicated to the current node yet, the request may be redirected to the leader.

> [!NOTE]
> See [snapshots](/docs/concepts/snapshots) for the snapshot lifecycle.

---

# Bootstrap

Returns the stream's latest snapshot (if any) plus all retained updates after the snapshot offset, packed as a `multipart/mixed` response. This is the recommended way to initialize a client that needs the complete current state of a stream.

<ParamField path="bucket" type="string" required>
 Bucket ID.
</ParamField>

<ParamField path="stream" type="string" required>
 Stream ID within the bucket.
</ParamField>

Bootstrap does not accept `?live=sse`. Combining the multipart body with an SSE event stream is rejected with `400`. To go live after bootstrap, finish the multipart response, then open a separate `GET /{bucket}/{stream}?offset=<Stream-Next-Offset>&live=sse`.

## Response

| Status | Meaning |
| ------ | --------------------------------------------- |
| `200` | Bootstrap response with snapshot and updates. |
| `400` | Invalid query parameters (including `live=sse`). |
| `404` | Stream not found or expired. |
| `410` | Requested offset has been trimmed. |

Response headers include:

| Header | Description |
| ------------------------ | ---------------------------------------------------------------- |
| `Content-Type` | `multipart/mixed; boundary=...` |
| `Stream-Next-Offset` | The offset after the last included update. |
| `Stream-Snapshot-Offset` | The snapshot offset (or `none` if no snapshot exists). |
| `Stream-Up-To-Date` | `true` if the response includes all data up to the tail. |
| `Stream-Closed` | Present and `true` if the stream is closed and fully caught up. |

## Response body

The body is a `multipart/mixed` message:

**Snapshot part**

The first part is the snapshot blob (or an empty part if no snapshot exists).

**Update parts**

Subsequent parts are individual update messages appended after the snapshot offset. For JSON streams, each update is a separate `application/json` part.

<RequestExample>
```bash Example
curl 'http://127.0.0.1:4437/demo/hello/bootstrap'
```
</RequestExample>

> [!TIP]
> After bootstrapping, switch to [SSE reads](/docs/api/read) with `live=sse` starting from the `Stream-Next-Offset` to receive real-time updates.

> [!NOTE]
> See [bootstrap](/docs/concepts/bootstrap) and [snapshots](/docs/concepts/snapshots) for the conceptual model.

---

# Delete stream

<ParamField path="bucket" type="string" required>
 Bucket ID.
</ParamField>

<ParamField path="stream" type="string" required>
 Stream ID within the bucket.
</ParamField>

## Response

| Status | Meaning |
| ------ | ----------------- |
| `204` | Stream deleted. |
| `404` | Stream not found. |

<RequestExample>
```bash Example
curl -X DELETE http://127.0.0.1:4437/demo/hello
```
</RequestExample>

> [!WARNING]
> Deletion is permanent. The stream's data will be asynchronously garbage-collected after the delete is committed.

---

# ursulactl

`ursulactl` manages the **logical** state of a running cluster over Ursula's admin and metrics HTTP APIs: which node leads which Raft groups, whether a node is caught up, and whether a leader will accept an amnesiac node back. It executes nothing on hosts. Physical lifecycle belongs to whatever owns the process: Helm and the StatefulSet controller on Kubernetes, systemd on hosts, OpenTofu for the infrastructure underneath. The verbs encode the safety properties an operator otherwise has to remember manually:

- before a node goes down, transfer every Raft group it leads to a healthy successor (`drain`)
- after it comes back, refuse to move on until `last_applied_index` has caught up to peers' `committed_index` (`wait`)
- abort rather than corner a group with no leader

A safe rolling restart wraps the platform's restart in these verbs, one node at a time. On Kubernetes:

```bash
ursulactl drain --config manifest.json --node 3
ursulactl prepare-restart --config manifest.json --node 3
kubectl delete pod ursula-2        # the platform restarts the pod
ursulactl wait --config manifest.json --node 3
ursulactl verify-cluster --config manifest.json
ursulactl undrain --config manifest.json --node 3
```

On bare metal the restart in the middle is `systemctl restart ursula` on the host. The drain planning and readiness logic is exercised under [deterministic simulation](/docs/architecture/overview).

> [!WARNING]
> The naive procedure of "restart followers, then leader" does **not** wait for `applied_index` to catch up between steps. Under `raft.wal.backend = "memory"` a target can come back as a voter while still missing committed entries, and a second restart pointed at a different node can corner the group with no live leader. Always `drain` before and `wait` after each node's restart.

## Verbs

| Verb | Effect |
|------|--------|
| `drain --node N` | Mark the node draining and transfer away every leadership it holds. The mark persists (the node attracts no leaderships) until `undrain`. `--dry-run` prints the transfer plan |
| `prepare-restart --node N` | After `drain`, detect the cluster WAL backend and arm stable non-target leaders for one empty-log rejoin when memory WAL requires it. Disk-WAL clusters need no rejoin permission |
| `undrain --node N` | Clear the drain mark so the node may hold leaderships again |
| `wait --node N` | Block until the node is a voter in every group and within `--lag-tolerance` of peers. Progress-gated: a node that keeps advancing is never timed out |
| `allow-rejoin --node N` | Arm one empty-log rejoin per group for a raft-memory node that lost its volatile log. Refused on disk-backed clusters |
| `status` | Per-node group counts and leadership distribution |
| `wait-ready` | Block until every node reports the expected group count and every group has a leader |
| `verify-cluster` | Require every configured voter to be present and caught up in two consecutive samples before the next rollout step |

Mutating verbs exit `0` on success and `2` on an abort (drain timeout, no safe transfer target, catch-up stall). Anything else is a configuration or transport error with a single-line, machine-greppable message.

## The admin plane

Nodes carry **no cluster-mutation surface on the network**. The mutating operator endpoints (raft snapshot/purge/membership/learners/leader-transfer/allow-next-revert, maintenance drain, cold-flush trigger) plus metrics are served on a separate **admin plane** bound to `server.admin_listen`, which defaults to loopback (`127.0.0.1:4438`). The public client plane (`:4437`) serves only stream traffic and read-only metrics.

`status` and `wait-ready` prefer each node's `http_url` because read-only metrics are available on the client plane, and fall back to `admin_url`. Mutating verbs always use `admin_url`.

ursulactl is a plain HTTP client and opens no tunnels itself. When the admin plane is not directly reachable, bring your own forward and point the manifest's `admin_url` at it. On Kubernetes, `kubectl port-forward` reaches the loopback-bound plane inside each pod:

```bash
kubectl port-forward pod/ursula-0 5441:4438 &
kubectl port-forward pod/ursula-1 5442:4438 &
kubectl port-forward pod/ursula-2 5443:4438 &
```

```toml
[[nodes]]
id = 1
admin_url = "http://127.0.0.1:5441"

[[nodes]]
id = 2
admin_url = "http://127.0.0.1:5442"

[[nodes]]
id = 3
admin_url = "http://127.0.0.1:5443"
```

On bare metal the same shape works with `ssh -N -L 5441:127.0.0.1:4438 admin@node1` per node.

## When to use ursulactl vs. the other surfaces

| Task | Tool |
|------|------|
| Day-2 logical operations: drain, observe, gate on readiness | **`ursulactl`** |
| Deployment, restarts, upgrades, topology | [Helm and OpenTofu](/docs/deploy-cluster) (systemd on bare metal) |
| Custom operator tooling | The admin-plane HTTP endpoints, reached over your own tunnel |

## Install

Build from the workspace alongside the server:

```bash
cargo build --release -p ursula-ctl --bin ursulactl
```

The binary lands at `target/release/ursulactl`. Drop it on your control machine. It does not need to run on the Ursula hosts themselves.

## Manifest format

Every verb accepts `--config <path>`. The manifest is **TOML, JSON, or YAML** (chosen by file extension, sniffed when read from stdin with `-`) and lists the cluster's nodes.

Prefer generating the manifest from whatever already knows the topology instead of writing it by hand. On Kubernetes the Helm chart renders one into its ConfigMap, and the URLs in it are in-cluster DNS, so pipe it to ursulactl running where those names resolve (the server image contains ursulactl):

```bash
kubectl get configmap ursula -o jsonpath='{.data.cluster-manifest\.json}' \
  | ursulactl status --config -
```

For infrastructure provisioned by OpenTofu, emit the manifest as a stack output or generated file (the same pattern as `deploy/eks`'s `generated-values.yaml`) rather than teaching ursulactl to read state files. `tofu output -json <name> | ursulactl status --config -` composes the same way.

Per-node fields, all optional except `id`:

- `admin_port` (default `4438`) or an explicit `admin_url`: the admin plane to reach, directly or through your forward.
- `host`: address shown in reports. Falls back to the admin URL's host.
- `http_url`: optional client-plane URL used by `status` and `wait-ready` for read-only metrics. Metrics fall back to `admin_url` when omitted.

## Restarting raft-memory nodes

On clusters running the volatile `raft.wal.backend = "memory"`, a restarted node rejoins with an **empty** log, and group leaders refuse that log reversion unless it was explicitly permitted. Run `prepare-restart --node N` after the drain and before the restart: it detects the reported WAL backend and asks every group's stable leader to accept one empty-log rejoin from the target when memory WAL requires it. It fails closed when nodes omit or disagree on their backend. Disk-WAL clusters report that no permission is needed. `allow-rejoin` remains the explicit recovery verb, not the normal rolling-restart step.

The permission must land on the node that is leader at that moment, since the endpoint answers `409 Conflict` from any other node (ursulactl resolves leadership per group and handles this). A node that is the membership initializer for some groups additionally refuses to start at all after losing its volatile log (the bootstrap-marker guard). That recovery is an explicit operator reset on the host, outside ursulactl's reach.

The rebuild after an amnesiac restart installs snapshots for every group and can take 10+ minutes. `wait` is progress-gated, so no timeout tuning is needed: a rebuild that keeps advancing is never timed out, and `--stall-timeout-secs` (default 90) only aborts a node that stops making progress.

## `status`

Per-node summary of Raft group count and leadership distribution, sourced from every node's `/__ursula/metrics`. Nodes whose metrics fail are reported with `metrics unavailable — …` rather than aborting the report, because `status` is meant to surface partial cluster health.

```bash
ursulactl status --config cluster.json
```

Sample output:

```
node 1 (10.0.0.1): groups=4 leaders={1: 2, 2: 2}
node 2 (10.0.0.2): groups=4 leaders={1: 2, 2: 2}
node 3 (10.0.0.3): groups=4 leaders={1: 2, 2: 2}
```

`leaders={…}` is the count of groups each node is leading from this reporter's perspective. Healthy clusters report the same distribution from every node.

## `wait-ready`

Block until every node reports `--expected-groups` Raft groups, each with a leader. Useful in CI / scripts after a deploy or a config change.

```bash
ursulactl wait-ready --config cluster.json --expected-groups 4
```

Exits non-zero with a one-line reason if the timeout passes (`cluster not ready after 120s: node 3 has 1 group(s) without a leader`).

## Underlying HTTP surface

For custom tooling, every verb maps onto a small set of HTTP endpoints on each node:

| Verb | Endpoint |
|------|----------|
| `status`, `wait-ready`, `wait` | `GET /__ursula/metrics` |
| `drain` / `undrain` (maintenance guard) | `POST /__ursula/leadership-shed/maintenance`, `DELETE` to clear |
| `drain` (transfer step) | `POST /__ursula/raft/{raft_group_id}/leader/transfer/{node_id}` |
| `allow-rejoin` | `POST /__ursula/raft/{raft_group_id}/nodes/{node_id}/allow-next-revert` per group, on the group's leader |

The transfer endpoint refuses with `409 Conflict` if the receiving node isn't the current leader of the group, and `400` if the target node isn't a voter. ursulactl uses this to refuse to attempt a transfer it cannot reason about.

---

# Operations

The first tool to reach for is [`ursulactl`](/docs/cli). It covers the raft-aware verbs operators run most often: drain, status, readiness gates. This page covers everything **around** it: the metrics shape ursulactl reads, the admin endpoints it (and your custom tooling) can call, and the operational policies ursulactl does not encode (backups, log levels). Deployment and process lifecycle belong to the platform, [Helm and OpenTofu](/docs/deploy-cluster).

## Tooling map

| Surface | When to reach for it |
|---------|---------------------|
| [`ursulactl`](/docs/cli) | Day-2 cluster ops over HTTP: drain, restart, status, wait-ready. |
| `/__ursula/metrics` and the `/__ursula/raft/...` admin endpoints | Custom tooling. ursulactl uses these underneath. The surface is small and stable enough to script directly. |

There is no Prometheus scrape and no general-purpose orchestrator yet.

## Metrics

```bash
curl http://127.0.0.1:4437/__ursula/metrics | jq .
```

The JSON snapshot covers per-core mailbox depth, append/read counters, latency histograms, per-group leader and `last_applied`, hot/cold bytes, cold-flush backlog, HTTP status counters, live-read watchers, and cold-write admission state. Start here when triaging slowness, lag, or `503`s. `ursulactl status` is a friendlier read of the leadership-related fields across the whole cluster.

## Admin endpoints

These are the primitives `ursulactl` and any custom operator tooling builds on. They live on the **admin plane** (`server.admin_listen`, loopback `127.0.0.1:4438` by default), not the public client plane. Nodes expose no cluster-mutation surface on the network. Reach them over a tunnel to the admin port (see [ursulactl's operation providers](/docs/cli#the-admin-plane)). The examples below assume a forward from local `4438` to the node's admin plane. Each call is local to one node. To act on every group, loop over the IDs in metrics.

```bash
# Force a cold flush for one stream (skip the timer)
curl -X POST http://127.0.0.1:4438/__ursula/flush-cold/demo/hello

# Trigger a Raft snapshot for one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/snapshot

# Purge log entries below the last snapshot index for one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/purge

# Add a learner (non-voting replica) to one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/learners/4

# Hand leadership of one group to another voter (used by `ursulactl drain`)
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/leader/transfer/2

# Replace the voter set of one group (call on the group's current leader)
curl -X POST "http://127.0.0.1:4438/__ursula/raft/42/membership?voters=1,2,3"

# Put this node into maintenance drain / lift it again
curl -X POST   http://127.0.0.1:4438/__ursula/leadership-shed/maintenance
curl -X DELETE http://127.0.0.1:4438/__ursula/leadership-shed/maintenance

# Permit the leader to accept one log revert from a wiped, rejoining node
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/nodes/4/allow-next-revert
```

The leader-transfer endpoint refuses with `409 Conflict` if the receiving node isn't the current leader and `400 Bad Request` if the target isn't a voter. This is why `ursulactl` is the safer way to chain these calls: it consults metrics first.

The membership endpoint has the same leader-only contract: it answers `409 Conflict` (with the current leader's id in the body) when the receiving node does not lead the group, and `400 Bad Request` without a non-empty `voters` query parameter. On success it returns the log index at which the new voter set was committed. It replaces the whole voter set in one call, so include every node that should remain a voter.

### Maintenance drain

`POST /__ursula/leadership-shed/maintenance` marks the node as draining: it stops campaigning and accepting transfers, and the background leadership balancer migrates its current leaderships to eligible peers one transfer per tick until the mark is lifted with `DELETE`. Both verbs return the node's leadership-shed state so tooling can confirm the transition took effect:

```json
{
  "bits": 1,
  "state": "maintenance-drain",
  "should_accept_transfer": false,
  "should_campaign": false,
  "should_shed_current_leaders": true
}
```

`ursulactl drain` wraps exactly this pair: mark first so the node stops re-acquiring groups, then transfer leaderships away explicitly instead of waiting for the balancer. The mark stays set until `ursulactl undrain` clears it. If a rollout ever dies mid-way, a node can be left refusing leadership. Run `ursulactl undrain` (or `DELETE` the mark directly) to recover. Peers read each other's shed state (`GET /__ursula/leadership-shed` on the cluster-plane listener) when deciding which nodes are eligible to pick up leaderships, so a drained node stops attracting groups cluster-wide, not just locally.

### Rejoining a wiped node

A node that lost its Raft log (disk replaced, data dir wiped) rejoins with a shorter log than the leader has recorded for it, which Raft treats as a fault. `POST /__ursula/raft/{group}/nodes/{node}/allow-next-revert` tells the group's leader to accept exactly one such log revert from that node and re-replicate from scratch. It is a per-group, one-shot permission: loop over all group IDs when re-admitting a fully wiped node, and expect the leader to answer `500` if the underlying Raft rejects the request.

`POST /__ursula/cluster-probe` also exists on the client plane, but it is machinery rather than an operator surface: nodes send each other heartbeat-sized bodies through it to verify cluster-plane egress, and it bypasses ingress admission.

## Tenant offboarding (bucket purge)

`DELETE /__ursula/purge/{bucket}` (admin plane) erases one tenant's stream content: every stream in every Raft group, the bucket itself, and its quota, followed by one immediate cold-GC pass that reclaims the streams' cold-object prefixes. It retains only the bucket's aggregate usage counters, because deleting a counter before an asynchronous meter observes it would erase committed billable work. The response is a completion report an operator can attest to:

```bash
curl -X DELETE http://127.0.0.1:4438/__ursula/purge/tenant-a
# {"bucket":"tenant-a","removed_streams":12,"groups_with_streams":[0,3,5],"cold_gc_entries_reclaimed":12}
```

Semantics:

- **Idempotent and resumable.** Purging an absent bucket returns the same report shape with zero counts. A purge interrupted mid-way converges on re-run: stream removal is a replicated command per group, and cold reclamation is a list-then-delete over object prefixes, so nothing depends on in-memory progress.
- **Isolated.** Other buckets' streams, offsets, checksums, and snapshots are untouched; the purged bucket name conceals as not-found afterwards and may be recreated. A recreated bucket continues the same monotonic usage counters.
- **Accounting residue.** The retained entry contains only aggregate counters and zero-valued content gauges, not stream data. Ursula does not currently provide an acknowledgement protocol for erasing it after an external meter persists the final delta. Use opaque bucket IDs if retaining the identifier itself would violate an erasure requirement.
- **Cold-GC interaction.** If the immediate GC pass fails (for example S3 is briefly unavailable), the entries stay queued and the background GC worker finishes reclamation; the purge itself is already durable at that point.
- **Backups.** A purged tenant may persist inside backups taken before the purge until those backups age out; restore replays whatever the manifest contains. Deleting old backup sets is part of completing an offboarding with erasure obligations.

## Cleaning S3

Cold data lives under the `storage.cold.root` prefix. Benchmark and test runs should use a date-stamped root so cleanup is one prefix delete with standard S3 tooling:

```bash
aws s3 rm "s3://my-ursula-bucket/ursula-test-20260518T000000Z" --recursive
```

There is no automatic retention policy. Application checkpoints and explicit stream-retention advances control which cold objects become eligible for GC.

## Backup and disaster recovery

`ursulactl` provides a verifiable whole-cluster backup and restore workflow:

```bash
# Create: one MessagePack snapshot object per raft group + manifest.json,
# into a local directory or s3://bucket/prefix.
ursulactl backup-create --config cluster.yaml --location s3://backups/ursula/2026-07-25

# Verify: manifest completeness, byte sizes, BLAKE3 checksums, and deep
# state-machine validation of every snapshot. Touches no cluster.
ursulactl backup-verify --location s3://backups/ursula/2026-07-25

# Restore: into a FRESH, EMPTY cluster with the same raft group count.
ursulactl restore --config new-cluster.yaml --location s3://backups/ursula/2026-07-25
```

The recovery contract:

- **Consistency boundary.** Each group export is the deterministic state-machine snapshot the raft snapshot path persists: internally consistent per group while writes continue. Cross-group consistency is not promised; the boundary is per stream. Acknowledged writes in the exporting replica's applied state are included whether or not they were cold-flushed, so **RPO is bounded by export time**, not by the cold-flush interval. `backup-create` asks every node and keeps the reply with the highest group commit index.
- **Fresh identity on restore.** Restore replays each snapshot as one replicated write on the target cluster, which keeps its own raft membership and log identity — nothing from the source cluster's raft metadata is reused. Groups that already hold buckets refuse the import with `409`; group-count mismatches fail closed before the first import. Group commit indexes and per-stream append counters restart under the new identity; stream bytes, offsets, record coordinates, attrs, close/TTL state, published snapshots, retention floors, and producer dedup states are preserved exactly.
- **Cold objects are part of the backup set.** Snapshots reference cold-tier chunks by object key; point the restored cluster at the same (or a copied) cold-store namespace. `backup-verify` validates snapshots but does not dereference cold objects.
- **Format compatibility.** The backup format is versioned (`format_version` in the manifest) independently from the server binary; tools refuse newer formats and servers refuse imports they cannot validate. Within `v0.x`, restore into the same minor version is the supported path.
- **RTO** is dominated by transferring and importing the group snapshots; each import is a single replicated write per group.

The recovery drill — write, snapshot, retain, close; back up; destroy the cluster; restore; verify checksums, coordinates, and boundaries; continue appending — runs in CI (`backup_restore_drill_preserves_streams_and_allows_continued_appends`).

What you can additionally rely on:

- **Quorum durability.** Acknowledged writes survive as long as a majority of voters survives. Three voters across AZs tolerate any single-AZ outage.
- **Cold-tier durability.** Once flushed to S3, a chunk inherits S3-grade durability. The unflushed window is bounded by the flush interval (seconds by default).
- **No on-disk migration.** `v0.x` does not promise stable on-disk formats. The runtime won't refuse to start on stale data, but it won't migrate either. Cross-version upgrades currently mean rebuild + replay from external truth.

Node-level loss: replace the host with the same `node_id` and cluster config, and it rehydrates from peers. Use `ursulactl wait-ready` afterwards to confirm the replacement is voting and caught up before declaring the recovery done. Total-cluster loss: restore the latest verified backup into a fresh cluster as above.

## Logs

`RUST_LOG=ursula=info,ursula_runtime=info,ursula_raft=info` is the baseline. Bump to `debug` for one crate when chasing a subsystem:

```bash
RUST_LOG=ursula_raft=debug ./target/release/ursula server ...
```

`debug` is verbose under sustained load, so redirect to a file.

---

# Observability

Ursula emits OpenTelemetry traces and metrics over OTLP, and keeps a built-in
JSON metrics endpoint for quick inspection. Telemetry export is **off by
default**: with no collector configured the server runs exactly as before and
the hot read/write path pays nothing.

## Enabling OTLP export

Set the standard OpenTelemetry OTLP exporter settings before starting `ursula`:

```bash
# Send traces and metrics to a collector (Tempo, Jaeger, the OTel Collector, …).
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4318"

# Optional: head-sampling ratio for new root traces (0.0–1.0, default 1.0).
export OTEL_TRACES_SAMPLER_ARG="0.1"

cat > /tmp/ursula-otel.toml <<'TOML'
[server]
listen = "0.0.0.0:8080"
TOML

ursula --preset default --config /tmp/ursula-otel.toml
```

The exporter uses OTLP over HTTP/protobuf (port `4318`), reusing the server's
HTTP stack rather than opening a second gRPC tree alongside Raft. When the
endpoint is unset, the process stays on the stderr `fmt` logger driven by
`RUST_LOG` (default `info`).

Spans batch-export continuously and metrics export on a fixed interval
(`OTEL_METRIC_EXPORT_INTERVAL`, milliseconds). A final flush on clean shutdown
is only a backstop. Don't rely on it to observe data.

### Identifying the node

Every span and metric carries the process's OpenTelemetry **resource**. The
server sets `service.name=ursula` and, when running with a Raft node id,
`service.instance.id=<node_id>` so a cluster's nodes are distinguishable. Add
more resource attributes (e.g. `host.name`, region, AZ) with the standard
variable. They stay bounded by cluster size, so this is a resource dimension,
not a high-cardinality label:

```bash
export OTEL_RESOURCE_ATTRIBUTES="host.name=$(hostname),deployment.environment=prod"
```

## Traces

A single request produces one boundary span (`http.append`, `http.read`, or
`http.head`) tagged with `bucket` and `stream`, never the payload. That span
is propagated across the internal actor mailboxes, so on-core work
(`core.append`, `core.read`, …) nests underneath it. When a follower forwards a
read to the leader, the W3C `traceparent` rides the Raft gRPC request, so the
follower and leader appear in the **same trace** end to end.

Trace context travels only in gRPC transport metadata. It is never written to
the Raft log, so state-machine determinism and replay are unaffected.

### Sampling and the hot path

Request-boundary spans (`http.*`) are `info` level, so production traces always
carry one span per request. The on-core spans below them (`core.append`,
`core.read`, …) are `debug` level: filtered out at the default `info` and
compiled out of release builds entirely (`release-max-info`, on by default), so
the hot read/write path pays nothing for them. Measured cost:

| Operation                                   | Cost    |
| ------------------------------------------- | ------- |
| `info` boundary span (export off, fmt only) | ~285 ns |
| `debug` span filtered at `info`             | ~1 ns   |
| cross-mailbox context capture               | ~5 ns   |

For high-throughput deployments, lower `OTEL_TRACES_SAMPLER_ARG` to bound
export volume.

### Seeing on-core detail

The `core.*` spans only materialize in a build without `release-max-info` (e.g.
a debug build), and only when their target is enabled. Use a **crate-scoped**
filter, never a global `debug`:

```bash
# Good: only Ursula's own crates emit debug spans.
export RUST_LOG="info,ursula_runtime=debug,ursula_raft=debug"
```

A global `RUST_LOG=debug` turns on debug spans/logs from OpenRaft, the
OpenTelemetry SDK, reqwest, and every other dependency, which floods the
collector and bloats the OTLP payload. Scope the filter to the `ursula_*`
crates instead.

> `tokio-console` needs tokio's `TRACE`-level instrumentation, which
> `release-max-info` compiles out. Build with
> `--no-default-features --features tokio-console` when using it.

## Metrics

Two surfaces expose the same counters:

- **OTLP**: when an endpoint is configured, runtime metrics are exported
  periodically as OpenTelemetry instruments (`ursula.appends.accepted`,
  `ursula.mutations.applied`, `ursula.raft_apply.ns`, `ursula.wal.records`,
  `ursula.group_mailbox.depth`, …), ready for Prometheus/Grafana via a
  collector.
- **JSON**: `GET /__ursula/metrics` returns the full per-core/per-group
  snapshot for quick inspection and is what `ursulactl status` reads.

Metrics are gathered on the hot path with lock-free per-core counters. The OTLP
layer is an *export-time bridge*: it reads a snapshot at the collection
interval rather than calling the metrics SDK per record, so enabling export
adds nothing to the append/read path.

The OTLP collection interval defaults to 60s. For a quick check, shorten it so
readings arrive without waiting (and without relying on the shutdown flush):

```bash
export OTEL_METRIC_EXPORT_INTERVAL=1000  # milliseconds
```

---

# Troubleshooting

Start at `/__ursula/metrics`. Most operational symptoms have a clear signal in the JSON snapshot.

## Diagnostic surface

```bash
curl http://NODE:4437/__ursula/metrics | jq .
```

Useful jq selectors:

| What you need | jq filter |
| ------------- | --------- |
| Per-group leader / term | `.raft_groups[] | {id, leader_id, current_term, last_applied}` |
| Hot bytes per group | `.raft_groups[] | {id, hot_bytes_total}` |
| Cold backpressure events | `.cold_backpressure_events_total` |
| Per-core mailbox depth | `.cores[] | {core_id, mailbox_pending}` |
| Live-read watcher counts | `.cores[] | {core_id, live_read_watchers_active}` |
| HTTP error counters | `.http.responses_by_status` |

Ursula does **not** expose `/healthz`, `/readyz`, Prometheus `/metrics`, or `/cluster/status`. The JSON snapshot is the source of truth.

## Startup failures

### `raft.wal.path is required when WAL backend is 'disk'`

Set `raft.wal.path` when `raft.wal.backend = "disk"`. Use `raft.wal.backend = "memory"` for volatile tests.

### `raft.node_id must be non-zero`

Static Raft mode requires a unique `raft.node_id` in the config file or a `--node-id` override.

### `raft.peers` must include this node id

The local `raft.node_id` must appear in `[[raft.peers]]`. The peer list always includes self.

### `storage.cold.s3.bucket is required when cold backend is 's3'`

Set `storage.cold.s3.bucket`, or set `storage.cold.backend = "none"` if you don't want cold storage.

### Port already in use

`server.listen` is taken. Public HTTP and inter-node gRPC share the same port unless `server.cluster_listen` is configured separately.

### I/O error on `raft.wal.path`

The directory must be writable. No lock file to clear, Ursula trusts the directory.

## Bootstrap and cluster join

### A fresh cluster never elects leaders

`/__ursula/metrics` on each node should show non-zero `leader_id` and a stable `current_term`. If groups stay leaderless:

- First start must include `raft.init_membership_per_group = true` (or `raft.init_membership = true`) on every voting node.
- Every peer URL must be reachable from every node.
- All peers must use the same `raft.wal.backend` mode. Mismatch causes silent join failures.

### Replacing or restarting a node leaves it leaderless

Restart with the **same** `raft.node_id` and peer list. A fresh `raft.wal.path` means rehydrating from peers, which can be slow on cold-storage-only history.

## Write path

### `503` with `Retry-After` on append

Cold-write backpressure for that group. Either the hot ring exceeds `storage.cold.max_hot_size_per_group` (cold flush isn't keeping up), or the cold backend is slow or erroring out.

Raise the per-group ceiling, lower `storage.cold.flush_interval`, raise `storage.cold.flush_max_concurrency`, or fix the cold backend.

### `409 Conflict` with `producer-expected-seq` / `producer-received-seq`

Out-of-order producer headers. Response tells you what was expected:

- `producer-expected-seq: N` is the next allowed sequence
- `producer-received-seq: M` is what the client sent

Common causes:

- Producer restart without bumping `Producer-Epoch`. Bump every restart.
- Two writers share the same `Producer-Id`. Use distinct IDs.
- A retry skipped a sequence number. Producer sequences must be contiguous within an epoch.

### `409 Conflict` from `Stream-Seq`

The supplied `Stream-Seq` is not lexicographically greater than the last accepted value. Re-read and re-derive, or use producer dedup instead.

### `404 Not Found` on `POST /{bucket}/{stream}`

The stream hasn't been created. Create with `PUT /{bucket}/{stream}` first. There is no implicit creation on POST.

### `400 Bad Request` on JSON appends

JSON streams are parsed and normalized on the server. Malformed JSON or empty arrays without `allow_empty_array` are rejected. Send valid JSON or switch to `application/octet-stream`.

## Read path

### `Stream-Up-To-Date: false`

More committed data exists past this response. Page forward with `Stream-Next-Offset`. Not an error.

### SSE drops after 30-60 s idle

A proxy or load balancer is closing idle TCP. Raise its idle timeout or enable TCP keepalive on the proxy-to-Ursula leg. Ursula doesn't emit SSE keepalive comments.

### SSE on a binary stream returns base64 text

Expected. SSE wire format is text-only, so binary stream data events carry raw base64 text and the `Stream-Sse-Data-Encoding: base64` header signals it. See [binary SSE](/docs/concepts/binary-sse).

### Reads stall for hundreds of ms for cold offsets

The first read of a cold offset triggers an S3 `GetObject` range read. It runs off the actor turn so it doesn't block other commands, but the request still pays S3 latency.

## Replication and consensus

### A group has no leader (`leader_id == 0`)

Either an election is in progress or quorum is lost for that group. Only streams hashed to that group are affected.

- Multiple nodes briefly claim leader: election is flapping. Check peer reachability and CPU pressure.
- No node claims leader: fewer than `n/2+1` voters are reachable. Restore reachability.

### One follower lags behind

State-machine apply runs on the follower's owner core. If CPU is pinned, `last_applied` trails. Check:

- Per-core mailbox depth (sustained queueing means saturation).
- Disk pressure if `raft.wal.backend = "disk"` (fsync latency).
- A pathological hot stream forcing constant cold flushes (`hot_bytes_total` per group).

### Manually trigger a snapshot

These are admin-plane endpoints (`server.admin_listen`, loopback `:4438` by default), reached over a tunnel (see [ursulactl's operation providers](/docs/cli#the-admin-plane)). The examples assume a forward from local `4438` to the node's admin plane.

```bash
curl -X POST http://127.0.0.1:4438/__ursula/raft/{group_id}/snapshot
curl -X POST http://127.0.0.1:4438/__ursula/raft/{group_id}/purge
```

Purge only after the snapshot replicates. Otherwise a slow follower may need log entries you just dropped.

## Cold flush issues

### Writes succeed but data never reaches S3

- `storage.cold.backend` is `none`, or set to `memory` when you expected `s3`.
- IAM permissions missing. Required: `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`, `s3:DeleteObject`.
- `storage.cold.s3.endpoint` unreachable.
- Cold flush worker stalled (`cold_backpressure_events_total` climbing).

If hot bytes grow unbounded, you'll eventually hit `storage.cold.max_hot_size_per_group` and writes start returning `503`. Fix the backend before that.

## Still stuck?

Open a [GitHub issue](https://github.com/tonbo-io/ursula/issues) with:

- `/__ursula/metrics` from every node (with timestamps)
- Config file and CLI overrides per node (redact credentials)
- Last ~200 log lines at `RUST_LOG=ursula_runtime=debug,ursula_raft=debug,ursula=info`

---

# Overview

## The problem

Building a durable stream directly on S3 hits two walls: S3 has no append (every write either pays per-PUT or batches and eats the latency), and its conditional writes are optimistic compare-and-set, so concurrent writers retry-storm with exponential latency ([Chroma's wal3 post](https://www.trychroma.com/engineering/wal3) covers the mechanics). The market splits on which wall to lean against. [S2](https://s2.dev/blog/intro) writes through S3 Express to get under 50 ms but pays around 7x per-GB, while [WarpStream](https://www.warpstream.com/blog/minimizing-s3-api-costs-with-warpstream) batches to S3 Standard and accepts 250 ms+ p50.

## How Ursula works

Ursula keeps S3 off the write path entirely. Writes go to a cluster of nodes that coordinates through [Raft consensus](https://en.wikipedia.org/wiki/Raft_(algorithm)). Once a majority of replicas has persisted a record, it is committed, with no compare-and-set races and no retry storms. S3 is the cold tier.

Two structural choices shape the rest of the design:

1. **[Thread-per-core](https://seastar.io/shared-nothing/) shard ownership.** Every stream is statically hashed to one specific CPU core on each node. That core handles all reads and writes for its streams, in order, with no cross-core synchronization on the hot path.
2. **[Multi-Raft](https://tikv.org/deep-dive/scalability/multi-raft/).** Ursula runs hundreds to thousands of small Raft groups per cluster, not one log per node. Each stream hashes to a group, each group pins to a core. An unhealthy follower for one group does not stall traffic for another.

Together, write throughput scales with healthy cores across the cluster instead of with any single leader's bandwidth, and failure domains stay per-group rather than per-node.

## The stack at a glance

Ursula is async Rust. The pieces that show up in the diagrams below:

- **axum** is the HTTP server library that terminates public client requests.
- **tokio** is the async runtime. Each core runs its own single-threaded executor (`tokio::current_thread`) so work on that core never migrates.
- **tonic** is the gRPC library used for peer-to-peer Raft traffic between nodes.
- **OpenRaft** is the Raft consensus implementation Ursula plugs into.

The Rust-named components in diagrams (`ShardRuntime`, `GroupActor`, `GroupEngine`, `StreamStateMachine`) are the structures inside one node, glossed in the sections below.

## Interface

The API is the [Durable Streams protocol](https://durablestreams.com/), an open MIT-licensed spec published by [Electric](https://electric-sql.com/): create a stream, append, read from any offset, tail in real time over Server-Sent Events (SSE), with exactly-once write semantics. Plain HTTP plus SSE, with no custom binary protocol or mandatory client library. See the [API overview](/docs/api/overview) and the [extensions spec](/docs/specs/extensions) for the append-batch path.

## Cluster topology

```
                          Clients (HTTP / SSE)
                                   │
                ┌──────────────────┼──────────────────┐
                v                  v                  v
          ┌──────────┐       ┌──────────┐       ┌──────────┐
          │  Node A  │       │  Node B  │       │  Node C  │
          └────┬─────┘       └────┬─────┘       └────┬─────┘
            cores 0..N         cores 0..N         cores 0..N
               │                  │                  │
               └─── gRPC peer-to-peer Raft traffic ──┘
                                   │
                                   v
                       ┌────────────────────────┐
                       │ Cold storage (S3)      │
                       │ per-stream chunks      │
                       └────────────────────────┘
```

Every stream is replicated to all voting members of its Raft group. Hot data lives in memory on each replica. Cold data lives in a shared object store any replica can read from on demand.

## Inside a node

```
   HTTP / SSE request
          │
          v
   route(bucket_id, stream_id)
          │
          v
   owner core
          │
          v
   Raft group
      │       │
      │       └── replicate to peer nodes
      │
      ├── hot state and live watchers
      │
      └── cold chunks in S3
```

A request enters the HTTP server, routing maps the bucket and stream name to one owner core and one Raft group, and the rest of the request runs inside that group. The group owns its hot bytes, live-tail watchers, producer dedup state, and Raft state machine, so mutable stream state does not cross cores after routing.

## The replaceable group engine

The `GroupEngine` trait is the seam between the runtime and a group's storage and replication strategy. Three engines ship today:

- **In-memory, non-replicated.** Used in tests and as a performance baseline.
- **Disk-backed Raft engine** (`raft.wal.backend = "disk"` with `raft.wal.path = "DIR"`). OpenRaft over a per-group write-ahead log of protobuf-framed records under `DIR/raft-log`, plus a shared per-core journal. Durable across restarts.
- **Memory-log Raft engine** (`raft.wal.backend = "memory"`). OpenRaft with an in-memory log store. Fast iteration, no persistence.

A factory selected at startup picks which engine each group opens, leaving the runtime code identical across modes.

## Write path

```
HTTP ---> Runtime ---> core mailbox ---> GroupActor
                                       │
                                       ├─ leader? ──no─---> 307 redirect to leader
                                       │
                                       v yes
                                  Raft.client_write
                                       │
                                       v
                            WAL append + fsync, replicate to followers
                                       │
                                       v commit
                                  StateMachine::apply
                                       │
                                       v
                          notify_read_watchers (same turn, no lock)
                                       │
                                       v
                                AppendResponse -> client
```

The leader writes the entry to its write-ahead log (WAL), fsyncs, and replicates to followers in parallel. Once a quorum has persisted the entry, the state machine applies it and the response goes back to the client. Producer dedup headers ([exactly-once writes](/docs/concepts/exactly-once-writes)) are checked inside the state machine, so retries of an already-applied append return the original offset rather than appending again. `Stream-Seq` provides a single-writer ordering guard for callers that prefer that style ([conditional writes](/docs/concepts/conditional-writes)).

## Storage layout

Each Raft group has two storage planes: a replicated log and a stream state machine.

```
              Raft group
                  │
        ┌─────────┴─────────┐
        v                   v
  replicated log       state machine
  command order        streams in this group
  quorum durability    offsets / metadata / dedup
                              │
                    ┌─────────┴─────────┐
                    v                   v
              hot ring memory      cold chunk refs
              recent bytes         S3 objects
```

The Raft log is the ordering and durability boundary for writes. A write is acknowledged only after a majority of that group's replicas persist the entry. The entry carries the stream command and metadata needed to apply it deterministically.

The stream data plane lives in the group's state machine. A group owns many streams, keyed by `(bucket_id, stream_id)`. For each stream it tracks metadata, offsets, producer dedup state, live snapshots, hot byte segments, and cold chunk references.

Recent bytes stay in an in-memory hot ring on every replica. When a group's hot bytes exceed the configured cold-flush threshold, a background worker uploads older contiguous segments to the cold backend, usually S3. After upload succeeds, Ursula commits a metadata update through Raft that publishes the cold chunk reference and advances the stream's hot start offset.

Reads assemble one logical stream from both tiers. The state machine first builds a read plan: which byte ranges are still hot and which ranges are cold chunk references. Hot bytes are copied from memory, and cold ranges are fetched with object-store range reads. Any replica that has applied the cold chunk metadata can serve those cold reads, as long as all nodes point at the same S3 bucket/prefix.

## Read path and cold offload

```
HTTP ---> Runtime ---> core mailbox ---> GroupActor
                                       │
                                       ├─ leader? ──no─---> forward to leader
                                       │
                                       v yes
                          engine.read_stream_parts (under state-machine lock)
                                       │
                                       v release lock
                       GroupReadStreamParts { plan, cold_store }
                                       │
                                       v tokio::spawn (off the actor turn)
                          materialize: hot bytes + S3 range reads
                                       │
                                       v
                                ReadStreamResponse -> client
```

Under the state-machine lock the runtime computes only a read plan: which segments are still in the hot ring and which live as cold chunks in S3. Then it releases the lock. Materialization, including any S3 `GetObject` range reads, runs in a separately spawned task, so a slow cold fetch never blocks other commands waiting on the same group. Followers serve replicated historical catch-up bytes from their local applied state when the read does not need a leader-owned access mutation. Tail-sensitive responses such as `HEAD`, open-stream `Stream-Up-To-Date: true`, and live reads stay on the leader path (see [offsets](/docs/concepts/offsets)).

## SSE live tailing

When a read finds no new data and the response carries `Stream-Up-To-Date: true`, the runtime registers a watcher inside the owning group actor. The watcher map is per-actor and mutated only inside that actor's turn, with no mutex and no awaiting while holding a lock. When a later append commits on the same group, the actor wakes the matching watchers in the same turn, deduplicates them by request shape, and dispatches materialization through the same fast path the read uses.

For deployment topology (voting layout across availability zones, non-voting replicas, S3 prefix layout), see the [operations guide](/docs/operations).

---

# Comparisons

Ursula is for streams that diverse clients reach over the public internet with plain HTTP. One durable timeline per resource, and many small streams rather than a few big ones.

Most streaming systems are built for a different shape. They assume an SDK client inside your network and a few high-throughput topics. That assumption decides most of the comparison below, so start there.

## Start with the access model

Two questions decide whether Ursula fits.

**Where do clients run, and how do they connect?** Ursula serves plain HTTP and SSE. A browser, a mobile app, a serverless function, or a third party can read, write, and tail a stream over the public internet with no SDK. Broker-style systems expect a client library inside the same network.

**What is one stream?** In Ursula one stream maps to one resource: a document, a session, a task, an agent run. You address it by URL and keep thousands of them. Broker-style systems multiplex many resources into a few topics and partitions, and you rebuild per-resource state downstream.

If your clients are in-network services and your data is a few high-throughput pipelines, Ursula is the wrong shape. If your clients are on the open internet and your data is many small per-resource timelines, the rest follows.

## Kafka / Redpanda

Kafka and Redpanda are built for high-throughput pipelines. Topics, partitions, consumer groups, and SDK clients inside the cluster network. They are mature and fast at that job.

They are not built for public-internet HTTP access or for thousands of small, independently addressable streams. There is no plain-HTTP read, write, and tail for an arbitrary client, and the unit of work is the topic, not the resource. Pick Kafka or Redpanda for a few high-throughput topics with in-network consumers. Pick Ursula for many small streams with open-internet clients.

## S2 / S2 Lite

S2 is an object-storage-backed stream API and the closest alternative to Ursula. Acknowledged writes go through S3 Express One Zone, and the cold tier stays on Express-class storage. Self-hosted S2 Lite is a single serving process. The managed product uses its own API, not Durable Streams.

S3 Express looks fine on its own. The point is that you do not need it. Ursula gets its write latency from an in-memory hot ring replicated by Raft quorum, which has nothing to do with the storage tier, and it keeps data on plain S3 Standard. In our 3-node benchmark Ursula commits appends faster than S2 Lite at every concurrency, since S2's per-append latency is bound by the S3 PUT round-trip. So S2 pays for a premium storage tier and still writes slower.

| | Ursula | S2 |
| --- | --- | --- |
| Protocol | Durable Streams over plain HTTP and SSE, portable clients | Custom API, locked to one vendor |
| Write latency | Single-digit to tens of ms p99 append | Hundreds of ms p99, bound by the S3 PUT |
| Storage tier | Plain S3 Standard, commodity per-GB price | S3 Express, around 7× the per-GB storage bill |
| Self-host | Raft cluster with quorum HA | Single process (Lite), no failover |

On a workload that accumulates terabytes in the cold tier, the storage gap is the larger line on your bill, and it grows every month. Pick S2 for a managed service when you can adopt its API and pay for Express-class storage. Pick Ursula for lower-latency writes on an open protocol with a commodity S3 bill.

## S3 directly

S3 is great for durable immutable blobs. It is a poor append log. Every append is a new object. Conditional writes are optimistic and retry-storm under load. Live readers have to poll.

Ursula uses S3 for the cold tier and snapshots, never the write path. Appends commit at Raft quorum in an in-memory hot ring, and a background flush moves older data to S3. One read serves hot and cold transparently. Use S3 directly for immutable blobs with no append or tail. Use Ursula when you need an append log with live readers.

## etcd

etcd is strongly consistent key-value over a single Raft log. It is excellent for small consistent state. Throughput is capped at one leader's serialization.

Ursula runs many small Raft groups across cores and nodes, so throughput scales with the cluster. There is no single-log order across the whole namespace, but each stream stays linearizable. Use etcd for small consistent cluster-wide state. Use Ursula for many independent durable streams.

## When to pick which

- Many small per-resource streams, open-internet HTTP clients, replay and live tail in one primitive: **Ursula**.
- A few high-throughput topics with consumer groups: **Kafka / Redpanda**.
- Managed service, willing to change API: **S2**.
- Durable immutable blobs, no append or tail: **S3 directly**.
- Small consistent key-value across the cluster: **etcd**.

---

# Durable Streams Protocol

> [!NOTE]
> This page is a verbatim mirror of the upstream Durable Streams Protocol specification, authored by ElectricSQL. The authoritative source is [github.com/durable-streams/durable-streams](https://github.com/durable-streams/durable-streams/blob/main/PROTOCOL.md). Please open issues and pull requests against that repository, not Ursula's. Ursula's own [extensions](/docs/specs/extensions) are documented separately.

**Document:** Durable Streams Protocol
**Version:** 1.0
**Author:** ElectricSQL

---

## Abstract

This document specifies the Durable Streams Protocol, an HTTP-based protocol for creating, appending to, and reading from durable, append-only byte streams. The protocol provides a simple, web-native primitive for applications requiring ordered, replayable data streams with support for catch-up reads, live tailing, and explicit stream closure (EOF). It is designed to be a foundation for higher-level abstractions such as event sourcing, database synchronization, collaborative editing, AI conversation histories, and finite response streaming.

## Copyright Notice

Copyright (c) 2025 ElectricSQL

## Table of Contents

1. [Introduction](#1-introduction)
2. [Terminology](#2-terminology)
3. [Protocol Overview](#3-protocol-overview)
4. [Stream Model](#4-stream-model)
 - 4.1. [Stream Closure](#41-stream-closure)
5. [HTTP Operations](#5-http-operations)
 - 5.1. [Create Stream](#51-create-stream)
 - 5.2. [Append to Stream](#52-append-to-stream)
 - 5.2.1. [Idempotent Producers](#521-idempotent-producers)
 - 5.3. [Close Stream](#53-close-stream)
 - 5.4. [Delete Stream](#54-delete-stream)
 - 5.5. [Stream Metadata](#55-stream-metadata)
 - 5.6. [Read Stream - Catch-up](#56-read-stream---catch-up)
 - 5.7. [Read Stream - Live (Long-poll)](#57-read-stream---live-long-poll)
 - 5.8. [Read Stream - Live (SSE)](#58-read-stream---live-sse)
6. [Offsets](#6-offsets)
7. [Content Types](#7-content-types)
8. [Caching and Collapsing](#8-caching-and-collapsing)
9. [Extensibility](#9-extensibility)
10. [Security Considerations](#10-security-considerations)
11. [IANA Considerations](#11-iana-considerations)
12. [References](#12-references)

---

## 1. Introduction

Modern web and cloud applications frequently require ordered, durable sequences of data that can be replayed from arbitrary points and tailed in real time. Common use cases include:

- Database synchronization and change feeds
- Event-sourced architectures
- Collaborative editing and CRDTs
- AI conversation histories and token streaming
- Workflow execution histories
- Real-time application state updates
- Finite response streaming (proxied HTTP responses, job outputs, file transfers)

While these patterns are widespread, the web platform lacks a simple, first-class primitive for durable streams. Applications typically implement ad-hoc solutions using combinations of databases, queues, and polling mechanisms, each reinventing similar offset-based replay semantics.

The Durable Streams Protocol provides a minimal HTTP-based interface for durable, append-only byte streams. It is intentionally low-level and byte-oriented, allowing higher-level abstractions to be built on top without protocol changes.

## 2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

**Stream**: A URL-addressable, append-only byte stream that can be read and written to. A stream is simply a URL. The protocol defines how to interact with that URL using HTTP methods, query parameters, and headers. Streams are durable and immutable by position. New data can only be appended.

**Offset**: An opaque, lexicographically sortable token that identifies a position within a stream. Clients use offsets to resume reading from a specific previously reached point.

**Content Type**: A MIME type set on stream creation that describes the format of the stream's bytes. The content type is returned on reads and may be used by clients to interpret message boundaries.

**Tail Offset**: The offset immediately after the last byte in the stream. This is the position where new appends will be written.

**Closed Stream**: A stream that has been explicitly closed by a writer. Once closed, a stream is in a terminal state: no further appends are permitted, and readers can observe the closure as an end-of-stream (EOF) signal. Closure is durable and monotonic - once closed, a stream remains closed.

## 3. Protocol Overview

The Durable Streams Protocol is an HTTP-based protocol that operates on URLs. A stream is simply a URL. The protocol defines how to interact with that URL using standard HTTP methods, query parameters, and custom headers.

The protocol defines operations to create, append to, read, close, delete, and query metadata for streams. Reads have three modes: catch-up, long-poll, and Server-Sent Events (SSE). The primary operations are:

1. **Create**: Establish a new stream at a URL with optional initial content (PUT)
2. **Append**: Add bytes to the end of an existing stream (POST)
3. **Close**: Transition a stream to closed state, optionally with a final append (POST with `Stream-Closed: true`)
4. **Read**: Retrieve bytes starting from a given offset, with support for catch-up and live modes (GET)
5. **Delete**: Remove a stream (DELETE)
6. **Head**: Query stream metadata without transferring data (HEAD)

The protocol does not prescribe a specific URL structure. Servers may organize streams using any URL scheme they choose (e.g., `/v1/stream/{path}`, `/{id}`, or domain-specific paths). The protocol is defined by the HTTP methods, query parameters, and headers applied to any stream URL.

Streams support arbitrary content types. The protocol operates at the byte level, leaving message framing and schema interpretation to clients.

**Independent Read/Write Implementation**: Servers **MAY** implement the read and write paths independently. For example, a database synchronization server may only implement the read path and use its own injection system for writes, while a collaborative editing service might implement both paths.

## 4. Stream Model

A stream is an append-only sequence of bytes with the following properties:

- **Durability**: Once written and acknowledged, bytes persist until the stream is deleted or expired
- **Immutability by Position**: Bytes at a given offset never change. New data is only appended
- **Ordering**: Bytes are strictly ordered by offset
- **Content Type**: Each stream has a MIME content type set at creation
- **TTL/Expiry**: Streams may have optional time-to-live or absolute expiry times
- **Retention**: Servers **MAY** implement retention policies that drop data older than a certain age while the stream continues. If a stream is deleted a new stream **SHOULD NOT** be created at the same URL.
- **Stream State**: A stream is either **open** (accepts appends) or **closed** (no further appends permitted). Streams start in the open state and transition to closed via an explicit close operation. This transition is **durable** (persisted) and **monotonic** (once closed, a stream cannot be reopened).

Clients track their position in a stream using offsets. Offsets are opaque to clients but are lexicographically sortable, allowing clients to determine ordering and resume from any point.

### 4.1. Stream Closure

Stream closure provides an explicit end-of-stream (EOF) signal that allows readers to distinguish between "no data yet" and "no more data ever." This is essential for finite streams where writers need to signal completion, such as:

- Proxied HTTP responses that have finished streaming
- Completed job outputs or workflow executions
- Finalized conversation histories or document streams

**Properties of stream closure:**

- **Durable**: The closed state is persisted and survives server restarts
- **Monotonic**: Once closed, a stream cannot be reopened
- **Idempotent**: Closing an already-closed stream succeeds (or returns a stable "already closed" response)
- **Observable**: Readers can detect closure and treat it as EOF
- **Atomic with final append**: Writers can atomically append a final message and close in a single operation

After closure, the stream's data remains fully readable. Only new appends are rejected.

**Stream-Closed Header Value:**

The `Stream-Closed` header uses the value `true` (case-insensitive) to indicate closure. Servers **MUST** treat the header as present only when its value is exactly `true` (case-insensitive comparison). Other values such as `false`, `yes`, `1`, or empty string **MUST** be treated as if the header were absent. Servers **SHOULD NOT** return error responses for non-`true` values. They simply ignore the header.

## 5. HTTP Operations

The protocol defines operations that are applied to a stream URL. The examples in this section use `{stream-url}` to represent any stream URL. Servers may implement any URL structure they choose. The protocol is defined by the HTTP methods, query parameters, and headers.

### 5.1. Create Stream

#### Request

```
PUT {stream-url}
```

Where `{stream-url}` is any URL that identifies the stream to be created.

Creates a new stream. If the stream already exists at `{stream-url}`, the server **MUST** either:

- return `200 OK` if the existing stream's configuration (content type, TTL/expiry, and closure status) matches the request, or
- return `409 Conflict` if it does not.

This provides idempotent "create or ensure exists" semantics aligned with HTTP PUT expectations.

**Closure status matching:** When checking for idempotent success (200 OK), servers **MUST** compare the `Stream-Closed` header in the request against the stream's current closure status. For example:

- `PUT /stream` (no `Stream-Closed`) to an **open** stream with matching config -> `200 OK`
- `PUT /stream` (no `Stream-Closed`) to a **closed** stream -> `409 Conflict` (closure status mismatch)
- `PUT /stream + Stream-Closed: true` to a **closed** stream with matching config -> `200 OK`
- `PUT /stream + Stream-Closed: true` to an **open** stream -> `409 Conflict` (closure status mismatch)

#### Request Headers (Optional)

- `Content-Type: <stream-content-type>`
 - Sets the stream's content type. If omitted, the server **MAY** default to `application/octet-stream`.

- `Stream-TTL: <seconds>`
 - Sets a relative time-to-live in seconds from creation. The value **MUST** be a non-negative integer in decimal notation without leading zeros, plus signs, decimal points, or scientific notation (e.g., `3600` is valid `+3600`, `03600`, `3600.0`, and `3.6e3` are not).

- `Stream-Expires-At: <rfc3339>`
 - Sets an absolute expiry time as an RFC 3339 timestamp.
 - If both `Stream-TTL` and `Stream-Expires-At` are supplied, servers **SHOULD** reject the request with `400 Bad Request`. Implementations **MAY** define a deterministic precedence rule, but **MUST** document it.

- `Stream-Closed: true` (optional)
 - When present, the stream is created in the **closed** state. Any body provided becomes the complete and final content of the stream.
 - This enables atomic "create and close" semantics for single-message or empty streams that are immediately complete (e.g., cached responses, placeholder errors, pre-computed results).
 - **Examples:**
 - `PUT /stream + Stream-Closed: true` (empty body): Creates an empty, immediately-closed stream (useful for "completed with no output" or error placeholders).
 - `PUT /stream + Stream-Closed: true + body`: Creates a single-shot stream with the body as its complete content (useful for cached responses, pre-computed results).

#### Request Body (Optional)

- Initial stream bytes. If provided, these bytes form the first content of the stream.

#### Response Codes

- `201 Created`: Stream created successfully
- `200 OK`: Stream already exists with matching configuration (idempotent success)
- `409 Conflict`: Stream already exists with different configuration
- `400 Bad Request`: Invalid headers or parameters (including conflicting TTL/expiry)
- `429 Too Many Requests`: Rate limit exceeded

#### Response Headers (on 201 or 200)

- `Location: {stream-url}` (on 201): Servers **SHOULD** include a `Location` header equal to `{stream-url}` in `201 Created` responses.
- `Content-Type: <stream-content-type>`: The stream's content type
- `Stream-Next-Offset: <offset>`: The tail offset after any initial content
- `Stream-Closed: true`: Present when the stream was created in the closed state

### 5.2. Append to Stream

#### Request

```
POST {stream-url}
```

Where `{stream-url}` is the URL of an existing stream.

Appends bytes to the end of an existing stream. Supports both full-body and streaming (chunked) append operations. Optionally closes the stream atomically with the append.

Servers that do not support appends for a given stream **SHOULD** return `405 Method Not Allowed` or `501 Not Implemented` to `POST` requests on that URL.

#### Request Headers

- `Content-Type: <stream-content-type>`
 - **MUST** match the stream's existing content type when a body is provided. Servers **MUST** return `409 Conflict` when the content type is valid but does not match the stream's configured type.
 - **MAY** be omitted when the request body is empty (i.e., close-only requests with `Stream-Closed: true`). When the request body is empty, servers **MUST NOT** reject based on `Content-Type` and **MAY** ignore it entirely. This ensures close-only requests remain robust even when clients/libraries attach default `Content-Type` headers.

- `Transfer-Encoding: chunked` (optional)
 - Indicates a streaming body. Servers **SHOULD** support HTTP/1.1 chunked encoding and HTTP/2 streaming semantics.

- `Stream-Seq: <string>` (optional)
 - A monotonic, lexicographic writer sequence number for coordination.
 - `Stream-Seq` values are opaque strings that **MUST** compare using simple byte-wise lexicographic ordering. Sequence numbers are scoped per authenticated writer identity (or per stream, depending on implementation). Servers **MUST** document the scope they enforce.
 - If provided and less than or equal to the last appended sequence (as determined by lexicographic comparison), the server **MUST** return `409 Conflict`. Sequence numbers **MUST** be strictly increasing.

- `Stream-Closed: true` (optional)
 - When present with value `true`, the stream is **closed** after the append completes. This is an atomic operation: the body (if any) is appended as the final data, and the stream transitions to the closed state in the same step.
 - If the request body is empty (Content-Length: 0 or no body), the stream is closed without appending any data. This is the only case where an empty POST body is valid.
 - Once closed, the stream rejects all subsequent appends with `409 Conflict` (see below).
 - **Close-only requests are idempotent**: if the stream is already closed and the request includes `Stream-Closed: true` with an empty body, servers **SHOULD** return `204 No Content` with `Stream-Closed: true`.
 - **Append-and-close requests are NOT idempotent** (without idempotent producer headers): if the stream is already closed and the request includes a body but no idempotent producer headers, servers **MUST** return `409 Conflict` with `Stream-Closed: true`, since the body cannot be appended. However, if idempotent producer semantics apply and the request matches the `(producerId, epoch, seq)` tuple that performed the closing append, servers treat it as a deduplicated success (see Section 5.2.1).

#### Request Body

- Bytes to append to the stream. Servers **MUST** reject POST requests with an empty body (Content-Length: 0 or no body) with `400 Bad Request`, **unless** the `Stream-Closed: true` header is present (which allows closing without appending data).

#### Response Codes

- `204 No Content`: Append successful (or stream already closed when closing idempotently)
- `400 Bad Request`: Malformed request (invalid header syntax, missing Content-Type, empty body without `Stream-Closed: true`)
- `404 Not Found`: Stream does not exist
- `405 Method Not Allowed` or `501 Not Implemented`: Append not supported for this stream
- `409 Conflict`: Content type mismatch with stream's configured type, sequence regression (if `Stream-Seq` provided), or **stream is closed** (when attempting to append without `Stream-Closed: true`)
- `413 Payload Too Large`: Request body exceeds server limits
- `429 Too Many Requests`: Rate limit exceeded

#### Response Headers (on success)

- `Stream-Next-Offset: <offset>`: The new tail offset after the append
- `Stream-Closed: true`: Present when the stream is now closed (either by this request or previously)

#### Response Headers (on 409 Conflict due to closed stream)

When a client attempts to append to a closed stream (without `Stream-Closed: true`), servers **MUST** return:

- `409 Conflict` status code
- `Stream-Closed: true` header
- `Stream-Next-Offset: <offset>`: The final offset of the closed stream (useful for clients to know the stream's final position)

This allows clients to detect and handle the "stream already closed" condition programmatically without parsing the response body. Servers **SHOULD** keep the response body empty or use a standardized error format. Clients **SHOULD NOT** rely on parsing the body to determine the reason for rejection.

**Error Precedence:** When an append request would trigger multiple conflict conditions (e.g., stream is closed AND content type mismatches), servers **SHOULD** check the stream's closed status first. This ensures clients receive the `Stream-Closed: true` header, enabling correct error handling. The recommended precedence is:

1. Stream closed -> `409 Conflict` with `Stream-Closed: true`
2. Content type mismatch -> `409 Conflict`
3. Sequence regression -> `409 Conflict`

### 5.2.1. Idempotent Producers

Durable Streams supports Kafka-style idempotent producers for exactly-once write semantics. This enables fire-and-forget writes with server-side deduplication, eliminating duplicates from client retries.

#### Design

- **Client-provided producer IDs**: Zero RTT overhead, no handshake required
- **Client-declared epochs, server-validated fencing**: Client increments epoch on restart. Server validates monotonicity and fences stale epochs
- **Per-batch sequence numbers**: Separate from `Stream-Seq`, used for retry safety
- **Two-layer sequence design**:
 - Transport layer: `Producer-Id` + `Producer-Epoch` + `Producer-Seq` (retry safety)
 - Application layer: `Stream-Seq` (cross-restart ordering, lexicographic)

#### Request Headers

All three producer headers **MUST** be provided together or none at all. If only some headers are provided, servers **MUST** return `400 Bad Request`.

- `Producer-Id: <string>`
 - Client-supplied stable identifier (e.g., "order-service-1", UUID)
 - **MUST** be a non-empty string. Empty values result in `400 Bad Request`
 - Identifies the logical producer across restarts

- `Producer-Epoch: <integer>`
 - Client-declared epoch, starting at 0
 - Increment on producer restart to establish a new session
 - Server validates that epoch is monotonically non-decreasing
 - **MUST** be a non-negative integer ≤ 2^53-1 (for JavaScript interoperability)

- `Producer-Seq: <integer>`
 - Monotonically increasing sequence number per epoch
 - Starts at 0 for each new epoch
 - Applies per-batch (per HTTP request), not per-message
 - **MUST** be a non-negative integer ≤ 2^53-1 (for JavaScript interoperability)

#### Response Headers

- `Producer-Epoch: <integer>`: Echoed back on success (200/204), or current server epoch on stale epoch (403)
- `Producer-Seq: <integer>`: On success (200/204), the highest accepted sequence number for this `(stream, producerId, epoch)` tuple. Enables clients to confirm pipelined requests and recover state after crashes.
- `Producer-Expected-Seq: <integer>`: On 409 Conflict (sequence gap), the expected sequence
- `Producer-Received-Seq: <integer>`: On 409 Conflict (sequence gap), the received sequence

#### Validation Logic

```
# Epoch validation (client-declared, server-validated)
if epoch < state.epoch:
  -> 403 Forbidden
  -> Headers: Producer-Epoch: <current epoch>

if epoch > state.epoch:
  if seq != 0:
    -> 400 Bad Request (new epoch must start at seq=0)
  -> Accept: update state.epoch = epoch, state.lastSeq = 0
  -> 200 OK (new epoch established)

# Same epoch: sequence validation
if seq <= state.lastSeq:
  -> 204 No Content (duplicate, idempotent success)
  -> Return the exact original byte and record ranges for this seq

if seq == state.lastSeq + 1:
  -> Accept, update state.lastSeq = seq
  -> 200 OK

if seq > state.lastSeq + 1:
  -> 409 Conflict
  -> Headers: Producer-Expected-Seq: <lastSeq + 1>, Producer-Received-Seq: <seq>
```

#### Response Codes (with Producer Headers)

- `200 OK`: Append successful (new data)
- `204 No Content`: Duplicate append (idempotent success, data already exists)
- `400 Bad Request`: Invalid producer headers (e.g., non-integer values, epoch increase with seq != 0)
- `403 Forbidden`: Stale producer epoch (zombie fencing). Response includes `Producer-Epoch` header with current server epoch.
- `409 Conflict`: Sequence gap detected. Response includes `Producer-Expected-Seq` and `Producer-Received-Seq` headers.

#### Bootstrap and Restart Flow

1. **Initial start (epoch=0)**:
 - Producer sends `(epoch=0, seq=0)`
 - Server accepts, establishes producer state

2. **Producer restart**:
 - Producer increments local epoch (0 -> 1), resets seq to 0
 - Sends `(epoch=1, seq=0)`
 - Server sees epoch > state.epoch, accepts, updates state

3. **Zombie fencing**:
 - Old producer (zombie) still sending `(epoch=0, seq=N)` gets 403 Forbidden
 - Response includes `Producer-Epoch: 1` header

#### Auto-claim Flow (for ephemeral producers)

For serverless or ephemeral producers without persisted epoch:

1. Producer starts fresh with `(epoch=0, seq=0)`
2. If server has `state.epoch=5`, returns 403 with `Producer-Epoch: 5`
3. Client can retry with `(epoch=6, seq=0)` to claim the producer ID

This is opt-in client behavior and should be used with caution.

#### Concurrency Requirements

Servers **MUST** serialize validation + append operations per `(stream, producerId)` pair. HTTP requests can arrive out-of-order. Without serialization, seq=1 arriving before seq=0 would cause false sequence gaps.

#### Atomicity Requirements

For persistent storage, servers **SHOULD** commit producer state updates and log appends atomically (e.g., in a single database transaction). Non-atomic implementations have a crash window where:

1. Data is appended to the log
2. Crash occurs before producer state is updated
3. On recovery, a retry may be re-accepted, causing duplicate data

**Recovery for non-atomic stores**: Clients can bump their epoch after a crash to establish a clean session. This trades "exactly once within epoch" for "at least once across crashes" which is acceptable for many use cases. Stores **SHOULD** document their atomicity guarantees clearly.

#### Producer State Cleanup

Servers **MAY** implement TTL-based cleanup for producer state:

- **In-memory stores**: 7 days TTL recommended, clean up on stream access
- **Persistent stores**: Retain as long as stream data exists (stronger guarantee)

After state expiry, the producer is treated as new. A zombie alive past TTL expiry can write again, which is acceptable for testing but persistent stores should use longer retention.

#### Stream Closure with Idempotent Producers

Idempotent producers can close streams using the `Stream-Closed: true` header. The behavior is:

- **Close with final append**: Include body, producer headers, and `Stream-Closed: true`. The append is deduplicated normally, and the stream closes atomically with the final append.
- **Close without append**: Include `Stream-Closed: true` with empty body. Producer headers are optional but if provided, the close operation is still idempotent.
- **Duplicate close**: If the stream was already closed by the same `(producerId, epoch, seq)` tuple, servers **SHOULD** return `204 No Content` with `Stream-Closed: true`.

When a closed stream receives an append from an idempotent producer:

- If the `(producerId, epoch, seq)` matches the request that closed the stream, return `204 No Content` (duplicate/idempotent success) with `Stream-Closed: true`
- Otherwise, return `409 Conflict` with `Stream-Closed: true` (stream is closed, no further appends allowed)

### 5.3. Close Stream

To close a stream without appending data, send a POST request with `Stream-Closed: true` and an empty body:

#### Request

```
POST {stream-url}
Stream-Closed: true
```

#### Response Codes

- `204 No Content`: Stream closed successfully (or already closed - idempotent)
- `404 Not Found`: Stream does not exist
- `405 Method Not Allowed` or `501 Not Implemented`: Append/close not supported for this stream

#### Response Headers

- `Stream-Next-Offset: <offset>`: The tail offset (unchanged, since no data was appended)
- `Stream-Closed: true`: Confirms the stream is now closed

This is the canonical "close-only" operation. For atomic "append final message and close", include a request body as described in Section 5.2.

### 5.4. Delete Stream

#### Request

```
DELETE {stream-url}
```

Where `{stream-url}` is the URL of the stream to delete.

Deletes the stream and all its data. In-flight reads may terminate with a `404 Not Found` on subsequent requests after deletion.

#### Response Codes

- `204 No Content`: Stream deleted successfully
- `404 Not Found`: Stream does not exist
- `405 Method Not Allowed` or `501 Not Implemented`: Delete not supported for this stream

### 5.5. Stream Metadata

#### Request

```
HEAD {stream-url}
```

Where `{stream-url}` is the URL of the stream. Checks stream existence and returns metadata without transferring a body. This is the canonical way to find the tail offset, TTL, expiry information, and **closure status**.

#### Response Codes

- `200 OK`: Stream exists
- `404 Not Found`: Stream does not exist
- `429 Too Many Requests`: Rate limit exceeded

#### Response Headers (on 200)

- `Content-Type: <stream-content-type>`: The stream's content type
- `Stream-Next-Offset: <offset>`: The tail offset (next offset after the current end)
- `Stream-TTL: <seconds>` (optional): Remaining time-to-live, if applicable
- `Stream-Expires-At: <rfc3339>` (optional): Absolute expiry time, if applicable
- `Stream-Closed: true` (optional): Present when the stream has been closed. Absence indicates the stream is still open.
- `Cache-Control`: See Section 8

#### Caching Guidance

Servers **SHOULD** make `HEAD` responses effectively non-cacheable, for example by returning `Cache-Control: no-store`. Servers **MAY** use `Cache-Control: private, max-age=0, must-revalidate` as an alternative, but `no-store` is recommended to avoid stale tail offsets and closure status.

### 5.6. Read Stream - Catch-up

#### Request

```
GET {stream-url}?offset=<offset>
```

Where `{stream-url}` is the URL of the stream. Returns bytes starting from the specified offset. This is used for catch-up reads when a client needs to replay stream content from a known position.

#### Query Parameters

- `offset` (optional)
 - Start offset token. If omitted, defaults to the stream start (offset -1).

#### Response Codes

- `200 OK`: Data available (or empty body if offset equals tail)
- `400 Bad Request`: Malformed offset or invalid parameters
- `404 Not Found`: Stream does not exist
- `410 Gone`: Offset is before the earliest retained position (retention/compaction)
- `429 Too Many Requests`: Rate limit exceeded

For non-live reads without data beyond the requested offset, servers **SHOULD** return `200 OK` with an empty body and `Stream-Next-Offset` equal to the requested offset. If the stream is closed, this response **MUST** also include `Stream-Closed: true` to signal EOF.

#### Response Headers (on 200)

- `Cache-Control`: Derived from TTL/expiry (see Section 8)
- `ETag: {internal_stream_id}:{start_offset}:{end_offset}`
 - Entity tag for cache validation
- `Stream-Cursor: <cursor>` (optional for catch-up, required for live modes)
 - Cursor to echo on subsequent long-poll requests to improve CDN collapsing. Servers **MAY** include this on catch-up reads. It is **required** for live modes when the stream is open (see Sections 5.7, 5.8). Servers **MAY** omit it when `Stream-Closed` is true. Clients **MUST** tolerate its absence when `Stream-Closed` is present.
- `Stream-Next-Offset: <offset>`
 - The next offset to read from (for subsequent requests)
- `Stream-Up-To-Date: true`
 - **MUST** be present and set to `true` when the response includes all data available in the stream at the time the response was generated (i.e., when the requested offset has reached the tail and no more data exists).
 - **SHOULD NOT** be present when returning partial data due to server-defined chunk size limits (when more data exists beyond what was returned).
 - Clients **MAY** use this header to determine when they have caught up and can transition to live tailing mode.
 - **Important:** `Stream-Up-To-Date: true` does **NOT** imply EOF. More data may be appended in the future. Only `Stream-Closed: true` indicates that no more data will ever arrive.
- `Stream-Closed: true`
 - **MUST** be present when the stream is closed **and** the client has reached the final offset **at the time the response is generated**. This includes:
 - Responses that return the final chunk of data, when the stream is already closed at response generation time, or
 - Responses with an empty body when the requested offset equals the tail offset of a closed stream (the canonical EOF signal).
 - When present, clients can conclude that no more data will ever be appended and treat this as EOF.
 - **SHOULD NOT** be present when returning partial data from a closed stream (when more data exists between the response and the final offset). In this case, `Stream-Closed: true` will be returned on a subsequent request that reaches the final offset.
 - **Timing note:** If a stream is closed **after** the final chunk was served (or cached), that chunk will not include `Stream-Closed: true`. Clients discover closure by requesting the next offset (`Stream-Next-Offset` from the previous response), which returns an empty body with `Stream-Closed: true`. This is the expected flow when closure occurs between chunk responses or when serving cached chunks.
 - Clients that need to know closure status before reaching the tail **SHOULD** use `HEAD` (see Section 5.5).

#### Response Body

- Bytes from the stream starting at the specified offset, up to a server-defined maximum chunk size.

### 5.7. Read Stream - Live (Long-poll)

#### Request

```
GET {stream-url}?offset=<offset>&live=long-poll[&cursor=<cursor>]
```

Where `{stream-url}` is the URL of the stream. If no data is available at the specified offset, the server waits up to a timeout for new data to arrive. This enables efficient live tailing without constant polling.

#### Query Parameters

- `offset` (required)
 - The offset to read from. **MUST** be provided.

- `live=long-poll` (required)
 - Indicates long-polling mode.

- `cursor` (optional)
 - Echo of the last `Stream-Cursor` header value from a previous response.
 - Used for collapsing keys in CDN/proxy configurations.

#### Response Codes

- `200 OK`: Data became available within the timeout
- `204 No Content`: Timeout expired with no new data
- `400 Bad Request`: Invalid parameters
- `404 Not Found`: Stream does not exist
- `429 Too Many Requests`: Rate limit exceeded

#### Response Headers (on 200)

- Same as catch-up reads (Section 5.6), plus:
- `Stream-Cursor: <cursor>`: Servers **MUST** include this header. See Section 8.1.

#### Response Headers (on 204)

- `Stream-Next-Offset: <offset>`: Servers **MUST** include a `Stream-Next-Offset` header indicating the current tail offset.
- `Stream-Up-To-Date: true`: Servers **MUST** include this header to indicate the client is caught up with all available data.
- `Stream-Cursor: <cursor>`: Servers **MUST** include this header when the stream is open. Servers **MAY** omit this header when `Stream-Closed` is true (cursor is unnecessary when no further polling is expected). Clients **MUST** tolerate its absence when `Stream-Closed` is present. See Section 8.1.
- `Stream-Closed: true`: **MUST** be present when the stream is closed (see Section 5.6 for semantics). A `204 No Content` with `Stream-Closed: true` indicates EOF.

**EOF Signaling Across Modes:**

Clients should treat **either** of the following as EOF, depending on the mode used:

- **Catch-up mode**: `200 OK` with empty body and `Stream-Closed: true`
- **Long-poll mode**: `204 No Content` with `Stream-Closed: true`
- **SSE mode**: `control` event with `streamClosed: true`

In all cases, `Stream-Closed` / `streamClosed` is the definitive EOF signal. The presence of `Stream-Up-To-Date` / `upToDate` alone does **not** indicate EOF - it only means the client has caught up with currently available data, but more may arrive.

#### Stream Closure Behavior in Long-poll Mode

When the stream is closed and the client is already at the tail offset:

- Servers **MUST NOT** wait for the long-poll timeout
- Servers **MUST** immediately return `204 No Content` with `Stream-Closed: true` and `Stream-Up-To-Date: true`

This ensures clients observing a closed stream do not have hanging connections waiting for data that will never arrive.

#### Response Body (on 200)

- New bytes that arrived during the long-poll period.

#### Timeout Behavior

The timeout for long-polling is implementation-defined. Servers **MAY** accept a `timeout` query parameter (in seconds) as a future extension, but this is not required by the base protocol.

### 5.8. Read Stream - Live (SSE)

#### Request

```
GET {stream-url}?offset=<offset>&live=sse
```

Where `{stream-url}` is the URL of the stream. Returns data as a Server-Sent Events (SSE) stream.

SSE mode supports all content types. Servers **MUST** include `stream-data-content-type` to identify the decoded `event: data` payload type. For `application/json` streams, this value is `application/x-ndjson`. For all other streams, this value is the stream's configured content type. For streams with `content-type: text/*` or `application/json`, data events carry UTF-8 text directly. For streams with any other `content-type` (binary streams), servers **MUST** automatically base64-encode data events and include the response header `stream-sse-data-encoding: base64`.

SSE responses **MUST** use `Content-Type: text/event-stream` in the HTTP response headers.

Clients **MUST** check `stream-sse-data-encoding` before interpreting `event: data`. If it is `base64`, clients decode the event data first, then interpret the decoded bytes according to `stream-data-content-type`. For `application/x-ndjson` SSE payloads, `data:` lines are transport lines, not record boundaries; clients MUST buffer until newline before parsing records.

#### Query Parameters

- `offset` (required)
 - The offset to start reading from.

- `live=sse` (required)
 - Indicates SSE streaming mode.

- `max_bytes` (optional)
 - Maximum bytes to read in one data batch before SSE framing. For non-base64 UTF-8 data, servers **MUST NOT** emit invalid UTF-8; if a read window ends inside a code point, servers shorten the emitted payload and set `streamNextOffset` to the actual emitted byte position. For `application/x-ndjson` SSE payloads, servers **SHOULD** prefer ending at the last complete newline-delimited record when one is available, but **MAY** emit a partial record to make progress.

#### Response Codes

- `200 OK`: Streaming body (SSE format)
- `400 Bad Request`: Invalid parameters
- `404 Not Found`: Stream does not exist
- `429 Too Many Requests`: Rate limit exceeded

#### Response Format

Data is emitted in [Server-Sent Events format](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format).

**Events:**

- `data`: Emitted for each batch of data
 - Each line prefixed with `data:`
 - The decoded data payload type is identified by the response header `stream-data-content-type`.
 - `data:` lines are transport framing and are not guaranteed to align with application message boundaries.
 - For binary streams (where `stream-sse-data-encoding: base64` is present), the `data` event payload represents bytes encoded using standard base64 per [RFC 4648](https://www.rfc-editor.org/rfc/rfc4648) (alphabet: A-Z, a-z, 0-9, +, /).
 - Servers **MAY** split the base64 text across multiple `data:` lines within the same SSE `data` event.
 - Clients **MUST** concatenate the `data:` lines for the event (per SSE rules) and **MUST** remove all `\n` and `\r` characters inserted between lines before base64-decoding.
 - The resulting string (after removing `\n` and `\r`) **MUST** be valid base64 text with length that is a multiple of 4 (or empty).
 - If a `data` event's byte payload length is 0, the base64 text **MUST** be the empty string.
 - Base64 encoding affects only `event: data` payloads. `event: control` events remain JSON as specified and are not encoded.
 - When the stream content type is `application/json`, implementations **MAY** batch multiple logical messages into a single SSE `data` event by streaming newline-delimited JSON across multiple `data:` lines, as in the example below.
- `control`: Emitted after every data event
 - **MUST** include `streamNextOffset`. See Section 8.1.
 - **MUST** include `streamCursor` when the stream is open. Servers **MAY** omit `streamCursor` when `streamClosed` is true (cursor is unnecessary when no reconnection is expected).
 - **MUST** include `upToDate: true` when the client is caught up with all available data. Note: `streamClosed: true` implies `upToDate: true` (a closed stream at the final offset is by definition up-to-date), so `upToDate` **MAY** be omitted when `streamClosed` is true.
 - **MUST** include `streamClosed: true` when the stream is closed and all data up to the final offset has been sent.
 - Format: JSON object with offset, cursor (when applicable), up-to-date status, and optionally closed status. Field names use camelCase: `streamNextOffset`, `streamCursor`, `upToDate`, and `streamClosed`.

**Example (normal data):**

```
event: data
data: {"k":"v"}
data: {"k":"w"}

event: control
data: {"streamNextOffset":"123456_789","streamCursor":"abc"}
```

**Example (final data with stream closure):**

```
event: data
data: {"k":"final"}

event: control
data: {"streamNextOffset":"123456_999","streamClosed":true}
```

Note: `streamCursor` is omitted when `streamClosed` is true, since clients must not reconnect after receiving a closed signal.

**Client Compatibility:** Clients **MUST** tolerate the absence of `streamCursor` (in SSE) and `Stream-Cursor` (in HTTP headers) when `streamClosed` / `Stream-Closed` is present. Implementations that assume cursor is always present will break when processing closed stream responses.

#### Stream Closure Behavior in SSE Mode

When the stream is closed:

- The final `control` event **MUST** include `streamClosed: true`
- After emitting the final control event, servers **MUST** close the SSE connection
- Clients receiving `streamClosed: true` **MUST NOT** attempt to reconnect, as no more data will arrive

If the stream is already closed when an SSE connection is established and the client's offset is at the tail:

- Servers **MUST** immediately emit a `control` event with `streamClosed: true` and `upToDate: true`
- Servers **MUST** then close the connection

**Example (binary stream with automatic base64 encoding):**

```
event: data
data: AQIDBAUG
data: BwgJCg==

event: control
data: {"streamNextOffset":"123456_789","streamCursor":"abc"}
```

#### Connection Lifecycle

- Server **SHOULD** close connections roughly every ~60 seconds to enable CDN collapsing
- Client **MUST** reconnect using the last received `streamNextOffset` value from the control event
- Client **MUST NOT** reconnect if the last control event included `streamClosed: true`

## 6. Offsets

Offsets are opaque tokens that identify positions within a stream. They have the following properties:

1. **Opaque**: Clients **MUST NOT** interpret offset structure or meaning
2. **Lexicographically Sortable**: For any two valid offsets for the same stream, a lexicographic comparison determines their relative position in the stream. Clients **MAY** compare offsets lexicographically to determine ordering.
3. **Persistent**: Offsets remain valid for the lifetime of the stream (until deletion or expiration)
4. **Unique**: Each offset identifies exactly one position in the stream. No two positions **MAY** share the same offset.
5. **Strictly Increasing**: Offsets assigned to appended data **MUST** be lexicographically greater than all previously assigned offsets. Server implementations **MUST NOT** use schemes (such as raw UTC timestamps) that can produce duplicate or non-monotonic offsets. Time-based identifiers like ULIDs, which combine timestamps with random components to guarantee uniqueness and monotonicity, are acceptable.

**Format**: Offset tokens are opaque, case-sensitive strings. Their internal structure is implementation-defined. Offsets are single tokens and **MUST NOT** contain `,`, `&`, `=`, `?`, or `/` (to avoid conflict with URL query parameter syntax). Servers **SHOULD** use URL-safe characters to avoid encoding issues, but clients **MUST** properly URL-encode offset values when including them in query parameters. Servers **SHOULD** keep offsets reasonably short (under 256 characters) since they appear in every request URL.

**Sentinel Values**: The protocol defines two special offset sentinel values:

- **`-1` (Stream Beginning)**: The special offset value `-1` represents the beginning of the stream. Clients **MAY** use `offset=-1` as an explicit way to request data from the start. This is semantically equivalent to omitting the offset parameter. Servers **MUST** recognize `-1` as a valid offset that returns data from the beginning of the stream.

- **`now` (Current Tail Position)**: The special offset value `now` allows clients to skip all existing data and begin reading from the current tail position. This is useful for applications that only care about future data (e.g., presence tracking, live monitoring, late joiners to a conversation). The behavior varies by read mode:

 **Catch-up mode** (`offset=now` without `live` parameter):
 - Servers **MUST** return `200 OK` with an empty response body.
 - Servers **MUST** include a `Stream-Next-Offset` header set to the current tail position
 - Servers **MUST** include `Stream-Up-To-Date: true` header
 - Servers **SHOULD** return `Cache-Control: no-store` to prevent caching of the tail offset
 - The response **MUST** contain no data messages, regardless of stream content

 **Long-poll mode** (`offset=now&live=long-poll`):
 - Servers **MUST** immediately begin waiting for new data (no initial empty response)
 - This eliminates a round-trip: clients can subscribe to future data in a single request
 - If new data arrives during the wait, servers return `200 OK` with the new data
 - If the timeout expires, servers return `204 No Content` with `Stream-Up-To-Date: true`
 - The `Stream-Next-Offset` header **MUST** be set to the tail position

 **SSE mode** (`offset=now&live=sse`):
 - Servers **MUST** immediately begin the SSE stream from the tail position
 - The first control event **MUST** include the tail offset in `streamNextOffset`
 - If no data has arrived, the first control event **MUST** include `upToDate: true`
 - If data arrives before the first control event, `upToDate` reflects the current state
 - No historical data is sent. Only future data events are streamed

 **Closed streams** (`offset=now` on a closed stream):
 - Regardless of the `live` parameter, servers **MUST** return immediately with the closure signal
 - The response **MUST** include `Stream-Closed: true` and `Stream-Up-To-Date: true` headers
 - The `Stream-Next-Offset` header **MUST** be set to the stream's final (tail) offset
 - For catch-up mode: `200 OK` with empty body
 - For long-poll mode: `204 No Content` (no waiting, immediate return)
 - For SSE mode: The first (and only) control event includes `streamClosed: true` and `upToDate: true`, then the connection closes
 - This ensures clients using `offset=now` can immediately discover that a stream has no future data

**Reserved Values**: The sentinel values `-1` and `now` are reserved by the protocol. Server implementations **MUST NOT** generate these strings as actual stream offsets (in `Stream-Next-Offset` headers or SSE control events). This ensures clients can always distinguish between sentinel requests and real offset values.

The opaque nature of offsets enables important server-side optimizations. For example, offsets may encode chunk file identifiers, allowing catch-up requests to be served directly from object storage without touching the main database.

Clients **MUST** use the `Stream-Next-Offset` value returned in responses for subsequent read requests. They **SHOULD** persist offsets locally (e.g., in browser local storage or a database) to enable resumability after disconnection or restart.

## 7. Content Types

The protocol supports arbitrary MIME content types. Most content types operate at the byte level, leaving message framing and interpretation to clients. The `application/json` content type has special semantics defined below.

**SSE Encoding:**

- SSE mode (Section 5.8) supports all content types. For streams with `content-type: text/*` or `application/json`, data events carry UTF-8 text natively. For all other content types, servers automatically base64-encode data events (see Section 5.8).

Clients **MAY** use any content type for their streams, including:

- `application/json` for JSON mode with message boundary preservation
- `application/ndjson` for newline-delimited JSON
- `application/x-protobuf` for Protocol Buffer messages
- `text/plain` for plain text
- Custom types for application-specific formats

### 7.1. JSON Mode

Streams created with `Content-Type: application/json` have special semantics for message boundaries and batch operations.

#### Message Boundaries

For `application/json` streams, servers **MUST** preserve message boundaries by storing each normalized message as one newline-delimited JSON record. `HEAD` responses report the configured stream content type (`application/json`). GET responses **MUST** return those records as newline-delimited JSON with `Content-Type: application/x-ndjson`, because that is the read representation.

#### Array Flattening for Batch Operations

When a POST request body contains a JSON array, servers **MUST** flatten exactly one level of the array, treating each element as a separate message. This enables clients to batch multiple messages in a single HTTP request while preserving individual message semantics.

**Examples (direct POST to server):**

- POST body `{"event": "created"}` stores one message: `{"event": "created"}`
- POST body `[{"event": "a"}, {"event": "b"}]` stores two messages: `{"event": "a"}`, `{"event": "b"}`
- POST body `[[1,2], [3,4]]` stores two messages: `[1,2]`, `[3,4]`
- POST body `[[[1,2,3]]]` stores one message: `[[1,2,3]]`

**Note:** Client libraries **MAY** automatically wrap individual values in arrays for batching. For example, a client calling `append({"x": 1})` might send POST body `[{"x": 1}]` to the server, which flattens it to store one message: `{"x": 1}`.

#### Empty Arrays

Servers **MUST** reject POST requests containing empty JSON arrays (`[]`) with `400 Bad Request`. Empty arrays in append operations represent no-op operations with no semantic meaning and likely indicate a client bug.

PUT requests with an empty array body (`[]`) are valid and create an empty stream. The empty array simply means no initial messages are being written.

#### JSON Validation

Servers **MUST** validate that appended data is valid JSON. If validation fails, servers **MUST** return `400 Bad Request` with an appropriate error message.

#### Response Format

GET responses for `application/json` streams **MUST** return `Content-Type: application/x-ndjson` with a body containing newline-delimited JSON messages from the requested offset range. `max_bytes` applies to the encoded byte stream and MAY end a response mid-record; clients MUST resume from `Stream-Next-Offset` and buffer incomplete trailing records before parsing.

```http
HTTP/1.1 200 OK
Content-Type: application/x-ndjson

{"event":"created"}
{"event":"updated"}
```

If no messages exist in the range, servers **MUST** return an empty body.

## 8. Caching and Collapsing

### 8.1. Catch-up and Long-poll Reads

For **shared, non-user-specific streams**, servers **SHOULD** return:

```
Cache-Control: public, max-age=60, stale-while-revalidate=300
```

For **streams that may contain user-specific or confidential data**, servers **SHOULD** use `private` instead of `public` and rely on CDN configurations that respect `Authorization` or other cache keys:

```
Cache-Control: private, max-age=60, stale-while-revalidate=300
```

This enables CDN/proxy caching while allowing stale content to be served during revalidation.

**Caching and Stream Closure:**

Catch-up chunks remain fully cacheable, including chunks at the tail of the stream. When a chunk is returned, it may or may not be the final chunk - this is unknown until the client requests the next offset.

The closure signal is discovered when the client requests the offset **after** the final data:

1. Client reads data and receives `Stream-Next-Offset: X` (the tail offset)
2. Client requests offset `X`
3. If stream is closed: server returns `200 OK` with **empty body** and `Stream-Closed: true`
4. If stream is open: server returns `200 OK` with empty body and `Stream-Up-To-Date: true` (or long-poll/SSE waits for data)

This design ensures:

- All data chunks are cacheable (a chunk that later becomes "final" was still valid data)
- The closure signal is a distinct request/response at the tail offset
- Cached chunks never become "stale" due to closure - clients simply make one more request to discover EOF

**ETag Usage:**

Servers **MUST** generate `ETag` headers for GET responses, except for `offset=now` responses. Clients **MAY** use `If-None-Match` with the `ETag` value on repeat catch-up requests. When a client provides a valid `If-None-Match` header that matches the current ETag, servers **MUST** respond with `304 Not Modified` (with no body) instead of re-sending the same data. This is essential for fast loading and efficient bandwidth usage.

**ETag and Stream Closure:** ETags **MUST** vary with the stream's closure status. When a stream is closed (without new data being appended), the ETag **MUST** change to ensure clients do not receive `304 Not Modified` responses that would hide the closure signal. Implementations **SHOULD** include a closure indicator in the ETag format (e.g., appending `:c` to the ETag when the stream is closed).

**Query Parameter Ordering:**

For optimal cache behavior, clients **SHOULD** order query parameters lexicographically by key name. This ensures consistent URL serialization across implementations and improves CDN cache hit rates.

**Collapsing:**

Clients **SHOULD** echo the `Stream-Cursor` value as `cursor=<cursor>` in subsequent long-poll requests. This, along with the appropriate `Cache-Control` header, enables CDNs and proxies to collapse multiple clients waiting for the same data into a single upstream request.

**Server-Generated Cursors:**

To prevent infinite CDN cache loops (where clients receive the same cached empty response indefinitely), servers **MUST** generate cursors on all live mode responses:

- **Long-poll**: `Stream-Cursor` response header
- **SSE**: `streamCursor` field in `control` events

The cursor mechanism works as follows:

1. **Interval-based Calculation**: Servers divide time into fixed intervals (default: 20 seconds) counted from an epoch (default: October 9, 2024 00:00:00 UTC). The cursor value is the interval number as a decimal string.

2. **Cursor Generation**: For each live response, the server calculates the current interval number and returns it as the cursor value.

3. **Monotonic Progression**: Servers **MUST** ensure cursors never go backwards. When a client provides a `cursor` query parameter that is greater than or equal to the current interval number, the server **MUST** return a cursor strictly greater than the client's cursor (by adding random jitter of 1-3600 seconds). This guarantees monotonic progression and prevents cache cycles.

4. **Client Behavior**: Clients **MUST** include the received cursor value as the `cursor` query parameter in subsequent requests. This creates different cache keys as time progresses, ensuring CDN caches eventually expire.

**Example Cursor Flow:**

```
# Client makes initial long-poll request
GET /stream?offset=123&live=long-poll

# Server returns cursor based on current interval (e.g., interval 1000)
< Stream-Cursor: 1000

# Client echoes cursor on next request
GET /stream?offset=123&live=long-poll&cursor=1000

# If still in same interval, server adds jitter and returns advanced cursor
< Stream-Cursor: 1050
```

**Long-poll Caching:**

CDNs and proxies **SHOULD NOT** cache `204 No Content` responses from long-poll requests in most cases. Long-poll `200 OK` responses are safe to cache when keyed by `offset`, `cursor`, and authentication credentials.

### 8.2. SSE

SSE connections **SHOULD** be closed by the server approximately every 60 seconds. This enables new clients to collapse onto edge requests rather than maintaining long-lived connections to origin servers.

## 9. Extensibility

The Durable Streams Protocol is designed to be extended for specific use cases and implementations. Extensions **SHOULD** be pure supersets of the base protocol, ensuring compatibility with any client that implements the base protocol.

### 9.1. Protocol Extensions

Implementations **MAY** extend the protocol with additional query parameters, headers, or response fields to support domain-specific semantics. For example, a database synchronization implementation might add query parameters to filter by table or schema, or include additional metadata in response headers.

Extensions **SHOULD** follow these principles:

- **Backward Compatibility**: Extensions **MUST NOT** break base protocol semantics. Clients that do not understand extension parameters or headers **MUST** be able to operate using only base protocol features.

- **Pure Superset**: Extensions **SHOULD** be additive only. New parameters and headers **SHOULD** be optional, and servers **SHOULD** provide sensible defaults or fallback behavior when extensions are not used.

- **Version Independence**: Extensions **SHOULD** work with any version of a client that implements the base protocol. Extension negotiation **MAY** be handled through headers or query parameters, but base protocol operations **MUST** remain functional without extension support.

### 9.2. Authentication Extensions

See Section 10.1 for authentication and authorization details. Implementations **MAY** extend the protocol with authentication-related query parameters or headers (e.g., API keys, OAuth tokens, custom authentication headers).

## 10. Security Considerations

### 10.1. Authentication and Authorization

Authentication and authorization are explicitly out of scope for this protocol specification. Clients **SHOULD** implement all standard HTTP authentication primitives (e.g., Basic Authentication [RFC7617], Bearer tokens [RFC6750], Digest Authentication [RFC7616]). Implementations **MUST** provide appropriate access controls to prevent unauthorized stream creation, modification, or deletion, but may do so using any mechanism they choose, including extending the protocol with authentication-related parameters or headers as described in Section 9.2.

### 10.2. Multi-tenant Safety

If stream URLs are guessable, servers **MUST** enforce access controls even when using shared caches. Servers **SHOULD** validate and sanitize stream URLs to prevent path traversal attacks and ensure URL components are within acceptable limits.

### 10.3. Untrusted Content

Clients **MUST** treat stream contents as untrusted input and **MUST NOT** evaluate or execute stream data without appropriate validation. This is particularly important for append-only streams used as logs, where log injection attacks are a concern.

### 10.4. Content Type Validation

Servers **MUST** validate that appended content types match the stream's declared content type to prevent type confusion attacks.

### 10.5. Rate Limiting

Servers **SHOULD** implement rate limiting to prevent abuse. The `429 Too Many Requests` response code indicates rate limit exhaustion.

### 10.6. Sequence Validation

The optional `Stream-Seq` header provides protection against out-of-order writes in multi-writer scenarios. Servers **MUST** reject sequence regressions to maintain stream integrity.

### 10.7. Browser Security Headers

When serving streams to browser clients, servers **SHOULD** include the following headers to prevent MIME-sniffing attacks, cross-origin embedding exploits, and cache-related vulnerabilities:

- `X-Content-Type-Options: nosniff`
 - Servers **SHOULD** include this header on all responses. This prevents browsers from MIME-sniffing the response content and potentially executing it as a different content type (e.g., interpreting binary data as HTML/JavaScript).

- `Cross-Origin-Resource-Policy: cross-origin` (or `same-origin`/`same-site`)
 - Servers **SHOULD** include this header to explicitly control cross-origin embedding. Use `cross-origin` to allow cross-origin access via `fetch()`, `same-site` to restrict to the same registrable domain, or `same-origin` for strict same-origin only. This prevents Cross-Origin Read Blocking (CORB) issues and protects against Spectre-like side-channel attacks.

- `Cache-Control: no-store`
 - Servers **SHOULD** include this header on HEAD responses and on responses containing sensitive or user-specific stream data. This prevents intermediate proxies and CDNs from caching potentially sensitive content. For public, non-sensitive historical reads, servers **MAY** use `Cache-Control: public, max-age=60, stale-while-revalidate=300` as described in Section 8.

- `Content-Disposition: attachment` (optional)
 - Servers **MAY** include this header for `application/octet-stream` responses to prevent inline rendering if a user navigates directly to the stream URL.

These headers provide defense-in-depth for scenarios where stream URLs might be accessed outside the intended programmatic fetch context (e.g., direct navigation, malicious cross-origin embedding via `<script>` or `<img>` tags).

### 10.8. TLS

All protocol operations **MUST** be performed over HTTPS (TLS) in production environments to protect data in transit.

## 11. IANA Considerations

### 11.1. Default Port

The default port for standalone Durable Streams servers is **4437/tcp** (with 4437/udp reserved for future use).

This port was selected from the IANA unassigned range 4434-4440. Standalone server implementations **SHOULD** use port 4437 as the default when no explicit port is configured. When Durable Streams is integrated into an existing web server or application framework, it **SHOULD** use the host server's port instead.

### 11.2. HTTP Headers

This document requests registration of the following HTTP headers in the "Permanent Message Header Field Names" registry:

| Field Name | Status | Reference |
| -------------------- | --------- | ------------- |
| `Stream-TTL` | permanent | This document |
| `Stream-Expires-At` | permanent | This document |
| `Stream-Seq` | permanent | This document |
| `Stream-Cursor` | permanent | This document |
| `Stream-Next-Offset` | permanent | This document |
| `Stream-Up-To-Date` | permanent | This document |
| `Stream-Closed` | permanent | This document |

**Descriptions:**

- `Stream-TTL`: Relative time-to-live for streams (seconds)
- `Stream-Expires-At`: Absolute expiry time for streams (RFC 3339 timestamp)
- `Stream-Seq`: Writer sequence number for coordination (opaque string)
- `Stream-Cursor`: Cursor for CDN collapsing (opaque string)
- `Stream-Next-Offset`: Next offset for subsequent reads (opaque string)
- `Stream-Up-To-Date`: Indicates up-to-date response (presence header)
- `Stream-Closed`: Indicates stream is closed / end-of-stream (presence header, value `true`)

## 12. References

### 12.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, [https://www.rfc-editor.org/info/rfc2119](https://www.rfc-editor.org/info/rfc2119).

[RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet: Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002, [https://www.rfc-editor.org/info/rfc3339](https://www.rfc-editor.org/info/rfc3339).

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, [https://www.rfc-editor.org/info/rfc8174](https://www.rfc-editor.org/info/rfc8174).

[RFC9110] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110, June 2022, [https://www.rfc-editor.org/info/rfc9110](https://www.rfc-editor.org/info/rfc9110).

[RFC9113] Thomson, M., Ed. and C. Benfield, Ed., "HTTP/2", RFC 9113, DOI 10.17487/RFC9113, June 2022, [https://www.rfc-editor.org/info/rfc9113](https://www.rfc-editor.org/info/rfc9113).

[RFC7617] Reschke, J., "The 'Basic' HTTP Authentication Scheme", RFC 7617, DOI 10.17487/RFC7617, September 2015, [https://www.rfc-editor.org/info/rfc7617](https://www.rfc-editor.org/info/rfc7617).

[RFC6750] Jones, M. and D. Hardt, "The OAuth 2.0 Authorization Framework: Bearer Token Usage", RFC 6750, DOI 10.17487/RFC6750, October 2012, [https://www.rfc-editor.org/info/rfc6750](https://www.rfc-editor.org/info/rfc6750).

[RFC7616] Shekh-Yusef, R., Ed., Ahrens, D., and S. Bremer, "HTTP Digest Access Authentication", RFC 7616, DOI 10.17487/RFC7616, September 2015, [https://www.rfc-editor.org/info/rfc7616](https://www.rfc-editor.org/info/rfc7616).

### 12.2. Informative References

[SSE] Hickson, I., "Server-Sent Events", W3C Recommendation, February 2015, [https://www.w3.org/TR/eventsource/](https://www.w3.org/TR/eventsource/).

---

**Full Copyright Statement**

Copyright (c) 2025 ElectricSQL

This document and the information contained herein are provided on an "AS IS" basis. ElectricSQL disclaims all warranties, express or implied, including but not limited to any warranty that the use of the information herein will not infringe any rights or any implied warranties of merchantability or fitness for a particular purpose.

---

# Durable Streams Extensions

**Document:** Durable Streams Protocol Extensions
**Status:** Draft
**Date:** 2026-07-18

---

## Abstract

This document defines extensions to the [Durable Streams Protocol](/docs/specs/durable-stream). It adds bucket namespacing, optional path affinity, snapshot and bootstrap semantics, multipart bootstrap delivery, mutable stream attributes, Ursula-specific append batching for high-throughput small-event writes, and stable record coordinates for JSON streams.

The snapshot and bootstrap design in sections 2 and 3 was proposed by [Loro](https://loro.dev/); Ursula adopts that design.

All base protocol semantics remain in effect. Where this document is silent, the base protocol governs.

## Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174].

## Table of Contents

- [Extension Capability Advertisement](#extension-capability-advertisement)
1. [Buckets](#1-buckets)
2. [Snapshots](#2-snapshots)
3. [Bootstrap](#3-bootstrap)
4. [Append Batch](#4-append-batch)
5. [Stream Attributes](#5-stream-attributes)
6. [JSON Record Coordinates](#6-json-record-coordinates)
7. [Ordinary SSE Compatibility](#7-ordinary-sse-compatibility)
8. [URL Encoding](#8-url-encoding)

---

## Extension Capability Advertisement

`Stream-Extensions` is a response header for advertising protocol extensions that are active for the target stream. Its value is a comma-separated list of extension tokens registered by the specifications that define them. A server MUST advertise only tokens whose extension semantics apply to the target stream; server-wide support for an extension is not sufficient. An extension specification defines its token and the responses on which an active implementation MUST advertise it.

Clients MUST ignore unrecognized extension tokens. A client that depends on an extension MUST verify its token before relying on extension-specific request parameters, response headers, or body semantics.

## 1. Buckets

Every stream belongs to a bucket. The stream URL takes the form `{base_url}/{bucket_id}/{stream_id}`.

### 1.1. Identifier Constraints

- `bucket_id`: MUST match `^[a-z0-9_-]{4,64}$`. Globally unique within the service.
- `stream_id`: Any UTF-8 string. MUST NOT exceed 122 bytes. MUST NOT contain `/`, `\0`, or `..`. On the bucketed `/` surface, the combined `{bucket_id}/{stream_id}` key MUST also not exceed 122 bytes.
- The literal local stream ID `streams` is reserved for the bucket listing endpoint `/{bucket_id}/streams`.

### 1.2. Create Bucket

```
PUT /{bucket_id}
```

Buckets MUST be created explicitly. `PUT /{bucket_id}/{stream_id}` MUST NOT implicitly create a bucket.

**Response Codes:**

- `201 Created`: Bucket created successfully.
- `400 Bad Request`: `bucket_id` is invalid.
- `409 Conflict`: Bucket already exists.

### 1.3. Get Bucket Metadata

```
GET /{bucket_id}
```

**Response Codes:**

- `200 OK`: Bucket exists. Returns a JSON object with at least `bucket_id` (string) and `streams` (integer, count of streams in the bucket).
- `400 Bad Request`: `bucket_id` is invalid.
- `404 Not Found`: Bucket does not exist.

### 1.4. List Bucket Streams

```
GET /{bucket_id}/streams?prefix={prefix}&after={cursor}&limit={n}
```

- `prefix` filters bucket-local stream IDs by prefix. When omitted, all bucket-local stream IDs are eligible.
- `after` is an exclusive cursor over bucket-local stream IDs.
- `limit` defaults to `1000` and must be in `1..=1000`.

The response body is:

```json
{
  "bucket_id": "agents",
  "prefix": "user-",
  "stream_count": 2,
  "streams": [
    {
      "stream_id": "user-1",
      "status": "Open",
      "content_type": "text/plain",
      "tail_offset": 42,
      "created_at_ms": 1735689600000,
      "last_write_at_ms": 1735689601000
    }
  ],
  "next_cursor": "user-2",
  "has_more": true
}
```

`streams` is sorted lexicographically by `stream_id`.

**Response Codes:**

- `200 OK`: Matching streams returned.
- `400 Bad Request`: `bucket_id` or query parameters are invalid.
- `404 Not Found`: Bucket does not exist.

### 1.5. Delete Bucket

```
DELETE /{bucket_id}
```

A bucket MUST be empty (zero streams) before deletion.

**Response Codes:**

- `204 No Content`: Bucket deleted.
- `400 Bad Request`: `bucket_id` is invalid.
- `404 Not Found`: Bucket does not exist.
- `409 Conflict`: Bucket is not empty.

### 1.6. Stream Operations

All stream operations defined in the base protocol apply under `{base_url}/{bucket_id}/{stream_id}`. When the bucket does not exist, the server MUST return `404 Not Found`.

### 1.7. Path Affinity

Path affinity is an optional routing extension for applications that need several independent streams to share one Ursula Raft group. Its extension token is `path-affinity-v1`.

The affinity form of a stream URL is:

```
{base_url}/{bucket_id}/{affinity_key}/{stream_id}
```

For example, `/workflow/run-42/journal` and `/workflow/run-42/queue` have the same affinity key and therefore route to the same Raft group. The ordinary two-segment form, such as `/workflow/run-42-journal`, has no affinity key and continues to hash the complete bucket and stream identity so unrelated streams are distributed across groups.

- `affinity_key` follows the same UTF-8 and path-safety constraints as `stream_id`.
- The complete `{bucket_id}/{affinity_key}/{stream_id}` identity MUST NOT exceed 122 bytes.
- Within one bucket, streams with the same `affinity_key` MUST route to the same Raft group.
- Streams with different affinity keys MAY still share a Raft group. Clients MUST NOT infer isolation, group IDs, or physical placement from different keys.
- Path affinity changes placement only. It does not by itself make separate HTTP operations atomic.
- `$transaction`, `append-batch`, `attrs`, `bootstrap`, `retention`, and `snapshot` are reserved local stream IDs under an affinity path because those names identify affinity-level operations or stream subresources on the two-segment form.

Responses for a stream addressed through the affinity form MUST advertise `path-affinity-v1` in `Stream-Extensions`. Responses for ordinary two-segment stream URLs MUST NOT advertise it merely because the server supports the extension.

### 1.8. Group-local Append Transaction

The `group-append-transaction-v1` extension atomically appends to existing streams that share one path affinity key:

```
POST /{bucket_id}/{affinity_key}/$transaction
Content-Type: application/json

{
  "operations": [
    {
      "stream": "journal",
      "content_type": "application/json",
      "payload_base64": "eyJ0eXBlIjoic3RlcF9jb21taXR0ZWQifQo=",
      "record_match": 44
    },
    {
      "stream": "queue",
      "content_type": "application/json",
      "payload_base64": "eyJ0eXBlIjoiYWNrIn0K",
      "record_match": 9
    }
  ]
}
```

The server MUST commit every operation in one Raft entry or commit none of them. A failed content-type check, producer check, sequence check, record precondition, quota check, closed-stream check, or missing-stream check MUST leave every targeted stream unchanged. The endpoint does not create streams, accept empty append payloads, or coordinate across affinity keys. The first version permits at most 64 operations and a 32 MiB request body.

Each operation accepts `stream`, `content_type`, `payload_base64`, and the optional fields `close_after`, `stream_seq`, `producer`, and `record_match`. `producer` uses the same `producer_id`, `producer_epoch`, and `producer_seq` fields as the producer extension. Payload bytes use standard padded base64 so the transaction envelope can carry arbitrary stream content without JSON reinterpretation.

The response is `200 OK` with a JSON array of append acknowledgements in request order. Any operation failure returns the normal Durable Streams error status for that failure, such as `412 Precondition Failed`, and no operation is visible. Responses MUST advertise both `path-affinity-v1` and `group-append-transaction-v1` in `Stream-Extensions`.

---

## 2. Snapshots

A snapshot is a materialized representation of a stream's content from offset `-1` (inclusive) to `snapshot_offset` (exclusive). Snapshots enable clients to skip full replay and resume from a compacted state.

### 2.1. Offset Conventions

- Offsets are opaque tokens as defined in the base protocol. Clients MUST use server-returned `Stream-Next-Offset` values.

Servers that expose snapshots and `/bootstrap` as message sequences MUST use one consistent retained-message boundary model across ordinary reads, snapshot-offset validation, and bootstrap responses. This extension does not require any particular binary framing format.

### 2.2. Publish Snapshot

```
PUT {stream_url}/snapshot/{snapshot_offset}
```

Creates a snapshot at the specified offset. The snapshot represents the materialized result of folding all updates in the range `[-1, snapshot_offset)`.

**Request Headers:**

- `Content-Type`: The content type of the snapshot blob. Servers MUST store this value and return it on subsequent reads. Defaults to `application/octet-stream`.
- `Stream-Snapshot-Match`: Optional digest of the currently visible snapshot. If supplied and it does not match, the server MUST reject the publish with `409 Conflict`.

**Response Codes:**

- `204 No Content`: Snapshot published successfully.
- `400 Bad Request`: `snapshot_offset` is invalid or not aligned to a committed message boundary.
- `404 Not Found`: Stream does not exist.
- `409 Conflict`: `snapshot_offset` exceeds the current tail, or the server cannot produce a consistent view at that offset.
- `410 Gone`: `snapshot_offset` is older than the current earliest retained offset.
- `413 Payload Too Large`: Snapshot exceeds the server's size limit.

**Idempotency:**

The server MUST return `Stream-Snapshot-Digest` for a successful publish. Repeating a publish at the same offset with the same content type and body MUST succeed and return the same digest. Publishing different content at an already-published offset MUST return `409 Conflict`.

**Retention Effect:**

Publishing a snapshot MUST NOT change the earliest retained offset. Retention is a separate, explicit operation:

```
PUT {stream_url}/retention/{retained_offset}
```

The server MUST reject a retention boundary that is not aligned to committed data, moves backward, or is newer than the latest published snapshot. After success, offsets less than `retained_offset` MUST return `410 Gone`.

For record-aware JSON streams, servers also expose:

```
PUT {stream_url}/snapshot?record={record}
PUT {stream_url}/retention?record={record}
```

The server resolves the stable record ordinal to the canonical byte boundary before applying the operation.

Publishing a new snapshot replaces the previously visible snapshot immediately. The superseded snapshot MAY be garbage-collected asynchronously after the new snapshot becomes visible. Clients MUST NOT depend on older snapshots remaining readable after overwrite.

**Concurrency:**

Snapshot creation MUST NOT block concurrent appends. The snapshot's consistency boundary is `snapshot_offset`. Updates at or beyond that offset MUST NOT be folded into the snapshot.

### 2.3. Read Latest Snapshot

```
GET {stream_url}/snapshot
```

Returns a redirect to the latest visible snapshot resource, if one exists.

**Response:**

- `307 Temporary Redirect` when a latest snapshot exists. The `Location` header points to `{stream_url}/snapshot/{snapshot_offset}`.
- `404 Not Found` when no snapshot exists.

### 2.4. Snapshot Metadata on HEAD

```
HEAD {stream_url}
```

Servers SHOULD include `Stream-Snapshot-Offset` when a latest visible snapshot exists for the stream.

**Response Headers:**

- `Stream-Snapshot-Offset`: The latest visible snapshot offset.
- `Stream-Snapshot-Digest`: A digest of the latest visible snapshot's content type and body.

**Semantics:**

- Absence of `Stream-Snapshot-Offset` means the stream has no visible snapshot.
- When present, the value MUST refer to the same latest visible snapshot that `GET {stream_url}/snapshot` would resolve to.

### 2.5. Read Snapshot

```
GET {stream_url}/snapshot/{snapshot_offset}
```

Returns the snapshot blob at the specified offset.

**Response Headers:**

- `Content-Type`: The content type stored at publish time.
- `Stream-Snapshot-Offset`: The snapshot offset.
- `Stream-Snapshot-Digest`: The snapshot digest returned at publish time.
- `Stream-Next-Offset`: The next offset after the snapshot.
- `Stream-Up-To-Date`: Boolean indicating whether the stream has updates beyond the snapshot.

**Response Codes:**

- `200 OK`: Snapshot exists.
- `404 Not Found`: Snapshot does not exist or has been garbage-collected.

### 2.6. Delete Snapshot

```
DELETE {stream_url}/snapshot/{snapshot_offset}
```

Deletes the snapshot identified by `snapshot_offset`, subject to bootstrap safety rules.

**Response Codes:**

- `404 Not Found`: Snapshot does not exist or has been superseded.
- `409 Conflict`: `snapshot_offset` refers to the latest visible snapshot and cannot be deleted because `/bootstrap` would become incomplete.

Implementations that expose only one visible snapshot MAY make superseded snapshots unreachable immediately after overwrite. In that case, deleting an older offset will return `404 Not Found`.

---

## 3. Bootstrap

Bootstrap provides single-request initialization: a snapshot (if any) plus all retained updates after the snapshot point, returned as a single ordered response that preserves per-message content types.

### 3.1. Request

```
GET {stream_url}/bootstrap
```

**Query Parameters:**

- `/bootstrap` is a one-shot initialization endpoint. It does not define any query parameters of its own.
- Servers SHOULD reject any `live` query parameter on `/bootstrap` with `400 Bad Request`.

### 3.2. Response

**Response Headers:**

- `Content-Type: multipart/mixed; boundary=<token>`
- `Stream-Snapshot-Offset`: The snapshot offset, or `-1` if no snapshot exists.
- `Stream-Next-Offset`: The next offset after all returned data.
- `Stream-Up-To-Date`: Boolean.

**Response Body:**

The body is an RFC 2046 `multipart/mixed` entity. Each MIME part is one logical bootstrap message. The multipart boundary, not any outer binary framing, defines the bootstrap message boundaries.

1. **First part**: Snapshot message. If a snapshot exists, the part body MUST be the raw snapshot bytes and the part `Content-Type` MUST equal the snapshot blob's stored content type.
2. **Subsequent parts**: Retained updates after the snapshot point, one update message per part. Each part body MUST be exactly one retained update message, and the part `Content-Type` MUST be the content type of that update. For streams with a fixed stream-level content type, all update parts will normally share that value.

If no snapshot exists, the server MUST still emit an empty first part, `Stream-Snapshot-Offset` MUST be `-1`, and that empty part's `Content-Type` MUST be `application/octet-stream`. Update parts then begin at the earliest retained offset.

Servers MUST NOT wrap the entire bootstrap response in an outer binary framing container. Each update part contains exactly one retained update message from the stream. Implementations MUST preserve the same retained-message boundaries they use for ordinary reads and snapshot-offset validation, and they MUST NOT reinterpret or reframe binary payloads when constructing bootstrap parts.

**Example:**

```http
HTTP/1.1 200 OK
Content-Type: multipart/mixed; boundary=rr-bootstrap-9f1c2e7a
Stream-Snapshot-Offset: 0000000000000012
Stream-Next-Offset: 0000000000000026
Stream-Up-To-Date: true
Cache-Control: no-store

--rr-bootstrap-9f1c2e7a
Content-Type: application/octet-stream

<snapshot-bytes>
--rr-bootstrap-9f1c2e7a
Content-Type: application/json

{"op":"set","path":["title"],"value":"hello"}
--rr-bootstrap-9f1c2e7a
Content-Type: application/json

{"op":"insert","path":["body",0],"value":"world"}
--rr-bootstrap-9f1c2e7a--
```

**Response Codes:**

- `200 OK`: Bootstrap data returned.
- `404 Not Found`: Stream does not exist.

### 3.3. Follow-Up Live Reads

Bootstrap is one-shot only:

- `/bootstrap` returns an ordered initialization payload once.
- `/bootstrap` does not support `live=sse` or `live=long-poll`.
- After applying the bootstrap response, clients MUST continue tailing through the ordinary stream read APIs using the returned `Stream-Next-Offset`.
- Clients SHOULD prefer `GET {stream_url}?offset=<offset>&live=sse` and fall back to `GET {stream_url}?offset=<offset>&live=long-poll` when SSE is unavailable.

### 3.4. Compatibility

- Regular `GET {stream_url}?offset=...` MUST NOT return snapshot bytes. Snapshots are only delivered through `/bootstrap`.
- When a server returns `410 Gone` for a read (because retention has advanced), clients SHOULD call `/bootstrap` to rebuild state, then continue tailing from the returned `Stream-Next-Offset` through the ordinary read APIs.
- If a server has begun returning `410 Gone` for a stream (i.e., earliest retained offset > `-1`), the server MUST ensure `/bootstrap` is available and can provide a snapshot covering that earliest retained offset.
- Clients MUST parse `/bootstrap` as an ordered message sequence. Message boundaries come from MIME parts.

---

## 4. Append Batch

Append Batch is an Ursula-specific implementation extension. It is not part of the base Durable Streams Protocol and clients MUST NOT assume it exists on other Durable Streams servers.

The extension reduces HTTP request overhead for workloads that append many small independent records. It is a transport batching mechanism only: each frame in the request is committed as an independent logical append with its own acknowledgement. Servers MUST NOT concatenate the frames into one stream record.

### 4.1. Request

Bucketed Ursula route:

```
POST /{bucket_id}/{stream_id}/append-batch
```

The request `Content-Type` applies to every frame and MUST match the stream's configured content type.

The request body is a sequence of binary frames:

```
uint32_be payload_len
payload bytes
uint32_be payload_len
payload bytes
...
```

`payload_len` is a 32-bit unsigned integer encoded in network byte order. It is the number of payload bytes that immediately follow. Zero-length frames are invalid.

Ursula currently accepts at most 512 frames per public append-batch request and a 32 MiB total body. Servers MAY reject larger batches with `413 Payload Too Large`.

### 4.2. Unsupported Per-Append Controls

The current Ursula append-batch endpoint does not support controls that require per-frame headers:

- `Stream-Closed`
- `Producer-Id`, `Producer-Epoch`, `Producer-Seq`
- `Stream-Record-Match`
- `Stream-Seq`

Clients that need producer fencing, idempotent producer sequence checks, stream preconditions, or atomic stream close MUST use the standard `POST {stream_url}` append operation.

### 4.3. Response

On a syntactically valid batch request, the response is `200 OK` with a JSON array. The array order matches the request frame order. Each element acknowledges one logical append:

```json
[
  { "status": 204, "stream_next_offset": 3 },
  { "status": 204, "stream_next_offset": 6 }
]
```

Acknowledgement fields:

- `status`: HTTP status code for the frame-level logical append.
- `stream_next_offset`: Present when the frame append succeeds or when an error can report the stream tail.
- `stream_closed`: Present and `true` if the append observes a closed stream.
- `error_code`: Present for failed frame-level appends.
- `error`: Human-readable failure detail for failed frame-level appends.

Malformed framing, missing `Content-Type`, unsupported headers, or an oversized request MAY fail the entire HTTP request with `400 Bad Request` or `413 Payload Too Large`.

Frame-level failures are reported inside the JSON array. A successful outer `200 OK` does not mean every frame committed. Clients MUST inspect every acknowledgement.

### 4.4. Ordering and Compatibility

Frames are submitted in request order and response acknowledgements preserve that order. For a single stream, successful frames advance the same stream offsets that standard `POST {stream_url}` would have produced.

All standard stream read APIs remain unchanged. Data appended through Append Batch is read through ordinary `GET`, long-poll, SSE, snapshot, and bootstrap paths using the same offsets and content type rules as standard appends.

---

## 5. Stream Attributes

Stream Attributes are an Ursula-specific metadata extension for mutable, application-owned stream attributes. They are separate from the base protocol's stream metadata returned by `HEAD {stream_url}`.

Attributes are not part of the append-only byte stream. Updating attributes MUST NOT change stream offsets, `ETag`, content type, closed state, snapshots, bootstrap payloads, or ordinary read results.

### 5.1. Attribute Object

The attribute document is a JSON object with the following optional fields:

```json
{
  "title": "Support session",
  "metadata": {
    "purpose": "customer-support",
    "environment_id": "env_019e2590d33f711fabf42f2857cecd8a",
    "agent": {
      "id": "agent_019e390add9f7bac9b6cc806db46fcbd",
      "version": 2
    }
  }
}
```

Fields:

- `title`: Optional string. Interpreted by the application.
- `metadata`: Optional JSON object for arbitrary application metadata.

Ursula does not define first-class semantics for application concepts such as agents or environments. If an application wants to associate such information with a stream, it SHOULD store that information inside `metadata`.

Unknown top-level fields are ignored: servers MUST accept attribute objects containing unrecognized top-level fields and MUST NOT store them. Clients that need additional fields MUST place them inside `metadata`.

The encoded attribute object MUST NOT exceed 16 KiB. Servers MUST reject larger attribute documents with `400 Bad Request`. Attributes are replicated with stream state, so the limit keeps log entries and snapshots small.

An empty attribute object (`{}`) is equivalent to no attributes.

### 5.2. Create Stream With Attributes

Attributes MAY be supplied when creating a stream:

```http
PUT {stream_url}
Stream-Attrs: {"title":"Support session","metadata":{"purpose":"customer-support"}}
```

`Stream-Attrs` is an optional request header whose value is the JSON attribute object.

When a stream already exists, the create request is idempotent only if the existing stream attributes match the supplied attributes after empty-object normalization. A mismatch MUST return `409 Conflict`, the same as other create-configuration mismatches.

### 5.3. Read Attributes

```http
GET {stream_url}/attrs
```

**Response:**

- `200 OK`: Returns the current attribute object as `application/json`.
- `404 Not Found`: The stream does not exist or has expired.

When no attributes are set, the response body is `{}`.

### 5.4. Replace Attributes

```http
PUT {stream_url}/attrs
Content-Type: application/json

{"title":"Renamed session","metadata":{"purpose":"debugging"}}
```

The request body replaces the complete attribute object. Partial merge semantics are not defined by this extension.

**Response Codes:**

- `204 No Content`: Attributes were replaced, or the submitted object already matched the stored attributes.
- `400 Bad Request`: The request is missing `Content-Type`, the content type is not `application/json`, the body is not a valid attribute JSON object, or the attribute object exceeds the size limit.
- `404 Not Found`: The stream does not exist or has expired.

Submitting `{}` clears the stream attributes. Attributes MAY be updated after a stream is closed because stream closure only forbids further appends.

---

## 6. JSON Record Coordinates

The JSON Record Coordinates extension adds a stable logical message coordinate to streams configured with `Content-Type: application/json`. It is a pure superset of the base protocol: offsets remain the canonical exact-resume cursor, ordinary offset reads remain byte-oriented, and clients that do not understand record coordinates continue to operate unchanged.

This extension registers the token `json-record-coordinates-v1`. It is active only for an `application/json` stream for which the server maintains the record model defined below. Responses to `HEAD` and stream operations targeting such a stream MUST advertise this token in `Stream-Extensions`. A server MUST NOT advertise this token for a stream on which the extension is inactive, including a stream with a non-JSON content type.

Record-aware clients MUST verify this token before relying on record query parameters or response headers and MUST reject a record-aware response that omits it.

### 6.1. Record Model

Each logical JSON message has a zero-based unsigned integer record ordinal. Ordinals are assigned consecutively in committed stream order and are scoped to one stream.

For one stream:

- `F` is the first retained record ordinal.
- `N` is the next record ordinal and record tail.
- Retained records occupy `[F, N)`.
- `O(r)` is the canonical Durable Streams offset at the first byte of record `r`.

A new empty stream has `F = N = 0`. An ordinal MUST NOT be reused or changed after it becomes externally visible. Retention MAY advance `F`, but MUST NOT renumber surviving records.

Record ordinals are independent from offsets, `Stream-Seq`, idempotent-producer sequence numbers, and sequence fields inside application JSON values. Clients MAY perform arithmetic on ordinals, but MUST continue to treat offsets as opaque tokens.

The JSON-mode normalization rules in the base protocol define the records:

- A non-array JSON value creates one record.
- A top-level JSON array creates one record per array element after exactly one level of flattening.
- An empty array creates no records on `PUT` and remains invalid on `POST`.

An implementation MUST preserve each normalized message boundary and its ordinal across hot storage, WAL, cold flush, compaction, snapshot, restart, and retention. Coalescing physical storage objects MUST NOT coalesce the logical record coordinate space.

### 6.2. Client Event Time

This extension does not assign a server timestamp to records. Applications that need event time SHOULD include one client-supplied timestamp in each JSON value, for example a `captured_at` field in a telemetry event.

Client event times MAY be out of order and MUST NOT change committed record order, ordinals, canonical bytes, or offsets. Their HTTP metadata representation and any secondary index over them are outside this extension.

### 6.3. Create and Append Acknowledgements

When `PUT` creates a JSON stream with initial messages, or `POST` appends one or more JSON messages, successful responses MUST include:

- `Stream-Record-Start`: Inclusive ordinal of the first record created by this operation.
- `Stream-Record-Next`: Exclusive ordinal after the last record created by this operation.

For an operation that creates no records, `Stream-Record-Start` and `Stream-Record-Next` MUST both equal the current record tail.

All messages from one JSON request body MUST commit atomically and receive one contiguous ordinal range, or the operation MUST fail without making any ordinal visible. Concurrent appenders MUST receive disjoint ranges in committed stream order.

When an idempotent-producer retry is deduplicated, the server MUST return the original `Stream-Record-Start` and `Stream-Record-Next` values.

For example, if the current record tail is `42`:

```http
POST {stream_url}
Content-Type: application/json

[
  {"type":"navigation_started"},
  {"type":"network_response","status":500}
]
```

The successful response includes:

```http
Stream-Extensions: json-record-coordinates-v1
Stream-Record-Start: 42
Stream-Record-Next: 44
Stream-Next-Offset: <opaque-offset>
```

### 6.4. Conditional Append by Record Tail

A writer MAY require the current record tail to match an expected value:

```http
Stream-Record-Match: 42
```

The server MUST accept the append only when the current `N` equals the supplied unsigned integer. A mismatch MUST return `412 Precondition Failed` with the current `Stream-Record-Next` and `Stream-Next-Offset`. This precondition composes with, but does not replace, `Stream-Seq` or idempotent-producer headers.

### 6.5. Record Metadata on HEAD

For a JSON stream implementing this extension, `HEAD {stream_url}` MUST include:

- `Stream-Record-First`: First retained record ordinal `F`.
- `Stream-Record-Next`: Next record ordinal `N`.

### 6.6. Record-Aware Read Parameters

A record-aware read selects its starting position with exactly one of:

- `record=<ordinal>`: Start at the specified record boundary.
- `record=now`: Start at the current record tail.
- `tail_records=<count>`: Start at `max(F, N - count)`.

These parameters are mutually exclusive with each other and with `offset`. `tail_records` MUST be a non-negative integer.

The following rules apply:

- `record < F` MUST return `410 Gone` with `Stream-Record-First` and `Stream-Record-Next`.
- `record = N` is a valid up-to-date empty read.
- `record > N` MUST return `400 Bad Request` with `Stream-Record-Next`.
- `tail_records=0` is equivalent to `record=now`.

Record-aware reads MAY also specify `max_records=<positive-integer>`. `max_records` limits the number of complete records returned. `max_records` without a record-aware start parameter MUST return `400 Bad Request`.

`max_bytes` and `max_records` MUST NOT appear in the same request. A record-aware read that includes `max_bytes` MUST return `400 Bad Request`. This keeps the base protocol's partial-byte behavior separate from the extension's complete-record guarantee.

### 6.7. Record-Aligned Read Responses

A successful record-aware catch-up or long-poll response MUST:

- begin at a complete record boundary;
- contain only complete normalized JSON messages;
- return `Content-Type: application/x-ndjson` by default;
- return at most `max_records` messages when supplied;
- include `Stream-Record-First`, `Stream-Record-Start`, `Stream-Record-Next`, and `Stream-Next-Offset`.

`Stream-Record-Start` is the resolved starting ordinal. `Stream-Record-Next` is the continuation ordinal after the last complete returned record. `Stream-Next-Offset` MUST identify the same logical boundary as `Stream-Record-Next`.

An empty successful response MUST set `Stream-Record-Start` and `Stream-Record-Next` to the resolved starting ordinal.

Example:

```http
HTTP/1.1 200 OK
Content-Type: application/x-ndjson
Stream-Extensions: json-record-coordinates-v1
Stream-Record-First: 0
Stream-Record-Start: 42
Stream-Record-Next: 44
Stream-Next-Offset: <opaque-offset>

{"type":"navigation_started"}
{"type":"network_response","status":500}
```

Ordinary `GET {stream_url}?offset=...` behavior is unchanged and MAY still end in the middle of a JSON record as defined by the base protocol.

### 6.8. Optional Record Envelope View

A record-aware client MAY request `record_view=envelope`. This parameter MUST be combined with a record-aware start parameter.

The response MUST use:

```http
Content-Type: application/vnd.durable-stream-records+ndjson
```

Each response line MUST be a JSON object containing:

- `record`: Record ordinal.
- `value`: The original normalized JSON value.

Example:

```json
{"record":42,"value":{"captured_at":"2026-07-18T10:29:59.900Z","type":"navigation_started"}}
{"record":43,"value":{"captured_at":"2026-07-18T10:30:00.100Z","type":"network_response","status":500}}
```

The envelope is an alternate read representation only. It MUST NOT change canonical stream bytes, offsets, record ordinals, ETags for the canonical representation, snapshots, bootstrap data, or ordinary reads.

### 6.9. Live Reads and SSE

Record start parameters compose with `live=long-poll` and `live=sse`.

SSE control events for record-aware reads MUST add:

```json
{
  "streamNextOffset": "<opaque-offset>",
  "streamFirstRecord": 0,
  "streamNextRecord": 44,
  "upToDate": true
}
```

For the default JSON view, SSE data events retain the base protocol's JSON-mode representation and MAY contain multiple newline-delimited messages. Control-event record coordinates MUST refer to the boundary after every complete message represented before that control event.

With `record_view=envelope`, each SSE `event: data` MUST contain exactly one envelope object and the response MUST include `Stream-Data-Content-Type: application/vnd.durable-stream-record+json`.

A reconnecting client MAY use either the last `streamNextRecord` or `streamNextOffset`. Offset-based reconnection remains the base-protocol path.

### 6.10. Retention, Snapshots, and Bootstrap

Retention MUST advance `F` only at a record boundary. A trimmed ordinal MUST return `410 Gone`; surviving records keep their original ordinals.

When a snapshot advances the earliest retained offset, that offset MUST map to one record boundary. Snapshot publication at an intra-record offset MUST return `400 Bad Request`. Snapshot and bootstrap responses SHOULD include `Stream-Record-First` and `Stream-Record-Next` when the underlying stream implements this extension. Bootstrap update parts MUST preserve one logical JSON message per update part.

Implementations MAY reclaim ordinal index entries only after the corresponding records fall before `F` and are no longer required by a visible snapshot or bootstrap response.

### 6.11. Append Batch Integration

For JSON streams implementing this extension, each successful Append Batch frame receives its own contiguous record range. Frame acknowledgements defined in Section 4.3 MUST additionally include:

- `stream_record_start`;
- `stream_record_next`.

One outer Append Batch request remains transport batching, not an atomic multi-frame transaction. A failed frame assigns no visible record ordinal; successful later frames continue from the last committed record tail.

### 6.12. Compatibility and Non-Goals

A base Durable Streams client continues to append JSON values or arrays, read by offset, and receive canonical NDJSON. It MAY ignore all `Stream-Record-*` and `Stream-Extensions` headers.

This extension does not define:

- schema validation or schema evolution;
- indexes over arbitrary JSON fields;
- SQL, Arrow, Parquet, or DataFusion integration;
- consumer groups;
- per-record arbitrary byte headers;
- authentication or authorization;
- a bidirectional append-session transport;
- record framing for arbitrary binary content types.

### 6.13. Correctness Requirements

Implementations MUST preserve these invariants:

1. Every successfully appended normalized JSON message receives exactly one ordinal.
2. Ordinals are contiguous in committed stream order and are never reused.
3. One atomic append receives one contiguous ordinal range or no range.
4. A deduplicated append returns its original ordinal range.
5. `O(r)` always identifies the first canonical byte of record `r`.
6. Offset reads and record reads expose the same canonical JSON sequence.
7. Record-aware reads never return partial records.
8. Flush, compaction, snapshot, restart, and retention never change a surviving record's ordinal.
9. `N - F` equals the number of retained logical messages.
10. Base-protocol clients observe no semantic change.

Conformance tests SHOULD cover JSON array flattening, concurrent append ordering, idempotent retry, record-tail preconditions, partial offset reads, record-aligned reads, retention, snapshot/bootstrap boundaries, cold flush, restart, and live-read reconnection.

---

## 7. Ordinary SSE Compatibility

Ordinary SSE behavior, including binary stream handling, is defined by `docs/specs/durable-stream.md`.

- For binary streams, ordinary `event: data` payloads are raw base64 text and the response MUST include `stream-sse-data-encoding: base64`.
- Extension endpoints MUST NOT redefine the payload shape of ordinary binary SSE `event: data` frames unless they document an explicit endpoint-local contract.

---

## 8. URL Encoding

`snapshot_offset` values originate from `Stream-Next-Offset` and MAY contain characters requiring URL encoding. Clients and servers MUST apply standard URL encoding/decoding when using offset values in path segments.