# Ursula 0.9 - full documentation > Open-source Distributed Durable Streams over HTTP, backed by S3. These docs are for Ursula 0.9.x (0.9.0). The latest docs are at https://ursula.tonbo.io/llms.txt. Source: https://ursula.tonbo.io --- # Ursula > [!NOTE] > Ursula is built by

Tonbo. Ursula is a self-hosted, distributed server for the replayable, append-only event timelines behind document edits, agent runs, workflows, and chat. It speaks the [Durable Streams Protocol](/docs/0.9/specs/durable-stream) over plain HTTP and SSE. Quorum-replicated in-memory front-ends give you single-digit-millisecond appends and live tail on top of S3 durability, with no separate broker, no batched 250 ms commits, and no S3 Express bill. - **HTTP-native.** `PUT` creates, `POST` appends, `GET` replays or tails with long-poll/SSE. Any HTTP client is a valid client. - **One timeline per resource.** A stream per document, session, workflow, room, or agent run, instead of a few high-throughput pipelines. - **Thread-per-core × multi-Raft.** Each stream hashes to one Raft group and one owner core. Hot-path requests touch that core only, with no cross-core synchronization. Hundreds to thousands of small groups co-exist per node, so a slow follower for one group never stalls another. - **Hot ring + S3 cold tier.** Writes commit at Raft quorum in an in-memory ring. Older segments flush to S3 in the background. A single `GET` transparently spans both tiers. ## Try it Start a single throwaway node with Docker (or build from source, see [Install](/docs/0.9/install)): ```bash docker run --rm -p 4437:4437 ghcr.io/tonbo-io/ursula:0.9.0 ``` Create a stream, append bytes, and read them back: ```bash curl -X PUT http://127.0.0.1:4437/demo curl -X PUT http://127.0.0.1:4437/demo/hello curl -X POST http://127.0.0.1:4437/demo/hello \ -H 'Content-Type: application/octet-stream' \ --data-binary 'hello world' curl 'http://127.0.0.1:4437/demo/hello?offset=-1' ``` Continue with the [Quick Start](/docs/0.9/quick-start) to replay and tail streams live, and [Deploy a Cluster](/docs/0.9/deploy-cluster) when you're ready for the production three-voter shape. ## Learn more - **[Why Ursula](/docs/0.9/why-ursula):** the four properties Ursula keeps that other servers force you to trade - **[Architecture](/docs/0.9/architecture/overview):** thread-per-core, multi-Raft, hot/cold tiers - **[Protocol Spec](/docs/0.9/specs/durable-stream):** Durable Streams plus [Ursula's extensions](/docs/0.9/specs/extensions) - **[ursulactl](/docs/0.9/cli):** the operator CLI for a running cluster (leadership drain, status, readiness gates) ## Credits - **[ElectricSQL](https://electric-sql.com/)** for the original Durable Streams Protocol that Ursula implements. - **[Loro](https://loro.dev/)** for the snapshot and replay extension design that Ursula adopted on top of the base protocol. --- # Why Ursula ## What Ursula keeps A new generation of event streams lives outside the broker network. Document editors, agents, and durable workflows need timelines that browsers, mobile apps, and serverless functions can read, write, and tail over the public internet. That asks for HTTP-native, distributed, S3-backed infrastructure, not the SDK-locked, single-network shape Kafka-style brokers were built for. The [Durable Streams Protocol](/docs/0.9/specs/durable-stream) nails that wire format, but its reference server is a single process: a node loss is data loss. The other servers we evaluated each force you to give up one of four things this primitive deserves to keep: - **Open-source self-hosting.** - **Low write latency** (sub-50 ms appends, no batching window required). - **Plain S3 economics** (cold tier on standard S3, no S3 Express tier, no per-GB SaaS markup). - **Quorum-replicated durability** (acknowledged writes survive a single-node failure). Ursula keeps all four. For the head-to-head numbers, see [How Ursula compares](/docs/0.9/competitive-comparison). ## One timeline per entity Most event systems multiplex many entities into shared channels and rebuild per-entity state downstream. Ursula inverts that: each document, session, task, room, or agent run gets its own durable timeline. Writers append directly to it and readers resume from its offsets. Treat it as a per-entity durable log runtime, purpose-built for that pattern rather than a general event backbone. ## What you get from one stream per entity - **Replayable recovery.** When a worker, sandbox, or agent restarts, replay its log and continue. - **Live tails with simple clients.** Catch-up reads, long-poll, or SSE, all over plain HTTP. - **Lifecycle in the same primitive.** Snapshots, bootstrap, and TTL ride the timeline instead of being separate infrastructure. ## The tradeoff Ursula makes one timeline per entity cheap. In exchange, it isn't the most general primitive for cross-system event distribution or arbitrary stream processing. Keeping the per-entity model simple is what makes recovery, inspection, and operation tractable for long-running application state. For latency and durability numbers, see [Architecture](/docs/0.9/architecture/overview) and [Competitive comparison](/docs/0.9/competitive-comparison). --- # Install This page gets a **single Ursula node** running on your machine so you can try the API. There are two ways to do it: | Pick | When | | ---- | ---- | | **Docker** | You just want to try Ursula. One command, no toolchain. | | **Build from source** | You want local binaries, plan to hack on Ursula, or can't run Docker. | Both start a single, non-replicated node on port `4437` that keeps nothing across restarts. The source build runs the in-memory engine. The Docker image runs a single Raft node whose WAL lives in a temporary directory inside the container. For a production cluster, see [Deploy a Cluster](/docs/0.9/deploy-cluster). ## Docker Every release publishes a multi-arch image to GHCR: ```bash docker run --rm -p 4437:4437 ghcr.io/tonbo-io/ursula:0.9.0 ``` The image runs as a non-root user and contains both `ursula` (the server) and `ursulactl` (the operator CLI). A container that runs a node of a multi-node cluster must set `server.cluster_id` in its configuration, the same on every node (see [Configuration](/docs/0.9/configuration#server-and-runtime)). Images are published only by the reviewed stable-release workflow, so pin the version you want to run. ## Build from source You need rustup, a C compiler, and `pkg-config`. The repository pins the toolchain in `rust-toolchain.toml`, so rustup selects the right nightly automatically. ### Installing the prerequisites (macOS, Debian/Ubuntu) On macOS: ```bash brew install rustup pkg-config rustup-init -y ``` On Debian or Ubuntu: ```bash sudo apt-get update sudo apt-get install -y build-essential pkg-config curl curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` Clone, build, and start the server: ```bash git clone https://github.com/tonbo-io/ursula.git cd ursula cargo build --release -p ursula -p ursula-ctl ./target/release/ursula server ``` The server is now listening on `127.0.0.1:4437`. The build also produces `target/release/ursulactl`, the operator CLI you will use once a cluster is up. ## Verify With the node running, from another terminal: ```bash curl http://127.0.0.1:4437/__ursula/metrics ``` A JSON snapshot of runtime state means the node is up. ## Next - [Quick Start](/docs/0.9/quick-start): create streams, append, replay, and tail live over SSE - [Deploy a Cluster](/docs/0.9/deploy-cluster): the production three-voter shape with Helm - [Configuration](/docs/0.9/configuration): persistence, presets, and S3 cold storage --- # Quick Start This page assumes a single node is already running on `127.0.0.1:4437`. Starting one is a single Docker or Cargo command, see [Install](/docs/0.9/install). The default node keeps nothing across restarts. For disk-backed persistence, set `raft.wal.path` (see [Configuration](/docs/0.9/configuration)). A multi-node cluster also needs `server.cluster_id`, the same on every node. Drive the HTTP API with `curl` from another terminal. ## Validate the bucket ID (optional) Buckets are implicit: a bucket exists once a stream is created in it, so this call is optional. It only validates the bucket ID (`201`, or `400` for an invalid one): ```bash curl -X PUT http://127.0.0.1:4437/demo ``` ## Create a stream ```bash curl -X PUT http://127.0.0.1:4437/demo/hello ``` ## Append data ```bash curl -X POST http://127.0.0.1:4437/demo/hello \ -H 'Content-Type: application/octet-stream' \ --data-binary 'first message' curl -X POST http://127.0.0.1:4437/demo/hello \ -H 'Content-Type: application/octet-stream' \ --data-binary 'second message' ``` Each successful append returns `204 No Content` with a `Stream-Next-Offset` header. Add `Producer-Id` / `Producer-Epoch` / `Producer-Seq` if you need [exactly-once retries](/docs/0.9/concepts/exactly-once-writes). ## Read everything from the beginning ```bash curl -i 'http://127.0.0.1:4437/demo/hello?offset=-1' ``` The body contains the appended bytes. Response headers include `Stream-Next-Offset`, `Stream-Up-To-Date`, and an `ETag`. ## Subscribe for live updates Open a second terminal: ```bash curl -N 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse' ``` `-N` keeps `curl` from line-buffering the SSE stream. Append more data from the first terminal and you'll see it arrive immediately as `event: data` lines. Binary streams are delivered as raw base64 text in `event: data` payloads (`Stream-Sse-Data-Encoding: base64`). See [binary SSE](/docs/0.9/concepts/binary-sse) for details. ## Inspect runtime state For a multi-node cluster the canonical day-2 tool is [`ursulactl`](/docs/0.9/cli). It speaks to every node, summarises leadership, and wraps restarts in safe drain and catch-up steps. For a single local node you can either point it at a one-line manifest: ```bash cat > /tmp/local.json <<'JSON' {"nodes": [{"id": 1, "client_url": "http://127.0.0.1:4437", "host": "127.0.0.1"}]} JSON ursulactl status --config /tmp/local.json ``` Or hit the underlying JSON endpoint directly: ```bash curl http://127.0.0.1:4437/__ursula/metrics ``` The raw endpoint is also what `ursulactl` consumes. Use it directly when you want the full snapshot or are building custom tooling. ## Next steps - [Deploy a Cluster](/docs/0.9/deploy-cluster): the production three-voter shape with Helm - [Configuration](/docs/0.9/configuration): persistence, presets, and S3 cold storage in one place - [ursulactl](/docs/0.9/cli): the operator CLI you'll use once a cluster is up - [API overview](/docs/0.9/api/overview): the HTTP surface Ursula currently exposes - [Streams](/docs/0.9/concepts/streams): the core stream abstraction and lifecycle - [Architecture](/docs/0.9/architecture/overview): thread-per-core, multi-Raft internals --- # Clients Ursula speaks plain HTTP and Server-Sent Events. There is no required client library. Any HTTP client in any language works. The examples elsewhere in these docs use `curl` because it's universal. The same routes, headers, and query parameters apply to every other client. ## Minimal examples ```bash # create a bucket and stream curl -X PUT http://127.0.0.1:4437/demo curl -X PUT http://127.0.0.1:4437/demo/hello # append curl -X POST http://127.0.0.1:4437/demo/hello \ -H 'Content-Type: application/octet-stream' \ --data-binary 'hello world' # catch-up read curl 'http://127.0.0.1:4437/demo/hello?offset=-1' # live tail curl 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse' ``` ```python import requests base = "http://127.0.0.1:4437" requests.put(f"{base}/demo") requests.put(f"{base}/demo/hello") requests.post( f"{base}/demo/hello", headers={"Content-Type": "application/octet-stream"}, data=b"hello world", ) # catch-up read resp = requests.get(f"{base}/demo/hello", params={"offset": -1}) print(resp.content) # live tail with SSE with requests.get( f"{base}/demo/hello", params={"offset": -1, "live": "sse"}, stream=True, ) as r: for line in r.iter_lines(): if line: print(line.decode()) ``` ```ts const base = "http://127.0.0.1:4437"; await fetch(`${base}/demo`, { method: "PUT" }); await fetch(`${base}/demo/hello`, { method: "PUT" }); await fetch(`${base}/demo/hello`, { method: "POST", headers: { "Content-Type": "application/octet-stream" }, body: "hello world", }); // catch-up read const data = await (await fetch(`${base}/demo/hello?offset=-1`)).text(); // live tail with native EventSource const es = new EventSource(`${base}/demo/hello?offset=-1&live=sse`); es.addEventListener("data", (e) => console.log(e.data)); ``` ## From a browser The examples above assume same-origin access. A page served from a different origin than the gateway needs the gateway to allow that origin — see [cross-origin reads](/docs/0.9/security#cross-origin-reads-opt-in): ```bash ursula gateway ... --cors-allowed-origin https://app.example.com ``` Two things to know once it is on: - **Read your continuation headers.** The gateway sends `Access-Control-Expose-Headers: *`, so `Stream-Next-Offset` and friends are readable. Without exposure a browser can read one page and never advance, which looks like the stream ending. If a `fetch` succeeds but `response.headers.get("stream-next-offset")` is `null`, the origin is not allowing your origin. - **`EventSource` cannot send `Authorization`.** It has no header API, so it only works against `public_read` streams. For a private live tail, use `fetch` and read the body: ```ts const response = await fetch(`${base}/${bucket}/hello?offset=-1&live=sse`, { headers: { Authorization: `Bearer ${token}` }, }); const reader = response.body!.pipeThrough(new TextDecoderStream()).getReader(); ``` A subscription ends when its credential expires, and says so: the final frame is `event: credential-expired`. Treat it as a reconnect signal rather than end of stream — refresh the token and re-read from the last `Stream-Next-Offset` you saw. Any other termination deserves the same handling. ## Notes for client implementers - After every read and append, the server returns `Stream-Next-Offset`. Track it. Use it as the `offset` query parameter on the next read. Don't construct offsets manually. They're opaque. - For binary streams over SSE, `data` events carry raw base64 text and the response includes `Stream-Sse-Data-Encoding: base64`. Decode the data event first, then interpret the bytes using `Stream-Data-Content-Type`. See [Binary SSE](/docs/0.9/concepts/binary-sse). - For exactly-once writes, send `Producer-Id`, `Producer-Epoch`, `Producer-Seq` headers and retry on network errors. The server deduplicates. See [Exactly-once writes](/docs/0.9/concepts/exactly-once-writes). - For conditional writes, use `Stream-Seq`: the server accepts an append only when its value is bytewise greater than the last one accepted on the stream. Several writers can extend one log safely by sending the number of events they read. See [Conditional writes](/docs/0.9/concepts/conditional-writes). --- # Deploy a Cluster Production Ursula is a static-membership Raft cluster: three voting nodes across availability zones, a durable Raft log per node, and a shared S3 bucket for the cold tier. The recommended way to run it is **Kubernetes via the Helm chart**. Bring a Kubernetes cluster with a StorageClass for the voters' volumes and an S3 bucket for the cold tier. The chart installs Ursula into it. If you are not running Kubernetes, see [bare metal](#without-kubernetes-bare-metal--vms) at the end. ## Helm The chart and images are published to GHCR on every release. One command starts a three-voter cluster: ```bash helm install ursula oci://ghcr.io/tonbo-io/charts/ursula --version 0.9.0 \ --set server.clusterId=my-cluster ``` The chart defaults to the `ghcr.io/tonbo-io/ursula` image pinned to the chart's `appVersion`. The default install runs three voters and 64 Raft groups, with durable per-pod Raft logs on PVCs, a headless peer Service, a client Service, and a quorum-protecting PodDisruptionBudget. Each voter listens on two ports. The client port (`server.ports.client`, `4437`) serves the stream API, and the client Service exposes only that port. The cluster port (`server.ports.cluster`, `4439`) carries Raft between voters, and only the headless peer Service names it. The chart renders each peer's `url` on the cluster port and its `client_url` on the client port, so leader redirects send clients to a client port. `server.clusterListen=false` serves both planes on the client port instead. The admin plane stays on pod loopback (`127.0.0.1:4438`). For production, add shared S3 cold storage and workload identity: ```yaml s3: bucket: my-ursula-bucket region: us-east-1 prefix: ursula-prod coldStorage: enabled: true serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ursula-s3 ``` With cold storage enabled the chart also turns on cold-chunk compaction and lets Ursula pick S3 for Raft snapshots (`snapshotStore.backend: auto`). Set `coldStorage.compaction.enabled=false` or `snapshotStore.backend=inline` to opt out. For MinIO or another S3-compatible backend, set `s3.endpoint`. Prefer workload identity over static S3 credentials. To use a registry mirror, set `global.image.repository`, `global.image.tag`, and optionally `global.imagePullSecrets`. `server.replicaCount` controls the voter set for a **fresh** cluster and supports `1`, `3`, and `5`. Production should use `3`, or `5` only when tolerating two simultaneous voter failures justifies the larger write quorum. Changing it on an initialized cluster is not safe Raft voter reconfiguration. Safe scaling needs the future operator workflow. ### On AWS (EKS) The chart's [`examples/production-eks.yaml`](https://github.com/tonbo-io/ursula/blob/main/charts/ursula/examples/production-eks.yaml) is a three-zone starting point: three voters on `gp3` volumes, S3 cold storage and snapshots, and three gateways. It needs: - an EKS cluster with nodes in three availability zones, - the EBS CSI driver and a StorageClass named `gp3`, - an S3 bucket, - an IAM role for the Ursula service account (IRSA or EKS Pod Identity), limited to its S3 prefix. The [chart README](https://github.com/tonbo-io/ursula/tree/main/charts/ursula#prerequisites) lists the details. Replace the bucket and the role ARN in the example, then install it: ```bash helm install ursula oci://ghcr.io/tonbo-io/charts/ursula --version 0.9.0 \ --namespace ursula --create-namespace -f production-eks.yaml helm test ursula --namespace ursula ``` ## Verify ```bash helm test ursula ``` Gate on this test, not on `helm install --wait`. Helm does not wait for a StatefulSet with the `OnDelete` update strategy, so `--wait` can report the release deployed before any voter is ready. The test mounts the chart-generated cluster manifest and runs `ursulactl wait-ready`. It requires the expected group inventory and leaders, a complete uniform voter set in each maintenance report, no uncertain maintenance fence, and a 200 response from every node's `/__ursula/ready` probe. To query a node directly: ```bash kubectl port-forward svc/ursula 4437:4437 curl http://127.0.0.1:4437/__ursula/metrics ``` ## Production notes - Run three voters across three availability zones, each with its own zonal persistent volume for the Raft log. Never use ephemeral voter data in production. The chart always gives each voter pod a `raft-data` volume claim. - Shared S3 is required for any multi-node cluster: replicas must be able to read chunks flushed by any leader. - Add an `AbortIncompleteMultipartUpload` lifecycle rule to every cold bucket, for example after 1 day. Ursula never aborts the multipart upload of a cold write that a crash or shutdown interrupted, and S3 bills its parts until they are aborted. See [Incomplete multipart uploads](/docs/0.9/operations#incomplete-multipart-uploads). - Set `server.clusterId`, which the chart renders as `server.cluster_id`, the id every `HEAD` answer carries as `Ursula-Cluster-Id`. The chart refuses to render without it, for any replica count. Name the environment and the region, such as `prod-us-east-1`, and give every deployment one client can reach its own id, since two Kubernetes clusters can run the same namespace and release name. Keep the id when a restore replaces a deployment. Each node's WAL records the id, and a node whose configured id differs refuses to start. - Put stateless gateway replicas behind authenticated TLS ingress and keep voter and peer Services private. Ursula itself has no TLS or auth (see [Security](/docs/0.9/security)). - Set `server.networkPolicy.enabled=true` on a CNI that enforces NetworkPolicy. The policy admits only the release's voters to the cluster port, and the sources in `server.networkPolicy.clientIngressFrom` to the client port (every source when the list is empty) plus the release's gateway and Helm test pods. See [Kubernetes NetworkPolicy](/docs/0.9/security#kubernetes-networkpolicy). - Kubernetes rolling updates do not transfer Raft leaders on their own. For an operationally safe restart, wrap the platform's restart in [ursulactl's maintenance verbs](/docs/0.9/cli): `ursulactl drain` the node, let Kubernetes restart the pod, then `ursulactl wait --node N` and `undrain` before moving to the next one. See [Operations](/docs/0.9/operations) for day-2 work. - The StatefulSet readiness probe calls `GET /__ursula/ready` and removes a pod from service during WAL disk pressure. Cluster-wide Raft readiness still comes from `helm test` or `ursulactl wait-ready`. The mutating admin surface is bound to pod loopback (`127.0.0.1:4438`) and is reached with `kubectl port-forward`. - `helm uninstall` leaves the graceful-rollout state ConfigMap (`-rollout-state`, for example `ursula-rollout-state`) behind. Delete it before you install again under the same name, so the rollout hook does not resume a rollout of the old release. - Clusters that serve the [SQLite VFS](/docs/0.9/examples/sqlite-vfs) take its appends of at most 4 MiB at any preset. Its snapshots of up to 32 MiB go in one request, so keep per-group admission caps of at least 64 MiB (`standard` or `large` preset) for them to be admitted. Larger snapshots go up in parts and need a cold backend. ## Without Kubernetes (bare metal / VMs) The same static-membership model works by hand. Every node runs the same config file, and only the node ID differs: ```toml [server] listen = "0.0.0.0:4437" cluster_listen = "0.0.0.0:4439" cluster_id = "prod-us-east-1" [raft] group_count = 256 init_membership_per_group = true [raft.wal] path = "/var/lib/ursula" [storage.cold] backend = "s3" root = "ursula-prod" [storage.cold.s3] bucket = "my-ursula-bucket" region = "us-east-1" [[raft.peers]] node_id = 1 url = "http://10.0.0.1:4439" client_url = "http://10.0.0.1:4437" [[raft.peers]] node_id = 2 url = "http://10.0.0.2:4439" client_url = "http://10.0.0.2:4437" [[raft.peers]] node_id = 3 url = "http://10.0.0.3:4439" client_url = "http://10.0.0.3:4437" ``` Start each node with its own ID: ```bash ursula --config /etc/ursula/ursula.toml --node-id 1 # node 2 and 3 likewise ``` Peers dial each other's `url` on the cluster port (`4439`), and a follower redirects clients to the leader's `client_url` on the client port (`4437`). Let only the voters reach port `4439`, with a security group or a firewall. Without `server.cluster_listen`, both planes share `server.listen` and each peer's `url` names that port (see [Network planes](/docs/0.9/security#network-planes)). `server.cluster_id` is required with `raft.peers`. Use the same value on every node and a different one for every deployment a client can reach, such as the environment and the region. `init_membership_per_group = true` bootstraps each group on the first start of a fresh cluster. Keep it on every node across restarts: a replica that ever held its group never initializes it again, and an empty replica initializes a group only when every configured voter reports it empty and no leader (see [Rejoining a wiped node](/docs/0.9/operations#rejoining-a-wiped-node)). Verify the cluster with a one-file manifest: ```bash cat > cluster-manifest.json <<'JSON' { "nodes": [ {"id": 1, "http_url": "http://10.0.0.1:4439", "client_url": "http://10.0.0.1:4437", "host": "10.0.0.1"}, {"id": 2, "http_url": "http://10.0.0.2:4439", "client_url": "http://10.0.0.2:4437", "host": "10.0.0.2"}, {"id": 3, "http_url": "http://10.0.0.3:4439", "client_url": "http://10.0.0.3:4437", "host": "10.0.0.3"} ] } JSON ursulactl wait-ready --config cluster-manifest.json --expected-groups 256 ursulactl status --config cluster-manifest.json ``` `http_url` is the node's Raft address, so it names the cluster port. `client_url` is where `ursulactl` reads metrics and readiness on the client plane. Every config key (peers, listeners, WAL, cold-tier tuning) is documented in [Configuration](/docs/0.9/configuration). --- # Configuration Everything configurable in Ursula lives on this page. Ursula is configured by a TOML config file, an optional resource preset, and a small set of CLI overrides, merged in this order: ```text built-in defaults < preset defaults < config file < CLI overrides ``` The normal startup command is: ```bash ursula --config /etc/ursula/ursula.toml --node-id 1 ``` With no config file, `ursula` falls back to the `default` preset, which starts a single-node in-memory runtime without Raft, and nothing survives a restart. That's what [Install](/docs/0.9/install) uses. Every other preset, and a config file loaded without `--preset default`, runs Raft on the disk WAL described under [Raft](#raft). ## CLI flags | Flag | Default | Notes | | ---- | ------- | ----- | | `--config FILE` | searches `./ursula.toml`, `/etc/ursula/ursula.toml`, then user config paths | TOML config file | | `--preset PRESET` | `default` when no config file is found, otherwise none | Resource preset: `default`, `dev`, `tiny`, `small`, `standard`, or `large` | | `--node-id ID` | config file or preset | Overrides `raft.node_id` | CLI `--node-id` intentionally overrides any `raft.node_id` value in the config file so the same file can be mounted on every node. Helm derives it from the StatefulSet ordinal. ## Resource presets `--preset` supplies a base configuration before the config file is merged, and config-file values win. Pick by node size: | Preset | Intended use | Main defaults | | ------ | ------------ | ------------- | | `default`, `dev` | Local development | Single-node in-memory runtime without Raft | | `tiny` | Memory-bound tests or very small nodes | 64 groups, 64 MiB cold cache, 4 MiB cold flushes, 2 cold writes, 8 MiB hot/admission caps, 64 MiB HTTP in-flight body budget | | `small` | Small nodes or cost-sensitive tests | 128 groups, 64 MiB cold cache, 4 MiB cold flushes, 2 cold writes, 16 MiB hot/admission caps, 64 MiB HTTP in-flight body budget | | `standard` | Production baseline | 256 groups, 256 MiB cold cache, 8 MiB cold flushes, 4 cold writes, 64 MiB hot/admission caps, 256 MiB HTTP in-flight body budget | | `large` | Larger nodes with more cache and write headroom | 512 groups, 512 MiB cold cache, 16 MiB cold flushes, 8 cold writes, 128 MiB hot/admission caps, 512 MiB HTTP in-flight body budget | ## Server and runtime | Config key | Default | Purpose | | ---------- | ------- | ------- | | `server.listen` | `127.0.0.1:4437` | Client-plane listener: the stream API, metrics, readiness and usage | | `server.cluster_listen` | none | Separate cluster-plane listener for Raft gRPC between voters. Unset, the cluster plane shares `server.listen`. Set, every `[[raft.peers]]` entry needs a `client_url` | | `server.admin_listen` | `127.0.0.1:4438` | Admin-plane listener for the mutating operator routes, plus metrics, readiness and usage. Keep it on loopback and reach it through a tunnel | | `server.cluster_id` | none | The deployment's id, sent as `Ursula-Cluster-Id` on every `HEAD`: 1 to 64 characters from `A-Z`, `a-z`, `0-9`, `.`, `_` and `-`. Required when `raft.peers` is set, the same on every node and different for every deployment a client can reach. A single node without it generates `u` followed by 32 hex digits at its first start and keeps it in its WAL. A single node without `raft.wal.path` gets a new one on every start. The WAL records the id, and a node whose configured id differs refuses to start. Peers check it: every Raft gRPC request carries the sender's id, and a node refuses a peer with another id (see [Peers of another cluster](/docs/0.9/operations#peers-of-another-cluster)) | | `server.http_inflight_body_size` | `256MiB` (`64MiB` in the `tiny` and `small` presets) | Write body bytes the node holds at once. A write that would exceed it answers `503` with `INGRESS_BODY_BYTES_LIMIT_REACHED`. A snapshot commit's copy of parts is charged against it too, from 32 MiB to 56 MiB depending on the budget, and starts only while 32 MiB stays free beside it. With a cold backend it must be at least `64MiB`, or the node refuses to start | | `runtime.cpu_affinity` | automatic | On Linux, `true` forces pinning and `false` disables it; omitted pins only when cpuset size equals core_count | | `runtime.core_count` | `available_parallelism` | Owner worker threads; Linux automatically pins only when the allowed cpuset matches this count. Journal writers and the main Tokio runtime use additional threads | | `runtime.live_read_max_waiters_per_core` | `65536` | SSE waiter cap per core. `0` disables the limit | | `observability.tokio_console` | `false` | Enables Tokio console when the binary is built with the feature | A node refuses a request body over 32 MiB with `413` and names that limit in `Ursula-Max-Request-Bytes`. The limit is the protocol's and is fixed. The gateway's `--max-request-body-bytes` (the chart's `gateway.maxRequestBodyBytes`) is at most 32 MiB, and a gateway or a chart set above it refuses to start, since the node refuses such a body before reading it and the gateway would answer `502` instead of a typed `413`. With a cold backend, a larger snapshot goes up [in parts](/docs/0.9/api/publish-snapshot#upload-in-parts) instead. Without `server.cluster_listen`, `server.listen` serves both the client and the cluster plane, and each `[[raft.peers]].url` points at that address. Leader redirects then name the same URL, so clients must be able to reach it too. With `server.cluster_listen` set, `server.listen` serves only the client plane. Each `[[raft.peers]].url` points at the peer's cluster listener, and each `[[raft.peers]].client_url` at its client listener. A follower redirects a client to the leader's `client_url`, so only peers need to reach the cluster listener. A node refuses to start when a peer lacks `client_url`. `ursula gateway` follows a redirect to an upstream directly when its `--upstream` equals the leader's `client_url`, and searches its upstreams for the leader otherwise. See [Security](/docs/0.9/security#network-planes) for what each plane serves. ```toml [server] listen = "0.0.0.0:4437" cluster_listen = "0.0.0.0:4439" [[raft.peers]] node_id = 1 url = "http://10.0.0.1:4439" client_url = "http://10.0.0.1:4437" ``` ## Raft | Config key | Default | Purpose | | ---------- | ------- | ------- | | `raft.node_id` | none | This node's stable ID, usually overridden per host with `--node-id` | | `raft.append_transport_budget_bytes` | `134217728` | Node-wide charged append bytes per direction, shared across cores and peers; excludes tonic/socket buffers | | `raft.group_count` | `core_count × 16` | Total Raft groups. Higher values improve stream-to-group hash spread | | `raft.wal.path` | none | Directory of the Raft WAL. Required when `raft.peers` is set. Raft logs live under `PATH/raft-log`, and removing the directory wipes state cleanly. Without it, a single node runs its WAL in a temporary directory that is removed on clean shutdown | | `raft.wal.fsync` | `never` | When journal appends reach stable storage. `never` acknowledges ordinary data appends from the page cache. Membership batches are always fsynced before acknowledgment. `always` acknowledges a batch after `fsync`. Votes, each group's log state and the node's run state are always written with `fsync` | | `raft.wal.segment_size` | `64MiB` | Size at which each core's journal starts a new segment. Purge deletes whole segments, so this is also how much disk space comes back at a time. At least `4KiB` | | `raft.wal.cache_size` | `256MiB` | Recent Raft entries kept in memory across the node, split evenly between its groups. Older entries, such as a lagging follower's catch-up, are read from disk | | `raft.wal.min_available_size` | `512MiB` | Reject writes and mark `/__ursula/ready` unavailable below this much free WAL-filesystem space. `0` disables the guard | | `raft.wal.resume_available_size` | `1GiB` | Free space required before disk-pressure admission clears. Must exceed the minimum | | `raft.peers` | empty | Static gRPC peer list, with one `[[raft.peers]]` entry per voter, including this node. Each entry has `node_id`, `url` (the address peers dial for the cluster plane) and `client_url` (the client-plane address leader redirects name, required with `server.cluster_listen` and `url` by default). It is the only source of peer addresses: keep it the same on every node, and change it there to move a node ([Changing a peer address](/docs/0.9/operations#changing-a-peer-address)) | | `raft.init_membership_per_group` | `false` | Per-group membership bootstrap. One voter per group initializes it once every configured voter reports the group empty and no leader. A replica that ever held its group never initializes it again, so keep it `true` across restarts. The Helm chart sets `true` | The Raft WAL is always the per-core disk journal. A node with `raft.peers` set refuses to start without `raft.wal.path`. A single node without a path creates a fresh `ursula-wal-*` directory under the system temporary directory, logs its location at startup, and removes it on a clean shutdown (`SIGTERM` or Ctrl-C). A failed start, or a shutdown that cannot close the WAL, leaves the directory in place. Its streams survive no restart. Set `raft.wal.path` to keep them. With `fsync = "never"`, a process crash loses nothing because the page cache survives it, and a graceful shutdown `fsync`s every journal before it exits. A host crash or power loss can drop the unsynced tail of that replica's journal. The replica restarts from the frames that still verify and reports that it is recovering. Each of its groups stays behind a recovery gate: it does not campaign or vote until it has applied a fresh barrier from the group's leader, so it never helps elect a leader that lacks a write it acknowledged. While at most a minority of a group's voters lost their tail, the other voters hold every acknowledged write and the group keeps serving. If a majority lost it at once, the group has no leader and refuses writes until an operator accepts the loss (see [Operations](/docs/0.9/operations#recovering-after-a-host-crash)). With `fsync = "always"`, a host crash loses no acknowledged write. Every batch waits for an `fsync`, which costs write latency. A host crash can still leave a complete invalid frame after the last `fsync` in a journal's newest segment. Recovery cannot tell it from media corruption, so it gates that core's groups as above. A group gated on a majority of its voters, or a single-voter group, then waits for an operator although nothing was lost (see [Operations](/docs/0.9/operations#recovering-after-a-host-crash)). ## Cold storage Ursula keeps recent data in an in-memory hot ring on every replica and flushes older segments and snapshot blobs to a cold backend. Multi-node clusters should use S3 or an S3-compatible object store so every replica can read chunks flushed by any leader. All replicas must point at the same shared bucket. ```toml [storage.cold] backend = "s3" root = "ursula-prod-20260518" [storage.cold.s3] bucket = "my-ursula-bucket" region = "us-east-1" # endpoint = "http://127.0.0.1:9000" # access_key_id = "AKIA..." # secret_access_key = "..." # session_token = "..." # server_side_encryption = "aes256" # default; "aws-kms" or "none" # kms_key_id = "arn:aws:kms:..." # customer managed key for "aws-kms" ``` When `access_key_id` and `secret_access_key` are omitted, Ursula uses the standard AWS SDK credential chain (instance profile, environment, profile, and so on). Prefer that over static keys. Set `endpoint` for S3-compatible stores such as MinIO, R2, or TOS. Every cold-tier object write requests server-side encryption by default (`server_side_encryption = "aes256"`, i.e. SSE-S3) — free on AWS S3, and Raft snapshot uploads inherit the same setting. `aws-kms` switches to SSE-KMS, using the AWS managed key unless `kms_key_id` names a customer managed key. MinIO honors SSE headers only when a KMS/KES is configured; for a MinIO deployment without one, set `server_side_encryption = "none"` explicitly or writes fail with an SSE error. | Config key | Default | Purpose | | ---------- | ------- | ------- | | `storage.cold.backend` | `none` | `none`, `memory`, or `s3` | | `storage.cold.root` | none | Prefix prepended to every cold key. Use a date-stamped value for benchmark runs so cleanup can't touch production data | | `storage.cold.s3.bucket` | none | Required when `backend = "s3"` | | `storage.cold.s3.server_side_encryption` | `aes256` | SSE mode for every object write: `aes256` (SSE-S3), `aws-kms`, or `none` | | `storage.cold.s3.kms_key_id` | none | Customer managed KMS key; only valid with `aws-kms` | | `storage.cold.s3.probe_timeout` | `5s` | Deadline of the snapshot-store health probe, retries included. A probe whose first attempt fails fast still has time for one retry, while an attempt that hangs runs into this deadline. Must be non-zero | | `storage.cold.s3.unhealthy_ticks` | `3` | Bad health ticks, with no good tick between them, before the node sheds its leaderships. Must be non-zero | | `storage.cold.s3.heal_ticks` | `2` | Good health ticks in a row before a node that shed may lead again. Must be non-zero | | `storage.cold.flush_interval` | `1s` | Background flush worker tick interval | | `storage.cold.flush_size` | `8MiB` | Target bytes flushed per group per tick | | `storage.cold.flush_max_concurrency` | `4` | Parallel cold writes in flight | | `storage.cold.max_hot_size_per_group` | `64MiB` | Backpressure ceiling: when a group's hot bytes exceed this, new writes get HTTP `503` until the flush catches up. Explicit `0` disables the cap | | `storage.cold.flush_max_hot_age` | `5m` | A stream's hot tail older than this is flushed whole even when its group is below the flush threshold. `0` disables | | `storage.cold.compaction_enabled` | `true` | Background compaction: merges small chunks that flushes recorded as compaction debt (no bucket listing) and rewrites packed slices once a stream holds 64 of them | | `storage.snapshot.backend` | `auto` | `auto` (S3 snapshots when `storage.cold.backend = "s3"`, inline otherwise), `inline`, or `s3`. A node without `raft.peers` keeps its snapshots inline | | `storage.snapshot.s3_timeout` | `10s` | Per-attempt timeout of an S3 snapshot-store request. The snapshot store has its own timeout and retries, apart from `storage.cold.s3.timeout` and `max_retries`. The timeout covers one attempt, and a PUT sends the whole object in one attempt, so it must also cover a healthy GET or PUT of the largest snapshot. A follower that installs a snapshot sends up to four requests one after another within `raft.install_snapshot_timeout`: a GET of the leader's object, a PUT of its copy, a second PUT under a new name when the first one's name is taken, and a HEAD that checks the copy. Each request can take every attempt to its timeout and every backoff before a retry, 22 seconds with the defaults, so the copy takes at most 88 seconds of the default 120. On a cluster node with S3 snapshots, the server refuses to start when the four requests can take longer than `raft.install_snapshot_timeout`, with `SnapshotCopyOutlastsInstall`. A single node keeps its snapshots inline and is not checked. Must be non-zero | | `storage.snapshot.s3_max_retries` | `1` | Retries of an S3 snapshot-store request. The first retry waits up to 2 seconds, the second up to 4, each later one twice as long up to 120 seconds. Counts toward the same bound as `storage.snapshot.s3_timeout` | | `raft.snapshot_log_budget` | `1GiB` | Node Raft-log budget of the snapshot driver. A group snapshots once its log since the last snapshot reaches the larger of this budget over twice the group count (at most `16MiB`) and twice its last snapshot's size. Above three quarters of the budget a pressure pass snapshots the groups that hold the most log first, one at a time, until the node would fall to half the budget. Past the budget it runs more builds at once, in even steps up to `raft.snapshot_build_max_concurrency` before twice the budget. With the default of 2 that is two builds past the budget. With 4 it is two past the budget, three past 1.33 times it and four past 1.67 times it. Over twice the budget a cluster node refuses writes with a body until its log is below 1.5 times the budget. A node without `raft.peers` never refuses them for its log | | `raft.snapshot_backstop_logs` | `100000` | Far entry backstop of the snapshot driver. A group snapshots once it applied this many entries since its last snapshot, whatever its log bytes. Entry count bounds a group's log index in memory and its replay at restart, so lower it for groups of many small entries. It is the same on a node without `raft.peers` as on a cluster node. 0.8 kept a single node's groups within 5,000 entries, and `5000` keeps that bound | | `raft.snapshot_build_max_concurrency` | `2` | Snapshot builds the node runs at once. Each build holds its group's captured state (the hot window, at most `storage.cold.max_hot_size_per_group`, and the streams' metadata) and, while it uploads, its compressed snapshot, about as large again, and it sends its snapshot-store requests one after another. The snapshot driver starts a build only while fewer run than the node's log calls for: one below `raft.snapshot_log_budget` and more past it. Builds it started past the budget finish after the log falls below it. A snapshot an operator triggers while every build slot is taken is deferred. Set `1` to keep one build at a time, as before 0.7.4. Raise it, to `4` for example, when the node keeps reaching twice the budget with every build in use (`raft_snapshot_builds_in_flight_max` at this value) and its memory holds that many builds of its largest groups. The server refuses to start with `0` or more than `16` | | `raft.snapshot_pressure_max_groups_per_tick` | `16` | Groups one snapshot driver pass starts while the node's log is below `raft.snapshot_log_budget`. Past the budget a pass starts this many for each build it keeps in flight. It also caps the snapshot GC passes one driver tick runs | | `raft.deleted_data_max_age` | the larger of `1h` and eight times `raft.install_snapshot_timeout` | How long a replica keeps a deleted, purged or expired stream in its current Raft snapshot and its Raft log after it applies the removal. The snapshot driver snapshots a group a quarter of this after its oldest removal that no snapshot covers, sooner when the removed bytes reach the group's snapshot threshold, and then purges the group's log through that snapshot. See [Deleted data](/docs/0.9/operations#deleted-data) for what the bound covers. With an S3 snapshot store, snapshot GC deletes the snapshot object a replica replaced once no send of it is in flight, within one `raft.install_snapshot_timeout` and a drive interval. A set value must be at least eight times `raft.install_snapshot_timeout` (16 minutes at the default), or the server refuses to start, which leaves that room with a margin of several installs | Hot bytes here are payload bytes: the hot window keeps no per-message bookkeeping, so admission and flush thresholds charge exactly the unflushed payload. Flush defaults are conservative, so tune them under load. A typical benchmark profile drops the interval to `200ms`, raises concurrency to `32`, and bumps the per-group ceiling. Advanced deployments can split the flush threshold and batch size with `storage.cold.flush_min_hot_size` and `storage.cold.flush_max_size`, but the single `storage.cold.flush_size` knob is the normal path. Once a group's accumulated hot payload reaches the threshold, Ursula packs eligible slices from that group into one immutable object up to the configured maximum; single-stream slices of 1 MiB or more keep the direct-object layout, and smaller ones are packed even when they are alone. The snapshot driver takes every Raft snapshot and always runs. It ticks every `storage.snapshot.drive_interval`, `5s` by default with S3 snapshots and `1s` with inline ones, and the server refuses to start with `0s`. With S3 snapshots, a health gate checks the store on each of its ticks. Each tick probes the store and reads the node's cold-flush writes. A tick is bad when the probe fails or misses its deadline, or when a write failed within the flush window and no write has succeeded since. The window is as long as the S3 client can take to give up on a write: `storage.cold.s3.timeout` for each of its `max_retries` plus one attempts, the backoff between them, and the flush worker's pause between passes. That is 55 seconds with the defaults. A tick in which writes both failed and succeeded neither ends a bad streak nor adds to it. One failed probe is not enough on its own, and once a later write succeeds a failed write weighs only on its own tick. A store that refuses writes but answers the probe is still caught. After `storage.cold.s3.unhealthy_ticks` bad ticks with no good tick between them the node stops campaigning and hands its leaderships to its peers. With the defaults that is 10 to 20 seconds into an outage when the probes fail, about 10 to 30 seconds when only writes fail (sooner when the store refuses them outright), and about a minute when every write attempt hangs until its timeout. While a tick is bad the node also holds routine snapshots back, though a log-pressure pass still runs. After `storage.cold.s3.heal_ticks` good ticks in a row it may lead again. For a node shed for failed writes, the good ticks start with the first write that succeeds after the last failed one. A node that has handed off every leadership flushes nothing, so its good ticks start only once its last failed write leaves the window, about a minute after that write with the defaults. A shedding node hands off at most four leaderships per tick, the groups with the most unflushed bytes first, and only to peers that may campaign. A peer that is draining, waiting at its recovery gate, or shedding for its own snapshot store, cluster egress or WAL disk is passed over. The gate asks each peer for its leadership-shed state and waits `governance.leadership_balance.peer_timeout` (`500ms` by default, must be non-zero) for the answer, the same timeout the leadership balancer uses. A peer that does not answer in time is treated as unable to campaign. A handoff is done once another node leads the group, and one that Raft abandons is handed off again on a later tick. A node with N leaderships therefore evacuates in about `ceil(N / 4)` ticks, for example 22 ticks or 110 seconds for 86 leaderships at the default interval. The cap is per gate. The cold-health gate (`governance.cold_health`, every `2s` by default) drains the same way and independently, so a node that sheds for both moves up to four leaderships per 2 seconds plus four per 5 seconds. The leadership balancer (`governance.leadership_balance`) also keeps moving leaderships off a node that holds more than its fair share, up to `max_per_tick` per balancer tick. The cluster-egress gate (`governance.cluster_probe`) still hands every leadership off at once, because a node that cannot reach a quorum cannot replicate, and it plans again on every tick for the ones still pending. Its questions to the peers travel the path it judges impaired, so a lost answer says little about a peer. It passes over a peer that answered that it may not campaign. A peer that did not answer may take a leadership's first handoff, and the Raft handoff check decides. A leadership whose handoff did not move goes only to a peer that answered that it may campaign, so a peer that never answers costs each leadership at most one stalled handoff per shed. On a tick when a peer answered that it may not campaign, the gate passes over the peers that did not answer too, because the shed may be cluster-wide. Raft snapshotting does not force a partial cold flush. Payload below the cold threshold stays in the bounded hot tail and is included in the next group snapshot, so a short snapshot interval does not turn small tails into small cold objects. Helm and the EC2 scripts map their deployment inputs onto these same keys, so this table applies to every deployment method. ## Complete examples Disk-backed single node (streams survive a restart): ```toml [server] listen = "127.0.0.1:4437" [raft] node_id = 1 group_count = 16 [raft.wal] path = "./data" ``` ```bash ursula --config ./ursula.toml ``` Three-node durable cluster on a single port per node (same file on each host, override the node ID per host). [Deploy a Cluster](/docs/0.9/deploy-cluster#without-kubernetes-bare-metal--vms) shows the same cluster with the cluster plane on its own port, which keeps Raft off the client port: ```toml [server] listen = "0.0.0.0:4437" cluster_id = "prod-us-east-1" [runtime] core_count = 16 [raft] node_id = 1 group_count = 256 init_membership_per_group = true [raft.wal] path = "/var/lib/ursula" [storage.cold] backend = "s3" root = "ursula-prod-20260518" [storage.cold.s3] bucket = "my-ursula-bucket" region = "us-east-1" [[raft.peers]] node_id = 1 url = "http://10.0.0.1:4437" [[raft.peers]] node_id = 2 url = "http://10.0.0.2:4437" [[raft.peers]] node_id = 3 url = "http://10.0.0.3:4437" ``` ```bash ursula --config /etc/ursula/ursula.toml --node-id 1 ``` See [Deploy a Cluster](/docs/0.9/deploy-cluster) for the full deployment walk-through. --- # Security > [!WARNING] > Ursula does not terminate TLS or authenticate clients or peers. Anyone who can reach a node's client port can read and write every stream. Anyone who can reach its cluster port can send it Raft traffic, and with a single port (no `server.cluster_listen`) that is the client port too. Run nodes on a private network. TLS termination belongs to a reverse proxy or ingress in front of them, since neither the nodes nor the gateway serve TLS. The gateway authenticates requests only when you enable its access control with `--auth-issuer` (see [Gateway access control](#gateway-access-control-opt-in)). The current `v0.x` security model is deliberately narrow. Ursula is built to slot behind your existing edge layer, not to be one. ## What Ursula does - **Quorum-acknowledged writes.** An append is acknowledged only after a majority of voters has replicated it. - **Per-group backpressure.** When a group's hot ring exceeds `storage.cold.max_hot_size_per_group`, appends to that group return `503` with `Retry-After` until cold flush catches up. Per-group, not global or per-client. - **Stream-level isolation.** Streams hash to disjoint Raft groups and disjoint owner cores. A hot stream on one group cannot starve writes on a different group on a different core. ## Network planes A node serves three planes, each on its own listener: | Plane | Listener | Routes | Who connects | | ----- | -------- | ------ | ------------ | | Client | `server.listen` | The Durable Streams protocol, plus read-only `/__ursula/metrics`, `/__ursula/ready` and `/__ursula/usage` | Applications, the gateway, and clients following a leader redirect | | Cluster | `server.cluster_listen`, or `server.listen` when it is unset | Raft gRPC (append entries, votes, snapshots, leader-read checks, group reads and writes one node forwards to another, leadership transfer), `/__ursula/leadership-shed` and `/__ursula/cluster-probe` | Other voters and learners, at the `url` of their `[[raft.peers]]` entry | | Admin | `server.admin_listen`, loopback `127.0.0.1:4438` by default | The mutating operator routes (Raft group operations, maintenance fences and drains, cold flush, bucket purge, backups) plus metrics, readiness and usage | Operators and `ursulactl` through a tunnel such as `kubectl port-forward` | None of the three authenticates its caller. The admin plane is protected by its loopback bind, not by a credential, so bind it to a reachable address only for a trusted in-cluster operator. Set `server.cluster_listen` so the client plane serves no Raft route. Each `[[raft.peers]]` entry then names the peer's cluster listener in `url` and its client listener in `client_url`, which is where a follower redirects a client. A client that can reach only the client port then cannot send Raft traffic, as long as the network also keeps it off the cluster port. The Helm chart sets `server.cluster_listen` by default (see [Kubernetes NetworkPolicy](#kubernetes-networkpolicy)). Every node of a deployment carries the same `server.cluster_id`. A check that makes peers compare it on every Raft RPC and refuse a peer with another id is planned for 0.8 ([#601](https://github.com/tonbo-io/ursula/issues/601)). It stops two deployments wired to each other by mistake from exchanging Raft traffic. It is not authentication: the id is not a secret, and anyone who can reach a cluster port can still send it. ## What Ursula does not do Handle the following outside Ursula: - **TLS / HTTPS.** The public listener serves plain HTTP. No built-in `rustls`. - **Inter-node encryption and authentication.** Peer gRPC (Raft heartbeats, append-entries, snapshots, and leader-read checks) runs over h2c on the cluster plane, and a node accepts it from any caller. Peers must share a private network. A client's write to a non-leader gets a `307` redirect to the group leader's client URL rather than being forwarded over gRPC. Only node-internal calls are forwarded between nodes. - **API authentication on nodes.** Ursula nodes themselves accept any caller with network reach. Bearer-token validation is available as an opt-in feature of the gateway (see below); node listeners must stay on a private network either way. - **Authorization / multi-tenancy on nodes.** Nodes enforce no per-user, per-bucket, or per-scope ACLs. The gateway's opt-in access control provides a bucket-level tenant boundary; anything finer stays upstream. - **Admin authentication.** The admin plane has no auth gate. Its loopback bind is the only thing that keeps it private. - **Per-client rate limiting.** A single noisy client can saturate a core's mailbox or a group's hot ring. - **Health/readiness endpoints.** `GET /__ursula/ready` is unauthenticated on the client plane and reports only readiness plus WAL free-space guard state. `/__ursula/metrics` on the same plane contains much more operational detail. Keep the client listener private. - **At-rest encryption beyond the cold tier.** Cold-tier S3 writes (including Raft snapshots) request SSE-S3 by default (`storage.cold.s3.server_side_encryption`, switchable to `aws-kms` or `none`). The hot ring is in memory; WAL and Raft log directories live on disk in plaintext — use full-disk encryption at the host level. Per-tenant KMS keys and client-side encryption are out of scope. CORS is permissive (`Access-Control-Allow-Origin: *`). Restrict at the proxy for browser traffic. Tenant offboarding has a first-class erasure path: the admin-plane [bucket purge endpoint](/docs/0.9/operations#tenant-offboarding-bucket-purge) removes a tenant's streams, bucket, and cold objects idempotently, leaving other tenants untouched. It deliberately retains aggregate monotonic usage counters so asynchronous accounting cannot miss committed work; see the operations note for the remaining identifier-erasure limitation. ## Gateway access control (opt-in) A shared or internet-facing deployment can enable OAuth resource-server checks on `ursula gateway`. The feature is off by default; without the flags the gateway keeps its original trusted pass-through behavior. ```bash ursula gateway \ --upstream http://ursula-0:4437 \ --auth-issuer https://issuer.example \ --auth-audience https://streams.example \ --auth-policy /etc/ursula/policy.toml ``` - **Authentication.** `Bearer` credentials are validated as RFC 9068 JWT access tokens: the header must declare `typ: at+jwt` (OIDC ID tokens are rejected), the signature must verify against the issuer's JWKS, and `iss`, `aud`, `sub`, `client_id`, `iat`, `exp`, and `jti` must all be present and valid. The JWKS location comes from `--auth-jwks-url` or RFC 8414 metadata discovery; keys are cached by `kid` and refetched on rotation. - **Tenant boundary.** The bucket is the top-level namespace and logical tenant boundary. The policy file declares each bucket's owners (issuer-qualified subjects) and whether anonymous reads are allowed: ```toml [[bucket]] id = "tenant-a" public_read = true owners = [{ issuer = "https://issuer.example", subject = "user-1" }] ``` - **Concealment.** Unknown buckets, private buckets probed by strangers, and write attempts without ownership all answer the same `404` a missing resource would, so a private stream's existence is not observable. - **Internal paths.** Every `/__ursula/*` path answers `404`, the gateway's own metrics at `/__ursula/gateway/metrics` included. The counters hold no tenant data, but they show traffic volume. `--auth-public-metrics` serves them to every caller, with or without a token, as a gateway without access control does. The gateway then does not check a presented token on that path, so an invalid or expired one gets the counters too. Set the flag only where untrusted clients cannot reach the gateway, or where a proxy in front of it denies `/__ursula/*`. Gateways before 0.7.4 served the metrics to every caller, with or without access control. - **Credential termination.** The gateway strips `Authorization` before forwarding; upstream nodes never see end-user credentials and must remain on a private network. - **Anonymous public reads.** `public_read` grants exactly the read-only actions (read, head, tail, snapshot read) to unauthenticated callers — never writes, deletes, or bucket administration. - **Subscriptions do not outlive their credential.** A live tail is one long-lived request, so a single admission check at connection time would turn a short-lived token into an unbounded read. The gateway ends an SSE subscription at the credential's `exp` with a final `event: credential-expired` frame, so a client can tell expiry from end-of-stream and re-authenticate. Anonymous reads on `public_read` buckets have no credential and so no deadline. Note that this bounds duration, not revocation: a token revoked before its `exp` keeps an open subscription until then. An access-controlled gateway can additionally enable per-tenant admission limits and usage accounting: ```bash ursula gateway ... \ --quota-policy /etc/ursula/quotas.toml \ --usage-log /var/log/ursula/usage.jsonl ``` - **Quotas** (`--quota-policy`): per-bucket request rate (429 with `Retry-After`), concurrent live-read connections, and request body size. Limits are gateway-process-local; a horizontally scaled deployment multiplies effective limits by replica count. Ursula's own `503` backpressure semantics are unchanged. Neither Ursula nor this gateway enforces per-bucket stream-count or retained-bytes limits; enforce them in your control plane from the aggregated `/__ursula/usage` counters. - **Usage** (`--usage-log`): per-tenant request, ingress, and egress byte counters aggregated by `(bucket, principal, action class)` and appended as sequence-numbered JSONL batches on `--usage-flush-secs` intervals. `--usage-chunk-bytes` adds a `chunks` counter alongside them: each append is counted as `ceil(bytes / unit)`, never below one, summed per request. It exists because that sum cannot be recovered afterwards — two appends of 5 KiB and 25 KiB and two of 10 KiB and 20 KiB agree on both request count and byte total while owing four units and three — so a deployment that charges per write unit has to be handed the sum rather than the ingredients. The unit size is a pricing choice and Ursula does not pick one; 10 KB and 25 KB are both in use. A batched append costs proportionally less than the same records sent individually, which is the intended incentive: batching is cheaper to serve. A failing sink delays reporting (batches queue and merge) but never blocks requests or drops counts. Egress is counted from actually streamed bytes, including SSE bodies. Committed-truth counters (append bytes surviving retries, retained bytes) come from Ursula's replicated state and are a separate, complementary ledger. ### Cross-origin reads (opt-in) `public_read` is only nominally public until the origin answers CORS: without it, browser JavaScript cannot read a public stream cross-origin. Allowed origins are deployment policy, so nothing is sent unless you list them. ```bash ursula gateway ... \ --cors-allowed-origin https://app.example.com \ --cors-allowed-origin https://studio.example.com ``` Pass `*` instead to allow any origin. Three properties are deliberate: - **Credentials are never allowed.** Ursula authenticates from an `Authorization` header the caller sets explicitly, which CORS does not treat as credentials, and no cookies are involved. So `*` grants no ambient access — a cross-origin page must still present its own bearer token. - **`Access-Control-Expose-Headers: *`.** A read carries its continuation in response headers (`stream-next-offset`, `stream-cursor`), and a browser cannot see those without exposure — an unexposed client can read one page and never advance. The wildcard is only honoured while credentials stay disallowed, which is the second reason they are. - **Preflight never consults the resource.** `OPTIONS` arrives without `Authorization`, so a per-bucket answer would tell an unauthenticated caller whether a private bucket exists. The preflight reply is identical for every path and is produced before authorization runs, which keeps concealment intact. > [!NOTE] > `EventSource` cannot set request headers, so a browser cannot open an SSE tail on a **private** stream with it. Use `fetch` with a `ReadableStream` and an `Authorization` header. `EventSource` is fine for `public_read` streams. ## Recommended deployment ``` Untrusted internet │ v ┌─────────────────────┐ │ Reverse proxy │ TLS, authn, per-client │ (nginx / Envoy / …) │ rate limiting, CORS └──────────┬──────────┘ │ plain HTTP, private network ┌────────────┼────────────┐ v v v client port (4437) ┌────────┐ ┌────────┐ ┌────────┐ │ Ursula │ │ Ursula │ │ Ursula │ │ node │ │ node │ │ node │ └────────┘ └────────┘ └────────┘ ↕ gRPC h2c between voters only, cluster port (4439) (Raft replication) ``` ### Checklist - **Bind to the private interface.** Set `server.listen = "10.0.0.X:4437"` or use a security group / firewall so the listener is unreachable from public networks. - **Split the cluster plane off.** Set `server.cluster_listen` and give every peer a `client_url`, then let only the voters reach the cluster port. - **Keep the admin plane on loopback.** Leave `server.admin_listen` at its loopback default and reach it through a tunnel. - **Terminate TLS at the proxy.** Ursula stays plain HTTP on the internal side. - **Authenticate at the proxy.** Validate the caller (OAuth2, mTLS, signed requests) and reject unauthenticated traffic before it reaches Ursula. - **Block internal paths from public traffic.** Deny `/__ursula/*` at the proxy. The client plane serves metrics, readiness and usage there. A gateway with access control answers `404` for them. - **Use IAM roles for S3.** Omit static `storage.cold.s3.access_key_id` / `storage.cold.s3.secret_access_key` values and let the AWS SDK credential chain discover credentials. - **Encrypt data volumes.** Apply full-disk encryption to `raft.wal.path`. - **Keep peer traffic private.** Never route gRPC peer traffic across the public internet. ## Kubernetes NetworkPolicy The Helm chart serves the cluster plane on its own port by default (`server.clusterListen: true`, `server.ports.cluster: 4439`). Peers dial that port through the headless peer Service. The client Service exposes only the client port, and the admin plane stays on pod loopback. Without a NetworkPolicy, any pod can still dial a voter's pod IP on the cluster port. A Service only decides what gets a stable name. Any pod in the cluster can still dial a pod IP on any port. Set `server.networkPolicy.enabled: true` to render a NetworkPolicy for the server pods, on a CNI that enforces NetworkPolicy: - The client port admits the sources in `server.networkPolicy.clientIngressFrom`, a list of NetworkPolicy peers. Empty, the default, admits every source. The release's gateway and Helm test pods are always admitted when they are enabled. List every other client: applications, anything that polls `/__ursula/metrics`, and an `ipBlock` for traffic that arrives through a `LoadBalancer` or `NodePort` client Service. - The cluster port admits only the release's own server pods. - The admin port needs no rule, because it listens on loopback. The chart refuses to render the policy when `server.adminListen` is not on loopback. ```yaml server: networkPolicy: enabled: true clientIngressFrom: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: apps ``` The policy selects only ingress, so egress stays open. Kubelet probes reach the client port from the node, which most CNIs admit whatever the policy says. Check yours before you restrict `clientIngressFrom`. ## Reporting vulnerabilities Open a GitHub Security Advisory on [tonbo-io/ursula](https://github.com/tonbo-io/ursula/security/advisories) rather than a public issue. --- # Resumable AI Stream > Stream LLM tokens through a durable timeline so clients can refresh, reconnect, and resume without losing a token. Instead of piping a model's output straight into one HTTP response, the generating worker appends tokens to a stream, and clients replay and tail that stream. A client that reconnects mid-generation replays the answer so far and continues from the same position, without losing or duplicating tokens. The upstream [`chat-aisdk` example](https://github.com/durable-streams/durable-streams/tree/main/examples) uses the same pattern. ## One stream per run Create a text stream per generation, with a TTL so finished runs clean themselves up: ```bash curl -X PUT http://127.0.0.1:4437/agent/run-42 \ -H 'Content-Type: text/plain' \ -H 'Stream-TTL: 86400' ``` ## Producer: append tokens, close at the end The worker that talks to the model appends each chunk. Producer headers make retries safe: a chunk re-sent after a network error is deduplicated instead of appended twice: ```ts const base = "http://127.0.0.1:4437"; let seq = 0; for await (const chunk of modelResponse) { await fetch(`${base}/agent/run-42`, { method: "POST", headers: { "Content-Type": "text/plain", "Producer-Id": "worker-1", "Producer-Epoch": "1", "Producer-Seq": String(seq++), }, body: chunk.text, }); } // Seal the stream: readers get an EOF signal, further appends return 409. await fetch(`${base}/agent/run-42`, { method: "POST", headers: { "Stream-Closed": "true" }, }); ``` If the worker crashes and restarts, bump `Producer-Epoch` and start `Producer-Seq` back at `0` (see [Exactly-once writes](/docs/0.9/concepts/exactly-once-writes)). ## Consumer: replay, then tail `offset=-1` replays everything appended so far, then stays live for new tokens. Reopening the page mid-generation therefore shows the partial answer immediately and continues streaming: ```ts const es = new EventSource(`${base}/agent/run-42?offset=-1&live=sse`); es.addEventListener("data", (e) => { answerEl.textContent += e.data; }); es.addEventListener("control", (e) => { if (JSON.parse(e.data).streamClosed) es.close(); // generation finished }); ``` Or watch it from a terminal: ```bash curl -N 'http://127.0.0.1:4437/agent/run-42?offset=-1&live=sse' ``` The stream is durable and readable by any number of clients, so the same transcript can be tailed from a second device, read after the fact, or consumed by another process. The producer does not need to know about its readers. ## Next - [Chat Room](/docs/0.9/examples/chat-room): the same pattern with JSON records and multiple writers - [Read stream](/docs/0.9/api/read): catch-up, long-poll, and SSE read modes - [Exactly-once writes](/docs/0.9/concepts/exactly-once-writes): producer epoch and sequence rules --- # Chat Room > One JSON stream per room: append messages, replay history from an offset, and tail new messages live. A chat room is a JSON timeline: every chat message is one JSON message in the stream, stored in commit order, and every participant replays history and tails the same stream. This mirrors the chat examples in the [upstream Durable Streams repo](https://github.com/durable-streams/durable-streams/tree/main/examples). ## One stream per room ```bash curl -X PUT http://127.0.0.1:4437/chat/room-7 \ -H 'Content-Type: application/json' ``` Each appended JSON object is stored as one line of NDJSON. Positions in the stream are offsets: opaque tokens that Ursula returns in `Stream-Next-Offset`. Clients store and pass back the offsets Ursula returned and never compute one themselves. ## Send a message ```bash curl -X POST http://127.0.0.1:4437/chat/room-7 \ -H 'Content-Type: application/json' \ --data '{"user":"ada","text":"hello","sent_at":"2026-07-23T09:00:00Z"}' ``` Each append returns the room's new tail in `Stream-Next-Offset`. Appending a top-level JSON array stores one message per element, which is useful for flushing an offline outbox in one request. To make send retries safe against double-posting, add the [exactly-once producer headers](/docs/0.9/concepts/exactly-once-writes) with one `Producer-Id` per device. JSON messages are validated on append and stored exactly as the sender wrote them, minus insignificant whitespace, so member order and number spelling survive. ## Replay history Read the room from its start, or from an offset the client stored earlier: ```bash curl -i 'http://127.0.0.1:4437/chat/room-7?offset=-1' ``` ```json {"user":"ada","text":"hello","sent_at":"2026-07-23T09:00:00Z"} {"user":"lin","text":"hey ada","sent_at":"2026-07-23T09:00:07Z"} ``` The body is NDJSON, one message per line. A response returns at most 8 MiB and can end mid-line: buffer an incomplete trailing line and continue from the response's `Stream-Next-Offset` until a response carries `Stream-Up-To-Date: true`. Loading only the last N messages needs a server-side paging primitive that does not exist yet; until it ships, replay from the start or from a stored offset, or keep a [snapshot](/docs/0.9/concepts/snapshots) of the recent history. ## Tail new messages live `offset=now` skips history and streams only messages committed from now on: ```ts const base = "http://127.0.0.1:4437"; const es = new EventSource(`${base}/chat/room-7?offset=now&live=sse`); let pending = ""; es.addEventListener("data", (e) => { pending += e.data + "\n"; const lines = pending.split("\n"); pending = lines.pop() ?? ""; for (const line of lines) { if (line) renderMessage(JSON.parse(line)); } }); ``` A data event may hold several messages, one per line; SSE `data:` lines are transport framing, not message boundaries. Control events publish `streamNextOffset`. Store it and reconnect with `offset=` to resume without missing or repeating a message. ## Next - [Resumable AI Stream](/docs/0.9/examples/resumable-ai-stream): the single-writer variant for streaming model output - [Offsets](/docs/0.9/concepts/offsets): how resumption works - [Browser Telemetry](/docs/0.9/examples/browser-telemetry): browser events in a JSON stream, replayed and tailed by offset --- # Browser Telemetry > Collect replayable browser events in a JSON stream and replay or tail them by offset. Browser telemetry is a natural fit for an `application/json` stream: the browser appends ordinary JSON over HTTP, Ursula stores each event as one message in commit order, and consumers replay or tail events from an offset. The runnable example is in [`examples/browser-telemetry`](https://github.com/tonbo-io/ursula/tree/main/examples/browser-telemetry). ## Data model Keep one timestamp: the time captured by the browser. It remains application data and may arrive out of order after offline buffering or retries. ```json { "captured_at": "2026-07-18T10:30:00.100Z", "type": "network_response", "status": 500 } ``` Ursula does not reorder the stream by `captured_at` and does not invent a second protocol timestamp. Commit order and offsets are the stream's only coordinates: an out-of-order `captured_at` never changes the committed order, the stored bytes or any offset. Event time is therefore not a stream read parameter (there is no `?captured_at=` read). A consumer that needs a time window reads by offset and filters on `captured_at` itself. ## Browser collector Point plain `fetch` at a same-origin route that proxies to the Ursula stream. The gateway owns browser authentication and CORS policy. ```js const events = [{ captured_at: new Date().toISOString(), type: "navigation", duration_ms: performance.getEntriesByType("navigation")[0]?.duration, }]; const response = await fetch("/telemetry/events", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(events), keepalive: true, }); console.log(response.headers.get("Stream-Next-Offset")); ``` One array append stores one message per element. A successful response returns the stream's new tail in `Stream-Next-Offset`. ## Replay and live tail Read events from the start of the stream: ```bash curl -i 'http://127.0.0.1:4437/telemetry/events?offset=-1' ``` The body is NDJSON, one event per line. A response returns at most 8 MiB and can end mid-line: buffer any incomplete trailing line and continue from the response's `Stream-Next-Offset`; `Stream-Up-To-Date: true` means the read reached the tail. Offsets are opaque: store and pass back the values Ursula returns, and compare them only with each other. For a live native `EventSource`, add `live=sse`. ## OpenTelemetry (OTLP-JSON) OpenTelemetry data exported as OTLP-JSON can go straight into a stream. Create one `application/json` stream per service, signal and day, and point the stock collector's `otlphttp` exporter at it: ```yaml exporters: otlphttp/ursula: traces_endpoint: http://ursula-gateway:4437/otel/traces-checkout-20261003 encoding: json compression: none processors: batch: send_batch_max_size: 2048 ``` Ursula does not decompress request bodies, so keep `compression: none`, and keep batches well under the 32 MiB request limit. Each export request is stored as one message. Replay and tail the stream by offset as above. --- # SQLite on Ursula `@tonbo/sqlite-ursula` gives a Node.js application a SQLite database that an Ursula stream replicates. Statements run on a local SQLite file. Each commit goes to a log on the same host and then to the stream in the background, and `await db.synced()` tells you when the stream holds it. Start a single node in a terminal of its own. It keeps nothing across restarts. ```bash docker run --rm -p 4437:4437 ghcr.io/tonbo-io/ursula:0.9.0 ``` Install the package, which needs Node.js 22.19 or later, and run: ```bash npm install @tonbo/sqlite-ursula ``` ```ts import { open } from "@tonbo/sqlite-ursula"; const db = await open("http://127.0.0.1:4437/demo/notes"); db.exec("CREATE TABLE IF NOT EXISTS notes(id INTEGER PRIMARY KEY, body TEXT)"); db.prepare("INSERT INTO notes(body) VALUES (?)").run("hello"); // committed on this host await db.synced(); // in the stream, and this process still owns it console.log(db.prepare("SELECT id, body FROM notes").all()); db.close(); ``` **The one rule:** before anything outside the process acts on what you wrote or read (a response, a queue acknowledgement, a call to another service), `await db.synced()`. | you | what happens | | --- | --- | | run a statement or COMMIT | local work: SQLite, plus one write to the log | | await `synced()` | one append round trip while writing, one leader read when idle | | lose the stream | commits continue, `synced()` waits, past `maxPendingBytes` writes fail with `backlog_full` | | are taken over | statements and `synced()` throw `UrsulaResetError` (`fenced`), and `reset` listeners run | | crash | commits the stream never acknowledged come back from the log at the next open and are resent | The local cache outlives the node. The quick-start node keeps nothing across restarts and sets no `server.cluster_id`, so a restarted node is a new deployment with a new random cluster id, and `open` refuses the old cache with `wrong_cluster`. Open with a fresh `dir`, or delete the stream's directory in the cache root (`demo-notes-*` in `~/Library/Caches/sqlite-ursula/0.8`, `~/.cache/sqlite-ursula/0.8` on Linux). For anything beyond a throwaway quick start, give the server a `server.cluster_id` and a Raft WAL path (see [Configuration](/docs/0.9/configuration)), so a restart keeps both the stream and its deployment. ## How it works `open` loads the `sqlite-ursula-vfs` extension into `node:sqlite`, brings the local cache up to date from the stream, claims the stream for this process and returns the database. Any host can rebuild the database from the stream: the same code on another host opens the same database, and fences this one. - **A commit is local.** At SQLite's commit point the extension appends the transaction's final pages to a log it owns, with no fsync and no network. Statements never wait on the stream. - **Durable means acknowledged.** One replicator per database sends logged commits in order, one append at a time, batching what accumulates. An append is resent unchanged until the stream answers it, however long an outage lasts. - **`synced()` is the output gate.** It resolves once the stream holds every commit made before the call, and a round trip that started after the call found this process still the owner. It never blocks the event loop. - **Ownership is explicit.** The last `open()` that reaches the stream owns it. Claims the extension makes on its own, after an outage for example, never take the stream from an owner that an explicit open installed. The design, with the test behind each guarantee, is in [`docs/architecture/sqlite-vfs.md`](https://github.com/tonbo-io/ursula/blob/main/docs/architecture/sqlite-vfs.md). ## Open options `open(url, options)` returns a promise of an `UrsulaDatabase`, a `node:sqlite` `DatabaseSync` subclass, so drizzle, kysely and Pi's driver take it. Options it does not know go to `DatabaseSync`. | option | default | meaning | | --- | --- | --- | | `dir` | `/sqlite-ursula/0.8` | The cache root, created when missing | | `token` | none | The bearer token requests carry. One per process: the last open's applies | | `signal` | 30 s | Bounds the open | | `maxPendingBytes` | 64 MiB | Compressed commits that may wait for the stream before writes fail with `backlog_full` | | `recreate` | `false` | When the stream is missing and the cache holds data, move the cache aside and create the stream again, instead of `not_found` | | `readOnly` | `false` | Claim nothing, see [Readers](#readers) | | `timeout` | 5000 ms | node:sqlite's busy timeout. The connections of one database share its files | - The first open of a stream in the process binds it. Later opens return another connection on the same binding with no network, concurrent opens share one, and worker threads call `open` too. - `db.close()` is synchronous. Once a database's last connection closes, replication drains in the background. There is no drain at exit: a script awaits `db.synced()` before it exits, and the log keeps what the stream lacks for the next open. - A stream is open under one `dir` at a time in a process. An open under another `dir` fails with `bound_elsewhere` while the first has connections open. `UrsulaDatabase` adds `name` (the stream's path, such as `/acme/notes`), `synced({ signal })`, `status()`, and `on("reset", listener)` and `off("reset", listener)`. ## Errors Every error the package throws for a failure of the extension has a `kind`, an `action` and a `cause`. Branch on `action`. Every other error, such as a constraint violation, is `node:sqlite`'s own. Network failures never throw from statements or `synced()`: they wait. | `action` | what to do | | --- | --- | | `retry` | Nothing was lost and the database stays usable. For `backlog_full`, await `synced()` and run the transaction again. For a failed `open`, call it again later | | `reopen` | Close every connection of the database and `open` it again | | `stop` | Another owner took the stream over. Stop using the database, and never reopen it in a loop: opening it takes the stream back | | `fix` | It cannot succeed as asked. Fix the URL, the setup, the stream or the transaction | | class | thrown by | kinds | | --- | --- | --- | | `UrsulaWriteRefusedError` | a write or COMMIT, nothing committed (`errcode` 7178) | `backlog_full`, `unsupported`, `too_large` | | `UrsulaResetError` | statements after a reset, `synced()`, `reset` listeners | `fenced`, `stale`, `rewound`, `deleted`, `local_damage`, `stream_state`, `internal` | | `UrsulaUnavailableError` | `open()` when its signal fires first, and `importSqlite()` when it cannot finish (action `fix`, since a failed import leaves the stream refusing opens) | the last condition (`unreachable`, `unavailable`, `rate_limited`, `unauthorized`, `payment_required`, `forbidden`), or `cancelled` | | `UrsulaSetupError` | `open()`, `importSqlite()`, and `synced()` or `status()` of a retired database | `invalid_url`, `invalid_argument`, `not_found`, `in_use`, `bound_elsewhere`, `local_setup`, `stream_format`, `server_too_old`, `wrong_cluster`, `stream_state`, `claim_races`, `not_bound`, `internal` | `unsupported` is a transaction the extension cannot capture: EXCLUSIVE locking mode, or a commit that leaves WAL mode or 4096-byte pages. Only that transaction fails. `too_large` is a commit whose pages take more than the commit limit, `URSULA_VFS_MAX_COMMIT_BYTES` (128 MiB by default). SQLite rolls the whole transaction back, nothing of it is logged or sent, and the database stays usable. Split the transaction, or raise the limit on every host that may own the database. A `VACUUM` or a rewrite of a large table writes most of the database in one transaction: raise the limit for it, run `PRAGMA incremental_vacuum(N)` in steps on a database created with `auto_vacuum = INCREMENTAL`, or `VACUUM INTO` a plain file and import that into a new stream with `importSqlite()`. ## Resets A reset ends a database. Pending `synced()` calls reject first, then the `reset` listeners run once, and every statement throws the reset, reads included (`errcode` 266 for a read, 778 for a write). A read transaction already open keeps its snapshot. `open` the stream again, or stop when the action is `stop`. The new open need not wait for the old database to close: it works on a copy of the cache, which carries every commit the old one made. | kind | cause | action | commits the stream never acknowledged | | --- | --- | --- | --- | | `fenced` | another owner claimed the stream | `stop` | kept, and sent if this process opens the stream again | | `stale` | the stream moved on while this process could not see it, such as an owner idle past producer expiry | `reopen` | discarded | | `rewound` | a restore or an accepted loss removed commits the stream had acknowledged | `reopen` | moved aside into `salvage/` by the reopen | | `deleted` | the stream was deleted | `fix` | kept: `open` with `recreate` | | `local_damage` | a local fsync failed, a WAL page read back wrong, or SQLite ended a commit without publishing it | `reopen` | kept, and the reopen replays them | | `stream_state` | another writer appended to the stream, the stream was closed, or the server lost acknowledged state | `fix` | kept | | `internal` | a bug inside the extension | `reopen` | kept | ## Provisional opens When the stream is unreachable (the name lookup or the connect fails, or takes more than 5 s), an open with a trusted local cache that claimed the stream before resolves at once, `provisional` in `status()`. Reads come from the cache, commits are local and pending, and `synced()` waits. Once the stream answers, the extension reads it from the cache's acknowledged state to its tail and claims it only when no other owner wrote or claimed there. The database is then `live`. If another owner took the stream meanwhile, the database resets (`fenced` or `stale`) with nothing sent, and no `synced()` ever resolved on the provisional state. Without a cache, the open waits for the stream until its signal fires. ## History changes and salvage A restore, an accepted loss, or a delete and recreate gives the stream a new history, and `Stream-Incarnation` changes. - When the stream kept every commit the cache acknowledged, the extension keeps the cache under the new history, with no reset (`attach.local: "rekeyed"` at an open). This covers a restart of a single node that lost nothing. - Otherwise it moves the cache aside and builds a new one from the stream. A bound database first resets `rewound`. - A cache is also moved aside when its db file is not its own (`replaced`), and with `recreate` (`recreated`). A moved cache goes into `salvage/` in the stream's cache directory as a plain SQLite database with its log and a `manifest.json`, and `status().salvage` lists it. The extension never reads or deletes it. Open the `db` file with any SQLite to look at the commits the new history lacks, and decide what to write again. ## Readers `readOnly: true` claims nothing and never fences the owner. Writes fail with `SQLITE_READONLY` (8). - In a process that owns the stream it returns a read-only connection on the owner's database, which sees the owner's latest commits. - In any other process it builds or catches up a reader cache of its own (`role: "reader"`, `reader-/` in the stream's cache directory) to the stream's tail at the open. It reads the stream as of then, and its `synced()` resolves at once. Close its connections and open again to read newer commits. - A closed stream opens. On an unreachable stream it opens an earlier reader cache as it is, and otherwise waits until the signal fires. A missing stream fails with `not_found`. ## Status `db.status()` returns one JSON object. The fields to watch: - `state`: `provisional`, `live`, `retiring`, `retired` or `reset`. `role`: `owner` or `reader`. - `commits`: `logged`, `acknowledged`, `pending` and `pending_bytes`. `pending_bytes` reaching `maxPendingBytes` means writes fail with `backlog_full`. - `replication.condition`: `null`, or why requests are retried: `retrying` (no definite answer yet), `unauthorized`, `forbidden`, `payment_required`, `request_too_large`, `redirected`, `wrong_cluster` or `rejected`. Commits continue under any condition until the backlog fills. - `attach.local`: where the open got the database: `resumed`, `rekeyed`, `built`, `salvaged` or `caught_up`. - `snapshot`: `offset`, `age_ms`, `failures`, `error`, `log_bytes` and `due_bytes`. A snapshot is due once `log_bytes` passes `due_bytes`, and a fresh host builds from the latest snapshot and the log after it. - `refused`, `salvage` and `reset`. The extension logs one logfmt line per event on stderr, such as `sqlite-ursula-vfs level=warn event=reset ...`. Alerts worth setting: - `replication.condition` other than `retrying` that stays: something in front of the stream refuses the writes. The condition says what. - `pending_bytes` growing for minutes: the stream is unreachable or slow. - Any reset, and any entry in `salvage`. - `snapshot.log_bytes` staying above a few times `due_bytes`, or `snapshot.failures` growing: snapshots are not published, and `snapshot.error` says why. Every rebuild then reads the whole log. ## Pi Durable Install Pi Durable next to the package: ```bash npm install @tonbo/sqlite-ursula @earendil-works/pi-durable ``` ```ts import { openUrsulaPiStorage } from "@tonbo/sqlite-ursula/pi"; const storage = await openUrsulaPiStorage("https://ursula.example.com/pi/harness-1", { closeTimeoutMs: 10_000 }); // hand `storage` to the Pi harness, and close it on shutdown ``` `openUrsulaPiStorage` opens the stream with `open` and runs Pi's own node driver on it. Every transaction and statement resolves only after `synced()`, so a Pi commit resolves once the stream holds it. - A commit refused with `backlog_full` commits nothing. From Pi 1.2, it waits for `synced()` and runs again, since Pi expects a storage to retry its own transient failures. On Pi 1.0 and 1.1 it throws Pi's `StorageRejected`, and the session continues. - A reset throws the `UrsulaResetError` itself, so the session stops. Open the storage again, or stop when its `action` is `stop`. - `close` waits up to `closeTimeoutMs` for `synced()`, then closes. On Pi 1.3, `Session.close` first waits for the storage calls in flight, which wait for the stream, so closing a session during an outage waits until the stream answers. Bound the shutdown in your host if it must end sooner. `@earendil-works/pi-durable` is an optional peer dependency in the range `>=1.0.0 <1.4.0`, and npm refuses the install with `ERESOLVE` outside it. ## Other hosts Any SQLite host can load the extension and call its SQL functions on the connection `loadUrsulaVfs()` returns: `ursula_open`, `ursula_synced`, `ursula_events`, `ursula_cancel`, `ursula_status`, `ursula_stats`, `ursula_set_token` and `ursula_import`. `ursula_open(url, dir, options_json, timeout_ms)` returns the path of the local file, which connections then open with plain SQLite. A host opens a reader's path with `SQLITE_OPEN_READONLY`. Errors read `: : `. The extension refuses to load into SQLite older than 3.50.4. A cache's files are not SQLite files outside the extension. The db file holds a marker in place of SQLite's header magic, and its WAL is `db-uwal`. The `sqlite3` shell, node:sqlite without the extension and a connection opened before the extension loaded fail with `SQLITE_NOTADB` (`file is not a database`) and change nothing. To read a database with plain SQLite, export it from a connection `open` returned, with `VACUUM INTO`. ## Import `importSqlite(url, file, { token, timeoutMs })` imports a plain SQLite database file into a new stream, as the stream's first snapshot, and returns `{ offset, digest, pages, bytes }`. Then `open(url)` it as any other stream. ```ts import { importSqlite, open } from "@tonbo/sqlite-ursula"; importSqlite("https://ursula.example.com/acme/notes", "/tmp/notes.db"); const db = await open("https://ursula.example.com/acme/notes"); ``` - The file must have 4096-byte pages, no `-wal` beside it and no hot journal. `VACUUM INTO` writes such a copy (after `PRAGMA page_size = 4096` when the database has other pages). The file is only read. - The stream must be missing, or empty. A stream of another content type fails with `stream_format`, anything else with `stream_state`, and nothing is published. A server before 0.8 fails with `server_too_old`. While the import runs, opens of the stream fail with `stream_state`. An import that fails after it claimed the stream leaves a stream that refuses opens and imports with `stream_state` until you delete or recreate it. - A file whose compressed snapshot is above the request limit goes up in parts, which needs a cold backend on the server. Without one it fails with `invalid_argument`, naming the limit. - The call blocks its thread until the import is done, for at most `timeoutMs` (10 minutes by default). ## Moving from 0.7 0.8 changes every format. A 0.8 server reads no 0.7 data, the 0.8 extension refuses 0.7 streams (`stream_format`) and 0.7 servers (`server_too_old`), and it never reads 0.7 caches. Move each database by export and import: 1. **Export with 0.7.** Close the owner cleanly, or open the database read-only, and export it with `VACUUM INTO`. 0.7 caches already use 4096-byte pages. ```ts // @tonbo/sqlite-ursula 0.7.x const db = open("https://old.example.com/acme/notes", { readOnly: true }); db.exec("VACUUM INTO '/tmp/notes.db'"); db.close(); ``` 2. **Import with 0.8** into a new stream on the 0.8 cluster, then open it: ```ts // @tonbo/sqlite-ursula 0.8.x importSqlite("https://new.example.com/acme/notes", "/tmp/notes.db"); const db = await open("https://new.example.com/acme/notes"); ``` From another host, call `SELECT ursula_import(url, file, options_json, timeout_ms)` with a positive `timeout_ms`. 3. **Keep the 0.7 cache** until you have checked the imported database. 0.7 appended every commit to the stream before the local WAL, so the cache holds nothing the 0.7 stream lacks, but once the 0.7 cluster is gone it may be the only copy. Code changes: - `open()` is asynchronous: `await open(url, options)`. A commit returns before the stream holds it, so add `await db.synced()` before every external effect. - The extension owns its files. The `path` option, `attach`, `setToken`, `status(file)`, `drainStats` and `instrument` are gone. Pass `token` to `open`, or use `URSULA_VFS_TOKEN_FILE`. - Six error classes become four. `UrsulaReattachError`, `UrsulaReplicationError` and `UrsulaFencedError` become `UrsulaResetError`, whose `action` includes `reopen`. A `402` is a replication condition in `status()` instead of `UrsulaPaymentRequiredError`. Network failures no longer throw. There is no poisoned state and no `outcome_unknown`. - `openUrsulaPiStorage(url, options)` takes no file. - `status()` has new fields: see [Status](#status). Operations changes: - `URSULA_VFS_RETRY_MS` is gone: appends are resent until a definite answer. `URSULA_VFS_MAX_FRAME_BYTES` becomes `URSULA_VFS_MAX_REQUEST_BYTES`. Commits up to the commit limit (`URSULA_VFS_MAX_COMMIT_BYTES`) go out, in parts above the request limit. `URSULA_VFS_SNAPSHOT_MIN_BYTES` is gone: a snapshot is due past the database's size or 8 MiB, whichever is larger. - The default cache root is `/sqlite-ursula/0.8`, and the `sqlite3` shell cannot open a cache. - The extension sends `User-Agent: sqlite-ursula-vfs/0.8.x`. Update any proxy, WAF or log rule keyed on it. - Multi-node clusters need `server.cluster_id`, and snapshots above 32 MiB compressed need a cold backend (see [What the cluster needs](#what-the-cluster-needs)). ## What the cluster needs - A 0.8 server. `HEAD` answers carry `Ursula-Protocol: 2` and `Ursula-Cluster-Id`, and the extension refuses a server without them. - A stream URL that reaches the stream's leader by itself: a gateway, or a single node. The extension never follows a redirect. - A cluster id per deployment (`server.cluster_id`). The extension remembers the id each cache was built on, and refuses (`wrong_cluster`) a host name that routes it to another deployment. - For databases whose compressed snapshots exceed 32 MiB, a cold backend, so snapshots go up in parts, up to 1 GiB. Without one, `snapshot.error` reports the limit, the log grows, and every rebuild reads it all. - Appends are at most 4 MiB, and snapshots up to 32 MiB go in one request. Clusters keep per-group admission caps of at least 64 MiB (`standard` or `large` preset), so such a publish is admitted. ## Layers in front of the stream Every append the extension sends, a claim included, carries a producer key and a digest of its body. A proxy or a gateway that sends an append again does no harm: the stream answers the repeat as a duplicate of the same write. A snapshot publish or a parts commit sent again is answered from its offset and digest. The extension never needs to know that a write applied nothing: it sends the same bytes again until the stream answers them. 0.8 servers and gateways send no `x-ursula-not-applied` header. Other clients on a shared deployment may send writes without producer headers, which a replayed request applies twice. Keep proxy retries off for stream writes on such a deployment (`proxy-next-upstream: "off"` in the chart's TLS ingress example). A layer that limits request bodies should send `Ursula-Max-Request-Bytes` with its limit on `HEAD`, and name it on its `413`, as `ursula gateway` does. The extension then cuts its requests to fit. A bare `413` makes it ask the stream's `HEAD` for the limit. ## TLS and tokens `https://` stream URLs work with the certificate authorities bundled with the extension (Mozilla's). For a private CA, point `URSULA_VFS_CA_FILE` at a PEM bundle: the extension then trusts only the CAs in it. Behind `ursula gateway` with access control (see [Security](/docs/0.9/security)), every request needs a bearer token. Pass `token` to `open`, or name a file that holds it, which the extension reads again whenever it changes and after a `401` (a Kubernetes projected token, or a file another process refreshes): ```bash export URSULA_VFS_TOKEN_FILE=/var/run/secrets/ursula/token ``` The token is process-wide in 0.8, and its audience must not depend on the host name, since every host name of a deployment shares one cache. A refused token is reread and the request sent again. A token refused twice blocks replication as `unauthorized` until it changes, and commits continue until the backlog fills. When the file holds no token as it is read, the extension keeps the last token it read and logs `token_file_unreadable`. Over `http://` the token crosses the network in clear. Use `https://` for any endpoint on another host. The extension logs `token_in_clear` once when it sends a token over `http://` to a host that is not a loopback address. ## Settings The extension reads these environment variables once per process: | variable | default | meaning | | --- | --- | --- | | `URSULA_VFS_TOKEN_FILE` | none | A file holding the bearer token, read again when it changes | | `URSULA_VFS_CA_FILE` | the bundled CAs | A PEM bundle of the certificate authorities to trust | | `URSULA_VFS_MAX_REQUEST_BYTES` | the server's | A request limit below the one `HEAD` advertises, for a layer in front of the stream that does not advertise its own | | `URSULA_VFS_MAX_COMMIT_BYTES` | 134217728 (128 MiB) | The most bytes of pages one commit may hold, from 4096 to 1 GiB. A larger commit fails as `too_large`. Writing a commit takes about three times the limit in memory for pages that do not compress, and readers need memory for the largest commit any owner logs | | `URSULA_VFS_IDLE_MS` | 90000 | How long a connection is kept after its last request. Keep it below the idle timeout of any NAT, load balancer or firewall on the path | | `URSULA_VFS_MIN_UPLOAD_KBPS` | 256 | The slowest upload, in KiB/s, an attempt's time limit allows: a request body may take 5 s plus its size at this rate | ## Platforms npm installs the prebuilt extension for your platform, `@tonbo/sqlite-ursula-`, as an optional dependency. It is prebuilt for Linux x64 and arm64 with glibc 2.28 or later (Debian 11, Ubuntu 20.04, Amazon Linux 2023 and newer), Linux x64 with musl (Alpine), and macOS 11 or later on Apple silicon. On other platforms, build the extension and point `SQLITE_URSULA_VFS` at it. The variable also overrides the prebuilt extension. ```bash cd clients/sqlite-vfs cargo build --release # target/release/libsqlite_ursula_vfs.so (.dylib on macOS) export SQLITE_URSULA_VFS=$PWD/target/release/libsqlite_ursula_vfs.so ``` Within 0.8.x the formats the extension writes to the stream and to its files do not change, so owners upgrade one at a time. Upgrade `@tonbo/sqlite-ursula` together with the extension. The [release notes](/docs/0.9/release-notes) list the [known limits](/docs/0.9/release-notes#sqlite-vfs). ## Related - [Snapshots](/docs/0.9/concepts/snapshots) - [Exactly-once writes](/docs/0.9/concepts/exactly-once-writes) - The design: [`docs/architecture/sqlite-vfs.md`](https://github.com/tonbo-io/ursula/blob/main/docs/architecture/sqlite-vfs.md) --- # Streams A stream is an append-only byte sequence addressed by a URL: `/{bucket}/{stream}`. Once data is written it cannot be modified or reordered. Only new data can be appended. ## Naming Stream IDs may contain any byte that is not `/`, `\0`, or `..`. Maximum stream ID length is 122 bytes, and the combined bucket-plus-stream path is also capped at 122 bytes. The literal name `streams` is reserved and cannot be used as a stream ID. A stream ID cannot start with `$` either; that prefix is reserved for bucket-level subresources (`400`). A workspace app might use `/workspace-a/doc-123`. An agent system might use `/agents/run-2026-05-13-abc`. Anything within the rules above is fair. ## Content type A stream's content type is set when it is created with `PUT` and defaults to `application/octet-stream` if no `Content-Type` header is supplied. `POST` never creates a stream: an append to a stream that does not exist answers `404`. Subsequent appends must declare the same content type, or the server rejects them with `409`. The content type rides along on reads so clients can dispatch on it without inspecting payloads. ## Lifecycle Every stream is in one of these states: - **Open.** Accepts appends. The default on creation. - **Closed.** Sealed by `Stream-Closed: true` on a POST. Readers receive an EOF signal. Further appends return `409`. Close is irreversible. - **Expired.** Past its `Stream-TTL` or `Stream-Expires-At` deadline. Reads, writes and deletes return `404`. The group's leader removes an expired stream within about a minute, and cold GC then deletes its cold objects. - **Deleted.** Removed by `DELETE`. Reads return `404`. `Stream-TTL` (seconds, relative) and `Stream-Expires-At` (RFC 3339 absolute) are mutually exclusive. Sending both yields `400`. `Stream-TTL` is a sliding window, not a deadline fixed at creation: - Every accepted append or close renews it; a deduplicated retry or a repeat close does not. - Reads renew it at most once per quarter of the TTL, so a stream that is only read can expire after about 0.75×TTL of idleness following the last read. `HEAD` does not renew it. - An SSE connection renews it when it opens, not while it waits for new data. Ursula does not close idle SSE connections (the base protocol expects servers to end them about every 60 seconds so readers reconnect), so an open subscription alone does not keep a stream with no writes alive. Each long-poll request renews like any other read. - Renewal never moves expiry earlier, even when the node that handled the request has a slower clock than the one that last renewed it. ## Related - [Buckets](/docs/0.9/concepts/buckets): how streams are grouped - [Snapshots](/docs/0.9/concepts/snapshots): compacting a long stream - [Bootstrap](/docs/0.9/concepts/bootstrap): efficient first-load for new clients --- # Buckets Streams are organized into buckets. A bucket is a namespace, like a folder that holds related streams under one URL prefix: ``` /{bucket}/{stream} ``` A collaborative editing app might use one bucket per workspace, with one stream per document: `/workspace-a/doc-1`, `/workspace-a/doc-2`. An agent platform might use one bucket per tenant: `/tenant-acme/run-xyz`. ## Naming Bucket IDs match the regex `[a-z0-9_-]{4,64}`: 4–64 bytes, lowercase ASCII letters and digits plus `_` and `-`. Uppercase letters, `/`, `.`, and most punctuation are rejected with `400`. The combined bucket-plus-stream URL path also has a 122-byte ceiling. With a 64-byte bucket name, you have 57 bytes left for the stream ID. ## Lifecycle Buckets are implicit. A bucket exists once a stream is created in it; there is no separate create step. - **`PUT /{bucket}`** validates the bucket ID and changes nothing: `201` for a valid ID, `400` for an invalid one. Clients that create the bucket first can keep doing so. - There is no bucket metadata read, stream listing, or bucket delete. Track your streams in your own application. - Removing a bucket and every stream in it is an operator action: the admin-plane [bucket purge](/docs/0.9/operations#tenant-offboarding-bucket-purge), which permanently fences the bucket ID: later stream creates in it answer `404`. ## Related - [Streams](/docs/0.9/concepts/streams) - [API: create bucket](/docs/0.9/api/create-bucket) --- # Offsets An offset is a position inside a stream. Clients use offsets to read from a specific point or resume where they left off. ## Format Offsets are opaque tokens. Ursula returns them as 20-character strings, for example `"00000000000000000042"`, but their content carries no meaning for clients. Take each offset from a server response and pass it back unchanged. To tell which of two offsets of the same stream comes first, compare them lexicographically. Don't parse offsets or compute new ones from them. Two special values are accepted on read requests: - `offset=-1`: start from the beginning of the stream. After a retention advance has trimmed the beginning, it answers `410 Gone` like any trimmed offset; continue from that response's `Stream-Next-Offset`, which is the earliest offset still readable. - `offset=now`: start from the stream's current tail. Useful for "only new data" subscriptions. ## Response headers After every read or append, the server returns two related headers: - **`Stream-Next-Offset`**: the offset to pass back on the next request. Every successful read and append response carries it (on SSE, in each control event's `streamNextOffset` instead of a header), except a repeat of a producer's older write, answered `204` with `Producer-Outcome: receipt-evicted` ([exactly-once writes](/docs/0.9/concepts/exactly-once-writes#how-dedup-works)). Error responses carry it only where the [read](/docs/0.9/api/read) and [append](/docs/0.9/api/append) status tables say so. - **`Stream-Cursor`**: set on live (long-poll, SSE) responses. A counter for caches, not a position: echo it as `?cursor=` on the next live request. Each response's cursor is past the one the request sent, so successive long-poll URLs differ and a cache can't serve the same empty response in a loop. Resume from `Stream-Next-Offset` (in SSE, the latest control event's `streamNextOffset`) in every read mode. `Stream-Cursor` carries no stream identity, so it doesn't reveal that a stream was recreated; see [Recreated streams](#recreated-streams). ## Stability across snapshots Offsets are stable across snapshot publishes. Publishing a snapshot at an offset doesn't renumber later offsets or delete earlier data. A separate retention advance trims the data before its offset. A read at a trimmed offset returns `410 Gone`, and its `Stream-Next-Offset` header is the earliest offset still readable, which `HEAD` also reports as `Stream-Retained-Offset`. ## Recreated streams An offset doesn't identify which stream it came from. If a stream is deleted and recreated at the same URL, an offset saved from the old stream may be accepted by the new one and return the new stream's data. `HEAD` returns a `Stream-Incarnation` token that changes when a stream is deleted and recreated, and when its history is rewritten: a restore from backup, which can rewind the stream to the backup's tail, or an operator's acceptance of lost writes after a crash. A client that keeps offsets across such a change should store the incarnation with them and compare it for equality before resuming. After a rewrite, `HEAD` can name the old token as `Stream-Previous-Incarnation` and an offset as `Stream-Inherited-Offset`: offsets below it, read under the old token, still hold. ## Related - [Read modes](/docs/0.9/concepts/read-modes) - [Snapshots](/docs/0.9/concepts/snapshots) - [Bootstrap](/docs/0.9/concepts/bootstrap): recovery when offsets you remember have been trimmed --- # Read Modes Streams support three read modes, all via `GET`: - **Catch-up**: `GET /b/s?offset=-1` returns all data from the given offset immediately. Use this to sync historical data. - **Long-poll**: `GET /b/s?offset=...&live=long-poll` returns immediately if data is available, otherwise holds the connection until new data arrives or a timeout. Good for simple polling loops. - **SSE**: `GET /b/s?offset=...&live=sse` opens a persistent Server-Sent Events connection. The server pushes new data as it arrives. This is the recommended mode for real-time frontends (`EventSource` in the browser works out of the box). --- # Exactly-Once Writes Network retries can produce duplicate appends. Ursula provides server-side deduplication so your application doesn't need its own bookkeeping. Three headers form the identity: - **`Producer-Id`**: a stable client identifier (UUID, hostname, etc.). - **`Producer-Epoch`**: bumped on producer restart. Must be ≥ the last epoch the server saw from this producer. - **`Producer-Seq`**: a per-epoch sequence number. Starts at `0` for a new epoch and must increase by exactly `1` per append. Both epoch and seq are capped at `2^53 − 1` so they survive a round-trip through JSON. ## How dedup works The server keeps one record per `Producer-Id` per stream: the current epoch, its highest applied sequence, and the receipt of the producer's newest write. On an append: - **Next in sequence** (`seq = last_seq + 1`): accepted with `200`. The response carries the write's receipt: `Stream-Offset` where its bytes start, `Stream-Next-Offset` where they end, and `Stream-Closed: true` when it closed the stream. - **Repeat of the newest write**: deduplicated and answered `204` with the original receipt. - **Repeat of an older write of the epoch**: deduplicated (never written twice) and answered `204` with `Producer-Outcome: receipt-evicted` and no range headers. - **Out of order** (seq skips ahead): rejected with `409`. The response carries `Producer-Expected-Seq`, the sequence the server wanted, and `Producer-Received-Seq`. A seq above `0` of a later epoch that is not claimed yet answers `400` with `Producer-Expected-Seq: 0`. - **Older epoch**: rejected with `403 Forbidden`, whether or not the write applied before. The response's `Producer-Epoch` is the current epoch. For a write of the epoch that the current epoch's claim closed, `Producer-Outcome: epoch-closed;by=claim` comes with `Producer-Closed-Epoch` and `Producer-Closed-Seq`: every seq up to the closed seq applied, and none above it ever will. For any older epoch, `Producer-Outcome: stale-epoch` says the server cannot tell which of its writes applied. A producer that gets `403` has been replaced by a newer instance and should stop writing. On `200` and `204`, `Producer-Seq` is the highest applied sequence of the epoch, not the request's own seq. When the outcome of an append is unknown, for example after a timeout or a `5xx`, send the same append again, unchanged. The answer to the repeat says what happened. A write may also carry `Producer-Digest`, the BLAKE3-256 hash of its body as 64 lowercase hex digits. The server checks it against the body (`400` on a mismatch) and makes it part of the write's identity: a different body under the key of the newest write, or of the current epoch's claim, answers `403` with `Producer-Outcome: key-conflict` and applies nothing. Without a digest, a repeat with a different body is answered as a duplicate. The full set of answers is in [write keys and typed answers](/docs/0.9/specs/extensions#5-write-keys-and-typed-answers). ## Restart and epoch hygiene When a producer crashes and restarts, **bump `Producer-Epoch`**. Otherwise the server still expects the next contiguous seq within the old epoch, and the producer (which has lost its in-memory seq counter) will collide. A new epoch starts the seq counter back at `0`. The write at seq `0` of a new epoch is its claim. When it closes an earlier epoch, its answer, and the answer to any repeat of it, carries `Producer-Closed-Epoch` and `Producer-Closed-Seq`, which say what of the old epoch applied. A create (`PUT`) with producer headers answers `400`. Create the stream, then claim an epoch. ## Per-stream state Dedup state is scoped per stream. The same `Producer-Id` writing to two different streams maintains two independent epoch/seq counters, so you do not have to elect a single global writer per producer. The server keeps the receipt of each producer's newest write only, and forgets older epochs once a higher epoch is claimed. A producer that has not written to a stream for 7 days is forgotten. A stream holds at most 4,096 producers: a new producer beyond that replaces the least recently seen producer idle for at least an hour, and when none has been idle that long its append is rejected with `429` and the error code `PRODUCER_LIMIT`. When the server forgets a producer, it raises the stream's epoch floor to that producer's epoch. A write that carries `Stream-Incarnation` never reaches an epoch at or below the floor that the stream may have forgotten: it answers `403` with `Producer-Outcome: producer-expired` and the floor as `Producer-Epoch`. Claim an epoch above the floor, starting at sequence `0`. Without `Stream-Incarnation`, a forgotten producer is new, as the base protocol says. ## When you don't need this For append-only workloads where each write is independent (event logging, audit trails), exactly-once headers are optional. Just POST without them. Add them when retries are real and double-applies would be visible. ## Related - [Conditional writes](/docs/0.9/concepts/conditional-writes): coordinate multiple writers with `Stream-Seq`, and combine it with producer headers to publish committed facts - [API: append](/docs/0.9/api/append) --- # Conditional Writes Writers that share a stream sometimes need to say "append this only if the stream is still what I read." Ursula does this with the base protocol's `Stream-Seq` header, which is a compare-and-set on a value the writers choose. There are no locks and no cross-stream transactions. ## How `Stream-Seq` is checked - Each stream stores the last accepted `Stream-Seq`. An append that carries `Stream-Seq` is accepted only when its value is strictly greater than the stored one, and the value is then stored. - The comparison is bytewise, not numeric: `"10"` sorts before `"9"`. Use fixed-width values, such as a counter zero-padded to 20 digits. - Values must be printable ASCII. A header with any other byte is ignored: the append is not checked and nothing is stored. - An equal or smaller value answers `409 Conflict` with the error code `STREAM_SEQ_CONFLICT`, and nothing from the request is committed. The `409` carries `Stream-Next-Offset` set to the stream's current tail and `Stream-Last-Seq` set to the stream's last accepted `Stream-Seq`, so a writer can retry from it without reading first. Only a stream that accepted a `Stream-Seq` refuses one, so a `Stream-Seq` `409` always carries `Stream-Last-Seq`. - The scope is the stream. A value is checked only against earlier values on the same stream, whoever sent them. - An append without `Stream-Seq` is not checked and leaves the stored value unchanged. - A `PUT` that creates the stream stores its `Stream-Seq` as the starting value without checking it. A `PUT` that finds the stream already there ignores the header. - Producer deduplication runs first. A retry of an append that was already accepted from the same `Producer-Id`, `Producer-Epoch` and `Producer-Seq` is answered as a duplicate, not `409`, even though its `Stream-Seq` is no longer greater than the stored value. A repeat of the producer's newest write carries its original `Stream-Next-Offset`, and a repeat of an older write carries none (see [Exactly-once writes](/docs/0.9/concepts/exactly-once-writes#how-dedup-works)). An append rejected with `409` uses up no `Producer-Seq`. - A `Stream-Seq` `409` carries neither `Stream-Closed` nor `Producer-Expected-Seq`. A closed stream answers `409` with `Stream-Closed: true`, and an out-of-order `Producer-Seq` answers `409` with `Producer-Expected-Seq`. - A `Content-Type` that differs from the stream's also answers `409`, with the code `CONTENT_TYPE_MISMATCH` and without `Stream-Last-Seq`. Types are compared after normalization, which keeps parameters, so `application/json; charset=utf-8` is not `application/json`. A `409` with `Stream-Last-Seq` comes only from the `Stream-Seq` check, and the recipes below treat every other `409` as an error. Rust clients parse it with `ursula_headers::StreamSeqConflict`. - A value is at most 256 bytes. ## Prefix-count compare-and-set This pattern lets several writers extend one log while each commit is validated against the whole log before it. 1. **Every writer, every append.** Every append to the stream, by every writer, carries a prefix-count `Stream-Seq`: an append without it is not checked and leaves the stored value unchanged, so a writer that read only the events before it could still win. Create the stream with a `PUT` without `Stream-Seq`, so no value is stored and the first commit's `0` is accepted; a `PUT` that seeds a value can make the first commit answer `409` on every retry. 2. **Commits.** Each append is one commit of one or more events. On an `application/json` stream, `POST` the commit as a JSON array: the elements are committed together, one message per event. Never append an empty commit. 3. **Read to the tail.** Read with `consistency=leader`, continuing from `Stream-Next-Offset`, until a response carries `Stream-Up-To-Date: true`. Count the events you read; call it `n`. `n` counts every event since the stream was created. Once retention has trimmed the start of the stream, those events can't be read (a read from `offset=-1` answers `410`), so keep the event count in every snapshot you publish and count on from it, or don't advance retention on a prefix-count stream. 4. **Append.** Validate the commit against what you read, then append it with `Stream-Seq` set to `n`, zero-padded to a fixed width. On a new stream, the first commit sends `0`. 5. **On a `Stream-Seq` `409`.** Another writer committed first. `Stream-Last-Seq` is the count before that writer's commit. Read on from where you stopped until `Stream-Up-To-Date: true`, count the `k` new events, validate again, and retry with `n + k`. ```bash # The writer read 3 events, ending with Stream-Up-To-Date: true. curl -i -X POST http://127.0.0.1:4437/demo/run-42 \ -H 'Content-Type: application/json' \ -H 'Stream-Seq: 00000000000000000003' \ --data-binary '[{"type":"step","n":4},{"type":"step","n":5}]' ``` Why it works: every commit holds at least one event, so the stored value is always the event count before the latest commit, and it grows with every commit. A count beats it only when it includes the latest commit, so a writer whose append is accepted has seen the whole log. When two writers send the same count, one wins and the other gets `409`. The read must end at the tail. A read shortened by `max_bytes` can stop inside a commit, and a count taken there can beat the stored value without the writer having seen the rest of that commit. A value above the true count, from a counting bug or a seeding `PUT`, is stored like any other. Every honest writer's count is then too small, and each of its retries answers `409`. Do not send the count after your own commit (the last position your events take). That is not a compare-and-set: with 3 events in the log, writer A appends 1 event with `4`, and writer B, which also saw only 3, appends 2 events with `5`. Both are accepted, and B's events were validated against a stale log. If the response to an append is lost, you can't tell whether it committed. Put an operation ID in each commit, and before retrying, read the new events and look for it. ## Publishing committed facts (outbox) When the facts are decided somewhere else, such as a database transaction or a [SQLite VFS](/docs/0.9/examples/sqlite-vfs) database, publish them to a stream after they commit: 1. **Number the commits** in the source of truth, and publish only facts that have committed, in commit order. 2. **Send both guards** on each append: `Producer-Id`, `Producer-Epoch` and `Producer-Seq` for retries, and `Stream-Seq` set to the commit's number, zero-padded. 3. **On takeover**, the new owner uses a higher `Producer-Epoch` and starts `Producer-Seq` again at `0`. If it republishes a fact the old owner already published, the `Stream-Seq` check answers `409` with `Stream-Last-Seq`, which here means "already published": go on to the next fact with the same `Producer-Seq`, or to the first fact after `Stream-Last-Seq`. Only a `409` with `Stream-Last-Seq` means that. Any other `409` is an error, and skipping on it would lose facts. 4. **Old owners.** Once the new owner's first append is accepted, appends with the old epoch answer `403`. Before that, an old owner can still publish only facts that committed, and `Stream-Seq` accepts each of those at most once and in commit order. 5. **Gaps.** If a fact can no longer be published, for example because the source pruned it, append a snapshot or reset marker that tells readers to resynchronize. A fact skipped once can never be appended later, because its number is no longer greater than the stored value. ## Guarding against a recreated stream (`Stream-Incarnation`) `Stream-Seq` and the producer headers are state of one stream. A stream deleted and recreated at the same URL (by a `DELETE` and a `PUT`, or by TTL expiry and a later `PUT`) starts without either, so they can't tell a client that its offsets, producer state or cached data belong to a stream that no longer exists. `Stream-Incarnation` does: - Every successful response that addresses a stream, except a `DELETE`, carries `Stream-Incarnation`, an opaque token of the stream's incarnation. Keep it with whatever you derived from the stream. - Send it back as a request header on any later request: appends, reads (including long-poll and SSE), `HEAD`, `DELETE`, a `PUT` of an existing stream, bootstrap, snapshot publish and read, retention. The server compares it for equality with the stream's current incarnation, atomically with the request (a write when it is applied), and answers `412 Precondition Failed` when the stream is another incarnation; a refused write applies nothing. The `412` carries the current token. - A request that would answer `404` (or, for a read, `410` or `416`) without the header still does. A `PUT` with the header for a missing stream answers `412` and creates nothing. - Without the header nothing changes. `412` never mixes with the `409`s above: a `Stream-Seq` or producer conflict is about the same stream; a `412` means the stream you knew is gone. Start over from the new stream: discard derived state, read from its start or bootstrap, and keep the new token. - **Prefix-count `Stream-Seq`**: several writers extend one log, and each commit must be validated against everything before it. - **Commit-number `Stream-Seq`**: one logical publisher, possibly failing over between owners, must not publish a fact twice or out of order. - **Producer headers** ([exactly-once writes](/docs/0.9/concepts/exactly-once-writes)): retries of the same append must not be written twice. - **`Stream-Incarnation`**: any client that keeps offsets, producer state or a cache across requests, so it never acts on a stream recreated at the same URL. Combine it with the others. - **Neither**: append-only workloads where every write is independent, such as event logging. Just `POST`. Ursula has no precondition on the stream's tail offset (`If-Match`). ## Related - [API: append](/docs/0.9/api/append) - [Exactly-once writes](/docs/0.9/concepts/exactly-once-writes) - [Durability and consistency](/docs/0.9/concepts/durability-and-consistency): what `consistency=leader` guarantees --- # Snapshots As a stream grows, replaying it from `offset=-1` becomes expensive. Snapshots solve this by storing a compacted representation of all data up to a given offset. ``` PUT /{bucket}/{stream}/snapshot/{offset} publish a snapshot, or commit its parts POST /{bucket}/{stream}/snapshot/{offset}/parts stage one part of a large snapshot PUT /{bucket}/{stream}/retention/{offset} advance retained history GET /{bucket}/{stream}/snapshot/{offset} read a specific snapshot, or a byte range of it HEAD /{bucket}/{stream} Stream-Snapshot-Offset names the latest snapshot ``` ## Who publishes Snapshots are an **application-level** operation. Ursula stores them but does not produce them. The writer that knows how to compute the merged state is responsible for publishing. A CRDT editor, for example, periodically computes the merged document and publishes it as the latest snapshot. An agent system might snapshot accumulated tool-call state. Application snapshots are separate from the Raft-internal snapshots Ursula takes for replication and recovery. The two never collide. ## Size and content type Every request body is at most `Ursula-Max-Request-Bytes`: 32 MiB from a node, lower through a gateway with a smaller `--max-request-body-bytes`. A snapshot up to that size is published in one request and kept inline in the stream's replicated state. On a cluster with a cold backend, bodies of at least `runtime.external_payload_min_size` (1 MiB by default) are stored as cold-tier objects instead: replicated state keeps only the offset, content type, digest and object reference, and reads and `/bootstrap` stream the body from the cold tier. With a cold backend, a larger snapshot of up to **1 GiB** (`Ursula-Max-Snapshot-Bytes`) goes up [in parts](/docs/0.9/api/publish-snapshot#upload-in-parts), each one request, which Ursula copies into one cold-tier object when the parts are committed. A superseded cold body is deleted five minutes after the publish that replaced it, so a read that started before still completes. The snapshot has its own content type (separate from the stream's), stored as `snapshot_content_type` and defaulting to `application/octet-stream`. A binary CRDT document can be snapshotted under `application/octet-stream` even when the stream itself carries `application/json` deltas. ## Checkpoint and retention Publishing a snapshot is safe on its own: it creates a replay checkpoint but leaves all stream history readable. After the application has verified the checkpoint, it may explicitly advance retention to the same or an older checkpoint boundary with `PUT /{bucket}/{stream}/retention/{offset}`. Only that operation makes earlier reads return `410 Gone` and makes old cold-tier data eligible for GC. On a JSON stream, snapshot and retention offsets must be message boundaries: an offset just after a message's LF, such as a `Stream-Next-Offset` Ursula returned. An offset inside a message answers `400`. When Ursula cannot read the byte before the offset to check it, it publishes nothing and answers `503`. Retention is exact, and it only moves forward: a boundary at or below the current one succeeds and changes nothing, so a repeated request is safe. ## Immutability Snapshots are immutable at an offset. A publish is keyed by its offset and digest: retrying the same body and content type returns the same `Stream-Snapshot-Digest` while that snapshot is the latest. A refused publish answers `409` naming the latest snapshot's offset and digest. A later offset there means the publish was superseded, the same offset means different content is published there. Snapshots cannot be deleted. To replace a snapshot, publish a new one at a higher offset. A snapshot read honors one byte range (`Range: bytes=a-b`), so a client can check a snapshot's header before it reads the whole body. The digest and the other snapshot headers always describe the whole snapshot. ## Related - [Bootstrap](/docs/0.9/concepts/bootstrap): fetch latest snapshot plus post-snapshot updates in one request - [API: publish snapshot](/docs/0.9/api/publish-snapshot) - [API: read snapshot](/docs/0.9/api/read-snapshot) --- # Bootstrap Bootstrap is the new-client initialization endpoint. A single request returns: 1. The latest snapshot (if any) 2. The updates after the snapshot (one part per record on a JSON stream, one part in all on any other stream), up to the stream's current tail or up to an honest stopping point ``` GET /{bucket}/{stream}/bootstrap ``` The response is `multipart/mixed`: - **First part.** The snapshot. If no snapshot exists, this part is present but empty. - **Subsequent parts.** Incremental updates from the snapshot offset (or from the earliest retained offset, if no snapshot exists). Each part has its own `Content-Type`: the snapshot part uses `snapshot_content_type`. Update parts use the stream's content type. The client does not need to know in advance whether a snapshot exists. The same parsing path handles both cases. ## Partial responses Bootstrap only serves updates that are still in the hot tier, and at most 8 MiB of them per response. When updates after the snapshot have already moved to cold storage, or there are more than 8 MiB of them, the response stops early: `Stream-Next-Offset` is where it stopped and `Stream-Up-To-Date` is absent. In the cold case, and for a non-JSON stream above 8 MiB, the response holds only the snapshot part; a JSON stream above 8 MiB stops at a record boundary. It never skips bytes. Clients therefore follow one rule: after applying the bootstrap response, read from `Stream-Next-Offset` with ordinary reads until `Stream-Up-To-Date: true`, then go live. A complete response simply makes that loop empty. ## When to use bootstrap vs catch-up | You want… | Use | | --- | --- | | Latest state, fastest first-load | `/bootstrap` | | Replay every byte from the beginning | `GET /{b}/{s}?offset=-1` | | Resume from a known offset that's still retained | `GET /{b}/{s}?offset=N` | | Resume from an offset the server has since trimmed (`410 Gone`) | `/bootstrap`, then continue from its `Stream-Next-Offset` | ## Live continuation Bootstrap does not accept `?live=sse`. Combining the multipart body with an SSE event stream is rejected with `400`. To go live after bootstrap, finish reading the bootstrap response, then open a separate `GET /{b}/{s}?offset=&live=sse`. SSE delivers any remaining catch-up data first, so this also covers a partial response. ## Related - [Snapshots](/docs/0.9/concepts/snapshots) - [Read modes](/docs/0.9/concepts/read-modes) --- # Durability and Consistency ## Durability An append is acknowledged once a **majority of voters** has replicated it. A single node failure cannot lose an acknowledged write. With the default `raft.wal.fsync = "never"`, a replica holds the newest appends in the page cache, so a host crash of a majority of voters at once can lose the writes acknowledged after their last `fsync`. Set `always` to acknowledge only after `fsync` (see [Configuration](/docs/0.9/configuration#raft)). Acknowledged data lives in replicated hot state across the cluster, and a background worker flushes older segments to S3 on a configurable interval, after which the data inherits S3-grade durability. In a cross-region deployment with a 5-second flush interval, per-message durability is approximately 9-10 nines. The window where acknowledged-but-unflushed data is at risk from a simultaneous multi-region failure is measured in seconds. ## Consistency Writes are **linearizable**. Each Raft group serializes appends through its current leader, which assigns a total order. Catch-up reads may be served by any replica that has applied the relevant stream state. Followers can return already replicated historical bytes locally because stream positions are immutable. Tail-sensitive reads still preserve protocol-visible semantics: `HEAD`, `Stream-Up-To-Date: true`, `Stream-Closed: true`, and `offset=now` are generated by the leader path unless the follower has applied a terminal closed state. Requests that need a write-side access transition, such as expiry or TTL touch, are also routed to the leader. A follower can be behind an acknowledged append. If a catch-up cursor exceeds its local tail, it checks that cursor through a quorum-confirmed leader read instead of returning `416` from its stale prefix. A genuinely invalid cursor still returns `416`; without a confirmed leader, the request redirects or returns a retryable `503`. A catch-up client recovering from an acknowledged append can request `consistency=leader`. That opt-in routes the ordinary GET through the owning Raft leader instead of treating a follower-local tail as authoritative. The leader first confirms with a majority of voters that it still leads (Raft ReadIndex) and waits until it has applied everything committed up to that point, so the read is **linearizable**: it reflects every append acknowledged before the read started, including appends acknowledged by a newer leader. A leader cut off from the majority redirects to the new leader or answers `503` with `Retry-After` instead of serving its stale state. The confirmation costs one round trip to the voters; concurrent leader reads to the same Raft group share rounds, and a read that arrives while a round is in flight waits for the next one. On the leader, a read waits for its confirmation before it reaches the group, so appends and other reads to the group do not queue behind it. A leader that misses a confirmation under load retries it for up to one election timeout before it answers `503`; a read that arrived during a round can wait for that round and its own, so up to two. A client should retry a `503` after `Retry-After`. It is not a live-read mode; long-poll and SSE already have leader-owned watcher semantics. The [Read path](/docs/0.9/specs/extensions#92-leader-consistency) section of the extensions specification states this contract normatively. `HEAD`, `GET …/bootstrap`, snapshot reads and the registration of a long-poll or SSE reader take the same leadership confirmation on the leader, as do `offset=now` on a `consistency=leader` read. A long-poll or SSE reader takes everything else it needs from that registration: its `offset=now` position, whether the stream exists, its content type and incarnation all come from the leader's state at or after the confirmed point. A long-poll also answers `416` there for an offset beyond the tail; an SSE session reports it as an `error` event from its first read. Its first read is served only by that leader while it still leads; a node that lost leadership in between redirects or answers `503` instead of asking a new leader that may not have applied everything yet. A long-poll that times out answers `204` at the requested offset. `offset=now` without `consistency=leader` skips the confirmation and uses the leader's applied state, so right after a leader change it may resolve to a tail that lags the committed one. With `consistency=leader`, `Stream-Up-To-Date: true` on a read means it reached the committed tail. On a local read it means the tail as the serving node knew it: a leader that has lost leadership without knowing it yet can mark a stale tail. A response without it is partial; keep reading from `Stream-Next-Offset`. ## Availability and durability (standard layout) Three voting replicas across availability zones, plus two non-voting replicas in a second region. | Property | What it means | |---|---| | **Write availability** | Writes continue as long as a majority of voters is healthy. The layout tolerates any single voting-AZ failure. Non-voting replicas hold extra copies but do not vote. | | **Read availability** | Any reachable replica can serve replicated historical catch-up data locally. Reads that need fresh tail metadata, leader-owned state changes, or live watcher ownership still track write availability per group, not per cluster. | | **Per-message durability** | ~9-10 nines with a 5-second flush interval. | | **Annual zero-loss probability** | ~3-4 nines. Probability of no data-loss events in a year. | --- # Binary SSE The SSE wire format is text-only: every `data:` field must be valid UTF-8. Ursula therefore advertises the decoded data payload type with `Stream-Data-Content-Type` on every SSE read. For `text/*` streams, data events carry UTF-8 text directly. For `application/json` streams, data events carry newline-delimited JSON and the response includes `Stream-Data-Content-Type: application/x-ndjson`. SSE `data:` lines are transport framing, not JSON record boundaries, so clients should buffer until newline before parsing records. For other content types (`application/octet-stream`, custom binary), Ursula base64-encodes the data event payload as raw text. It is not wrapped in a JSON envelope: ```text event: data data: AQIDBAUG data: BwgJCg== event: control data: {"streamNextOffset":"123456_789","streamCursor":"abc"} ``` The choice is automatic, determined by the stream's content type. There is no client opt-in. What you get is what the stream's content type implied. The response advertises base64 data with `Stream-Sse-Data-Encoding: base64`. When that header is absent, the `data` event payload is already UTF-8 text with the type identified by `Stream-Data-Content-Type`. ## Decoding For binary streams, concatenate the `data:` lines for an `event: data`, remove line breaks inserted by SSE framing, then base64-decode the resulting text. Interpret the decoded bytes according to `Stream-Data-Content-Type`. For text and JSON streams, the `data` field is already the payload text. JSON streams use newline-delimited JSON (`application/x-ndjson`), so buffer until newline before parsing. Browser `EventSource` works for both modes. Only the `data` handler differs. ## Related - [Read modes](/docs/0.9/concepts/read-modes) - [API: read](/docs/0.9/api/read) - [Length-prefixed framing](/docs/0.9/concepts/len-prefixed-framing): when one SSE event contains multiple application records --- # Length-Prefixed Framing Streams are raw byte sequences. The protocol does not impose message boundaries. Every append and every read returns whatever bytes are in flight, possibly mid-record. For streams carrying many small messages (CRDT operations, agent events), Ursula recommends a simple framing convention: ``` [4-byte big-endian length][payload bytes] ``` Each application-level append writes one framed record. Each read returns a concatenation of framed records that the client parses incrementally. ## Server is frame-agnostic The server **does not** validate or enforce frames. It stores and serves raw bytes. That means: - A read may end mid-frame. The client must handle partial frames and resume parsing on the next read. - An ill-formed frame (wrong length, truncated payload) is the producer's bug, not a server-rejectable error. - The framing convention is per-application. Writers and readers of a stream must agree on it. ## When you don't need framing - Streams using `application/json` typically use newline-delimited JSON or a single self-describing document. JSON has its own delimiters. - Streams holding one logical blob per stream don't need framing at all. Use framing when one stream carries many independently meaningful messages and the client wants to dispatch on each one without scanning. ## Related - [Streams](/docs/0.9/concepts/streams) - [Binary SSE](/docs/0.9/concepts/binary-sse): over SSE, each event is one delivery. Framing lets one event carry multiple records. --- # API overview Ursula exposes a small public HTTP API for durable append-only streams. Most users only need the `/{bucket}/{stream}` route family, which maps cleanly to buckets and stream IDs. This API is Ursula's implementation of the [Durable Streams Protocol](/docs/0.9/specs/durable-stream), defined by the durable-streams community, with additional route families for compatibility and deployment needs. ## Bucket operations | Method | Path | Description | | -------- | ----------------------------- | ------------------------------------------------------------ | | `PUT` | `/{bucket}` | [Create a bucket](/docs/0.9/api/create-bucket) (validates the ID; buckets are implicit) | There is no bucket metadata read, stream listing, or bucket delete. Removing a bucket is an operator action ([bucket purge](/docs/0.9/operations#tenant-offboarding-bucket-purge)), which permanently fences the bucket ID: later stream creates in it answer `404`. ## Stream operations | Method | Path | Description | | -------- | ----------------------------- | ------------------------------------------------------------ | | `PUT` | `/{bucket}/{stream}` | [Create a stream](/docs/0.9/api/create-stream) | | `POST` | `/{bucket}/{stream}` | [Append data or close](/docs/0.9/api/append) | | `GET` | `/{bucket}/{stream}` | [Read (catch-up, long-poll, SSE)](/docs/0.9/api/read) | | `HEAD` | `/{bucket}/{stream}` | [Get stream metadata](/docs/0.9/api/head-stream) | | `DELETE` | `/{bucket}/{stream}` | [Delete a stream](/docs/0.9/api/delete-stream) | ## Bootstrap and snapshots | Method | Path | Description | | -------- | --------------------------------------------- | ---------------------------------------------------- | | `GET` | `/{bucket}/{stream}/bootstrap` | [Snapshot + tail replay](/docs/0.9/api/bootstrap) | | `GET` | `/{bucket}/{stream}/snapshot/{offset}` | [Read snapshot at offset](/docs/0.9/api/read-snapshot) | | `PUT` | `/{bucket}/{stream}/snapshot/{offset}` | [Publish a snapshot](/docs/0.9/api/publish-snapshot) | | `GET` | `/{bucket}/{stream}/incarnation/{token}/chunk/{offset}` | [Read a chunk a cache can store](/docs/0.9/api/immutable-reads) | | `GET` | `/{bucket}/{stream}/incarnation/{token}/snapshot/{offset}` | [Read a snapshot a cache can store](/docs/0.9/api/immutable-reads) | ## Common request patterns ### Create a bucket and stream ```bash curl -X PUT http://127.0.0.1:4437/demo curl -X PUT http://127.0.0.1:4437/demo/hello ``` ### Append data ```bash curl -X POST http://127.0.0.1:4437/demo/hello \ -H 'Content-Type: application/octet-stream' \ --data-binary 'hello world' ``` ### Read from the beginning ```bash curl 'http://127.0.0.1:4437/demo/hello?offset=-1' ``` ### Subscribe with SSE ```bash curl 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse' ``` ## Errors Every error answer from a node or from the gateway carries the same JSON body, the shape the Durable Streams protocol uses for its subscription API errors: ```json {"error":{"code":"STREAM_NOT_FOUND","message":"the stream does not exist"}} ``` `error.code` is a stable code that names the refusal, so match it instead of parsing text. `error.message` says why for a person reading it. A `4xx` that only names what the request named says what was wrong with it, for example `{"error":{"code":"INVALID_HEADER","message":"stream-ttl must not contain leading zeros"}}`. Every other answer carries a fixed sentence per code, which names nothing of the cluster. The wording can change in any release, so never match it. It never echoes the request body or a credential. Ignore members you do not know, and treat a code you do not know by its status alone: a later 0.8 release may add both. The status and the typed headers keep their meaning beside it: a `413` names its limit in `Ursula-Max-Request-Bytes` or `Ursula-Max-Snapshot-Bytes`, a `503` carries `Retry-After`, a producer `403` carries `Producer-Outcome`. A `503` whose code is `NO_LEADER` means no leader of the stream's group is known yet. A layer in front of Ursula, such as a proxy, may answer with another body, so treat the code as optional and fall back on the status. A live SSE read that ends in error sends the same body as the data of an `event: error`. [Error bodies](/docs/0.9/specs/extensions#11-error-bodies) lists every code. Rust clients parse the body with `ursula_headers::ErrorBody`. The operator routes under `/__ursula/` keep their own answers. ## Write outcomes A write whose answer is lost, such as one that timed out or got a `5xx`, may have applied. A refusal says that its own request applied nothing only by its status or its error code, as [write outcomes](/docs/0.9/specs/extensions#111-write-outcomes) defines: a `4xx` other than `408`, or a `408` or `5xx` whose code proves it, such as `NO_LEADER` before a proposal. `OUTCOME_UNKNOWN`, any other `5xx` and a lost answer leave the outcome unknown. Resolve an unknown outcome by repeating the write with its key: the same `Producer-Id`, `Producer-Epoch`, `Producer-Seq` and body, and the same `Producer-Digest` and `Stream-Incarnation` when the first attempt carried them. The answer to the repeat is definite. A `200` or `204` means the write applied. A producer `403` carries `Producer-Epoch` and a `Producer-Outcome` that says what became of the write's epoch. A `409`, or a `400` with `Producer-Expected-Seq`, names the seq the server expects next. See [write keys and typed answers](/docs/0.9/specs/extensions#5-write-keys-and-typed-answers) for every answer and [exactly-once writes](/docs/0.9/concepts/exactly-once-writes) for the producer model. A write without producer headers has no key. Repeating it can apply it twice. ### Redirects and retries A node that does not lead the stream's Raft group answers a write with a `307` to the leader, or `503` when it knows no leader yet, before it reads the body. Once it has read the body it never redirects: a node that lost the leadership meanwhile answers `503` with `Retry-After`. The gateway follows Raft leader redirects itself, so its clients never get one: up to three hops in a row, none started later than 5 s after the first redirect, and through the upstreams it has not tried yet when a redirect names an address that is none of them, or one it already went to. When no upstream answers without redirecting within those limits, the gateway answers `503` with `Retry-After: 1`. Each hop sends the request again. A node redirects only a write it has not read, so a redirect never follows a proposal. A `503` can follow one, so repeat a write whose outcome is unknown with its producer key: the server answers a repeat of an applied write as a duplicate. A plain append without a producer can apply twice when it is repeated. Use a producer for exactly-once writes. For the same reason, a proxy or service mesh must not resend writes on its own. A resent write without producer headers applies twice. With ingress-nginx, set `nginx.ingress.kubernetes.io/proxy-next-upstream: "off"`, as the chart's [TLS ingress example](https://github.com/tonbo-io/ursula/blob/main/charts/ursula/examples/tls-ingress.yaml) does. In a service mesh, turn off retries for the stream routes on both sides of the gateway. Istio before 1.24, for example, resends a request up to twice by default when it gets a `503`, and any version resends requests on a route whose `retryOn` includes `503`, `5xx`, `gateway-error` or `reset`. Istio 1.24 to 1.26 restores the default `503` retry when istiod runs with `EXCLUDE_UNSAFE_503_FROM_DEFAULT_RETRY=false`, which `compatibilityVersion=1.23` sets. Set `retries: {attempts: 0}` on those routes in their `VirtualService`. ## Related concepts - [Read modes](/docs/0.9/concepts/read-modes): catch-up vs long-poll vs SSE - [Bootstrap](/docs/0.9/concepts/bootstrap): snapshot + tail recovery - [Snapshots](/docs/0.9/concepts/snapshots): snapshot lifecycle - [Exactly-once writes](/docs/0.9/concepts/exactly-once-writes): producer deduplication - [Conditional writes](/docs/0.9/concepts/conditional-writes): `Stream-Seq` as a compare-and-set --- # Create bucket Bucket ID. Must match `[a-z0-9_-]{4,64}`. ## Response | Status | Meaning | | ------ | -------------------------------------- | | `201` | The bucket ID is valid. Returned on every call, whether or not streams already exist in the bucket. | | `400` | The bucket ID is invalid. | > [!NOTE] > `PUT /{bucket}` validates the bucket ID and changes no state. Buckets are implicit: a bucket exists once a stream is created in it, so creating the bucket first is optional. It is kept for clients that do so. ```bash Example curl -X PUT http://127.0.0.1:4437/demo ``` --- # Create stream Bucket ID. Must match `[a-z0-9_-]{4,64}`; an invalid ID answers `400`. Stream ID within the bucket: 1 to 122 bytes, without `/`, `\0` or `..`, not starting with `$`, and `{bucket}/{stream}` together at most 122 bytes. `streams` is reserved. An invalid ID answers `400`. Content type of the initial payload (e.g. `application/json`). Becomes the stream's content type. Set to `true` to close the stream immediately after creation. Sliding time-to-live in seconds. Appends and closes restart the window; reads renew it at most once per quarter of the TTL. See [stream lifecycle](/docs/0.9/concepts/streams#lifecycle). Absolute expiration timestamp (RFC 3339). Mutually exclusive with `Stream-TTL`. Seeds the stream's stored `Stream-Seq`: later appends that carry `Stream-Seq` must be bytewise greater than it. A create never checks it, and a `PUT` that finds the stream already there ignores it. See [how `Stream-Seq` is checked](/docs/0.9/concepts/conditional-writes#how-stream-seq-is-checked). Optional precondition: the stream's incarnation token, as a previous response's `Stream-Incarnation` header returned it. The create applies only if the stream exists as that incarnation (it then answers as a create of an existing stream: `200` or `409`); a missing stream answers `412` and is not created. Without the header, the request behaves exactly as described here. A malformed value answers `400`. See [`Stream-Incarnation`](/docs/0.9/specs/extensions#24-snapshot-metadata-on-head-and-stream-incarnation). Optional initial payload. If provided, becomes the first entry in the stream. ## Response | Status | Meaning | | ------ | ------------------------------------------------------------------------- | | `201` | Stream created. | | `200` | Stream already exists (idempotent). | | `400` | Invalid bucket ID, invalid stream ID (including one that starts with `$`), invalid headers, producer headers (create the stream, then claim an epoch with an [append](/docs/0.9/api/append)), or bad JSON payload. | | `404` | The bucket was purged; its ID is permanently fenced. | | `409` | Stream already exists with a different content type, closed state or lifetime. | | `412` | `Stream-Incarnation` did not match: the stream is another incarnation (its token is in the response's `Stream-Incarnation`), or it does not exist. Nothing was created. | Response headers include `Location`, `Content-Type`, `Stream-Next-Offset`, `Stream-Incarnation`, and lifetime headers (`Stream-TTL` / `Stream-Expires-At`) when set. `Stream-Closed: true` is set if the create request also closed the stream. `ETag` is set on reads only. In an `application/json` stream, a top-level JSON array in the initial payload creates one message per element. ```bash Create empty stream curl -X PUT http://127.0.0.1:4437/demo/hello ``` ```bash Create with initial payload curl -X PUT http://127.0.0.1:4437/demo/hello \ -H 'Content-Type: application/json' \ --data-binary '{"msg": "first entry"}' ``` ```bash Create with TTL curl -X PUT http://127.0.0.1:4437/demo/ephemeral \ -H 'Stream-TTL: 3600' ``` > [!NOTE] > If a stream with the same ID already exists and has identical configuration, the response is `200 OK` (idempotent). If the existing stream differs in content type, closed state, TTL or expiry, the response is `409 Conflict`. A `PUT` answered `200` ignores its body and its `Stream-Seq`. --- # Append Bucket ID. Stream ID within the bucket. Must match the stream's content type (set when the stream was created with `PUT`) after normalization: parameters are trimmed, lowercased and joined with `; `. Required when the body is non-empty. A mismatch returns `409`. Set to `true` to close the stream after this append. A compare-and-set token. The append is accepted only when the value is bytewise greater than the last `Stream-Seq` accepted on this stream; it is then stored. An equal or smaller value answers `409 STREAM_SEQ_CONFLICT` with `Stream-Next-Offset` at the tail and `Stream-Last-Seq`, the stream's last accepted value. Without the header the append is not checked and the stored value is unchanged. Printable ASCII, at most 256 bytes. See [how `Stream-Seq` is checked](/docs/0.9/concepts/conditional-writes#how-stream-seq-is-checked). Stable producer identity (UUID, hostname, etc.) for [exactly-once writes](/docs/0.9/concepts/exactly-once-writes). Dedup state is per-stream. At most 256 bytes. Producer epoch. Bumped on producer restart. Must be ≥ the last epoch the server accepted for this `Producer-Id`. A new epoch resets the seq counter. Max value `2^53 − 1`. Producer sequence number. Starts at `0` for a new epoch and must increase by exactly `1` per append. Exact `(epoch, seq)` duplicates are silently deduplicated. Max value `2^53 − 1`. Optional, with producer headers: the BLAKE3-256 hash of the body as 64 lowercase hex digits. The server checks it against the body (`400` on a mismatch, nothing appended) and makes it part of the write's identity, so a different body under an applied key answers `403` with `Producer-Outcome: key-conflict`. See [write keys](/docs/0.9/specs/extensions#5-write-keys-and-typed-answers). Optional precondition: the stream's incarnation token, as a previous response's `Stream-Incarnation` header returned it. The request applies only if the stream is still that incarnation, checked atomically with the write; otherwise it answers `412` and changes nothing. Without the header, the request behaves exactly as described here. A malformed value answers `400`. See [`Stream-Incarnation`](/docs/0.9/specs/extensions#24-snapshot-metadata-on-head-and-stream-incarnation). The bytes to append. Must not be empty unless `Stream-Closed: true` is set (close-only request). ## Response | Status | Meaning | | ------ | ------------------------------------------------------------------ | | `204` | Append successful (default success response, no body). | | `200` | Append successful with body - returned when a `Producer-Id` was supplied and the append was not deduplicated, so the response carries producer ack headers. | | `307` | In a cluster, this node does not lead or host the stream's group. `Location` is the same request on the leader, or on a node that hosts the group. The node answers before it reads the body, and a redirect to the leader carries `x-ursula-raft-leader-id`, `x-ursula-raft-group` and `x-ursula-raft-term`. A redirect to a node that hosts the group carries `x-ursula-raft-group` alone. See [routing answers](/docs/0.9/specs/extensions#57-routing-answers). | | `400` | Empty body without `Stream-Closed: true`, missing content type, invalid JSON or UTF-8, a JSON message nested deeper than 127 levels, a `Producer-Id` or `Stream-Seq` longer than 256 bytes, a `Producer-Digest` that does not match the body, a `Producer-Seq` above `0` of a later epoch that is not claimed yet (with `Producer-Expected-Seq: 0`), or a header removed in 0.6.0 (see the [release notes](/docs/0.9/release-notes)). | | `403` | A producer refusal. The response carries `Producer-Epoch` and one `Producer-Outcome`: `epoch-closed;by=claim` or `stale-epoch` for a write of an older epoch, `key-conflict` for another body under an applied key, or `producer-expired` for a write with `Stream-Incarnation` that could reach an epoch the stream forgot (at or below its epoch floor). See [producer write answers](/docs/0.9/specs/extensions#53-producer-write-answers). | | `404` | Stream not found. | | `409` | Content type does not match the stream (`CONTENT_TYPE_MISMATCH`), stream is already closed (`STREAM_CLOSED`, with `Stream-Closed`, also for a close with producer headers that is not a repeat of the producer's own close), `Stream-Seq` not greater than the stored value (`STREAM_SEQ_CONFLICT`, with `Stream-Last-Seq` naming the stored value), or `Producer-Seq` out of order (`PRODUCER_SEQ_CONFLICT`, with `Producer-Expected-Seq`). | | `412` | `Stream-Incarnation` did not match: the stream was deleted and recreated. Nothing was appended; the response's `Stream-Incarnation` is the current token. | | `413` | Body larger than 32 MiB. `Ursula-Max-Request-Bytes` names the limit, and a gateway with a smaller limit names its own. | | `503` | Write admission refused the append (`WAL_DISK_PRESSURE`, `RAFT_LOG_PRESSURE` or `INGRESS_BODY_BYTES_LIMIT_REACHED`), or the stream's Raft group refused it with cold-write or uncommitted-bytes backpressure (`COLD_BACKPRESSURE` or `RAFT_UNCOMMITTED_BACKPRESSURE`). A node also answers `503 NoLeader` when it knows no leader for the group yet (with `x-ursula-raft-group`), when it knows no node that hosts the group, and when it lost the group's leadership while it read the body. Retry after the duration in `Retry-After`. | Every error answer carries a JSON body with its code and a human-readable message, such as `{"error":{"code":"STREAM_CLOSED","message":"..."}}`. See [errors](/docs/0.9/api/overview#errors). Any answer other than a `2xx` or a producer refusal of [extensions Section 5.3](/docs/0.9/specs/extensions#53-producer-write-answers) leaves the outcome unknown: repeat the same write. See [write outcomes](/docs/0.9/api/overview#write-outcomes). Response headers include `Stream-Incarnation` and `Stream-Next-Offset`, except on a repeat of a producer's older write, which answers `204` with `Producer-Outcome: receipt-evicted` and no range ([exactly-once writes](/docs/0.9/concepts/exactly-once-writes#how-dedup-works)). When a `Producer-Id` was supplied, the response carries `Producer-Epoch`, `Producer-Seq` (the epoch's highest applied seq) and `Stream-Offset`, where the write's bytes start, so the producer can confirm what was durably recorded. A claim that closed an earlier epoch also carries `Producer-Closed-Epoch` and `Producer-Closed-Seq`. `Stream-Closed: true` is set if this request closed the stream. `ETag` is set on reads only, not on appends. ```bash Append binary curl -X POST http://127.0.0.1:4437/demo/hello \ -H 'Content-Type: application/octet-stream' \ --data-binary 'hello world' ``` ```bash Append JSON curl -i -X POST http://127.0.0.1:4437/demo/hello \ -H 'Content-Type: application/json' \ --data-binary '{"captured_at":"2026-07-18T10:30:00.100Z","event":"click"}' ``` ```bash Append multiple JSON messages curl -i -X POST http://127.0.0.1:4437/demo/hello \ -H 'Content-Type: application/json' \ --data-binary '[{"event":"click"},{"event":"navigation"}]' ``` ```bash Close a stream curl -X POST http://127.0.0.1:4437/demo/hello \ -H 'Stream-Closed: true' ``` ## JSON message text Ursula validates every JSON body and stores each message as you wrote it, minus insignificant whitespace (spaces, tabs and line breaks outside strings), followed by one LF. Member order, duplicate members, number spelling such as `1.50e3` or `1e400`, and string escapes, including escapes of unpaired surrogates, are kept byte for byte, and every read returns that text. A top-level array is split into one message per element. A message may nest at most 127 levels. See [JSON message text](/docs/0.9/specs/extensions#json-message-text). Releases before this rule re-serialized messages; see the [release notes](/docs/0.9/release-notes). To append only if no other write has committed since you last read the stream, see [conditional writes](/docs/0.9/concepts/conditional-writes). --- # Read stream Bucket ID. Stream ID within the bucket. Starting offset: `-1` for the beginning, `now` for the current tail, or a `Stream-Next-Offset` value from an earlier response. On a long-poll or SSE read, the `Stream-Cursor` value from the previous live response. It lets caches tell successive live requests apart. It is not a position and never replaces `offset`. `local` (default) or `leader`, on catch-up reads only. `leader` makes the read linearizable: it includes every append acknowledged before the request, at the cost of one round trip from the group's leader to a majority of voters. Use it to read back your own acknowledged writes. Combining it with `live` answers `400`. See [Read path](/docs/0.9/specs/extensions#92-leader-consistency). Live mode: `sse` for Server-Sent Events, `long-poll` for long-polling. Omit for catch-up read. On an offset read, the maximum bytes to read in one response or SSE data batch. Every response is capped at 8 MiB; a larger value is clamped, and a value that is not a decimal integer is ignored. A shortened response has no `Stream-Up-To-Date` and may end inside a message, on a JSON stream too; continue from `Stream-Next-Offset`. For UTF-8 SSE data, Ursula may shorten the emitted batch so `Stream-Next-Offset` lands on a valid text boundary. ## Read modes No `live` parameter. Returns all available data from the given offset immediately. ```bash curl 'http://127.0.0.1:4437/demo/hello?offset=-1' ``` `live=long-poll`. Returns immediately if data is available, otherwise holds the connection until new data arrives or `timeout_ms` elapses (default 1000, at most 60000). ```bash curl 'http://127.0.0.1:4437/demo/hello?offset=00000000000000000042&live=long-poll' ``` `live=sse`. Opens a persistent Server-Sent Events connection. The server pushes data events as new entries are appended. It sends no heartbeats, so a proxy with an idle timeout may close a quiet connection; reconnect from the last control event's `streamNextOffset`. ```bash curl 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse' ``` Optional precondition: the stream's incarnation token, as a previous response's `Stream-Incarnation` header returned it. The read is answered only if the stream that serves it is still that incarnation; otherwise it answers `412`. A long-poll or SSE read with it ends when the stream it was opened against is deleted or recreated (a long-poll answers `404` or `412`; an SSE stream ends with an `error` event). Without the header, a long-poll still ends when the stream it registered against is recreated, usually with `416`; otherwise the request behaves exactly as described here. A malformed value answers `400`. See [`Stream-Incarnation`](/docs/0.9/specs/extensions#24-snapshot-metadata-on-head-and-stream-incarnation). ## Response | Status | Meaning | | ------ | ------------------------------------------------------------- | | `200` | Data returned. A catch-up read at the tail answers `200` with an empty body and `Stream-Up-To-Date: true`. | | `204` | Long-poll only: `timeout_ms` passed without new data, or the stream is closed and the read is at its tail (answered at once with `Stream-Closed: true`, which signals end of stream). | | `304` | `If-None-Match` matched the response's `ETag`, compared weakly. A read the server may gzip (JSON or NDJSON, at least 256 bytes) has a weak tag (`W/"…"`). | | `307` | In a cluster, this node could not serve the read as the stream's group leader (a long-poll or SSE read on a follower, a `consistency=leader` read on a node that lost leadership, or a read that would answer `404`, `410` or `416` on a leader that cannot confirm its leadership). `Location` is the same request on the leader. | | `400` | Invalid offset, live mode or `consistency`, `consistency=leader` with `live`, a read parameter removed in 0.6.0 (see the [release notes](/docs/0.9/release-notes)), or `incarnation` or `aligned`, which [immutable reads](/docs/0.9/api/immutable-reads) carry in the path instead. | | `404` | Stream not found or expired. | | `410` | Requested offset has been trimmed. `Stream-Next-Offset` is the earliest offset still readable, and `Stream-Incarnation` the token of the stream that answered. | | `412` | `Stream-Incarnation` did not match: the stream was deleted and recreated. The response's `Stream-Incarnation` is the current token. A read that would answer `404`, `410` or `416` without the header still does. | | `416` | Requested offset is beyond the stream's tail, on a catch-up read or a long-poll. `Stream-Next-Offset` is the tail, and `Stream-Incarnation` the token of the stream that answered. A `410` or `416` comes before the `Stream-Incarnation` precondition, so compare its token with yours: another token means the stream was replaced, not that your offset is wrong. | | `503` | The group has no known leader, the leader could not confirm its leadership in time (`consistency=leader` and live reads), or this node holds too many live readers. Retry after `Retry-After`. | Response headers include `Stream-Next-Offset`, `Stream-Incarnation`, `ETag`, `Stream-Up-To-Date`, `Stream-Closed`, and `Content-Type`; the SSE response and the long-poll `204` carry `Stream-Incarnation` too. Long-poll responses for an open stream also carry `Stream-Cursor`. Streams written with `Content-Type: application/json` are returned as newline-delimited JSON with response `Content-Type: application/x-ndjson`, one stored message per line, exactly as written minus insignificant whitespace ([JSON message text](/docs/0.9/specs/extensions#json-message-text)). On an offset read, `max_bytes` applies to the encoded byte stream, so a response can end mid-line. Resume from `Stream-Next-Offset` and buffer any incomplete trailing line before parsing. `HEAD` reports the configured stream content type, while read responses report the wire representation. ## SSE event format In SSE mode, the server sends: - **Data events** (`event: data`): stream payload in the `data` field. The response `Stream-Data-Content-Type` header identifies the data payload type (`application/x-ndjson` for JSON streams, the original content type for other streams). SSE `data:` lines are transport lines, not guaranteed message boundaries. JSON clients should buffer until newline before parsing messages. For binary streams, data is base64-encoded (controlled by the `Stream-Sse-Data-Encoding` header). - **Control events** (`event: control`): JSON metadata including the current offset and stream state. ```bash Catch-up curl 'http://127.0.0.1:4437/demo/hello?offset=-1' ``` ```bash Long-poll curl 'http://127.0.0.1:4437/demo/hello?offset=00000000000000000042&live=long-poll' ``` ```bash SSE tail curl 'http://127.0.0.1:4437/demo/hello?offset=-1&live=sse' ``` ```bash Tail future appends over SSE curl -N 'http://127.0.0.1:4437/demo/hello?offset=now&live=sse' ``` > [!NOTE] > See [read modes](/docs/0.9/concepts/read-modes), [binary SSE](/docs/0.9/concepts/binary-sse), and [offsets](/docs/0.9/concepts/offsets) for more details. --- # Head stream Bucket ID. Stream ID within the bucket. Optional precondition: the stream's incarnation token, as a previous response's `Stream-Incarnation` header returned it. The `HEAD` answers `200` only if the stream is still that incarnation; otherwise `412`, with the current token in `Stream-Incarnation`. Without the header, the request behaves exactly as described here. A malformed value answers `400`. See [`Stream-Incarnation`](/docs/0.9/specs/extensions#24-snapshot-metadata-on-head-and-stream-incarnation). ## Response | Status | Meaning | | ------ | ------------------------------------------------------ | | `200` | Stream found. | | `307` | In a cluster, this node could not answer as the stream's group leader. `Location` is the same request on the leader. | | `404` | Stream not found or expired. | | `412` | `Stream-Incarnation` did not match. The response's `Stream-Incarnation` is the current token. | | `503` | The group has no known leader, or the leader could not confirm its leadership in time. Retry after `Retry-After`. | `HEAD` is linearizable: the leader confirms with a majority of voters that it still leads before it answers, so the response reflects every append acknowledged before the request started. See [Durability and consistency](/docs/0.9/concepts/durability-and-consistency#consistency). Response headers include: | Header | Description | | ------------------------ | --------------------------------------------------------- | | `Content-Type` | The stream's content type. | | `Stream-Next-Offset` | The stream's tail: the offset the next append starts at. | | `Stream-Closed` | Present and `true` if the stream is closed. | | `Stream-Retained-Offset` | The earliest offset still readable. Reads below it answer `410 Gone`. Before any [retention](/docs/0.9/concepts/snapshots) advance, it is the start of the stream. | | `Stream-Snapshot-Offset` | The offset of the latest snapshot. Absent when the stream has no snapshot. Read the snapshot with [`GET …/snapshot/{offset}`](/docs/0.9/api/read-snapshot). | | `Stream-Snapshot-Digest` | The digest of the latest snapshot's content type and body, as its publish returned it. Present with `Stream-Snapshot-Offset`. | | `Stream-Incarnation` | An opaque token for this incarnation of the stream. It changes when the stream is deleted and recreated at the same URL; compare it only for equality. | | `Ursula-Protocol` | `2`. Present on every answer to a `HEAD`, `404`, `307` and `503` included. See [Server metadata](/docs/0.9/specs/extensions#10-server-metadata). | | `Ursula-Cluster-Id` | The deployment that served the request, as `server.cluster_id` names it. Present on every answer to a `HEAD`. A different id means another deployment, not a new history of the stream. | | `Ursula-Max-Request-Bytes` | The largest request body any request to the stream may carry: 33554432 (32 MiB) from a node, lowered by a gateway with a smaller limit. A larger request answers `413`. A larger snapshot goes up [in parts](/docs/0.9/api/publish-snapshot#upload-in-parts). See [Server metadata](/docs/0.9/specs/extensions#10-server-metadata). | | `Ursula-Max-Snapshot-Bytes` | The largest snapshot the node accepts in parts: 1073741824 (1 GiB), in at most 4,096 parts. Present only on a node with a cold backend. A gateway passes it unchanged: each part is a request under its own limit. | | `Stream-TTL` | The configured TTL in seconds (if a TTL was set). This is the sliding window length, not the time remaining. `HEAD` does not renew it. | | `Stream-Expires-At` | Expiration timestamp (if set). | | `Cache-Control` | `no-store`. | | `Stream-Cold-Hot-Start-Offset` | Diagnostic and unstable: may change or disappear in any release. Data below this offset has been flushed to cold storage. | ```bash Example curl -I http://127.0.0.1:4437/demo/hello ``` --- # Publish snapshot Publishes a new checkpoint blob at the given offset. The snapshot replaces any previously published snapshot, but does not delete stream history. Retention advances only through the separate `/retention` endpoint. Bucket ID. Stream ID. Stream offset this snapshot represents: the 20-character zero-padded decimal token. For an `application/json` stream the offset must be a message boundary: a `Stream-Next-Offset` that Ursula returned, or any offset just after a message's LF. Content type of the snapshot blob, stored separately from the stream's own content type. Defaults to `application/octet-stream`. The snapshot blob bytes, at most `Ursula-Max-Request-Bytes` (**32 MiB** from a node). Bodies are stored inline in the stream's replicated state. On a node with a cold backend, a body of at least `runtime.external_payload_min_size` (1 MiB by default) is streamed into the cold tier instead. A larger snapshot goes up [in parts](#upload-in-parts). Optional precondition: the stream's incarnation token, as a previous response's `Stream-Incarnation` header returned it. The request applies only if the stream is still that incarnation, checked atomically with the write; otherwise it answers `412` and changes nothing. Without the header, the request behaves exactly as described here. A malformed value answers `400`. See [`Stream-Incarnation`](/docs/0.9/specs/extensions#24-snapshot-metadata-on-head-and-stream-incarnation). ## Response | Status | Meaning | | ------ | --------------------------------------------------------------- | | `204` | Snapshot published successfully. | | `400` | Invalid offset or content type, or, on a JSON stream, an offset inside a message. | | `404` | Stream not found or expired. | | `409` | Another snapshot is the latest: a newer one (this publish is superseded) or a different one at the same offset. The response names it in `Stream-Snapshot-Offset` and `Stream-Snapshot-Digest`. A `409` without them means the offset is beyond the stream's tail. | | `410` | Snapshot offset is older than the retained stream history. | | `412` | `Stream-Incarnation` did not match: the stream was deleted and recreated. Nothing was published; the response's `Stream-Incarnation` is the current token. | | `413` | The body is above `Ursula-Max-Request-Bytes`, which the response names. For a commit of parts, the assembled snapshot is above `Ursula-Max-Snapshot-Bytes`, which the response names instead. | | `501` | A commit of parts on a node without a cold backend. | | `502` | The cold backend failed while storing the body or copying the parts. | | `503` | Retry after `Retry-After`. Nothing was published. The error body names the cause: `SNAPSHOT_COMMITS_BUSY` (every copy slot taken, or no room in the in-flight budget beside a request), `SNAPSHOT_COMMIT_IN_PROGRESS` (this commit runs now), `INGRESS_BODY_BYTES_LIMIT_REACHED`, `WAL_DISK_PRESSURE`, `RAFT_LOG_PRESSURE`, `NO_LEADER` (no leader is known), or `JSON_BOUNDARY_UNVERIFIED` (on a JSON stream, Ursula could not read the byte before the offset to check the boundary). | Every error answer carries a JSON body with its code and a human-readable message, such as `{"error":{"code":"SNAPSHOT_CONFLICT","message":"..."}}`. See [errors](/docs/0.9/api/overview#errors). A `204` carries `Stream-Snapshot-Offset`, `Stream-Snapshot-Digest`, and `Stream-Incarnation`. A publish is keyed by its offset and digest. Repeating the same offset, content type, and body returns the same digest while that snapshot is the latest. Reusing an offset with different content returns `409`. ```bash Example curl -X PUT 'http://127.0.0.1:4437/demo/hello/snapshot/00000000000000000042' \ -H 'Content-Type: application/json' \ --data-binary '{"state": "aggregated snapshot data"}' ``` ## Upload in parts On a node with a cold backend, a snapshot above `Ursula-Max-Request-Bytes` goes up in parts, up to `Ursula-Max-Snapshot-Bytes` (**1 GiB**) in at most 4,096 parts. `HEAD` advertises both limits. 1. Stage each part with `POST /{bucket}/{stream}/snapshot/{offset}/parts`. Its body is the part's bytes, at most `Ursula-Max-Request-Bytes`. The answer is `201` with the part's id in `Stream-Snapshot-Part`. Staging changes nothing a reader sees, and repeating it stages another part. The stream must exist (`404` otherwise), and the request may carry `Stream-Incarnation`. A part belongs to the stream's current incarnation, not to the offset in its path, which is checked and otherwise ignored. A part is charged against the node's in-flight body budget but is not refused for WAL or Raft-log pressure. A stream holds at most 8,192 staged parts that no commit used, and at most 2 GiB of them (see [staged parts](#staged-parts)). 2. Commit with `PUT /{bucket}/{stream}/snapshot/{offset}`, `Stream-Snapshot-Parts: ` and `Stream-Snapshot-Digest: `. The body lists the part ids in order, each followed by a line feed, no id twice. `Content-Type` is the snapshot's. A commit checks its key first, before it reads its body. A repeat of a published snapshot answers `204` with its digest, and a commit whose offset holds another snapshot, or that a later snapshot supersedes, answers `409` naming the latest, both even after the parts are gone. Otherwise Ursula copies the parts into one snapshot, checks its digest, and publishes it like any other snapshot, with the same answers. It answers `400` when a part is unknown, belongs to another stream or to an earlier incarnation of this one, or is listed twice, when the manifest does not match `Stream-Snapshot-Parts`, or when the digest differs. Whether Ursula deletes the parts depends on the answer: - After the publish's answer, when it is final for the commit's key (`204`, `400`, `404`, `409`, `410` or `412`), the parts are deleted, after the answer has gone out. - After a `400` for a digest that differs, the parts are deleted too: the copy that found the mismatch used them up. Upload them again. - After a `400` that copies nothing (an unknown part, another stream's part, a part listed twice, a malformed or oversized manifest), the parts stay, so a corrected commit can use them. - After a `503` or any other retryable answer, the parts stay. Repeat the same commit. The orphan sweep reclaims a part that no commit used once it is older than its grace, a day by default, whether its stream still exists or not, so commit within a day of staging the first part. Without a cold backend, parts, commits and discards answer `501`. ### Staged parts A stream holds at most 8,192 staged parts that no commit used, and at most 2 GiB of them. That is twice what one commit can use, so an abandoned upload and a full retry both fit. The parts of the stream's earlier incarnations count until they are deleted, so deleting and recreating a stream does not reset the bound. A part past either bound answers `429 Too Many Requests` before its body is read, with the error code `STAGED_PARTS_LIMIT`, the stream's `Stream-Incarnation` and the bound it hit: `Ursula-Max-Staged-Parts: 8192` or `Ursula-Max-Staged-Bytes: 2147483648`. The answer has no `Retry-After`, because a short wait does not free anything. A commit frees its parts once it has deleted them, a discard frees them at once, and the orphan sweep frees the parts it reclaims after its grace. Commit or discard the parts you staged before you stage another snapshot. A part counts its `Content-Length` while it uploads, and 32 MiB when it has none, so send `Content-Length` near the byte bound. Discard a stream's staged parts with `DELETE /{bucket}/{stream}/snapshot/parts`. It deletes every staged part of the stream's current incarnation, and the parts of its earlier incarnations staged more than a minute before it, and answers `204` with `Stream-Incarnation`. It may carry `Stream-Incarnation` (`412` when it does not match), answers `404` for a stream that does not exist, `501` without a cold backend and `502` with `COLD_STORE_FAILED` when the cold backend fails to list or delete the parts, and is answered by the stream's leader like a write. A part whose upload ends during a discard may or may not survive it. While a commit of the stream runs, a discard answers `503` with `Retry-After` and the error code `SNAPSHOT_COMMIT_IN_PROGRESS`. Through `ursulagw` a discard is authorized as a snapshot publish. Deleting a stream deletes its staged parts after the delete has answered, in a task the node owns: the parts of the incarnations the stream's leader knew of at any age, and the stream's other parts staged more than a minute before the delete. A recreated stream's next listing deletes the parts of its earlier incarnations staged more than a minute before it. These margins assume the nodes' clocks agree within a minute. When the delete of a commit's spent parts fails, they stay until a discard, a delete of the stream or the orphan sweep removes them. The stream's leader keeps the count in memory. It lists the stream's parts at the stream's first part after it starts to lead or after the orphan sweep reclaimed some of them, after every commit and discard, and at a refused part when its last listing is over a minute old. A new leader does not see the parts that were still uploading to the old one when it took over, so the stream can hold more than the bound by those parts until a commit, a discard, a delete or the sweep removes them. | Status | Meaning (part upload) | | ------ | --------------------- | | `201` | The part is staged. `Stream-Snapshot-Part` is its id. | | `404` | Stream not found or expired. | | `412` | `Stream-Incarnation` did not match. | | `413` | The part is above `Ursula-Max-Request-Bytes`, which the response names. | | `429` | The stream holds as many staged parts, or bytes of them, as it may. The response names the bound and the stream's `Stream-Incarnation`. | | `501` | The node has no cold backend. | | `502` | The cold backend failed while storing the part or listing the stream's parts. | A stream deleted and created again while its commit copies is not published to. A commit that sent `Stream-Incarnation` then answers `412`. One that did not answers `400` for an unknown part, as a commit sent after the new create does. A node copies at most two commits at once, and starts a copy only while a request of the full 32 MiB limit still fits in `server.http_inflight_body_size` beside it. Another commit answers `503` with `Retry-After` and the error code `SNAPSHOT_COMMITS_BUSY`. A copy is charged against the budget while it runs: 56 MiB at the default 256 MiB, 32 MiB on a node of 64 MiB, which reads fewer pieces ahead and so copies one commit at a time. > [!NOTE] > The node owns a commit once it starts copying. A client that goes away, or a gateway that times out, cancels none of it: the copy, the publish and the deletion of the parts run to their end. A repeat while the commit runs, and any other commit of the same stream incarnation, answers `503` with `Retry-After` and the error code `SNAPSHOT_COMMIT_IN_PROGRESS`, and a repeat after it answers `204` from the key. A copy takes time in proportion to the snapshot's size and the number of parts. On S3, 256 MiB in eight 32 MiB parts took about 3 seconds. Through `ursulagw`, whose `--response-header-timeout` is 30 seconds by default, a commit of many small parts can outlast the timeout. Repeat it until it answers, or send larger parts. ```bash Example curl -X POST 'http://127.0.0.1:4437/demo/hello/snapshot/00000000000000000042/parts' \ --data-binary @part-1.bin # 201, Stream-Snapshot-Part: 0000000000000000186f2b3a1c0d9e00-0000000000000007.part printf '%s\n%s\n' "$PART_1" "$PART_2" | curl -X PUT \ 'http://127.0.0.1:4437/demo/hello/snapshot/00000000000000000042' \ -H 'Content-Type: application/octet-stream' \ -H 'Stream-Snapshot-Parts: 2' \ -H "Stream-Snapshot-Digest: $DIGEST" \ --data-binary @- ``` ## Advance retention `PUT /{bucket}/{stream}/retention/{offset}` explicitly discards history before a published checkpoint. The boundary must be no newer than the visible checkpoint and, on a JSON stream, a message boundary (`400` otherwise, `503` when it cannot be checked). Retention only moves forward: a boundary at or below the current one succeeds, changes nothing and reports the current boundary, so a repeated request is safe. Otherwise retention is exact: the response's `Stream-Retained-Offset` is the requested offset. After a successful `204`, reads before the retained offset return `410 Gone`. This separate step lets clients publish and verify a checkpoint before making history unreachable. > [!NOTE] > Snapshots are immutable at a given offset. A byte-identical retry is idempotent; a different body at the same offset conflicts. Publishing does not advance retention. --- # Read snapshot `GET /{bucket}/{stream}/snapshot/{offset}` returns the snapshot blob at a specific offset. To read the latest snapshot, take its offset from the `Stream-Snapshot-Offset` header of [`HEAD`](/docs/0.9/api/head-stream). Bucket ID. Stream ID. Snapshot offset: the 20-character zero-padded decimal token returned by previous reads or `Stream-Snapshot-Offset` headers. Optional precondition: the stream's incarnation token, as a previous response's `Stream-Incarnation` header returned it. The read is answered only if the stream that serves it is still that incarnation; otherwise it answers `412`. Without the header, the request behaves exactly as described here. A malformed value answers `400`. See [`Stream-Incarnation`](/docs/0.9/specs/extensions#24-snapshot-metadata-on-head-and-stream-incarnation). Optional: one byte range of the snapshot, `bytes=a-b`, `bytes=a-` or `bytes=-n`. The answer is `206` with only those bytes and `Content-Range: bytes a-b/total`. Several ranges, a value that does not parse, or a request that also carries `If-Range` get the whole snapshot as `200`. A body in the cold tier is read only where the range needs it. | Status | Meaning | | ------ | -------------------------------------- | | `200` | Snapshot blob returned. | | `206` | The requested byte range returned. | | `304` | `If-None-Match` named the snapshot's `ETag`. The headers are those of the `200`. | | `307` | In a cluster, this node is not the stream's group leader. `Location` is the same request on the leader. | | `400` | The request carries `incarnation`, which [immutable reads](/docs/0.9/api/immutable-reads) carry in the path instead. | | `404` | Stream, snapshot, or offset not found. When the stream exists and the snapshot is not its latest, `Stream-Incarnation` is the stream's token. | | `412` | `Stream-Incarnation` did not match. The response's `Stream-Incarnation` is the current token. | | `416` | The range starts at or past the end of the snapshot. `Content-Range: bytes */total` names its size, and `Stream-Incarnation` the stream's token. | | `502` | The snapshot body is in the cold tier and could not be read. | | `503` | The group has no known leader, or the leader could not confirm its leadership in time. Retry after `Retry-After`. | Response headers include `Content-Type`, `Content-Length`, `Accept-Ranges: bytes`, `ETag` (the snapshot digest in quotes), `Stream-Next-Offset`, `Stream-Snapshot-Offset`, `Stream-Snapshot-Digest`, `Stream-Incarnation`, and `Stream-Up-To-Date: true` when the snapshot offset is the stream's tail. They describe the whole snapshot on a `206` too: the digest always covers the whole body. A `404` comes before the range is looked at, then `416`, then `412`. A revalidation with `If-None-Match` ignores the range and answers `412` or `304`. To read a snapshot through a cache, use [immutable reads](/docs/0.9/api/immutable-reads). A body stored in the cold tier is streamed from it. Snapshot answers are never compressed. ```bash Read specific offset curl 'http://127.0.0.1:4437/demo/hello/snapshot/00000000000000000042' ``` ```bash Read the first 56 bytes curl -H 'Range: bytes=0-55' \ 'http://127.0.0.1:4437/demo/hello/snapshot/00000000000000000042' ``` > [!NOTE] > Snapshot reads are served by the group's leader after it confirms with a majority of voters that it still leads, so the read reflects every snapshot publish acknowledged before the request. A follower answers `307` to the leader; follow the redirect. See [Durability and consistency](/docs/0.9/concepts/durability-and-consistency#consistency). > [!NOTE] > See [snapshots](/docs/0.9/concepts/snapshots) for the snapshot lifecycle. --- # Bootstrap Returns the stream's latest snapshot (if any) plus the retained updates after the snapshot offset, packed as a `multipart/mixed` response. When some of those updates are only in cold storage, or they exceed 8 MiB, the response is an honest partial: it stops early and the client reads the rest with ordinary reads. This is the recommended way to initialize a client that needs the complete current state of a stream. Bucket ID. Stream ID within the bucket. Bootstrap does not accept `?live=sse`. Combining the multipart body with an SSE event stream is rejected with `400`. To go live after bootstrap, finish the multipart response, then open a separate `GET /{bucket}/{stream}?offset=&live=sse`. Optional precondition: the stream's incarnation token, as a previous response's `Stream-Incarnation` header returned it. The read is answered only if the stream that serves it is still that incarnation; otherwise it answers `412`. Without the header, the request behaves exactly as described here. A malformed value answers `400`. See [`Stream-Incarnation`](/docs/0.9/specs/extensions#24-snapshot-metadata-on-head-and-stream-incarnation). ## Response | Status | Meaning | | ------ | --------------------------------------------- | | `200` | Bootstrap response with snapshot and updates. | | `307` | In a cluster, this node is not the stream's group leader. `Location` is the same request on the leader. | | `400` | Invalid query parameters (including `live=sse`). | | `404` | Stream not found or expired. | | `410` | Requested offset has been trimmed. | | `412` | `Stream-Incarnation` did not match. The response's `Stream-Incarnation` is the current token. | | `503` | The group has no known leader, or the leader could not confirm its leadership in time. Retry after `Retry-After`. | Bootstrap is served by the group's leader after it confirms with a majority of voters that it still leads, so the response reflects every append and snapshot acknowledged before the request. See [Durability and consistency](/docs/0.9/concepts/durability-and-consistency#consistency). Response headers include: | Header | Description | | ------------------------ | ---------------------------------------------------------------- | | `Content-Type` | `multipart/mixed; boundary=...` | | `Stream-Next-Offset` | The offset after the last included update, or the snapshot point when the response has no update parts. Continue reading from here. | | `Stream-Snapshot-Offset` | The snapshot offset, or `-1` if no snapshot exists. | | `Stream-Incarnation` | The incarnation token of the stream that served the response. | | `Stream-Up-To-Date` | `true` when the response reaches the stream's tail. Absent on a partial response. | | `Stream-Closed` | Present and `true` if the stream is closed and the response reaches its tail. | | `Cache-Control` | `no-store`. | ## Complete and partial responses Let `S` be the snapshot offset, or the first retained offset when there is no snapshot. Ursula returns one of two forms: - **Complete.** Every retained byte in `[S, tail)`, with `Stream-Next-Offset` at the tail and `Stream-Up-To-Date: true`. - **Partial.** The bytes in `[S, P)` for some `P` below the tail, with `Stream-Next-Offset: P` and no `Stream-Up-To-Date`. When bytes after `S` have already moved to cold storage, `P = S` and the response holds only the snapshot part. When the updates exceed 8 MiB, a JSON stream stops at the last message boundary within 8 MiB and always holds at least one message; any other stream answers the snapshot part alone. Neither form skips bytes. After a partial response, read from `Stream-Next-Offset` with ordinary catch-up reads until `Stream-Up-To-Date: true`, and apply those updates before treating the state as current. Bootstrap never reads cold storage itself. ## Response body The body is a `multipart/mixed` message: **Snapshot part** The first part is the snapshot blob (or an empty part if no snapshot exists). **Update parts** Subsequent parts hold the updates starting at the snapshot point, with the stream's content type. For JSON streams, each part is one JSON message. For any other stream, the updates come back as one part that may contain several messages; a part boundary is not a message boundary. ```bash Example curl 'http://127.0.0.1:4437/demo/hello/bootstrap' ``` > [!TIP] > After bootstrapping, catch up from `Stream-Next-Offset` if the response was partial, then switch to [SSE reads](/docs/0.9/api/read) with `live=sse` from the latest `Stream-Next-Offset` to receive real-time updates. > [!NOTE] > See [bootstrap](/docs/0.9/concepts/bootstrap) and [snapshots](/docs/0.9/concepts/snapshots) for the conceptual model. --- # Delete stream Bucket ID. Stream ID within the bucket. Optional precondition: the stream's incarnation token, as a previous response's `Stream-Incarnation` header returned it. The delete applies only if the stream is still that incarnation; otherwise it answers `412` and deletes nothing. Without the header, the request behaves exactly as described here. A malformed value answers `400`. See [`Stream-Incarnation`](/docs/0.9/specs/extensions#24-snapshot-metadata-on-head-and-stream-incarnation). ## Response | Status | Meaning | | ------ | ----------------- | | `204` | Stream deleted. | | `404` | Stream not found or expired. | | `412` | `Stream-Incarnation` did not match: the stream is another incarnation, which was not deleted. The response's `Stream-Incarnation` is its token. | ```bash Example curl -X DELETE http://127.0.0.1:4437/demo/hello ``` > [!WARNING] > Deletion is permanent. The stream's data will be asynchronously garbage-collected after the delete is committed. --- # ursulactl `ursulactl` manages the **logical** state of a running cluster over Ursula's admin and metrics HTTP APIs: which node leads which Raft groups, and whether a node is caught up. It executes nothing on hosts. Physical lifecycle belongs to whatever owns the process: Helm and the StatefulSet controller on Kubernetes, systemd on hosts, OpenTofu for the infrastructure underneath. The verbs encode the safety properties an operator otherwise has to remember manually: - before a node goes down, transfer every Raft group it leads to a healthy successor (`drain`) - after it comes back, refuse to move on until `last_applied_index` has caught up to peers' `committed_index` (`wait`) - abort rather than corner a group with no leader A safe rolling restart wraps the platform's restart in these verbs, one node at a time. On Kubernetes: ```bash ursulactl drain --config manifest.json --node 3 kubectl delete pod ursula-2 # the platform restarts the pod ursulactl wait --config manifest.json --node 3 ursulactl undrain --config manifest.json --node 3 ursulactl verify-cluster --config manifest.json ``` On bare metal the restart in the middle is `systemctl restart ursula` on the host. The drain planning and readiness logic is exercised under [deterministic simulation](/docs/0.9/architecture/overview). > [!WARNING] > The naive procedure of "restart followers, then leader" does **not** wait for `applied_index` to catch up between steps. A restarted voter that lost entries, for example after a host crash under `raft.wal.fsync = "never"` or with a replaced disk, stays out of elections until it has caught up with its group leader. Always drain before restart and wait for catch-up before moving to the next voter. ## Verbs | Verb | Effect | |------|--------| | `drain --node N` | Mark the node draining and transfer away every leadership it holds. The mark persists (the node attracts no leaderships) until `undrain`. `--dry-run` prints the transfer plan | | `undrain --node N` | Clear the drain mark so the node may hold leaderships again | | `wait --node N` | Block until the node is a voter in every group and within `--lag-tolerance` of peers. Progress-gated: a node that keeps advancing is never timed out | | `pin-incarnations` | Save required process identities in a manifest for subsequent commands | | `status` | Per-node group counts and leadership distribution | | `wait-ready` | Wait for complete maintenance evidence and local serving health on every node | | `verify-cluster` | Require every configured voter to be present and caught up in two consecutive samples before the next rollout step | | `verify-quorum` | Confirm every configured group with a fresh ReadIndex round and require every replica to apply its fixed prefix under the same leader term | | `verify-survivors` | Confirm fresh prefixes on two observed survivors while preserving the three-voter configuration; does not certify full redundancy or physical fencing | | `backup-create` | Export every Raft group into a backup at a local directory or `s3://bucket/prefix` | | `backup-verify` | Check a backup's manifest, sizes, checksums and every group snapshot without contacting a cluster | | `restore` | Restore a verified backup into a fresh, empty cluster with the same group count, once the cluster confirms it holds every cold object the backup references | | `recover-group --group G` | Explain the recovery plan for one stalled group and send nothing: the floor, why it cuts or not, each replica's expected log and voter sets, and the voters that did not answer. With `--accept-loss`, accept the unsynced loss: plan the group from every reachable voter, with a majority of every voter set a replica reports, send the plan to its leader first and then to the other voters with the leader's plan term, and wait until its history event applied on every replica in the leader's membership, naming the replicas outside it. It stops when a replica leads with a live quorum and no plan was sent. It waits while a gate may still open without a plan, and once it sent a plan, while a replica leads with a live quorum or follows a live leader it did not reach. It never plans again while a replica leads above the last plan's term. A plan that wedges in two observations in a row, or that its leader refuses, or that replicas refuse so that it cannot win, is planned again, up to `--max-plans`. The leader's refusal that no plan fixes stops it. A member that refuses so is left out of the next plan. A `leader_not_voter` or `recovery_point_behind` refusal plans again from a new read. It refuses a plan below a committed or applied index any replica it reached reports, unless a voter's log holds that entry, in which case the plan cuts to it. When the leader's answer is lost, it reads the leader's recorded plan and goes on. `--accept-timeout-secs` bounds one call. See [Recovering after a host crash](/docs/0.9/operations#recovering-after-a-host-crash) | Mutating verbs exit `0` on success and `2` on an abort (drain timeout, no safe transfer target, catch-up stall). Anything else is a configuration or transport error with a single-line, machine-greppable message. ## Process identity across commands Each CLI invocation pins the first observed server instance. A process change makes subsequent observations fail, and mutations retain the original `x-ursula-process-incarnation` header. To preserve the same plan across invocations: ```bash ursulactl pin-incarnations --config cluster.json > pinned-cluster.json ursulactl drain --config pinned-cluster.json --node 3 ``` The saved nodes carry `expected_process_incarnation`. After an admitted Pod replacement, `pin-incarnations --replace-node 3` can bind that one replacement. Every survivor must still match its saved identity. The Helm rollout persists these pins and the admitted source/replacement Pod UIDs in state schema 3. Resuming the same replacement validates its saved process instead of refreshing it after a container restart. It stops on a missing saved plan, changed survivor, or another unadmitted replacement. `pin-incarnations` requires a process incarnation from every node, including an explicitly selected replacement. Nodes that do not report one are refused. `verify-quorum` records the identity of every observed process, and reservation steps compare that map with the saved plan. Instance identity does not serialize concurrent executors or prove a failed host is physically fenced. ## Maintenance executor admission A reviewed cell reservation consumer can add one immutable `maintenance_fence` object at the manifest root: `reservation_id` and `executor_id` are 32 lowercase hexadecimal characters, and `generation` is a nonzero cell-wide monotonically increasing integer. Every saved node carries that same token as `expected_maintenance_fence`; inconsistent or partially populated plans are rejected. The lifecycle commands require saved process identities for every configured voter: ```bash ursulactl activate-maintenance-fence --config admitted-cluster.json ursulactl drain --config admitted-cluster.json --node 3 # After replacement and reconciliation of its physical lifecycle: ursulactl retire-maintenance-fence --config admitted-cluster.json ursulactl verify-quorum --config admitted-cluster.json --expected-groups 256 ``` These commands install or retire the supplied token; they do not acquire a shared reservation, choose generations, authorize Pod deletion, or establish provider-host fencing. A multi-node command may stop after partial acknowledgement. Keep the external operation reserved, resume its same identities, and require all relevant acknowledgements before advancing. Expiry is not authority to select another voter. A new executor uses the higher generation admitted by the external CAS operation; CLI never adopts a token discovered in metrics or refreshes it after 412/428. Activation waits for admitted mutation handlers and then confirms prior messages were processed by each captured local Raft API queue. Caller cancellation leaves its admitted handler running; abnormal handler termination latches that process closed. Failed or cancelled activation/retirement stays closed until the same pending lifecycle completes or an externally admitted higher generation activates. Retirement retains its generation and cannot be reopened by the same token. This confirms submission ordering, not completion of asynchronous replication, snapshot work, or physical operations. Fresh all-replica prefix certification and external lifecycle reconciliation remain required before releasing the reservation. New processes start `unclaimed`, explicitly uncertified until admission. An explicit `pin-incarnations --replace-node` can bind only that selected unclaimed replacement while preserving the token. Other voters must still match the active saved authority. Once activated, all mutating admin requests require `x-ursula-maintenance-fence` as `reservation_id:generation:executor_id` in addition to the process-incarnation header. Metrics report `maintenance_fence` and `maintenance_fence_uncertain`. `verify-quorum` schema 3 records `maintenance_executor_certified` only when every observed process matches the same caller-supplied active token. A subsequent proof can inspect the same retired token and reports `maintenance_executor_retired_certified` only when all observed processes have retired it. A mix of active and retired nodes certifies neither state. These flags certify process-local admission or retirement, not the external reservation or physical fencing. Existing chart and Cloud consumers are uncertified until their shared reservation integration is complete. ## The admin plane Nodes carry **no cluster-mutation surface on the network**. The mutating operator endpoints (raft snapshot/purge/membership/leader-transfer, maintenance drain, cold-flush trigger, bucket purge) plus metrics and usage are served on a separate **admin plane** bound to `server.admin_listen`, which defaults to loopback (`127.0.0.1:4438`). The public client plane (`:4437`) serves only stream traffic and read-only metrics and usage. Metrics and readiness use an explicit `metrics_url` when configured, then `client_url`, then `admin_url`. The client and admin planes serve them. The Raft `http_url` is never used for them, since a cluster listener serves neither. The chart's rollout hook sets `metrics_url` to its Pod-bound admin tunnel so it can inspect a replacement before cluster DNS becomes reachable. Mutating verbs always use `admin_url`. ursulactl is a plain HTTP client and opens no tunnels itself. When the admin plane is not directly reachable, bring your own forward and point the manifest's `admin_url` at it. On Kubernetes, `kubectl port-forward` reaches the loopback-bound plane inside each pod: ```bash kubectl port-forward pod/ursula-0 5441:4438 & kubectl port-forward pod/ursula-1 5442:4438 & kubectl port-forward pod/ursula-2 5443:4438 & ``` ```toml [[nodes]] id = 1 admin_url = "http://127.0.0.1:5441" [[nodes]] id = 2 admin_url = "http://127.0.0.1:5442" [[nodes]] id = 3 admin_url = "http://127.0.0.1:5443" ``` On bare metal the same shape works with `ssh -N -L 5441:127.0.0.1:4438 admin@node1` per node. ## When to use ursulactl vs. the other surfaces | Task | Tool | |------|------| | Day-2 logical operations: drain, observe, gate on readiness | **`ursulactl`** | | Deployment, restarts, upgrades, topology | [Helm](/docs/0.9/deploy-cluster) (systemd on bare metal) | | Custom operator tooling | The admin-plane HTTP endpoints, reached over your own tunnel | ## Install Build from the workspace alongside the server: ```bash cargo build --release -p ursula-ctl --bin ursulactl ``` The binary lands at `target/release/ursulactl`. Drop it on your control machine. It does not need to run on the Ursula hosts themselves. ## Manifest format Every verb accepts `--config `. The manifest is **TOML, JSON, or YAML** (chosen by file extension, sniffed when read from stdin with `-`) and lists the cluster's nodes. Prefer generating the manifest from whatever already knows the topology instead of writing it by hand. On Kubernetes the Helm chart renders one into its ConfigMap, and the URLs in it are in-cluster DNS, so pipe it to ursulactl running where those names resolve (the server image contains ursulactl): ```bash kubectl get configmap ursula -o jsonpath='{.data.cluster-manifest\.json}' \ | ursulactl status --config - ``` For infrastructure provisioned by OpenTofu, emit the manifest as a stack output or a generated file rather than teaching ursulactl to read state files. `tofu output -json | ursulactl status --config -` composes the same way. Per-node fields, all optional except `id`: - `admin_port` (default `4438`) or an explicit `admin_url`: the admin plane to reach, directly or through your forward. - `host`: address shown in reports. Falls back to the admin URL's host. - `http_url`: the node's Raft URL (its `[[raft.peers]].url`). It is the default `host` and part of a pinned plan. Nodes take each other's addresses from `[[raft.peers]]`, never from it. - `client_url`: the node's client-plane URL (its `server.listen`), used for metrics and readiness when no `metrics_url` is set. The chart's manifest sets it. - `metrics_url`: optional metrics and readiness endpoint, for example the same tunnel as `admin_url`. It overrides that access without changing `http_url`. ## Restarting a voter that lost its log A voter that shut down cleanly comes back with every entry it acknowledged and only catches up on what it missed. A voter that lost entries (a host crash under `raft.wal.fsync = "never"`, a journal I/O error, a wiped or replaced disk) needs no ursulactl verb either: it stays behind its recovery gate and stays a voter, and each group leader catches it up by itself (see [Rejoining a wiped node](/docs/0.9/operations#rejoining-a-wiped-node)). Keep the node drained and run `wait` until it caught up in every group. A group whose majority lost entries at once waits for an operator instead (see [Recovering after a host crash](/docs/0.9/operations#recovering-after-a-host-crash)). The catch-up of a voter that lost its disk installs snapshots for every group and can take 10+ minutes. `wait` is progress-gated, so no timeout tuning is needed: a catch-up that keeps advancing is never timed out, and `--stall-timeout-secs` (default 90) only aborts a node that stops making progress. ## `status` Per-node summary of Raft group count and leadership distribution, sourced from every node's `/__ursula/metrics`. Nodes whose metrics fail are reported with `metrics unavailable — …` rather than aborting the report, because `status` is meant to surface partial cluster health. ```bash ursulactl status --config cluster.json ``` Sample output: ``` node 1 (10.0.0.1): groups=4 leaders={1: 2, 2: 2} node 2 (10.0.0.2): groups=4 leaders={1: 2, 2: 2} node 3 (10.0.0.3): groups=4 leaders={1: 2, 2: 2} ``` `leaders={…}` is the count of groups each node is leading from this reporter's perspective. Healthy clusters report the same distribution from every node. ## `wait-ready` Wait until every node reports `--expected-groups` initialized Raft groups with leaders, a complete uniform voter set in `raft_maintenance`, and no uncertain maintenance fence. Each node must also return 200 from `/__ursula/ready`, read from the same address as its metrics. ```bash ursulactl wait-ready --config cluster.json --expected-groups 4 ``` Exits non-zero with the maintenance report or serving refusal when the timeout passes. Connection failures are retried until the timeout, so the wait can start while a node restarts. Every poll observes the processes afresh. Only a node whose process differs from its pinned `expected_process_incarnation`, or an invalid response, stops the wait early with its original cause. Healthy survivors can serve while a replica rebuilds, but `wait-ready` still rejects a second maintenance operation until full redundancy returns. ## Underlying HTTP surface For custom tooling, every verb maps onto a small set of HTTP endpoints on each node: | Verb | Endpoint | |------|----------| | `status`, `wait` | `GET /__ursula/metrics` | | `wait-ready` | `GET /__ursula/metrics` and `GET /__ursula/ready` | | `drain` / `undrain` (maintenance guard) | `POST /__ursula/leadership-shed/maintenance`, `DELETE` to clear | | `drain` (transfer step) | `POST /__ursula/raft/{raft_group_id}/leader/transfer/{node_id}` | | `backup-create` | `GET /__ursula/backup/info`, `GET /__ursula/backup/group/{raft_group_id}` | | `restore` | `GET /__ursula/backup/info`, `POST /__ursula/backup/group/{raft_group_id}/cold-check`, `POST /__ursula/backup/group/{raft_group_id}/import` | The transfer endpoint returns a typed rejection: `409 Conflict` for a changed leader or a recovering target, `400` for an invalid target, `404` for an unregistered group, and `500` for a stopped Raft. Drain re-observes and re-plans the two `409` cases; other errors stop the operation. ## Backup and restore `backup-create`, `backup-verify` and `restore` implement the backup workflow in [Backup and disaster recovery](/docs/0.9/operations#backup-and-disaster-recovery). A backup location is a local directory or `s3://bucket/prefix`. ```bash ursulactl backup-create --config cluster.json --location s3://backups/ursula/2026-10-07 ursulactl backup-verify --location s3://backups/ursula/2026-10-07 ursulactl restore --config new-cluster.json --location s3://backups/ursula/2026-10-07 ``` - `backup-create` writes one compact MessagePack snapshot per Raft group and `manifest.json`. It asks every node for each group and keeps the answer with the highest commit index. `--created-unix-ms` overrides the manifest timestamp. - `backup-verify` checks that the manifest lists every group once, and each object's size, BLAKE3 checksum and snapshot. It reads only the backup. - `restore` verifies the backup, refuses a target whose backup format or group count differs, and then asks the target to check every group's cold references against its cold store. Only when nothing is missing does it import the groups, one replicated write each. A target group that already holds data refuses the import. Restore into a fresh, empty cluster. `backup-create` and `restore` take `--http-timeout-secs` (default `30`). The cold check looks up every object a group references, 32 at a time, in one request per group, so a group with tens of thousands of cold objects needs a longer timeout. An `s3://` location uses the AWS SDK credential chain and `AWS_REGION`. For MinIO or another S3-compatible store, also set `AWS_ENDPOINT_URL`. The backup format is 5, the format of Ursula 0.9. It changes only when what a backup holds changes, not with every format epoch. `ursulactl` refuses a backup or a cluster of another backup format, such as a 0.8 backup (format 4). Each command exits `0` on success and non-zero with an error that names the failed check. A refused `restore` imports nothing. ## `verify-quorum` This source provides a bounded observation for recovery measurements and maintenance tooling. Supply the deployed group count explicitly; the example below uses 256 groups: ```bash ursulactl verify-quorum --config cluster.json --expected-groups 256 --timeout-secs 120 ``` The command requires each configured voter exactly once, every configured group on every voter, complete uniform membership, no learners, and eligible local participation reports. It confirms each group's leader through a new outbound ReadIndex probe, then waits for all replicas to apply the captured prefix under that leader's term. Continuous writes can advance without moving this target. One absolute deadline covers metrics requests, RPCs and waits. The JSON output includes the observation start/end times, each group's leader/term/prefix and every replica's applied index. The prefix is confirmed through the incarnation-bound admin route `GET /__ursula/raft/{group}/quorum`. The CLI uses shared admin protocol types and has no dependency on `ursula-raft`. Quorum proof and self-election requests require process identity; servers without the admin protocol are unsupported. A successful observation is not a maintenance reservation, a continuously refreshed health claim or physical fencing permission. Independent maintenance workflows still need to serialize disruptions and bind replacement to the actual Pod/process/host incarnation. ## `verify-survivors` After one voter becomes unavailable, this source can separately observe the two survivors. Keep all three voters in the manifest and explicitly exclude exactly one known node: ```bash ursulactl verify-survivors --config cluster.json --excluded-node-id 3 --expected-groups 256 --timeout-secs 120 ``` Every group must still have the complete uniform three-voter membership, no learners and open participation gates on both observed replicas. Both survivors must agree on a reachable leader and apply each newly confirmed fixed prefix. The command rejects a second missing replica, an unknown exclusion or a changed topology. It never shrinks the configured membership to fit observations. The output is scoped as `surviving_quorum`, records the configured and observed voter sets, and always reports `full_redundancy_restored=false`. Excluding a voter from observation does not prove its old process or host is stopped. Hard fencing and an exclusive maintenance reservation remain prerequisites for physical replacement, and the complete `verify-quorum` check is required before any subsequent disruption. ### Shared reservation proposals Reservation stores have one schema, version 1, for Pod replacement, host recovery and candidate restaging. Host inventory is optional data, and publishing it keeps the version. `ursulactl` refuses a store with any other version. `reservation-propose --cell cell.json --snapshot before.json --request request.json` produces a whole-ConfigMap CAS proposal without contacting Kubernetes. `reservation-acknowledge` takes the same inputs plus `--response response.json` and validates the exact successful update receipt. Conflicts stop without refreshing authority, and missing stores do not initialize themselves. The policy covers sticky, one-source planned Pod replacement, current-token fresh prefix admission and all-retired completion after the original Pod UID is gone. Its inputs are operational evidence, not authentication. The chart's `maintenance-rollout.sh` Job drives planned Pod replacement through it. No component drives host recovery or calls a provider to terminate an instance, and this offline interface alone grants no disruption authority. The full protocol is documented in `docs/architecture/cluster-membership.md`. --- # Operations The first tool to reach for is [`ursulactl`](/docs/0.9/cli). It covers the raft-aware verbs operators run most often: drain, status, readiness gates. This page covers everything **around** it: the metrics shape ursulactl reads, the admin endpoints it (and your custom tooling) can call, and the operational policies ursulactl does not encode (backups, log levels). Deployment and process lifecycle belong to the platform, such as [Helm](/docs/0.9/deploy-cluster). ## Tooling map | Surface | When to reach for it | |---------|---------------------| | [`ursulactl`](/docs/0.9/cli) | Day-2 cluster ops over HTTP: drain, catch-up waits, status, wait-ready. | | `/__ursula/metrics` and the `/__ursula/raft/...` admin endpoints | Custom tooling. ursulactl uses these underneath. The surface is small and stable enough to script directly. | There is no Prometheus scrape and no general-purpose orchestrator yet. ## Metrics ```bash curl http://127.0.0.1:4437/__ursula/metrics | jq . ``` The JSON snapshot covers per-core mailbox depth, append and apply counters, cumulative timings (`*_ns`), each group's leader, term and applied index (`raft_groups`), hot bytes waiting for cold flush, cold-flush uploads and errors, WAL physical/reclaim/recovery work, WAL disk-pressure admission, SSE counters, live-read waiters, and cold-write backpressure. Start here when triaging slowness, lag, or `503`s. [Troubleshooting](/docs/0.9/troubleshooting#diagnostic-surface) lists jq selectors for the common fields. `GET /__ursula/ready` provides the local serving signal with a typed `reason` on refusal, on the client and admin planes alike. Healthy surviving voters can serve while a voter that lost entries catches up and remains unready. Maintenance also requires a complete `raft_maintenance` report and certain maintenance authority from metrics. `ursulactl wait-ready` checks both the metrics and this existing readiness endpoint. `ursulactl status` presents leadership fields across the cluster. ### Peers of another cluster Every Raft gRPC request a node sends carries the id its WAL holds (`server.cluster_id`) and its `raft.node_id` as the `ursula-cluster-id` and `ursula-node-id` request metadata. That covers replication, votes, snapshots, leadership transfers, recovery probes, forwarded reads and HEADs, and forwarded bucket purges. A node checks the metadata before it decodes the request and refuses a peer with another id with `FAILED_PRECONDITION`, before any Raft RPC runs. The refusal names the refusing node's id, so the sender learns it as well. A mistyped `server.cluster_id` or a peer URL that points at another deployment therefore fails loudly instead of exchanging Raft traffic between two clusters, and a misconfigured node of another cluster cannot move this cluster's Raft terms. The cluster id is public, since every `HEAD` answer carries it as `Ursula-Cluster-Id`, so the check guards against misconfiguration, not against a hostile sender: keep the cluster plane on a private network. What each side does with a refusal: - **Counts it.** `/__ursula/metrics` counts every refusal, both ways, in `raft_grpc_peer_cluster_refusals`. - **Logs it once.** The node logs each refused peer once at error level, with its node id, cluster id and address, while it keeps being refused, not once per RPC. - **Reports configured peers.** `/__ursula/ready` and `/__ursula/metrics` list, under `peer_cluster_mismatches`, the peers of `raft.peers` refused within the last minute, with their `node_id` and the `cluster_id` they sent or answered with. A peer this node dialed always counts. A request this node refused counts when its sender claims a configured node id and, where that peer's URL names an IP address, comes from that address. Senders outside `raft.peers` are counted and logged, never listed. - **Keeps its readiness.** A refusal never changes readiness or its `reason`. One misconfigured node does not take its healthy peers out of service, and an unready node still names its own cause, such as `recovery_stalled`. - **Backs off.** OpenRaft treats a refusing peer as unreachable. The node keeps its connection to it, since the answer is final rather than a broken channel. - **Leaves it out of handoffs.** `GET /__ursula/leadership-shed` answers carry the node's `cluster_id`, and a node counts a peer's answer only when the id is its own. A node of another cluster at a peer's address therefore never becomes a handoff target, even before any Raft traffic reached it. - **Forwards answer `503`.** A read, HEAD or bucket purge a follower forwards to a leader of another cluster answers the client `503` with `Retry-After`, as for a leader it cannot reach. Nothing ran on the refusing node. A request that names no cluster id comes from a node before format epoch 4 or a stray client. It is refused with the Raft protocol-mismatch text, which such a node reads as a [format-epoch mismatch](#upgrading-to-07). The node counts it in `raft_grpc_unidentified_peer_refusals` and logs each sender address once. It does not make the node unready, so a misrouted older node cannot take a healthy one out of service. A node that itself contacts a configured peer of another format epoch still becomes unready with `format_epoch_mismatch`. At startup, a node checks that its WAL holds its configured `server.cluster_id` before it contacts any peer, and its [format-epoch probe](#upgrading-to-07) reports a configured peer of another cluster at error level and starts anyway. A refusal lasts until an operator fixes the configuration: correct `server.cluster_id` or the peer URL on the wrong node. A node whose WAL already holds another id refuses to start, so give a replaced node an empty WAL (see [Rejoining a wiped node](#rejoining-a-wiped-node)). `cluster_id` is in both `/__ursula/ready` and `/__ursula/metrics`, so tooling can check that every node of a deployment reports the same id: ```bash for node in NODE1 NODE2 NODE3; do curl -s http://$node:4437/__ursula/ready | jq -c '{cluster_id, peer_cluster_mismatches}' done ``` ## Admin endpoints These are the primitives `ursulactl` and any custom operator tooling builds on. They live on the **admin plane** (`server.admin_listen`, loopback `127.0.0.1:4438` by default), not the public client plane. Nodes expose no cluster-mutation surface on the network. Reach them over a tunnel to the admin port (see [ursulactl's operation providers](/docs/0.9/cli#the-admin-plane)). The examples below assume a forward from local `4438` to the node's admin plane. Each call is local to one node. To act on every group, loop over the IDs in metrics. Every admin mutation requires `x-ursula-process-incarnation`, pinned from `GET /__ursula/metrics` before the operation. Missing identity returns `428`; a restarted target returns `412` before the handler changes state. Keep the same identity for the entire plan and stop on either response. This precondition is public instance identity, not authentication or an exclusive maintenance reservation. Updated `ursulactl` attaches it automatically and refuses to refresh it during an operation. ```bash # Observe once for this operation; do not refresh after a 412. process_incarnation=$(curl --fail http://127.0.0.1:4438/__ursula/metrics | jq -er .process_incarnation) # Force a cold flush for one stream (skip the timer) curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/flush-cold/demo/hello # Trigger a Raft snapshot for one group curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/42/snapshot # Purge log entries through index 1000 for one group, at most through its # last snapshot. Answers 503 with Retry-After while a snapshot of the group # builds or a purge of it has not landed. Send it again then. curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/42/purge?upto=1000" # Hand leadership of one group to another voter (used by `ursulactl drain`) curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/42/leader/transfer/2 # Replace the voter set of one group (call on the group's current leader) curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/42/membership?voters=1,2,3" # Put this node into maintenance drain / lift it again curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/leadership-shed/maintenance curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X DELETE http://127.0.0.1:4438/__ursula/leadership-shed/maintenance # Send one group's recovery plan to this node, a member of a plan whose leader answered plan term 9, # naming the log its metrics show (see Recovering after a host crash) curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \ -d '{"leader": 2, "point": 1234, "cut": false, "step": {"follow": {"leader_term": 9}}, "expected": {"last_log_index": 1230, "last_log_term": 7, "current_term": 8}}' \ -X POST http://127.0.0.1:4438/__ursula/raft/42/recovery/accept-unsynced-loss # The same for a plan that cuts to an entry another replica reported committed, at index 1240 # of term 7 led by node 3: every request of the plan names that entry curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \ -d '{"leader": 2, "point": 1240, "cut": true, "point_entry": {"term": 7, "node_id": 3}, "step": {"follow": {"leader_term": 9}}, "expected": {"last_log_index": 1241, "last_log_term": 7, "current_term": 8}}' \ -X POST http://127.0.0.1:4438/__ursula/raft/42/recovery/accept-unsynced-loss ``` The snapshot endpoint answers once the group has a snapshot of everything it had applied when the request arrived. It waits while a build of the group runs or other snapshot builds hold all of the node's build permits, and answers `503` with `Retry-After: 1` when no build of the group can start within 10 seconds (see [Manually trigger a snapshot](/docs/0.9/troubleshooting#manually-trigger-a-snapshot)). The leader-transfer endpoint refuses with `409 Conflict` if the receiving node isn't the current leader and `400 Bad Request` if the target isn't a voter. This is why `ursulactl` is the safer way to chain these calls: it consults metrics first. The group leader keeps every group at its configured voters: if the membership endpoint drops a configured voter, the leader adds it back as a learner and promotes it again once it has caught up (see [Recovering after a host crash](#recovering-after-a-host-crash)). Membership changes are for repair, not for shrinking a group. The membership endpoint has the same leader-only contract: it answers `409 Conflict` (with the current leader's id in the body) when the receiving node does not lead the group, and `400 Bad Request` without a non-empty `voters` query parameter. On success it returns the log index at which the new voter set was committed. It replaces the whole voter set in one call, so include every node that should remain a voter. It also removes every learner outside the new voters in the same change. ### Maintenance drain `POST /__ursula/leadership-shed/maintenance` marks the node as draining: it stops campaigning and accepting transfers, and the background leadership balancer migrates its current leaderships to eligible peers one transfer per tick until the mark is lifted with `DELETE`. Both verbs return the node's leadership-shed state so tooling can confirm the transition took effect: ```json { "bits": 1, "state": "maintenance-drain", "should_accept_transfer": false, "should_campaign": false, "should_shed_current_leaders": true } ``` `ursulactl drain` wraps exactly this pair: mark first so the node stops re-acquiring groups, then transfer leaderships away explicitly instead of waiting for the balancer. The mark stays set until `ursulactl undrain` clears it. If a rollout ever dies mid-way, a node can be left refusing leadership. Run `ursulactl undrain` (or `DELETE` the mark directly) to recover. Peers read each other's shed state (`GET /__ursula/leadership-shed` on the cluster-plane listener) when deciding which nodes are eligible to pick up leaderships, so a drained node stops attracting groups cluster-wide, not just locally. ### Graceful voter shutdown On the first SIGTERM or Ctrl-C, the voter marks itself maintenance-drained to prevent re-acquiring leadership. It keeps Raft transport available while attempting to hand off each leadership to an eligible voter (see the handoff rule under [Recovering after a host crash](#recovering-after-a-host-crash)). It waits for observed leadership changes for up to five seconds, then closes Raft transport and drains HTTP listeners for up to ten seconds. A connection still open after that, such as one from a peer that lost power, is abandoned. It then stops its Raft groups, `fsync`s every journal and records a clean shutdown, so the next start reads the journals strictly and the node holds every entry it acknowledged. The whole shutdown has a deadline of twenty seconds from the first signal. If it has not finished by then, the process exits without recording a clean shutdown, and the next start treats the run as a crash. A second signal also exits at once without recording a clean shutdown. The chart gives voter pods thirty seconds by default. This reduces planned-stop election pauses. It cannot hand off leadership on SIGKILL or OOM. `runtime.node_memory_abort_cap_size` is an emergency process-abort threshold, not HTTP admission control. Exceeding it calls `abort()` without graceful leadership handoff, like an OOM kill. It does not return `503`. Leave room below the container memory limit for normal operation and recovery, and use write admission and backpressure to manage load. ### Restarting a voter Restart one voter at a time: 1. `ursulactl drain --node N` moves every leadership away and keeps the node from taking one back. 2. Restart the process through the platform (Helm, systemd). The graceful shutdown above keeps every journal intact. 3. `ursulactl wait --node N` returns once the node is a voter of every group again and has caught up. 4. `ursulactl undrain --node N` lets it hold leaderships again. Do not stop another voter before `wait` returns. Two overlapping outages of a three-voter group leave it without a quorum even if the restarts were issued separately. The chart's graceful-rollout hook runs the same sequence for every pod. ### Changing a peer address Raft membership names node IDs only. Every node dials its peers at the `url` its own `[[raft.peers]]` lists, and a follower redirects clients to the leader's `client_url` there. A peer's address therefore changes with the configuration, and no admin step migrates it. Keep `[[raft.peers]]` the same on every node. - **A new address behind the same name** needs no change. Name peers by DNS names that follow each node, as the chart's headless Service does. - **A new name that the old one still reaches** rolls like any restart. Change `[[raft.peers]]` on every node and restart the nodes one at a time ([Restarting a voter](#restarting-a-voter)). - **A listener that moves**, such as a new cluster port, `server.cluster_listen` turned on or off, or a host without a stable name, needs every voter restarted together. Stop every voter, change `[[raft.peers]]` on every node, and start them all. The cluster serves nothing until a quorum is up, which takes about as long as one restart. Restarting one node at a time does not work here. Until the last node restarted, some nodes cannot reach each other, and a group can lose its quorum. Rolling listener moves are tracked in [#735](https://github.com/tonbo-io/ursula/issues/735). With the chart, `server.ports.cluster` and `server.clusterListen` move every listener. Run that `helm upgrade` with `server.updateStrategy=OnDelete` and `server.gracefulRollout.enabled=false`, since the graceful-rollout job restarts voters one at a time. Then delete every voter pod at once, and turn the graceful rollout back on with the next upgrade. `global.clusterDomain` rolls like any restart while the old domain still resolves. A node that a group's membership names but `[[raft.peers]]` does not list is never dialed. Each node logs it once, and the group reports `unconfigured_member` in the maintenance report ([`unconfigured_member`](/docs/0.9/troubleshooting#unconfigured_member)). ### WAL routing configuration A WAL root is bound to `runtime.core_count` and `raft.group_count` in `/raft-log/topology.bin`. Startup holds the WAL lock and checks both values before recovering any group or writing the run state. A mismatch, missing record on an existing WAL, or corrupt record refuses startup. Keep both values explicit and unchanged when moving a persistent volume or resizing its host. Without explicit values, `core_count` follows the process's available parallelism and `group_count` defaults to that parallelism times 16 (resource presets set a fixed group count). Helm defaults to 4 and 64. Changing cores changes the journal directory of some groups. Changing group count changes stream routing. Neither is an online resharding operation. For a mismatch, configure the stored counts again. The error reports them. Do not manufacture a record from the new configuration or remove the check's files. A replica whose record is missing or corrupt can be rebuilt on an empty WAL directory from a healthy quorum, with the cluster's routing counts. Follow the rejoin procedure below and wait for full catch-up before replacing another replica. A single node has no quorum to rebuild from, so a fresh WAL does not preserve its data. ### Rejoining a wiped node A voter whose WAL directory was wiped or replaced, for example after a lost disk, starts with an empty log for every group. For each group: - **It does not initialize the group again.** A per-group membership initializer (`raft.init_membership_per_group = true`, the chart default) runs `Initialize` only when every other configured voter answers that it holds nothing for the group either and no leader is seen. A fresh install still bootstraps this way. One voter holding entries or following a leader means the group exists, and the node waits to be replicated to. It waits for every voter, not a quorum, so a fresh install with a voter that never comes up stays uninitialized (it logs "waits for every voter before initializing"). - **It refuses replication until the leader proves a quorum without it.** The node lost its vote with its disk, so it cannot tell the current leader from a stale one. It refuses appends, heartbeats, snapshots and votes until the group's leader confirms a fresh ReadIndex barrier with a quorum of the other voters. The node then records that leader's vote as its vote floor and accepts replication. In a 3-voter group that quorum needs both other voters up. While the wiped voter waits and another voter is down, the group has no quorum and cannot commit writes. - **It stays behind the recovery gate.** It does not campaign and refuses every vote once it knows the group holds entries, until it has applied a fresh barrier from the group's leader (see [Recovering after a host crash](#recovering-after-a-host-crash)). - **The leader catches it up in place.** The leader sees the node answer that it lacks entries it had acknowledged. It rewinds its replication to the node and sends it its log again, and the node stays a voter throughout. Leadership handoffs (the balancer, the commit-stall watchdog) skip the node until it holds every committed entry. While the other voters are up, no operator step is needed. `ursulactl wait --node N` returns once the node caught up in every group and opened its gates, and `ursulactl verify-cluster` shows it unready until then. Do not stop another voter before that. If a majority of a group's voters lost their disks at once, the voters that kept their disks are not a quorum. No leader can prove a quorum to the wiped replicas, so they never obtain a vote floor, and `accept-unsynced-loss` cannot replace lost voting history. The group stays stopped. Restore the latest verified backup into a fresh cluster (see [Backup and disaster recovery](#backup-and-disaster-recovery)). If every voter lost its disk, the bootstrap probe sees only empty voters and initializes the groups again, empty: every acknowledged write is lost, and cold objects written before stay in object storage, unreferenced. Start such a cluster over on a new object-storage prefix, as for any fresh install. ### A log below its snapshot record A replica refuses to open a group whose log holds an entry, or whose commit marker names one, at or after the index of its restored snapshot record but below that snapshot in log-id order. It logs `LogBelowSnapshot` with the snapshot and the entry, and the group stays closed on that replica while its other replicas keep serving it. No such entry can be committed. In 0.9 a log reaches this state only through a bug or disk corruption, because the WAL is synced before every snapshot record. To recover, rejoin the node as in [Rejoining a wiped node](#rejoining-a-wiped-node), after the checks of its other groups in [Recovering after a host crash](#recovering-after-a-host-crash). ### Recovering after a host crash With the default `raft.wal.fsync = "never"`, a node acknowledges ordinary data appends from the page cache. Membership batches are always fsynced before acknowledgment, so a full-cluster power loss cannot erase the voter configuration. A process crash loses nothing. A host crash or power loss, or a journal I/O failure, can cost the node the unsynced tail of its journals. The node detects it on the next start from its run state (`wal_recovery` in `GET /__ursula/metrics`, the `ursula.wal.recovering` gauge) and keeps the frames that still verify. A node whose run state file is missing while its journals hold records is treated the same way. Every group the node held enters the recovery gate, which is recorded in the node's metadata and survives further restarts: - The node does not campaign, takes no leadership transfer and grants no vote. A node that led the group starts as a follower. It still accepts appends from a leader of its current term or a later one. - It asks the group's leader for a fresh ReadIndex barrier, applies the log up to the barrier's index and then opens the gate. Until then readiness answers `503` with reason `recovery_gate_closed`, and `recovery_gates` in `GET /__ursula/metrics` lists the gated groups. - The leader sees that the node lost entries it had acknowledged, rewinds its replication to the node and sends it its log again. The node stays a voter, and nothing is proposed to repair it. The node's answer still counts as its acknowledgement of the leader, so the leader keeps its quorum while the node catches up. The leader logs a warning naming the group and the node, and `ursula.raft.follower_log_reversions` counts it per node. A node that shows up there without a host crash or a lost disk may have a disk that drops synced writes. Leadership handoffs (draining, balancing, a node that sheds its leaderships, the commit-stall watchdog, the admin transfer endpoint) go only to a voter that acknowledged the leader within the last 1.5 seconds (the minimum election timeout) and holds every committed entry. A handoff whose target has not taken over within 6 seconds (two maximum election timeouts) is abandoned: the leader steps down and campaigns again. While at most a minority of a group's voters lost their tail, the other voters hold every acknowledged write and nothing is lost. Wait for every group to open the gate on a node before you stop another voter. **A majority of a group's voters lost their tail at once.** If the group's leader kept its log, it stays leader, catches the gated replicas up and its barrier opens them as above. The gated replicas renew its lease before they caught up. If they cannot catch up, for example when one needs a snapshot while object storage is unavailable, the leader stays live: writes are accepted and can time out with an unknown outcome instead of answering `503`, and `recover-group` refuses to fence the group while it leads with a live quorum. Otherwise the gated voters refuse every vote, so the group elects no leader and writes answer `503`. After `30s` without a leader barrier and without applying anything, each gated replica reports its group stalled: readiness reason `recovery_stalled`, the group under `recovery_stalled_groups` in `GET /__ursula/ready` and under `recovery_gates.stalled` in the metrics, the `ursula.raft.recovery_gates{status="stalled"}` gauge, and an error log naming the group. Writes acknowledged after the last `fsync` of the surviving replicas may be gone. Only an operator can accept that, with a recovery plan. The plan elects one replica as the group's leader, and that leader starts a new history of the group: every stream of the group gets a new `Stream-Incarnation`, and every producer epoch claimed before is closed (see [history rewrites](/docs/0.9/specs/extensions#24-snapshot-metadata-on-head-and-stream-incarnation)). For each stalled group: `ursulactl recover-group --config cluster.toml --group 42` explains the plan it would send for one group and sends nothing: the floor, why it cuts or not, each replica's log and voter sets the plan expects, and the voters that did not answer. With `--accept-loss` it runs the steps below: it reads every voter, computes the plan, sends it to the plan's leader first and then to every other replica that answered, and waits until the plan is satisfied. It stops with an error when a replica leads the group with a live quorum and no plan was sent. While a replica's gate may still open without a plan, it waits, and once it sent a plan, it waits while a replica leads with a live quorum or follows a live leader it did not reach. It plans again when a plan wedges in two observations in a row (see below), when the plan's leader refuses it, or when replicas refuse it so that the plan's leader and the replicas that took it are no majority, up to `--max-plans` plans. It never plans again while a replica leads at a term above the last plan's term: that leader was elected under the plan. When the plan's leader refuses for a reason no plan fixes, `missing_vote_floor` or `step_mismatch`, it stops with an error. A member that refuses so is left out of the next plan, but the voter sets it reports still count, and a plan that a majority took stands. A `leader_not_voter` refusal changed nothing: the membership moved since the tool read it, so it reads the replicas again and plans again. Once the plan is satisfied, it names the replicas it reached outside the leader's membership, which keep the replaced history. When the plan's leader does not answer in time, it reads the plan the leader recorded and sends the other replicas that plan term. `--accept-timeout-secs` bounds one call, which a cut or a demotion restart makes slow ([ursulactl](/docs/0.9/cli)). By hand: 1. On every node that holds a replica of the group, learners included, read `GET /__ursula/metrics`, and under `raft_groups` the group's `recovery_gate`, `last_log_term`, `last_log_index`, `current_term`, `committed_index`, `recovery`, `leads_live`, `voter_configs`, `committed_voter_configs` and `maintenance.running`. `voter_configs` are the voter sets of the replica's membership, two while a change is joint. `committed_voter_configs` are the voter sets of the membership it keeps when a plan cuts its log back to its committed entry. Plan without a replica whose `maintenance.running` is `false`: its group stopped, and it reports no membership. Keep its `last_applied_index` for step 2. Read also `committed_term` and `committed_node`, which with `committed_index` name a replica's committed entry, and `last_applied_index`. Go on only when, in every voter set any replica reports in either field, a majority of its voters answer, only while no replica's gate is `awaiting_barrier`, `catching_up` or `cutting`, and only while no replica reports `leads_live`: such a replica may still open through a leader's barrier without losing anything, is running a plan's cut, or leads the group. A replica whose own metrics name it leader without `leads_live` lost its quorum: the plan demotes it. 2. Compute the plan from the replicas that answered: - `floor`: the highest `current_term` among the replicas whose gate is not open or whose `recovery` is not `null`. - The leader: among the replicas some voter set names, the one with the greatest last log id, highest `last_log_term` first, then highest `last_log_index`. An empty log (`null`) ranks last. Its `last_log_index` is the plan's `point`, and `cut` is `false`. - When that replica's `voter_configs` and `committed_voter_configs` name more than one voter and its `last_log_term` is below `floor`, its log may hold an entry that a lost leader overwrote. When its own `voter_configs` do not name it, it cannot campaign. In both cases `cut` is `true`, the leader is the replica with the greatest `committed_index` among those whose own `committed_voter_configs` name them, and its `committed_index` is the `point`. Every replica then keeps only the entries it knew were committed. - When `cut` is `true` and a replica, a learner say, reports a `committed_index` above the `point`, the voters may still hold that entry in their logs although their markers lie below it. Then the leader is instead the replica with the greatest `committed_index` among those whose own `committed_voter_configs` and `voter_configs` both name them and whose `last_log_index` reaches that index. The `point` is that index, and `point_entry` names the entry: `{"term": committed_term, "node_id": committed_node}` of the replica that reported it. The leader refuses the plan unless its log holds exactly that entry at the `point`. - Stop when any replica that answered, learners and replicas whose group stopped included, reports a `committed_index` or a `last_applied_index` above the `point`: the plan's leader would overwrite that replica's committed entries. - `highest_term`: the highest `current_term` among all the replicas that answered. 3. Send the plan to the leader first: `{"leader", "point", "cut", "step": {"lead": {"highest_term": H}}, "expected": {"last_log_index", "last_log_term", "current_term"}}`, where `H` is `highest_term` and `expected` is what you read on the leader. A cut to a reported entry adds `"point_entry": {"term", "node_id"}`, the same in every request of the plan. Leave it out otherwise. The leader answers `outcome` `leads` and `leader_term`, the plan term. 4. Then send it to every other replica that answered, with `"step": {"follow": {"leader_term": T}}` where `T` is the `leader_term` the leader answered, and `expected` as you read it on that replica. Each answers `follows`. Take `T` from the leader's answer, never from the metrics. A replica refuses a `T` below the `current_term` in its `expected` with `plan_term_below_vote`, but it cannot tell a wrong `T` above it. Send no `follow` before the leader answered. A replica that takes the plan before its leader can elect the plan's leader at a term below the plan term, before that leader took the plan. The leader then keeps leading below the plan term, and the plan never completes. No replica elects a leader that lacks the plan even then ([#643](https://github.com/tonbo-io/ursula/issues/643)). `ursulactl` and the script below always send to the leader first. 5. When the plan cuts, each replica first stops voting and taking replication, restarts its group in the process with the log cut back to its committed entry, and answers once the restarted group holds the plan. A replica that leads or campaigns when it takes a plan restarts its group the same way, so it drops that leadership or campaign. The plan's leader also restarts when its vote term is below the plan term, and starts with the plan term as its vote, so its first election is above the plan term. The plan term is the higher of `highest_term` and the leader's own vote term, so it is at least the vote term of every replica that holds the plan. From the moment it takes the plan, the plan's leader campaigns but votes for no other candidate, and while it leads it serves nothing until the plan's history event applies. A replica that holds the plan votes only for the plan's leader, and only in an election at a term above the plan term: an election the leader started after it took the plan. So no replica elects a leader that has not taken the plan, even one that campaigns while the plan reaches the other replicas. It also refuses the plan's leader while the leader's log holds an entry above the `point` at or below the plan term, and a replica that holds the plan takes replication only from a leader above the plan term. A plan's leader that restarts after it lost entries comes back with a closed gate instead: it votes and campaigns for no one, and the group waits for another plan. A leader elected before the plan at or below the plan term, which may still believe it leads, thus never extends the plan's leader, and none of its entries is applied. A leader at a higher term could only have been elected by replicas that hold no plan: an open replica refuses a leader whose log is behind its own, and a closed one votes for no one, so it cannot get the plan's leader elected over its entries either. Once elected, the leader proposes the group's history event before anything else and answers `503` with `Retry-After` to writes and reads until the event applies. On each replica the plan is satisfied once it applied the event: `history.event_index` above the plan's `point` and `recovery` back to `null` in its metrics. The plan is recorded in the node's metadata, so a restart keeps it until then, and the leader serves nothing until its record is gone. A replica that lost power again after taking the plan keeps its vote restricted to the plan's leader across restarts, and once the event applies its gate stays closed until a leader's barrier opens it. Leaders after that hold the event, so a later leader change starts no new history. If the plan's leader loses entries it had replicated, for example in a crash right after its election, the other replicas' logs are longer than its own and they elect no one. The plan wedges. After `30s` in which a replica holding the plan applied nothing, it reports its group stalled like a closed gate does: readiness reason `recovery_stalled`, and `stalled` as its `recovery_gate`. Plan again. The new plan replaces the first, and since the wedged leader's failed elections raised its vote term, it usually cuts every log back to its committed entry. `ursulactl recover-group --accept-loss` plans again by itself when it sees, in two observations in a row, a replica holding the plan with a longer log than the plan's leader, or reporting the group stalled. ```bash # group 42: one admin URL per node with a replica of the group, learners included, through your forwards admins=(http://127.0.0.1:5441 http://127.0.0.1:5442 http://127.0.0.1:5443) for admin in "${admins[@]}"; do curl -sS --fail "${admin}/__ursula/metrics" | jq -c --arg admin "${admin}" ' .process_incarnation as $incarnation | .raft_groups[] | select(.raft_group_id == 42) | {admin: $admin, process_incarnation: $incarnation, node_id, running: .maintenance.running, voters: .voter_configs, kept: .committed_voter_configs, gate: .recovery_gate, recovery, committed_index, committed_term, committed_node, last_applied_index, expected: {last_log_index, last_log_term, current_term}}' done > replicas.jsonl jq -sc 'def holds($n): any(.[]; . == $n); def votes($cut): .node_id as $n | any((if $cut then .kept else .voters end)[]; holds($n)); . as $all | map(select(.running)) | (map(.node_id)) as $answered | ([.[] | (.voters + .kept)[] | select(length > 0)] | unique) as $sets | if all($sets[]; (map(select(. as $n | $answered | holds($n))) | length) * 2 > length) then . else error("a reported voter set has no majority of its voters answering") end | ($sets | add | unique) as $voters | (map(select(.gate != "open" or .recovery != null) | .expected.current_term) | max // 0) as $floor | map(select(.node_id as $n | $voters | holds($n))) as $candidates | ($candidates | max_by([.expected.last_log_term // -1, .expected.last_log_index // -1, votes(false), .node_id])) as $longest | ((($longest.voters + $longest.kept) | add // [] | unique | length) == 1) as $single | ((($single | not)) and ((($longest.expected.last_log_term // 0) < $floor) or ($longest | votes(false) | not))) as $cut | (if $cut then ($candidates | map(select(votes(true))) | max_by([.committed_index // -1, .node_id])) else $longest end) as $leader | if $leader == null then error("no replica the plan may elect is a voter of the membership it would campaign under") else . end | {leader: $leader.node_id, cut: $cut, point: (if $cut then $leader.committed_index else $leader.expected.last_log_index end // 0)} as $plain | ($all | map(select(.committed_index != null and .committed_term != null)) | max_by([.committed_index, .node_id])) as $known | (if $cut and $known != null and $known.committed_index > $plain.point then ($candidates | map(select(votes(true) and votes(false) and (.expected.last_log_index // -1) >= $known.committed_index)) | max_by([.committed_index // -1, .node_id])) as $reaches | if $reaches == null then $plain else {leader: $reaches.node_id, cut: true, point: $known.committed_index, point_entry: {term: $known.committed_term, node_id: $known.committed_node}} end else $plain end) as $plan | if any($all[]; ([.committed_index // 0, .last_applied_index // 0] | max) > $plan.point) then error("a replica knows a later committed entry than the plan would keep") else . end | sort_by(.node_id != $plan.leader)[] | . + {plan: ($plan + {expected})}' replicas.jsonl > plan.jsonl # Review plan.jsonl, then send it: the leader first, then every other replica with the # leader's plan term. Stop at the first refusal. leader_term="" while read -r replica; do if [ -z "${leader_term}" ]; then body=$(jq -c --argjson highest "$(jq -s 'map(.expected.current_term) | max' replicas.jsonl)" \ '.plan + {step: {lead: {highest_term: $highest}}}' <<<"${replica}") else body=$(jq -c --argjson term "${leader_term}" '.plan + {step: {follow: {leader_term: $term}}}' <<<"${replica}") fi answer=$(curl -sS --fail-with-body -H "x-ursula-process-incarnation: $(jq -r .process_incarnation <<<"${replica}")" \ -H "content-type: application/json" -d "${body}" \ -X POST "$(jq -r .admin <<<"${replica}")/__ursula/raft/42/recovery/accept-unsynced-loss") status=$? echo "${answer}" [ "${status}" -eq 0 ] || break leader_term=${leader_term:-$(jq -r .leader_term <<<"${answer}")} done < plan.jsonl ``` Do not plan again while a replica reports itself leader at a term above the last plan's term: it was elected under that plan, and a member that stalled under it is cut off, not wedged. If a live leader elected before a plan keeps a minority of replicas that took the plan, they take no replication from it, since its term is at or below the plan term. Transfer the leadership away from that leader ([`POST /__ursula/raft/{group}/leader/transfer/{node}`](#admin-endpoints)). Every transfer raises the term by one: once a leader's term is above the `leader_term` those replicas show under `recovery`, they follow it. A refusal answers `409` with a typed body: `{"raft_group_id", "node_id", "refusal": {"reason", ...}, "message"}`. If the leader refuses the plan, nothing changed: read the metrics again and plan again. A replica that refuses after the leader took the plan does not undo it. A replica whose `reason` is `follows_leader` with the plan's leader as `leader` refuses because the replicas that took the plan elected it, and it rejoins through it. For any other refusal, read the metrics again and plan again once the group has no leader. A refusal with `reason` `not_stalled` means the replica may still open through a leader's barrier: wait until its gate reports `stalled` before you plan again. The script plans one group. For many stalled groups, as after a rack power loss, run it or `ursulactl recover-group --accept-loss` for each, one group at a time. The call is refused with `409 Conflict`, and changes nothing, when: - the replica lost its vote and has not established a durable vote floor. Restore connectivity to an intact quorum so it can obtain a fresh barrier. Accepting unsynced data loss cannot authorize a replica to acknowledge an unknown leader. - the replica leads the group with a quorum that acknowledged it within the last 3 seconds, or follows a leader whose replication reached it within the last 3 seconds, or its gate awaits or applies a leader's barrier or runs a plan's cut. The group has a live leader or may still recover without loss. - the request's `step` is not the replica's: `lead` goes to the plan's leader only, and `follow` to every other replica. - a `follow` names a plan term below the `current_term` its `expected` names (`plan_term_below_vote`): the term did not come from the plan's leader. - the plan cuts a replica whose `voter_configs` and `committed_voter_configs` name only one voter (`single_voter_cut`): no other voter could have overwritten an entry it holds, so its log is kept whole. The tool decides the cut the same way, from the voter sets of the replica with the longest log. - the replica is taking another plan (`plan_in_flight`). Wait for its answer and observe again. - the replica is the plan's leader and is no voter of the membership it would campaign under (`leader_not_voter`): its `voter_configs`, or its `committed_voter_configs` when the plan cuts. The membership moved since you read the metrics: read them again and plan again. Only the plan's leader checks membership. Another replica takes the plan whatever its own membership says, since the leader counts its vote under the leader's membership. - the replica no longer holds the log `expected` names, because it moved since you read the metrics. A replica that campaigns, one with an open gate or the plan's leader, raises its `current_term` with every election it starts, so only its last log id is compared. Read the metrics again and plan again. - the replica committed or applied an entry above the plan's `point`. The plan picked the wrong leader: read the metrics again and plan again. - the replica is the plan's leader and its log does not continue from the `point`: its last log index, or its committed index when the plan cuts. When the plan cuts to a later entry another replica reported committed, the leader must hold exactly that entry at the `point`. - the plan cuts and the replica knows no committed entry, or no longer holds the committed entry it recorded. - the plan cuts and its cut cannot run when the group restarts, because the log no longer holds the entry it keeps. The replica drops the plan, records its log as recovering so that its gate stays closed across restarts, and answers `409`. Wait until the gate reports `stalled`, then read the metrics again and plan again. **Across a membership change.** A membership entry is synced before it is acknowledged, also under `never`, so a power loss cannot drop it from a replica that holds it. The replicas may still report different voter sets when you plan: - A change that only a minority appended, such as a promotion the leader sent only to the new voter before it was cut off, has three outcomes. A cut removes it. Without a cut, the plan's leader either holds it and commits it, or lacks it and overwrites it. - A cut can also undo a committed change. A replica's commit marker is not synced on its own, so after a crash it can sit below a change the group committed. The replica reports the change in `voter_configs` but keeps the membership before it, which `committed_voter_configs` shows. A removed voter can come back, and a promoted voter can go back to being a learner. After a cutting plan, check the group's membership and make the change again. Count every voter set, as step 1 says, and pick the leader by the membership it campaigns under, as step 2 says. A plan's leader whose cut leaves it no voter of the membership it keeps drops the plan and answers `cut_left_no_vote`: plan again. Every replica that applies the plan's history event holds the leader's membership. Every replica that holds a plan and does not lead answers client reads of the group (catch-up, live reads, `HEAD`, snapshots and forwarded reads) with `503` and `Retry-After` at once until it applied the event, as the plan's leader does after a short wait, so a replica outside the leader's membership, which never receives the event, never serves the history the plan replaced. Its `/__ursula/usage` and `/__ursula/metrics` keep answering. If it is a configured voter, the leader adds it back as a learner and it catches up from the plan's leader. If its log committed entries above the plan's point, it cannot follow: rejoin its node as in [Rejoining a wiped node](#rejoining-a-wiped-node), which affects every group on that node. The same holds for a replica that no plan reached while two or more plans ran in a row, or whose committed history a later plan dropped: it refuses the leader's appends and snapshots, since nothing shows that the new history kept its committed entries. It logs the refused snapshot with `UnrelatedHistory` when two or more plans ran, and with `CommittedOverwrite` when one plan's history replaced its own below its commit marker. Its own committed entries above the first such plan's point are gone for good: that plan replaced them, and wiping the node only stops it from waiting for them. Rejoining as a wiped node is an operator action, and it wipes every group on the node. Before you wipe, check every other group the node holds a replica of in `/__ursula/metrics`. Do not wipe while the node may hold the only copy of another group's acknowledged writes: a group whose other voters are gated, stalled or down, or whose leader is this node. Do not wipe while another group on the node would lose its quorum without this replica: while it is rebuilt, each of its groups needs a quorum of the other voters, and a group whose other voters are gated stalls. Recover or wait for those groups first. A voter set that names nodes that never come back can block every plan. The tool needs a majority of each set's voters, because it cannot rule out that the set committed, and the plan's leader must win one. When those nodes are gone for good, no plan can prove what the group committed: restore the group from a backup. **A replica that cannot lead knows more is committed.** A learner, or a promoted voter whose cut would make it a learner again, can know a later committed entry than every replica that could lead. Commit markers are not synced on their own, so after a power loss the voters can still hold that entry in their logs while their markers lie below it. The tool then plans a cut that keeps every log up to that entry, led by a voter whose log reaches it, and the leader refuses the plan unless it holds exactly that entry. When no voter it reached holds the entry, or a replica reports a committed or applied index above every log that could lead, the tool stops with an error naming that replica: a plan from any voter would overwrite those committed entries. Each replica also refuses an append, or a snapshot install, that would replace entries at or below its commit marker. Waiting does not help, since the voters' logs no longer hold the entry. Bring back a voter that was not reached, if one may hold it, and plan again. Otherwise restore the group from a backup. A body of the 0.7 form answers `422`. A second plan replaces a first one that was never satisfied, as long as `floor` counts the replicas the first one holds. A voter that was never gated does not report `recovery_stalled`. Its readiness answers `503` with `raft_replica_unready`, and `raft_maintenance.group_issues` shows `leader_unknown` for the group. With `fsync = "always"`, acknowledged appends survive a host crash, but the newest journal segment can still hold bytes written after the last `fsync`: an unacknowledged batch, or a commit or truncate marker. Recovery refuses damage in sealed segments. In the newest segment it removes an incomplete final frame without gating anything. A complete frame that fails verification, such as a hole, a zero-filled tail or a torn sector inside an unacknowledged batch, looks the same as media corruption of an acknowledged write. Recovery therefore records the recovery gate of every group on that core before it cuts the frame, and those groups rejoin as described above. A single-voter group has no other replica to recover from, so it needs a plan after every host crash under `never`, once its gate reports the group stalled, and every such plan gives every stream of the group a new `Stream-Incarnation`. `Stream-Previous-Incarnation` and `Stream-Inherited-Offset` on `HEAD` then tell clients how much of each stream survived. Use `fsync = "always"` for single-voter groups whose clients keep state. Under `always` it needs the call only after a host crash that leaves a complete invalid frame in the newest segment. So does a group whose voters lose power together, for example in a rack power loss, when that gates a majority of them. No acknowledged write is lost in these cases, but only an operator can open the gates. `POST /__ursula/cluster-probe` is served on the cluster plane, but it is machinery rather than an operator surface. Nodes send each other probe bodies through it at their peer `url` to verify cluster-plane egress. The cluster plane runs no ingress admission, so a node under write pressure still answers it. ## Memory without retention Per-stream replicated state, on every replica, is at most about 32 KiB plus bounded producer state, beyond the unflushed payload itself, without retention. It grows neither with the number of messages nor with the length of the cold history. The bounds that make this hold are always on: one receipt per producer with 7-day idle-producer expiry and a 4,096-producer cap, unique stream incarnations with incarnation-scoped cold objects, external payload locators committed first and indexed after, and a compact hot window with no per-message bookkeeping, so cold admission and the flush thresholds charge hot payload only. Snapshot bodies of at least `runtime.external_payload_min_size` go to the cold tier when a cold backend is configured, up to 1 GiB uploaded in parts. Retention stays available as a storage-cost tool, but it is not the way to bound memory. The per-group gauges under `group_state_gauges` in `/__ursula/metrics` show what each group holds. ## Usage export `GET /__ursula/usage` returns committed counters and current gauges from local replicated state, aggregated by bucket across this node's Raft groups: ```json { "version": 1, "node_id": 1, "write_unit_bytes": 10240, "buckets": { "tenant-a": { "committed_append_bytes": 24576, "committed_records": 3, "committed_write_units": 4, "retained_bytes": 24576, "stream_count": 1 } }, "raft_group_count": 1, "staged_snapshot_parts": [ { "raft_group_id": 0, "term": 7, "counting_finished": true, "bytes": { "tenant-a": 33554432 } } ] } ``` The raw byte and record counters are pricing-neutral facts. `committed_write_units` is a convenience derived meter: every committed, non-deduplicated create or append contributes at least one unit and rounds its canonical payload up independently by `write_unit_bytes`. Consumers must validate both `version` and `write_unit_bytes` before interpreting it; the unit is data in the contract, not encoded into the counter name. The same unit is persisted in Raft snapshots, and a build refuses to restore a snapshot carrying a different unit. Changing it therefore requires a new contract version and an explicit state migration, never a configuration edit. Ursula does not attach currency, a price, or an account owner. Counters are monotonic and survive snapshots, stream deletion, and bucket purge. `retained_bytes` and `stream_count` are current-state gauges and may decrease. A node serves local applied state, so replicated consumers should poll every voter and use their own completeness policy rather than treating one response as a cluster-wide linearizable read. `staged_snapshot_parts` reports the staged snapshot parts that no commit has used yet, separately from the replicated fields: for each Raft group the answering node leads, the group, the node's term and the bytes of staged parts per bucket. A new term's totals start at the node's first part, commit or discard in the group, or when its Raft metrics show it leads, whichever comes first. The node then lists the group's part directories in the background and reports `counting_finished: false` until that listing ends. From then on it follows every part it stages, every commit, discard and delete, and every part the orphan sweep reclaims. The bytes are exact except for parts that were still uploading to the previous leader when the term began, parts a failed delete may have removed, and a part the sweep deletes while the seed listing runs, each until its stream is listed again. A collector polls every voter. For each group it keeps the report with the highest term, so a deposed leader that has not yet seen the new term is not counted twice. It treats the total as partial when a kept group has not finished counting, or when no report names one of the `raft_group_count` groups, for example a group without a leader. `ursula_proto::admin::UsageReport::staged_part_bytes` applies these rules. `node_id` is the answering node's id. Ursula bounds staged parts per stream only. A bound per bucket, or a charge for staged parts, is the operator's policy. ## Tenant offboarding (bucket purge) `DELETE /__ursula/purge/{bucket}` erases one tenant's complete bucket domain: every stream in every Raft group, the bucket, cold chunks and indexes, external payloads, and any uncommitted object-store staging orphan below `{bucket}/`. The replicated purge also installs a permanent bucket-erasure fence in every group. It retains only the bucket's aggregate usage counters, because deleting a counter before an asynchronous meter observes it would erase committed billable work. Purge is served only on the admin listener (`server.admin_listen`, loopback `127.0.0.1:4438` by default); the client listener answers `404`. On Kubernetes the chart keeps the admin listener on pod loopback, so reach it through `kubectl port-forward pod/ 4438:4438` or `kubectl exec`; any one voter accepts the request for the whole cluster. The response is a completion report an operator can attest to: ```bash curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X DELETE http://127.0.0.1:4438/__ursula/purge/tenant-a # {"bucket":"tenant-a","removed_streams":12,"groups_with_streams":[0,3,5],"cold_gc_entries_reclaimed":12,"cold_gc_pending_entries":0,"cold_gc_complete":true,"cold_gc_error":null,"bucket_prefix_absent":true} ``` Semantics: - **Retry on `503`.** A Raft group whose leader is unknown or changing answers `503` with `Retry-After`, never a redirect to a peer. Because purge is idempotent, retry the same request. - **Idempotent and resumable.** Re-purging an erased bucket returns the same report shape with zero removals. A purge interrupted mid-way converges on re-run: the permanent fence and stream removal are replicated commands per group, while cold reclamation and the complete bucket-prefix erasure are repeatable object-store operations. - **Permanently fenced.** Commands linearized after purge cannot create the bucket or append to its streams. The fence survives Raft snapshots and restore; data-plane access continues to conceal the erased namespace as `404 Not Found` instead of recreating it. - **Isolated.** Other buckets' streams, offsets, checksums, snapshots, and object-store prefixes are untouched. Every cold object, shared packs included, lives under its own bucket's `{bucket}/` prefix. - **Accounting residue.** The retained entry contains only aggregate counters and zero-valued content gauges, not stream data. Ursula does not currently provide an acknowledgement protocol for erasing it after an external meter persists the final delta. Use opaque bucket IDs if retaining the identifier itself would violate an erasure requirement. - **Physical absence proof.** `cold_gc_entries_reclaimed` is diagnostic only. After a second replicated purge proves every group's bucket-specific GC queue is empty, Ursula recursively deletes the complete `{bucket}/` erasure domain and lists the authoritative object store to prove it contains no files. Only `cold_gc_complete=true`, `cold_gc_pending_entries=0`, `bucket_prefix_absent=true`, and no error constitute the proof. If object storage is unavailable or still lists a file, the response remains incomplete and a later idempotent request retries the proof. - **Backups.** A purged tenant may persist inside backups taken before the purge until those backups age out; restore replays whatever the manifest contains. Deleting old backup sets is part of completing an offboarding with erasure obligations. - **Raft snapshots and logs.** Each replica drops the purged streams from its current Raft snapshot and its Raft log within `raft.deleted_data_max_age`. See [Deleted data](#deleted-data) for what that bound covers. ## Deleted data Deleting a stream and purging a bucket remove the stream from its group's state at once. An expired stream reads as gone at once and leaves its group's state within about a minute, as described below. Copies of its records can stay for a while in each replica's Raft snapshot, its Raft log and the log's files, and in older snapshot objects of an S3 snapshot store. They are cleaned up in the background. `raft.deleted_data_max_age` (1 hour by default) bounds how long they stay. Within that time after a replica applies the removal: - The replica's current snapshot no longer holds the stream. That covers the snapshot in memory, the snapshot file in the Raft log directory and, with an S3 snapshot store, the snapshot object that file names. - The replica's Raft log holds no entry with the stream's records. - No file in the replica's Raft log directory holds the stream's records, as long as the journal can copy the entries it keeps fast enough. A purged entry's bytes stay in its journal segment until the segment is deleted. A segment that has held such bytes for a quarter of `raft.deleted_data_max_age` since the purge is sealed, the few entries other groups still keep in it and in the segments before it are copied into the newest segment at the rate below, and the segments are deleted. At the defaults that takes far less than the rest of the bound. With a short `raft.deleted_data_max_age`, a slow drive interval and large live logs to copy, it can take longer. - The snapshot store no longer holds the snapshot objects the replica used before. Removed bytes count toward a group's next snapshot, so a large removal leaves the snapshot and the log within minutes. A small removal makes its group take a snapshot a quarter of `raft.deleted_data_max_age` after the replica applied it, and the replica then purges its Raft log through that snapshot. For a large and a small removal alike, the segments that held the purged entries go a quarter of `raft.deleted_data_max_age` after that purge, plus the time the copies take. Cold chunks, cold indexes and external payloads are deleted by cold GC, usually within seconds, and a bucket purge reports when they are gone. Each group's leader removes the streams that expired in its group in the background, every minute, or every quarter of `raft.deleted_data_max_age` if that is shorter. Each time it writes up to 16 small `ExpireStreams` Raft entries per group, and starts no more after 250 ms. Each entry removes up to 256 expired streams on every replica, earliest expiry first. A pass that stopped at its 16 entries or its 250 ms in a group runs again a second later, with 50 ms per group so that client writes keep their latency, until the group's backlog is gone. A pass that stopped on an error waits for the usual interval. No other write removes an expired stream it does not name, while a request that names one removes it at once. The leader removes a stream once its own clock is a second past the stream's expiry, so keep the nodes' clocks in sync. As for a deletion, the time counts from when a replica applies the entry that removed the stream. The removal also queues the stream's cold objects for cold GC and takes the stream out of `retained_bytes` and `stream_count` in `/__ursula/usage`. `ttl_expiry_sweeps` in `/__ursula/metrics` counts the entries the node's leaders wrote, `ttl_expiry_swept_streams` the streams they removed, and `ttl_expiry_sweep_due_groups` the groups the latest pass left with expired streams. Freeing segments this way copies the live entries of quiet groups forward about once per quarter of `raft.deleted_data_max_age`, which shows in `wal_rewritten_bytes`. The snapshot driver drives the copies at most once per `storage.snapshot.drive_interval`, an eighth of `raft.wal.segment_size` per core each time, so a mass removal does not saturate the disk. At the defaults that is 8 MiB per second per core with inline snapshots and 1.6 MiB per second with an S3 snapshot store. A group that keeps more than a quarter of a segment there is reported in `wal_lagging_groups`, and the driver snapshots it first, or purges its log through its snapshot when it applied nothing since. `wal_dirty_segments` counts the segments that hold purged entries. The first passes after an upgrade or a restart find all the old purged entries at once and copy the live entries of every kept segment forward, up to about twice a core's live log, and snapshot the groups reported lagging, before the journal settles. The time counts per replica, from when that replica first applies the removal. A replica that is down when the removal commits keeps its copies until it is back. It then applies the removal from the leader's log, or installs a newer snapshot, and the time counts from there. For the snapshot and the log, a restart after the replica applied the removal does not extend the bound: a removal it applies again while it replays its log at startup is due at once, and it purges its log through the snapshot it restored. The log's files follow the purge. Purged entries in a journal segment keep their age across a plain restart, since it is read from the segment's last change, but two cases count from the restart. Entries the replica purges only after it restarts leave the files a quarter of `raft.deleted_data_max_age` after that purge. A restart that rewrites the journal's segments, after a host crash under `raft.wal.fsync = "never"` or when the journal ended in a damaged frame, starts their age again. One exception: a replica that loses the removal from its log in a host crash under `raft.wal.fsync = "never"` learns it again from the leader, and its time counts from then. The snapshot driver enforces the bound and reports its gauges. It runs on every node that runs Raft, a single node included, and cannot be turned off. A single node keeps its snapshots inline whatever `storage.snapshot.backend` says, and never refuses writes for its log's pressure. While the driver purges a group's log through a snapshot, the group takes no new snapshot until that purge lands, which is usually at once. An operator's `POST /__ursula/raft/{group}/snapshot` waits for it, and answers `503` with `Retry-After` when the wait passes 10 seconds. An operator's `POST /__ursula/raft/{group}/purge?upto=` follows the same rule: while a snapshot of the group builds, OpenRaft has not published the one it built, or an earlier purge of the group has not landed, it answers `503` with `Retry-After`. Send it again then. Under write pressure the driver first snapshots the groups that free the most log, and groups that owe a removal wait their turn among the due groups. They lead the driver's passes again once the node's log is below three quarters of its budget. With an S3 snapshot store, each node writes its snapshot objects under `/node-/group-/`, each under a name no object had before, and only that node deletes them. A node that installs a snapshot from a peer writes its own copy of the object before it uses it, so no node ever uses another node's object for long: a peer reads the object only while the leader's send of it is in flight. What a node may still use of its own objects lives in its memory: the object its current snapshot names, the object a build or an install copy is writing, and the object a send of its snapshot names until that send returns. A snapshot GC pass lists the node's directory for the group and deletes every object in it that the node no longer uses. So a node deletes the object it replaced at its next pass once no send of it is in flight, which a send's `raft.install_snapshot_timeout` bounds, and with it whatever an earlier build or install left behind. A write the store answered with an error can still land later, so a node runs another pass one `raft.install_snapshot_timeout` after such a write and after a restart. GC reads no modification times and assumes nothing about clocks. Each build runs a GC pass as it ends. The snapshot driver runs the passes a group still owes after its snapshot builds, and holds them back while the node's log is under pressure or the store health gate reports trouble. `snapshot_gc_deleted_objects` in `/__ursula/metrics` counts the objects GC deleted, and `snapshot_gc_failed_passes` the passes that failed, which run again later. A node restores only a snapshot object of its own directory. Started with another node's Raft log directory or with another `storage.snapshot.s3_prefix`, it refuses to open each group whose snapshot names another object, and logs `ForeignKey` with the object and its own directory. Start it with its `raft.wal.path` emptied to rejoin as a wiped node ([Rejoining a wiped node](#rejoining-a-wiped-node)). The bound does not cover: - Retention advances. Records a stream's retention trims are not a removal and leave the snapshots and the log on the group's usual cadence. - A leader whose follower lags. A leader purges no log entry a follower still needs, so a slow or unreachable follower holds the leader's log through the removal until it catches up or the leader sends it a snapshot. - A group whose Raft is not running on the replica, for example while the replica restarts or after it stopped on a storage error. The time keeps counting, and the driver cleans the group up once its Raft runs again. Until then its live log entries also keep the journal segments they share with other groups. - Journal segments while the snapshot store is in trouble. While the store health gate holds routine snapshots back and the node's log is not under pressure, a group that keeps more than a quarter of such a segment and applied entries since its snapshot is not snapshotted, so the segment stays until the store recovers or the log comes under pressure. - A replica that is down for longer than `raft.deleted_data_max_age`. It catches up when it is back, and its time counts from when it applies the removal. - Sustained write pressure. While the node's log stays above three quarters of its budget, the driver snapshots the groups that free the most log first and holds snapshot GC back, and the time can run past the bound. - A node that is down. Its snapshot objects stay until it is back and runs its GC passes. - A node that left the cluster. It keeps `/node-/`, with every snapshot object it wrote. No other node uses them. Delete that prefix once the node is stopped for good. - Object versioning. On a bucket with versioning, a deleted snapshot object stays as a noncurrent version until a lifecycle rule expires it. - Backups taken before the removal. Delete them to complete an erasure. - Freed disk blocks and memory. Ursula deletes and replaces files but does not overwrite the blocks they used, and read caches keep blocks of a deleted stream until they are evicted. Use encrypted volumes and buckets where that matters. - A group whose committed application failed. It keeps its whole log for replay until it is repaired. `raft_deleted_data_oldest_age_ms` in `/__ursula/metrics` reports how long ago the oldest removal on the node applied whose stream its current snapshot or its Raft log still holds, and `raft_deleted_data_overdue_groups` counts the groups whose snapshot still holds one past a quarter of `raft.deleted_data_max_age`. `deleted_data` lists each such group with what it still holds: `owed` or `replayed` while its snapshot holds the stream, with the bytes and age, and `purging` while only its log does, with the age and the index the log is purged through. Alert when `raft_deleted_data_oldest_age_ms` nears half of `raft.deleted_data_max_age`. ## Cleaning S3 Cold data lives under the `storage.cold.root` prefix. Benchmark and test runs should use a date-stamped root so cleanup is one prefix delete with standard S3 tooling: ```bash aws s3 rm "s3://my-ursula-bucket/ursula-test-20260518T000000Z" --recursive ``` There is no automatic retention policy. Application checkpoints and explicit stream-retention advances control which cold objects become eligible for GC. After a retention advance, each group's leader deletes the stream's cold objects that lie wholly below the retained offset (chunks and offloaded external payloads, in whole 64 MiB cold-index pages, so up to one page just below the retained offset can remain until retention moves past it) once a 5-minute grace has passed, so reads planned before the advance still complete; its cold-index cursor visits 16 streams per group per minute, so on groups with many streams collection can lag further. The grace restarts after a leader change. Retention is a storage-cost tool only: replica memory does not depend on it (see [Memory without retention](#memory-without-retention)). Cold GC deletes a path a queued entry names only if it lies in the entry's own namespace: an exclusive chunk or an external payload of a stream in the group, or a pack of the group, each in the entry's own bucket. It refuses any other path, such as a snapshot object, a cold-index page or another bucket's object, and it refuses an entry for a whole stream that is not of the entry's bucket or is placed in another group. Only a corrupt or hand-edited backup can queue these. A refused path or stream stays in the store, the entry's other paths are deleted as usual, and the entry leaves the queue, so GC does not stall. Each refusal is logged as an error and counted in `cold_gc_foreign_paths_refused` in `/__ursula/metrics`. Alert on any increase, and check the backup the group was restored from. The check is on the namespace, not on liveness. It does not prove that nothing references an object of the right shape in the group's own namespace. Ursula's own entries are safe because each one is queued only after the state drops its references, and every object name is unique. A leader proposes an object it uploaded only in the leadership it uploaded it in. When that leadership ends between the upload and the proposal, even by a re-election of the same node, the proposal is refused before anything of it is written and the leader deletes the object at once. Objects of writes whose outcome is unknown stay until the orphan sweep finds them unreferenced and more than a day old. ### Incomplete multipart uploads Ursula writes large cold objects with S3 multipart uploads. A crash or shutdown in the middle of a cold write leaves an incomplete upload behind. Ursula never aborts these uploads, and S3 bills for their uploaded parts until they are aborted. Snapshot commits from parts make long cold writes more common, so these leftovers show up more often. Add an `AbortIncompleteMultipartUpload` lifecycle rule to every cold bucket, for example after 1 day: ```bash aws s3api put-bucket-lifecycle-configuration --bucket my-ursula-bucket \ --lifecycle-configuration '{ "Rules": [{ "ID": "abort-incomplete-multipart-uploads", "Status": "Enabled", "Filter": {}, "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 1} }] }' ``` This call replaces the bucket's whole lifecycle configuration. If the bucket has other rules, add this rule to them. MinIO and other S3-compatible stores need their own equivalent rule. Check the store's lifecycle documentation for how to expire incomplete multipart uploads. ## Backup and disaster recovery `ursulactl` provides a verifiable whole-cluster backup and restore workflow: ```bash # Create: one MessagePack snapshot object per raft group + manifest.json, # into a local directory or s3://bucket/prefix. ursulactl backup-create --config cluster.yaml --location s3://backups/ursula/2026-07-25 # Verify: manifest completeness, byte sizes, BLAKE3 checksums, and deep # state-machine validation of every snapshot. Touches no cluster. ursulactl backup-verify --location s3://backups/ursula/2026-07-25 # Restore: into a FRESH, EMPTY cluster with the same raft group count and the # source cluster's server.cluster_id. ursulactl restore --config new-cluster.yaml --location s3://backups/ursula/2026-07-25 ``` An `s3://` location uses the AWS SDK credential chain and `AWS_REGION`. For MinIO or another S3-compatible store, also set `AWS_ENDPOINT_URL`, for example `AWS_ENDPOINT_URL=http://minio.internal:9000`. The recovery contract: - **Consistency boundary.** Each group export is the deterministic state-machine snapshot the raft snapshot path persists: internally consistent per group while writes continue. Cross-group consistency is not promised. The boundary is per stream. Acknowledged writes in the exporting replica's applied state are included whether or not they were cold-flushed, so **RPO is bounded by export time**, not by the cold-flush interval. `backup-create` asks every node and keeps the reply with the highest group commit index. - **`ursulactl restore` is the only supported restore.** Do not restore a node or a cluster from volume-level copies, such as disk or EBS snapshots of the Raft WAL or the data directories. A cluster brought back from such copies keeps its history ids, so its streams keep their `Stream-Incarnation` tokens while their tails go back, and new writes then put other bytes at offsets the old cluster already served under the same tokens. Clients that kept state, and caches that stored immutable reads, take those bytes for the ones they hold. A single rolled-back volume also breaks Raft itself. A restore through `ursulactl restore` starts a new history, so every token changes. - **Keep the cluster id.** Give the target cluster the source cluster's `server.cluster_id` (Helm `server.clusterId`) before its first start. Clients keep the id with their state and treat a different id as another deployment, so a restore under a new id looks to them like a move to another cluster, not a restore. A target node whose WAL was created with another id refuses to start, so set it before the nodes first run. - **Fresh identity on restore.** Restore replays each snapshot as one replicated write on the target cluster, which keeps its own raft membership and log identity. Nothing from the source cluster's raft metadata is reused. Groups that already hold buckets refuse the import with `409`, and group-count mismatches fail closed before the first import. Group commit indexes and per-stream append counters restart under the new identity. Stream bytes, offsets, close/TTL state, published snapshots, retention floors, and producer dedup states are preserved exactly. - **A restore is a new history.** Each group's import starts a new history of the group, so every restored stream answers with a new `Stream-Incarnation`. A client that holds a stream's token from the source cluster gets `412` with the new token, and `HEAD` names the source token as `Stream-Previous-Incarnation` with the restored tail as `Stream-Inherited-Offset` (or, for a stream the source had not written since a rewrite of its own, the token from before that rewrite): what the client derived from bytes below it still holds. Every producer epoch claimed on the source is closed: a producer's next new write of it answers `403` with `Producer-Outcome: epoch-closed;by=history` and the last seq the backup kept, and the producer claims a new epoch. The import requires `x-ursula-backup-blake3`, the BLAKE3 digest of the group's backup object, which `ursulactl restore` sends. A repeat of an import that applied answers `200` with `"already": true` and starts no second history, so a restore interrupted by a timeout or a leader change can run again. - **Cold objects are part of the backup set.** Snapshots reference cold-tier chunks, cold-index pages and external payloads by object key. The backup holds the references, not the objects. The restored cluster's cold store must hold the same objects under the same keys: point it at a copy (see [Copying cold objects](#copying-cold-objects)). `backup-verify` validates snapshots but does not dereference cold objects. `restore` asks the target cluster, before the first import, to check every group's references against its own cold store (`POST /__ursula/backup/group/{group}/cold-check`). If any object is missing, it stops with nothing imported and prints how many are missing and the first keys. - **Format compatibility.** Backups have a format version of their own: `format_version` 5 in the manifest and in `/__ursula/backup/info`, and the first field of every group object, which is compact MessagePack. It changes only when what a backup holds changes (the manifest, the group snapshots or the cold objects they reference), not with every format epoch. Ursula 0.9 writes and reads backup format 5. It does not restore backup format 4, which Ursula 0.8 wrote. `ursulactl` refuses a manifest or a target cluster of another backup format before the first import, and a server refuses to import a group of another backup format (`400`). - **RTO** is dominated by transferring and importing the group snapshots. Each import is a single replicated write per group. The recovery drill runs in CI (`backup_restore_drill_preserves_streams_and_allows_continued_appends`): write, snapshot, retain and close, back up, destroy the cluster, restore, verify checksums, offsets and boundaries, and continue appending. What you can additionally rely on: - **Quorum durability.** Acknowledged writes survive as long as a majority of voters survives. Three voters across AZs tolerate any single-AZ outage. - **Cold-tier durability.** Once flushed to S3, a chunk inherits S3-grade durability. The unflushed window is bounded by the flush interval (seconds by default). - **Format epoch 5, no migration.** The Raft WAL, snapshots, the object-storage prefix and the Raft gRPC protocol carry format epoch 5. A node refuses data and peers of another format epoch and does not migrate them. Backups have a format of their own (see Format compatibility above). Node-level loss: replace the host with the same `node_id` and cluster config. Each group's leader catches the replacement up from the other voters, and it stays a voter (see [Rejoining a wiped node](#rejoining-a-wiped-node)). `ursulactl wait --node N` returns once the replacement has caught up in every group, and `ursulactl verify-cluster` then requires every configured voter to be present and caught up. `ursulactl wait-ready` checks the whole cluster: leaders for every group, complete voter sets and `200` from every node's `/__ursula/ready`. Run it before declaring the recovery done. Total-cluster loss: restore the latest verified backup into a fresh cluster as above. ### Copying cold objects A restored cluster reads cold data under the keys the source cluster wrote, relative to `storage.cold.root`. Copy the source's objects into the target's root after the target has started and written its `URSULA_FORMAT_EPOCH` marker, and before `restore`. Skip two things: - `URSULA_FORMAT_EPOCH`, the marker. The target's own marker stays. - `snapshots/`, the source's Raft snapshot objects. The target writes its own there. With a custom `storage.snapshot.s3_prefix` inside the cold root, skip that prefix instead. With the AWS CLI, from the source root to the target root: ```bash aws s3 sync s3://my-bucket/source/cold/ s3://my-bucket/target/cold/ \ --exclude 'URSULA_FORMAT_EPOCH' --exclude 'snapshots/*' ``` For MinIO or another S3-compatible store, add `--endpoint-url`. The copy must match the backup, so the source must not change its cold objects in between. Stopping writes is not enough: an idle cluster still compacts small cold chunks and deletes the ones it replaced a few minutes later. If the source cluster still runs, stop its nodes once the backup verifies, then copy. `restore` checks that the copy is complete. Never point two clusters at one prefix, and never point a new cluster at the source's prefix in place. After a restore both clusters hold the same streams under the same keys, but each one deletes cold objects by its own state alone: the cold GC after a delete, a retention advance or a compaction, the orphan sweep for unreferenced objects older than a day, and each node's snapshot GC of its own directory under `snapshots/`. Two clusters on one prefix delete objects the other still reads. A copy also leaves the source cluster and its prefix as they were, so the source stays a way back until clients switch. ## Upgrading to 0.9 **0.9.0 is a breaking release.** It changes the storage and protocol format to format epoch 5 and the backup format to 5. A 0.9 node refuses epoch-4 Raft WAL directories and object-storage prefixes at startup and refuses 0.8 peers, and a 0.8 node refuses 0.9 peers. 0.9 does not restore 0.8 backups. There is no in-place upgrade, no rolling upgrade, no backup migration and no rollback. If you cannot move your data yet, stay on 0.8.2. 1. **Install 0.9 fresh.** Use empty PVCs and never-used prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.8, or a failed upgrade attempt, wrote to. Give the 0.9 cluster its own `server.cluster_id` (Helm `server.clusterId`). Its streams are new streams, so a client that compares cluster ids then knows that its 0.8 state does not carry over. 2. **Move your data through your applications.** Read the streams you keep from the 0.8 cluster with your own clients and write them to the 0.9 cluster through the HTTP API. A stream written again is a new stream: a new `Stream-Incarnation`, offsets from its first written byte, and only the producer states, TTL and published snapshots that the application sets again. 3. **Switch the clients** to the 0.9 cluster. A client that keeps state, such as offsets or producer sequences, starts it again from the streams of the 0.9 cluster. 4. **Drop the removed snapshot settings** from any config and Helm values you carry over from 0.8. Remove `raft.snapshot_logs_since_last` (Helm `raft.snapshotLogsSinceLast`) and any `storage.snapshot.drive_interval = "0s"` (Helm `snapshotStore.driveIntervalMs: 0`). 0.9 refuses both at startup, and the chart's schema refuses them at render. The snapshot driver always runs. To keep the 5,000-entry bound a 0.8 single node had, set `raft.snapshot_backstop_logs = 5000` (Helm `raft.snapshotBackstopLogs`). 5. **Check the snapshot store's request budget.** S3 snapshot requests no longer follow `storage.cold.s3.timeout` and `max_retries`. They use `storage.snapshot.s3_timeout` (`10s`) and `storage.snapshot.s3_max_retries` (`1`), Helm `snapshotStore.s3TimeoutMs` and `snapshotStore.s3MaxRetries`. A follower's install copy, up to four of these requests, must fit inside `raft.install_snapshot_timeout`. A carried-over cluster config with S3 snapshots and an install timeout below that bound, 88 seconds with the defaults, is refused at startup with `SnapshotCopyOutlastsInstall`. Raise `raft.install_snapshot_timeout` or lower the snapshot store's timeout or retries. The timeout covers one attempt, and a PUT sends the whole object in one attempt, so keep it above a healthy PUT of your largest snapshot. 6. **Optionally scope credentials per node.** With an S3 snapshot store, a node writes and deletes only under `/node-/` and reads the rest of `/`, so an IAM policy per node can enforce that. 7. **Retire the 0.8 cluster** once no client reads it. Then delete its volumes and its object-storage prefixes, among them its snapshot objects under `/group-/`, where `` is the 0.8 cluster's `storage.snapshot.s3_prefix`. 0.9 never reads or deletes them. `helm upgrade` from 0.8 stops at the first pod. The new pod finds the 0.8 data on its volume and exits at startup, so no second voter is replaced and the refused pod writes nothing. Recover with `helm rollback`. With `OnDelete`, also delete the refused pod after the rollback so that it starts from the 0.8 template again. The 0.8 cluster and its prefix stay as they were, so they remain a way back until clients switch. ## Upgrading to 0.8 **0.8.0 is a breaking release.** It changes the storage and protocol format to format epoch 4. A 0.8 node refuses epoch-3 Raft WAL directories and object-storage prefixes at startup, refuses 0.7 peers, snapshots and backups, and `ursulactl restore` refuses 0.7 backups. There is no migration, no in-place upgrade, no rolling upgrade and no rollback. If you cannot move your data yet, stay on 0.7.6. 1. **Install fresh.** Use empty PVCs and never-used prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.7, or a failed upgrade attempt, wrote to. 2. **Set the cluster id.** Set `server.cluster_id` on every node of a multi-node cluster, the same on every node and different for every deployment a client can reach. The chart requires `server.clusterId` for every install, a single replica included. 3. **Choose the network planes before the first install.** The chart serves Raft on `server.ports.cluster` (`4439`) by default. Do not change `server.clusterListen` or `server.ports.cluster` on an initialized cluster. Outside the chart, set `server.cluster_listen` and `client_url` together on every node. 4. **Check the inflight body budget.** A node with a cold backend needs `server.http_inflight_body_size` of at least 64 MiB, or it refuses to start with `InflightBodyBudgetTooSmall`. 5. **Lower the gateway's request limit.** A gateway whose `--max-request-body-bytes` (`gateway.maxRequestBodyBytes`) is above 33554432 refuses to start. Larger snapshots go up in parts. 6. **Move data by export and re-import.** The applications that own the data read it from 0.7 and write it into 0.8. For the SQLite VFS, export each database with 0.7 using `VACUUM INTO` and import the file with 0.8 ([Moving from 0.7](/docs/0.9/examples/sqlite-vfs#moving-from-07)). 7. **Update clients and proxies.** Error answers have a JSON body, `x-ursula-not-applied` is gone, a leadership loss after a proposal answers `503` with `OUTCOME_UNKNOWN`, and `Stream-Incarnation` changes on a restore or an accepted loss. A proxy in front of Ursula follows the intermediary rules of [write outcomes](/docs/0.9/specs/extensions#111-write-outcomes). Also check clients and tooling for these changes: - A snapshot publish above 32 MiB sent straight to a node answers `413`. Send it in parts. - A failed cold-tier read answers `502` `COLD_STORE_FAILED`, where 0.7 answered `500`, and a write that needs a cold backend the node lacks answers `501` `NO_COLD_STORE`, where 0.7 answered `503`. - `Producer-Seq` on `200` and `204` is the epoch's highest applied seq, not the request's. - A create with producer headers answers `400`. Create the stream, then claim an epoch. - A producer close of a closed stream answers `409`. Only a close without producer headers is idempotent there. - A retention request at or below the retained offset answers `204` and changes nothing, where 0.7 answered `409`. - An SSE session ends on a delete and recreate, since it is pinned to the token its first response announced. - Read and snapshot `ETag`s have a new form. - `/__ursula/metrics` no longer reports `receipts`, `receipt_items` or `max_receipt_items_per_stream`. - The gateway no longer uses a system proxy from `HTTP_PROXY` for its upstreams. - The chart's Helm test pod's name label is now `-test`. - `ursulactl` reads metrics at a manifest node's `client_url`, never at `http_url`. Add `client_url` to custom manifests. 8. **Update recovery runbooks.** `accept-unsynced-loss` takes a recovery plan, and `ursulactl recover-group --accept-loss` computes it. A replica whose committed entries a plan dropped waits for its node to be rejoined as a wiped node. See [Recovering after a host crash](#recovering-after-a-host-crash) for the checks before a wipe. Every behavior change, with what to change for each area, is listed in the [0.8.0 release notes](/docs/0.9/release-notes#080). 0.8.1 upgrades 0.8.0 in place with an ordinary rolling restart (`helm upgrade`), with no fresh install. See the [0.8.1 release notes](/docs/0.9/release-notes#081). 0.8.2 upgrades 0.8.0 and 0.8.1 in place the same way, once your Helm values no longer set the indexer it removes. See the [0.8.2 release notes](/docs/0.9/release-notes#082). ## Upgrading to 0.7 **0.7.0 is a breaking release.** It changes the Raft WAL format (format epoch 3) and removes the memory WAL, so the per-core disk journal is the only Raft WAL. 0.7 cannot read 0.6 data: a node refuses 0.6 Raft WAL directories and object-storage prefixes at startup, `ursulactl restore` refuses 0.6 backups, and a node will not join a cluster with 0.6 nodes. There is no in-place upgrade, no automatic migration and no rollback. - **If you run 0.6 with data you need to keep, stay on 0.6.2.** 0.7 provides no migration path from 0.6. - **Deploy 0.7 fresh:** empty PVCs and a never-used object-storage prefix. - **`helm upgrade` from 0.6 stops before a second voter is replaced.** Recover with `helm rollback`. Every behavior change is listed in the [0.7.0 release notes](/docs/0.9/release-notes#070). 0.7.6 upgrades 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4 or 0.7.5 in place with an ordinary rolling restart (`helm upgrade`), with no fresh install. See the [0.7.6 release notes](/docs/0.9/release-notes#076). To install 0.7: 1. **Install fresh.** Install with a new `s3.prefix` and empty PVCs. With cold storage off and S3 snapshots, also use a new `snapshotStore.prefix`. 2. **Never reuse a prefix** that an earlier release, or a failed upgrade attempt, wrote to. 3. **Update the configuration.** - Remove `raft.wal.backend`, `raft.wal.allow_volatile_multi_peer`, `raft.memory_bootstrap_marker_dir` and `raft.rejoin_probe`. A config file that still sets one fails to load. - Set `raft.wal.path` on every node of a multi-node cluster. 0.6 defaulted to the memory WAL, and a 0.7 node with `raft.peers` refuses to start without a WAL path. - Choose `raft.wal.fsync` (Helm `raft.walFsync`). The default `never` acknowledges a write once a quorum of voters holds it in the page cache, and a host crash can cost a replica its unsynced tail (see [Recovering after a host crash](#recovering-after-a-host-crash)). Set `always` to keep the 0.6 disk WAL behavior of one `fsync` per batch. - With Helm, remove `raft.storageMode`, `raft.allowVolatileMultiPeer` and `persistence.enabled`. Every pod gets a PVC for its WAL. The values schema refuses unknown keys under `raft` and `persistence`. - Keep `runtime.core_count` and `raft.group_count` explicit and unchanged (see [WAL routing configuration](#wal-routing-configuration)). 4. **Update runbooks and tooling.** `ursulactl` 0.7 removes the memory-WAL restart verbs `prepare-restart`, `restart-quiesce-capability`, `finish-prepared-restart`, `abort-prepared-restart`, `prepare-amnesiac-restart`, `repair-restarted-voter`, `prepare-recovery-handoff` and `classify-amnesiac`. A planned restart is drain, restart, wait and undrain ([Restarting a voter](#restarting-a-voter)), and a voter that lost entries is rebuilt by its group leaders. Custom tooling that calls admin mutations must send `x-ursula-process-incarnation` ([Admin endpoints](#admin-endpoints)). A values file that still sets a removed chart value fails validation, and nothing changes. Otherwise `helm upgrade` from 0.6 replaces the first pod, with the default `RollingUpdate` strategy and also with `OnDelete` plus the graceful rollout. That pod finds the 0.6 data on its volume and exits at startup, so no second voter is replaced. Recover with `helm rollback`. With `OnDelete`, also delete the refused pod after the rollback so that it starts from the 0.6 template again. The refused pod wrote nothing, so the old pods are unaffected. How a 0.7 node refuses data of another release, and what it prints, is listed under [Startup failures](/docs/0.9/troubleshooting#startup-failures). Every startup refusal happens before the node writes anything: - **Raft WAL directory.** `{raft.wal.path}/raft-log/FORMAT_EPOCH` records the epoch. A directory that holds data without the marker, or a marker of another epoch, is refused. The marker is written after every check passes. - **Object storage.** `URSULA_FORMAT_EPOCH` sits at `storage.cold.root` when the cold backend is S3, otherwise at the S3 snapshot namespace (`storage.snapshot.s3_prefix`) when snapshots live on S3. A namespace that holds objects without the marker is refused, so it must be used only by this cluster. With a memory cold store and inline snapshots there is no object-storage check. - **Peers.** Before it writes the markers, a node with configured peers asks each one, in parallel (2 seconds to connect plus 2 seconds for the answer, about 4 seconds at most), whether it speaks Raft protocol 3. A peer that refuses ends startup. An unreachable peer is skipped, because a fresh cluster starts its pods in parallel. - **Readiness.** Any later Raft protocol mismatch, inbound or outbound, is logged at error level and makes `/__ursula/ready` answer `503` with `"reason": "format_epoch_mismatch"` until the node restarts. This also holds for a 0.6 node that a 0.7 node contacts, including through the startup probe. Keep the peer lists of 0.6 and 0.7 nodes apart. - **Backups.** Backups use format version 3. `ursulactl restore` and `backup-create` refuse a cluster that reports another version, and a server refuses to import a group of another format epoch (`400`). Bucket IDs `snapshots` and any ID starting with `__` are reserved (`400`): `snapshots` is the default raft-snapshot namespace in the object store. ## Logs `RUST_LOG=ursula=info,ursula_runtime=info,ursula_raft=info` is the baseline. Bump to `debug` for one crate when chasing a subsystem: ```bash RUST_LOG=ursula_raft=debug ./target/release/ursula server ... ``` `debug` is verbose under sustained load, so redirect to a file. ## Committed application failures A stopped group requires [corrected-code replay](/docs/0.9/apply-failure-recovery). Retain the original WAL and referenced snapshot objects. Never skip a committed record to recover readiness. --- # Recovering stopped Raft groups A panic or invariant violation while applying a committed command stops that local Raft group. Other groups remain running. Isolation relies on Rust panic unwinding, so a process abort, OOM kill or machine failure is outside it. The failed command receives no successful application response, and the applied index does not advance past it. A normal stream rejection (for example, deleting a missing stream) remains a deterministic application result and does not stop the group. `/__ursula/metrics` reports `raft_groups[].apply_failure` with the log term, index, failure kind and diagnostic message. The affected group's `maintenance.running` is false. The maintenance report names the group with `apply_stopped`, so `ursulactl wait-ready` and rollouts refuse another disruption. The node's `/__ursula/ready` stays 200, its other groups keep serving, and planned leadership handoffs of those groups continue. Requests to the stopped group answer a typed `503`. Invariant violations can have partially modified memory: that state must not be read, flushed, or made into a snapshot. A previously captured snapshot only contains its earlier successful prefix. ## Supported recovery: corrected-code replay This procedure requires the original durable WAL and snapshot/cold objects. It does not repair physical corruption or recover data lost beyond the chosen WAL fsync policy. Do not truncate a committed record, edit its bytes, reset its applied index, copy a snapshot made from failed memory, or use an unsynced-loss operation to bypass a poison record. 1. Record the group, failing log term/index, binary revision and diagnostics on every replica. Stop writes to affected streams. A failed request may already be committed, so retain its producer sequence and idempotency information. 2. Preserve the WAL directories, snapshot pointers and referenced cold objects. Have a corrected binary ready before stopping the affected server to take a consistent filesystem backup. Keep node IDs, membership and storage paths unchanged. Do not delete the failed group's data to make the pod healthy. A restart with the faulty binary keeps the stopped group stopped and the other groups serving, so it is safe but does not repair anything. 3. Correct the deterministic application bug, keeping command encoding and semantics compatible with the retained history. Reproduce the failing record on an isolated copy. Verify the last acknowledged records and the failing command are replayed exactly once into fresh in-memory state. A deployment of the same faulty code is expected to stop at the same record. 4. Deploy the corrected binary to the affected replicas using their original volumes. A group with a live quorum can resume by normal replay and catch-up. If every replica stopped, bring back enough corrected replicas for quorum. Never create a replacement cluster or silently skip a committed command. 5. Before restoring traffic, require no `apply_failure`, running/recovery-ready group metrics, a current leader and applied progress through the recorded failing index. Read and compare previously acknowledged payloads and the failed request using its original identity before deciding whether to retry. If the failure occurs during initial WAL replay, the owner retains an `ApplyStopped` terminal group instead of a live Raft handle. Other groups can start and serve. Metrics retain the failing committed term and index and the last successfully applied prefix. Maintenance readiness refuses disruption while the node stays ready. Requests to the stopped group return `ApplyStopped` without repeatedly opening its WAL. A corrected compatible binary must restart and replay the original history before that group can serve again. Ordinary configuration, WAL-open and snapshot-restore errors still fail startup. They are not application failures and are not silently converted to isolated groups. There is no online skip/quarantine command in this recovery contract. Repair requires a corrected compatible binary. If none is available, keep the group stopped and preserve its data. Rolling a snapshot or command format to an incompatible binary is not a recovery method. ## Local drill Run `cargo test -p ursula-raft --lib apply_failure_tests`. It uses real disk WALs with `fsync=always`, a partial-mutation panic and an invariant violation, and a three-replica in-process Raft network. It verifies stopped-group isolation, unchanged applied boundary, retained records, same-code replay failure with healthy-group reads and writes on the same owner thread, and payload preservation after corrected-code replay. The three-replica drill waits for a surviving follower to elect a new leader and encounter the committed poison through normal Raft replication. The deterministic simulation `a_poison_record_isolates_its_group_until_corrected_replay` (`RUSTFLAGS="--cfg madsim" cargo test -p ursula-sim --lib disk_tests`) runs the same poison on every voter of one group while another group shares each node's core journal. It restarts a node with the faulty code and then replays every journal with corrected code. These are application-failure drills, not power-loss or Kubernetes Chaos qualifications. ## WAL retention cost A stopped group is excluded from snapshot selection and the lagging-snapshot queue. Reclaim moves each retained payload once into an immutable `frozen-*.seg` file beside the journal segments, using the existing journal encoding and disk interface. The shared journal keeps checksummed references. Later reclaim passes copy only those references, so the stopped group does not hold healthy groups' segments or repeatedly copy its payloads. The archive is synced and its directory entry made durable before the journal reference is synced. Only then can reclaim discard the original records. Recovery verifies every referenced archive and the exact retained log IDs. Missing or damaged archives fail startup. Segment continuity checks still apply. Preserve the entire WAL directory, including the frozen files, during repair. `wal_stopped_rewritten_bytes` counts payload bytes moved into archives. `wal_frozen_reference_rewritten_bytes` counts later reference copying. `wal_physical_bytes` includes retained archive files. Alert on `ursula.raft.apply_stopped` and monitor retained disk space while repairing the binary. The payload counter should stop growing once that group's retained records have been archived, even while healthy groups keep writing. Corrected code replays the same committed entries through the references. Archives are collected after no remaining physical journal record references them and the old segments' deletion is durable. Cleanup failures are retried without blocking healthy journal reclamation. During an unverified-tail repair run, cleanup waits until the affected segment has been durably reclaimed. A later clean startup validates the durably repaired journal before collecting orphans. A binary that predates frozen references rejects them on replay, including in the newest segment. Once references have been written, repair with compatible code rather than downgrading or deleting the archives. --- # Observability Ursula emits OpenTelemetry traces and metrics over OTLP, and keeps a built-in JSON metrics endpoint for quick inspection. Telemetry export is **off by default**: with no collector configured the server runs exactly as before and the hot read/write path pays nothing. ## Enabling OTLP export Set the standard OpenTelemetry OTLP exporter settings before starting `ursula`: ```bash # Send traces and metrics to a collector (Tempo, Jaeger, the OTel Collector, …). export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4318" # Optional: head-sampling ratio for new root traces (0.0–1.0, default 1.0). export OTEL_TRACES_SAMPLER_ARG="0.1" cat > /tmp/ursula-otel.toml <<'TOML' [server] listen = "0.0.0.0:8080" TOML ursula --preset default --config /tmp/ursula-otel.toml ``` The exporter uses OTLP over HTTP/protobuf (port `4318`), reusing the server's HTTP stack rather than opening a second gRPC tree alongside Raft. When the endpoint is unset, the process stays on the stderr `fmt` logger driven by `RUST_LOG` (default `info`). Spans batch-export continuously and metrics export on a fixed interval (`OTEL_METRIC_EXPORT_INTERVAL`, milliseconds). A final flush on clean shutdown is only a backstop. Don't rely on it to observe data. ### Identifying the node Every span and metric carries the process's OpenTelemetry **resource**. The server sets `service.name=ursula` and, when running with a Raft node id, `service.instance.id=` so a cluster's nodes are distinguishable. Add more resource attributes (e.g. `host.name`, region, AZ) with the standard variable. They stay bounded by cluster size, so this is a resource dimension, not a high-cardinality label: ```bash export OTEL_RESOURCE_ATTRIBUTES="host.name=$(hostname),deployment.environment=prod" ``` ## Traces A single request produces one boundary span (`http.append`, `http.read`, or `http.head`) tagged with `bucket` and `stream`, never the payload. That span is propagated across the internal actor mailboxes, so on-core work (`core.append`, `core.read`, …) nests underneath it. When a follower forwards a read to the leader, the W3C `traceparent` rides the Raft gRPC request, so the follower and leader appear in the **same trace** end to end. Trace context travels only in gRPC transport metadata. It is never written to the Raft log, so state-machine determinism and replay are unaffected. ### Sampling and the hot path Request-boundary spans (`http.*`) are `info` level, so production traces always carry one span per request. The on-core spans below them (`core.append`, `core.read`, …) are `debug` level: filtered out at the default `info` and compiled out of release builds entirely (`release-max-info`, on by default), so the hot read/write path pays nothing for them. Measured cost: | Operation | Cost | | ------------------------------------------- | ------- | | `info` boundary span (export off, fmt only) | ~285 ns | | `debug` span filtered at `info` | ~1 ns | | cross-mailbox context capture | ~5 ns | For high-throughput deployments, lower `OTEL_TRACES_SAMPLER_ARG` to bound export volume. ### Seeing on-core detail The `core.*` spans only materialize in a build without `release-max-info` (e.g. a debug build), and only when their target is enabled. Use a **crate-scoped** filter, never a global `debug`: ```bash # Good: only Ursula's own crates emit debug spans. export RUST_LOG="info,ursula_runtime=debug,ursula_raft=debug" ``` A global `RUST_LOG=debug` turns on debug spans/logs from OpenRaft, the OpenTelemetry SDK, reqwest, and every other dependency, which floods the collector and bloats the OTLP payload. Scope the filter to the `ursula_*` crates instead. > `tokio-console` needs tokio's `TRACE`-level instrumentation, which > `release-max-info` compiles out. Build with > `--no-default-features --features tokio-console` when using it. ## Metrics Two surfaces expose the same counters: - **OTLP**: when an endpoint is configured, runtime metrics are exported periodically as OpenTelemetry instruments (`ursula.appends.accepted`, `ursula.mutations.applied`, `ursula.raft_apply.ns`, `ursula.wal.records`, `ursula.group_mailbox.depth`, …), ready for Prometheus/Grafana via a collector. - **Cluster id**: `ursula.cluster.info` is `1` with the node's `server.cluster_id` as its `cluster_id` label, so dashboards and alerts tell deployments apart. `ursula.raft.peer_cluster_refusals` counts the Raft RPCs refused because the peer is in another cluster, both ways, and `ursula.raft.unidentified_peer_refusals` those refused because they named no cluster id. Alert on any increase (see [Peers of another cluster](/docs/0.9/operations#peers-of-another-cluster)). - **JSON**: `GET /__ursula/metrics` returns the full per-core/per-group snapshot for quick inspection and is what `ursulactl status` reads. For redundancy alerts, inspect the JSON metric `raft_maintenance.group_issues`. Alert on `incomplete_voter_set`, `unconfigured_member` or `joint_membership`, and treat a missing maintenance report as unavailable evidence. Also alert on nonempty `raft_maintenance.node_issues`. These fields remain strict when healthy survivors return 200 from `/__ursula/ready` while a replica catches up. They are JSON report fields, not new OTLP instrument names. ```bash curl --fail http://NODE:4437/__ursula/metrics | jq ' .raft_maintenance | {node_issues, group_issues}' ``` Staged snapshot parts are reported separately: the JSON field `staged_snapshot_part_bytes` and the OTLP gauge `ursula.snapshot_parts.staged_bytes` are the bytes of unused parts in the groups the node leads. For a cluster total, read `/__ursula/usage` from every voter and keep the highest term per group (see [usage export](/docs/0.9/operations#usage-export)). Bounding or billing staged parts beyond Ursula's per-stream bound is the operator's policy. Metrics are gathered on the hot path with lock-free per-core counters. The OTLP layer is an *export-time bridge*: it reads a snapshot at the collection interval rather than calling the metrics SDK per record, so enabling export adds nothing to the append/read path. For WAL operations, the JSON snapshot includes per core the journal's size and segments (`wal_physical_bytes`, `wal_segments`), rotations (`wal_rotations`), segments deleted and their bytes (`wal_reclaims`, `wal_reclaimed_bytes`), the time reclaim took and the passes that stopped on an error (`wal_reclaim_ns`, `wal_reclaim_failures`), live entry bytes copied out of old segments (`wal_rewritten_bytes`), and the groups reported lagging to the snapshot driver with the segments kept only for them (`wal_lagging_groups`, `wal_pinned_segments`), and the segments that hold purged or otherwise dead entries (`wal_dirty_segments`), which the journal frees within a quarter of `raft.deleted_data_max_age`. A core that keeps writing always has some, so `wal_dirty_segments` is rarely 0 there and is no signal to alert on. Reads report entries served from the cache and read from disk (`wal_cache_hits`, `wal_cache_misses`) and the frames and bytes read (`wal_disk_reads`, `wal_disk_read_bytes`), per core and per group. Each group reports its cached bytes and indexed entries (`wal_cache_bytes`, `wal_indexed_entries`). Startup scan work is under `wal_recovery_ns`, `wal_recovery_records`, `wal_recovery_bytes`, and `wal_recovery_live_entries`. Free-space admission is reported separately as `wal_available_bytes`, its two watermarks, `wal_disk_pressure`, and `wal_disk_stat_errors`. These are bounded per-node, per-core and per-group series. For the snapshot driver, the JSON snapshot includes the node's unsnapshotted Raft log and the most it has held (`raft_log_bytes`, `raft_log_bytes_high_watermark`), its pressure level against `raft.snapshot_log_budget` (`raft_snapshot_pressure_level`: 0 below three quarters of the budget, 1 up to the budget, 2 past it), the snapshot builds a pass keeps in flight at that level (`raft_snapshot_pressure_builds`), and the builds that run now and the most that ran at once (`raft_snapshot_builds_in_flight`, `raft_snapshot_builds_in_flight_max`). Its pressure passes that planned a build or changed the groups held out (a group whose trigger went unclaimed is held out for 0.5 s, doubling for each such trigger in a row up to 16 s, until a trigger of it is claimed) add up in `raft_snapshot_pressure_passes`, the builds they started in `raft_snapshot_pressure_groups`, and their time in `raft_snapshot_pressure_pass_ns`. For deleted data (see [Deleted data](/docs/0.9/operations#deleted-data)), the JSON snapshot reports how long ago the oldest removal applied whose stream a snapshot or a Raft log on the node still holds (`raft_deleted_data_oldest_age_ms`, 0 for none) and the groups whose snapshot holds one past a quarter of `raft.deleted_data_max_age` (`raft_deleted_data_overdue_groups`). `deleted_data` lists each group whose replica still holds removed streams, with their state, bytes and age. Both gauges come from the snapshot driver, so they stay at 0 in the in-memory mode, which runs no Raft. Alert when `raft_deleted_data_oldest_age_ms` nears half of `raft.deleted_data_max_age`. Each group's leader removes expired streams in the background (see [Deleted data](/docs/0.9/operations#deleted-data)). `ttl_expiry_sweeps` counts the `ExpireStreams` entries the node's leaders committed, each of which removes up to 256 expired streams, and `ttl_expiry_swept_streams` the expired streams they removed. `ttl_expiry_sweep_due_groups` reports the groups the latest pass left with expired streams. A group the pass stopped at its 16 entries or its 250 ms is swept again a second later, with 50 ms per group, so the gauge returns to 0 once its backlog drains. A value that stays above 0 across many passes means a group expires streams faster than its re-runs remove them, that its commits are slow enough to reach the sweep's time per group, or that the sweep stopped on an error, such as a group whose history is pending or whose apply stopped. A sweep an error stopped runs again only at the usual interval. All three work in the in-memory mode too. With an S3 snapshot store, `snapshot_gc_deleted_objects` counts the snapshot objects the node's snapshot GC deleted from its own directory. `snapshot_gc_failed_passes` counts the GC passes that failed, for example because the store refuses deletes, and a failed pass runs again later, backed off up to 15 minutes. Alert when it keeps rising: while GC fails, old snapshot objects that hold deleted streams stay in the store. `cold_gc_foreign_paths_refused` counts the refusals of cold GC. Cold GC refuses a path of a queued entry that is not an exclusive chunk, a pack or an external payload of the entry's bucket that the group owns, and an entry for a stream that is not of the entry's bucket or is placed in another group (see [Cleaning S3](/docs/0.9/operations#cleaning-s3)). A pass that runs an entry again counts its refusals again, so the value is not a number of distinct paths. It stays at 0 unless a corrupt or hand-edited backup was restored. Alert on any increase. `open_backoff_groups` counts the groups whose engine did not open because the snapshot store did not answer its restore, and `per_group_open_backoff` is 1 for each of them. Such a group answers `GROUP_RESTARTING` until its core opens it again, after a wait of 1 to 30 seconds. Alert when it stays above 0: the node cannot read its snapshots. The OTLP collection interval defaults to 60s. For a quick check, shorten it so readings arrive without waiting (and without relying on the shutdown flush): ```bash export OTEL_METRIC_EXPORT_INTERVAL=1000 # milliseconds ``` --- # Troubleshooting Start at `/__ursula/metrics`. Most operational symptoms have a clear signal in the JSON snapshot. ## Diagnostic surface ```bash curl http://NODE:4437/__ursula/metrics > metrics.json ``` Useful jq selectors. Arrays named `per_core_*` are indexed by core, and arrays named `per_group_*` by Raft group ID. ```bash # Leader, term and applied index of each group. current_leader is null while the group has no leader. jq '.raft_groups[] | {raft_group_id, current_leader, current_term, last_applied_index}' metrics.json # Groups without a leader jq '[.raft_groups[] | select(.current_leader == null) | .raft_group_id]' metrics.json # Hot bytes waiting for cold flush, per group jq '.per_group_cold_hot_bytes' metrics.json # Writes refused by cold-write backpressure, in total and per group jq '{total: .cold_backpressure_events, per_group: .per_group_cold_backpressure_events}' metrics.json # Mailbox depth and capacity, per core jq '{depths: .mailbox_depths, capacities: .mailbox_capacities}' metrics.json # Live reads waiting for new data, per core jq '.per_core_live_read_waiters' metrics.json # SSE streams opened and SSE error events jq '{opened: .sse_streams_opened, errors: .sse_error_events}' metrics.json # WAL disk pressure jq '{available: .wal_available_bytes, minimum: .wal_min_available_bytes, pressure: .wal_disk_pressure}' metrics.json # WAL reclaim and restart work jq '{physical: .wal_physical_bytes, segments: .wal_segments, reclaims: .wal_reclaims, lagging: .wal_lagging_groups, recovered_bytes: .wal_recovery_bytes, recovery_ns: .wal_recovery_ns}' metrics.json # How the WAL opened, and the groups behind a recovery gate jq '{opening: .wal_recovery, gates: .recovery_gates}' metrics.json # Local Raft issues that make the node unready or block maintenance jq '.raft_maintenance | {node_issues, group_issues}' metrics.json # This node's cluster id, and Raft traffic refused for another one jq '{cluster_id, refusals: .raft_grpc_peer_cluster_refusals, unidentified: .raft_grpc_unidentified_peer_refusals, peers: .peer_cluster_mismatches}' metrics.json ``` `GET /__ursula/ready` is the process readiness probe. It answers `200` when the node may serve, and otherwise `503` with a typed `reason`. When several apply, it reports the first of: | `reason` | Meaning | | -------- | ------- | | `format_epoch_mismatch` | The node exchanged a Raft RPC with a peer of another format epoch. It stays unready until it restarts | | `wal_disk_pressure` | The WAL filesystem is below `raft.wal.min_available_size`, or its free-space check failed | | `recovery_stalled` | A group's recovery gate on this node is stalled. The groups are listed under `recovery_stalled_groups` | | `recovery_gate_closed` | A group's recovery gate on this node is closed until the replica has applied a fresh barrier from the group's leader | | `raft_replica_unready` | A local replica is not a healthy voter. `raft_maintenance.group_issues` names the issue | Ursula does **not** expose `/healthz`, `/readyz`, Prometheus `/metrics`, or `/cluster/status`. Use the JSON snapshot for detailed diagnosis and `ursulactl wait-ready` for cluster-wide Raft readiness. ### Serving while a replica catches up `raft_replica_unready` means local Raft participation is not safe for serving. Inspect `raft_maintenance.group_issues` for `local_replica_not_voter`, `membership_not_applied`, `not_applied`, `apply_lag` or `leader_unknown`. A wiped replica stays 503 while its recovery gate is closed and it catches up. It stays a voter throughout. Healthy surviving voters can return 200 during the repair. A voter leaving a joint membership also stays unready once any configuration excludes it. A replica whose own gate is open reports `leader_unknown` for a group that has no leader, for example a group stalled because a majority of its voters are gated. A 200 response does not certify full redundancy. Keep maintenance blocked until `ursulactl wait-ready` succeeds. Alert separately on incomplete voter sets in the maintenance report, as described in [Observability](/docs/0.9/observability). ### `503` with `APPLY_STOPPED` from one group A committed command failed to apply and stopped that group on this replica. The node stays ready and serves its other groups. Its maintenance report names the group with `apply_stopped`, so `ursulactl wait-ready` refuses disruption until the group recovers. Follow [Recovering stopped Raft groups](/docs/0.9/apply-failure-recovery) and never skip the committed record. ### `503` with `GROUP_RESTARTING` from one group The group's engine on this node did not open. There are two causes. **The snapshot store did not answer the group's restore.** With an S3 snapshot store, opening a group reads the snapshot object its record names. When the store does not answer that read, the open fails after the whole restore attempt, and the node logs `the persisted snapshot object does not restore` at error level with the group and the object's key, and `the snapshot store did not answer the group's restore` at warn level with how many opens failed in a row. The node then holds the group closed for a wait that doubles per failed open, from 1 second up to 30 seconds, and opens the group again by itself. During the wait and during that reopen, it answers the group's requests with this code at once. `open_backoff_groups` in `/__ursula/metrics` counts the groups that wait. An open runs in a task of the core that owns the group, so the core's other groups keep serving. A request that reaches the group during its first open waits for the open, which the store can hold until the S3 client gives up, up to 22 seconds for each request with the default `storage.snapshot.s3_timeout` and `s3_max_retries`. Fix the node's reachability of the snapshot store: its endpoint, credentials and network path. The group opens at the first open after the store answers. A node whose store does not answer at startup starts anyway and keeps those groups closed until then. **The previous engine is still stopping.** The group's engine on this node is stopping or restarting, and the previous engine's Raft work has not ended yet. The node opens the group only once that work ends, because it could still change the group's state. A request that arrives meanwhile is refused at once, so the code shows up briefly during any restart of the group. Retry the request. If the answer persists for more than 30 seconds, the previous engine is stuck. Its shutdown logged `EngineWorkOutlivedShutdown` at error level, with the group and how many of the engine's state-machine workers, snapshot builders, log readers, network clients and snapshot installs still ran. Collect the node's logs and restart the node. ## Startup failures ### WAL topology mismatch or missing topology `topology.bin` binds a WAL root to `runtime.core_count` and `raft.group_count`. For a mismatch, restore the stored counts reported in the error. A missing record on an existing WAL or a corrupt record also refuses startup. Do not replace it with the current configuration. See [WAL routing configuration](/docs/0.9/operations#wal-routing-configuration) for rebuilding a replica. ### `raft.wal.path is required when raft.peers is set` Every node of a static cluster keeps its Raft WAL in `raft.wal.path`. Set it to a directory on a persistent volume. Only a single node without `raft.peers` may omit it, and then its WAL lives in a temporary directory that is removed on clean shutdown. ### `server.cluster_id is required when raft.peers is set` Every node of a static cluster names its deployment with `server.cluster_id`. Set the same value on every node, and a different one for every deployment a client can reach. With the Helm chart, set `server.clusterId`. The chart refuses to render without it. See [Configuration](/docs/0.9/configuration#server-and-runtime). ### `the WAL at … belongs to cluster …` The node's WAL records the cluster id of its first start, and the configured `server.cluster_id` differs. A disk does not move between clusters. Set `server.cluster_id` to the stored id the error names, or start the node on an empty WAL and let it rejoin as a [wiped node](/docs/0.9/operations#rejoining-a-wiped-node). ### `raft.node_id must be non-zero` Static Raft mode requires a unique `raft.node_id` in the config file or a `--node-id` override. ### `raft.peers` must include this node id The local `raft.node_id` must appear in `[[raft.peers]]`. The peer list always includes self. ### `storage.cold.s3.bucket is required when cold backend is 's3'` Set `storage.cold.s3.bucket`, or set `storage.cold.backend = "none"` if you don't want cold storage. ### Port already in use `server.listen`, `server.cluster_listen` or `server.admin_listen` is taken. The client plane and inter-node gRPC share `server.listen` unless `server.cluster_listen` is set. ### A node at a new address stays unready Every node dials its peers at the `url` its own `[[raft.peers]]` lists. A node that moved, or a node whose peers still list its old address, gets no Raft traffic from the peers that list the wrong one. Its groups catch up on nothing, and `ursulactl wait` does not return. Check that `[[raft.peers]]` is the same on every node and names each node where it listens now, then restart the nodes whose list was wrong. A move of a listener, such as a new cluster port, needs every voter restarted together ([Changing a peer address](/docs/0.9/operations#changing-a-peer-address)). ### `unconfigured_member` The maintenance report names `unconfigured_member` for a group whose Raft membership names a node that is not one of its configured voters, as a voter or a learner. No node dials a member that `[[raft.peers]]` does not list, and each node logs that once: ```text raft.peers lists no node 4, a member of a Raft group: not dialing it; list the node in raft.peers or remove it from the group's membership ``` The node keeps serving, but maintenance readiness fails, so `ursulactl drain` and `wait` stop. The group leader never removes such a member by itself. List the node in `[[raft.peers]]` again, or send the group's configured voters to the group's leader with `POST /__ursula/raft/{group}/membership?voters=` ([Admin endpoints](/docs/0.9/operations#admin-endpoints)). The endpoint replaces the voter set and removes every learner outside it in the same change, so a member left behind as a learner goes too. ### `raft WAL directory … holds data without a format marker` The data directory was written by Ursula 0.5.x or earlier (format epoch 1). This release reads format epoch 5 only and has no in-place upgrade. Start on an empty `raft.wal.path` and a new object-storage prefix, or run the release that wrote it. ### `raft WAL directory … is format epoch N` The directory's `FORMAT_EPOCH` marker names another epoch. This release reads format epoch 5 only. Format epoch 4 is Ursula 0.8, whose data moves to this release through your applications ([Upgrading to 0.9](/docs/0.9/operations#upgrading-to-09)). Run the release that wrote it, or start on an empty directory. ### `object storage s3://… is not empty and has no URSULA_FORMAT_EPOCH marker` The cold root (or, with cold storage off, the S3 snapshot namespace) holds objects from Ursula 0.5.x or from another application. Point `storage.cold.root` (Helm: `s3.prefix` or `coldStorage.prefix`), or `storage.snapshot.s3_prefix` when only snapshots use S3, at an empty prefix used only by this cluster. ### `object storage s3://… is format epoch N` The prefix's `URSULA_FORMAT_EPOCH` marker names another epoch. Use a new prefix and never reuse one across epochs. The data of a 0.8 cluster (format epoch 4) moves to this release through your applications, not by copying its objects ([Upgrading to 0.9](/docs/0.9/operations#upgrading-to-09)). ### `journal … uses Ursula WAL version N` A Raft WAL journal of another format epoch: version 1 is Ursula 0.5.x, version 2 is 0.6, version 3 is 0.7, version 4 is 0.8. This release reads format epoch 5 only. The directory marker normally refuses the directory first, and the journal check is the backstop. Start on an empty `raft.wal.path`. ### `peer N (…) refused Raft protocol 5` A configured peer runs a release of another format epoch (Ursula 0.8 speaks protocol 4, 0.7 protocol 3, 0.6 protocol 2, 0.5.x protocol 1). A node cannot join a cluster of another format epoch. Install this release as a new cluster. ### `/__ursula/ready` answers `503` with `format_epoch_mismatch` The node exchanged a Raft RPC with a peer on another format epoch: a configured peer that was unreachable during the startup probe, or a node of this cluster on another release that sent this node a Raft RPC. The node stays not ready until it restarts. Remove the mismatched node from the peer lists and restart. A sender that names no cluster id, such as a 0.7 node of another deployment, does not make this node unready (see [Peers of another cluster](/docs/0.9/operations#peers-of-another-cluster)). ### `refusing Raft traffic with a peer of another cluster` The node and a peer have different `server.cluster_id` values: one of them is mistyped, or a peer URL points at a node of another deployment. Each refuses the other's Raft traffic. The log line names the peer's node id, cluster id and address. `/__ursula/ready` and `/__ursula/metrics` list a configured peer under `peer_cluster_mismatches` with the id it reported, and `raft_grpc_peer_cluster_refusals` counts the refusals. The refusals never change readiness: an unready node's `reason` is its own cause. Find the node whose id differs from the rest of the deployment and correct its `server.cluster_id` or the peer URL. A node whose WAL already holds the wrong id refuses to start with the corrected one, so restart it on an empty WAL and let it rejoin as a [wiped node](/docs/0.9/operations#rejoining-a-wiped-node). The healthy nodes stop listing the peer a minute after its last refusal. See [Peers of another cluster](/docs/0.9/operations#peers-of-another-cluster). ### `refusing Raft traffic from a sender that names no cluster id` A node before format epoch 4, or a client that is not an Ursula node, sent a Raft RPC to this node, usually because a peer list of another deployment names this node's address. The sender is refused and gets a format-epoch refusal. This node stays ready and counts it in `raft_grpc_unidentified_peer_refusals`. Remove this node's address from the other deployment's peer list. ### `/__ursula/ready` answers `503` with `recovery_gate_closed` The node's logs may be missing entries it acknowledged: it started after a host crash or power loss under `raft.wal.fsync = "never"`, after a journal I/O error, or without its run state file, or with an emptied WAL directory. Under `raft.wal.fsync = "always"`, a host crash gates the groups of a core whose newest journal segment ended in a complete invalid frame, although no acknowledged write was lost. Each gated group waits until it has applied a fresh barrier from the group's leader, and its leader catches the replica up meanwhile, which stays a voter. `recovery_gates` in `/__ursula/metrics` lists the gated groups. This normally clears in seconds once a leader is reachable. See [Recovering after a host crash](/docs/0.9/operations#recovering-after-a-host-crash). ### `/__ursula/ready` answers `503` with `recovery_stalled` A gated group on this node has seen no leader barrier and applied nothing for `30s`. A majority of the group's voters are probably gated, so the group has no leader and refuses writes. The groups are listed under `recovery_stalled_groups`. Restore connectivity to a healthy quorum first. If the stalled replicas retain their voting history, an operator can accept the loss of writes acknowledged after the last `fsync` with a recovery plan, sent through `POST /__ursula/raft/{group}/recovery/accept-unsynced-loss` to the plan's leader first and then to every other replica that answers, with the leader's plan term (`ursulactl recover-group --accept-loss` does both). The plan starts a new history of the group, so every stream of the group gets a new `Stream-Incarnation`. A voter of the group that was never gated does not report `recovery_stalled`. It answers `raft_replica_unready` with `leader_unknown` for the group. See [Recovering after a host crash](/docs/0.9/operations#recovering-after-a-host-crash). ### `accept-unsynced-loss` answers `409 Conflict` Nothing changed. A replica that lost its voting history must first obtain a durable vote floor from a fresh quorum proof, and the call cannot replace that proof. Restore connectivity to intact voters. The call also refuses when the replica leads or follows a live leader, when its gate awaits or applies a leader's barrier (it may still open without losing anything) or runs a plan's cut, when the request's `step` is not the replica's (`lead` for the plan's leader, `follow` for every other replica), when the plan's leader is not a voter, when the replica no longer holds the log the plan's `expected` names, when it committed or applied an entry above the plan's `point`, when it is the plan's leader and its log does not continue from the `point` or, with a `point_entry`, does not hold exactly that entry there, or when the plan's cut cannot run because the log no longer holds the entry it keeps. The answer's body names the `refusal` by its `reason`. Read `/__ursula/metrics` on every replica again and plan again with what it shows, once the group has no leader and, after a `not_stalled` refusal, once the gate reports `stalled`. Send each member the `leader_term` from the leader's answer, never a term read from the metrics. A body of the 0.7 form answers `422`. ### Writes and reads answer `503` after a recovery plan The plan's leader serves nothing until the group's new history event applies. It proposes the event once it is elected and has applied its first entry, so this normally clears within a second. If it persists, check that the plan's leader is running and that every other replica that holds the plan can reach it: a replica with a plan votes only for the plan's leader. If that leader lost entries it had replicated, the members refuse it and the group stalls again: plan again with the replicas' current metrics. ### `LogBelowSnapshot`: a group's log contradicts its restored snapshot record The replica found an entry, or a commit marker, at or after its snapshot record's index from an older term, and keeps the group closed. The group's other replicas keep serving it. Rejoin the node as a wiped node after checking its other groups. See [A log below its snapshot record](/docs/0.9/operations#a-log-below-its-snapshot-record). ### I/O error on `raft.wal.path` The directory must be writable and owned by exactly one Ursula process. Each core journal holds an advisory `journal.lock` containing the owner PID. Do not remove it while that process is alive; stop the conflicting process or fix the volume mount. A stale lock file is harmless because ownership is enforced by the OS lock, not the file's existence. ## Backup and restore ### `backup manifest format_version N` or `target cluster speaks backup format N` The backup or the cluster uses another backup format. This release reads and writes backup format 5, the format of Ursula 0.9. A backup of another format, such as a 0.8 backup (format 4), cannot be restored, and a cluster that reports another backup format cannot be backed up or restored into. Move 0.8 data through your applications instead (see [Upgrading to 0.9](/docs/0.9/operations#upgrading-to-09)). ### `… cold objects this backup references are missing from the target cluster's cold store` The backup references cold objects that the target's `storage.cold.root` does not hold. `restore` stopped before the first import, so the target is still empty. Copy the source cluster's cold objects into the target's root (see [Copying cold objects](/docs/0.9/operations#copying-cold-objects)) and run `restore` again. The error names the first missing keys. A key under `cold-index/` is an index page, so everything that page lists is unchecked until the page is there. ## Bootstrap and cluster join ### A fresh cluster never elects leaders `/__ursula/metrics` on each node should show a non-null `current_leader` and a stable `current_term` for every group in `raft_groups`. If groups stay leaderless: - First start must include `raft.init_membership_per_group = true` (or `raft.init_membership = true`) on every voting node. - Every peer URL must be reachable from every node. ### Replacing or restarting a node leaves it leaderless Restart with the **same** `raft.node_id` and peer list. A fresh `raft.wal.path` means rehydrating from peers, which can be slow on cold-storage-only history (see [Rejoining a wiped node](/docs/0.9/operations#rejoining-a-wiped-node)). ## Write path ### `503` with `Retry-After` on append Inspect the error body's `error.code`. `WAL_DISK_PRESSURE` means the WAL filesystem is below `raft.wal.min_available_size` (or its free-space stat failed); the node also becomes unready and attempts to yield its leaders. Free space or expand the volume. Admission resumes only at `raft.wal.resume_available_size`, preventing probe flapping. `RAFT_LOG_PRESSURE` means the node's unsnapshotted Raft log is over twice `raft.snapshot_log_budget`. The node refuses writes with a body until snapshots bring the log below 1.5 times the budget. Before it gets there, the snapshot driver runs more snapshot builds at once the further the log is past the budget, up to `raft.snapshot_build_max_concurrency`. When writes still reach the limit, the node's metrics show how far the log got (`raft_log_bytes_high_watermark`), whether the driver used every build (`raft_snapshot_builds_in_flight_max`), and how long a build's upload takes (`raft_snapshot_build_ns` over `raft_snapshot_builds`). If every build was in use, raise `raft.snapshot_build_max_concurrency` (the chart's `raft.snapshotBuildMaxConcurrency`) or the budget. Each build holds its group's captured state and, while it uploads, the compressed snapshot, so raise the builds only as far as the node's memory holds that many builds of its largest groups. Set it to `1` to keep one build at a time, as before 0.7.4. Only a cluster node answers `RAFT_LOG_PRESSURE`. A node without `raft.peers` never refuses writes for its log in 0.8.x. `INGRESS_BODY_BYTES_LIMIT_REACHED` means the write bodies in flight on the node leave too little of `server.http_inflight_body_size` for this one. `SNAPSHOT_COMMITS_BUSY` means that the node copies as many snapshot commits from parts as it may, two at once. `SNAPSHOT_COMMIT_IN_PROGRESS` means that the node copies this same commit now: repeat it, and it answers once the commit ends. `snapshot_commits_busy` in `/__ursula/metrics` counts the first. `WAL_DISK_PRESSURE`, `RAFT_LOG_PRESSURE` and `INGRESS_BODY_BYTES_LIMIT_REACHED` are write admission refusals. Other `503`s come from the stream's Raft group, and their code names the cause. `COLD_BACKPRESSURE` means that the group's hot ring would exceed `storage.cold.max_hot_size_per_group`: cold flush isn't keeping up, or the cold backend is slow or erroring out. Raise the per-group ceiling, lower `storage.cold.flush_interval`, raise `storage.cold.flush_max_concurrency`, or fix the cold backend. `RAFT_UNCOMMITTED_BACKPRESSURE` means that the group's write bytes that are proposed but not yet applied would exceed `raft.max_uncommitted_size_per_group`. `UPLOAD_BUDGET_EXHAUSTED` comes from `ursulagw`: its request body budget (`--body-budget-bytes`) stayed spent for `--body-budget-wait-ms`, so it sent nothing on. Bodies of at most `--small-body-max-bytes` have headroom of their own, so this hits large bodies under heavy upload load. Writes resume once the uploads under way end. The gateway's `upload_refusals_budget` counter shows these refusals. See the gateway's [request body budget](https://github.com/tonbo-io/ursula/blob/main/crates/ursula-gateway/README.md#request-body-budget). A `503` also answers a write while the group has no known leader, for example during an election, and on a node that does not host the stream's group and knows no node that does. A client resolves a `503` like any unknown outcome: it repeats the write with its producer key. See [write outcomes](/docs/0.9/api/overview#write-outcomes). ### `408` with `BODY_TOO_SLOW` from the gateway No byte of the body reached `ursulagw` for `--body-stall-timeout`, or less than `--body-min-bytes-per-sec` arrived over one such window, so the gateway cut the body and sent nothing on. With the defaults (30 s, 1 KiB/s) this does not happen to a SQLite commit, since the extension gives up on a commit after 10 s. Over HTTP/1.1 the gateway closes the connection with the `408`, so a client still sending usually meets the closed connection instead. Keep `--body-stall-timeout` above 10 s, or a commit still being sent can meet that closed connection on every attempt. The gateway's `upload_refusals_too_slow` counter shows the cuts. ### `409 Conflict` with `producer-expected-seq` / `producer-received-seq` Out-of-order producer headers. Response tells you what was expected: - `producer-expected-seq: N` is the next allowed sequence - `producer-received-seq: M` is what the client sent Common causes: - Producer restart without bumping `Producer-Epoch`. Bump every restart. - Two writers share the same `Producer-Id`. Use distinct IDs. - A retry skipped a sequence number. Producer sequences must be contiguous within an epoch. ### `409 Conflict` from `Stream-Seq` The supplied `Stream-Seq` is not bytewise greater than the last value accepted on this stream: another writer committed first, or the values are not fixed-width (`"10"` sorts before `"9"`). The response's `Stream-Next-Offset` is the tail. Read what you missed, re-derive the value and retry. See [Conditional writes](/docs/0.9/concepts/conditional-writes). ### `404 Not Found` on `POST /{bucket}/{stream}` The stream hasn't been created. Create with `PUT /{bucket}/{stream}` first. There is no implicit creation on POST. ### `400 Bad Request` on JSON appends JSON streams are validated on the server. Malformed JSON, invalid UTF-8, messages nested deeper than 127 levels, and empty arrays without `allow_empty_array` are rejected. Send valid JSON or switch to `application/octet-stream`. ## Read path ### No `Stream-Up-To-Date` on a read The response is partial: cut by `max_bytes` or the 8 MiB cap, or served by a follower, which never sends it with data. Read on from `Stream-Next-Offset`. Not an error. ### SSE drops after 30-60 s idle A proxy or load balancer is closing idle TCP. Raise its idle timeout or enable TCP keepalive on the proxy-to-Ursula leg. Ursula doesn't emit SSE keepalive comments. ### SSE on a binary stream returns base64 text Expected. SSE wire format is text-only, so binary stream data events carry raw base64 text and the `Stream-Sse-Data-Encoding: base64` header signals it. See [binary SSE](/docs/0.9/concepts/binary-sse). ### Reads stall for hundreds of ms for cold offsets The first read of a cold offset triggers an S3 `GetObject` range read. It runs off the actor turn so it doesn't block other commands, but the request still pays S3 latency. ## Replication and consensus ### A group has no leader (`current_leader: null`) Either an election is in progress or quorum is lost for that group. Only streams hashed to that group are affected. - Multiple nodes briefly claim leader: election is flapping. Check peer reachability and CPU pressure. - No node claims leader: fewer than `n/2+1` voters are reachable, or a majority of the group's voters are behind their recovery gates. Restore reachability. For gated voters, see [Recovering after a host crash](/docs/0.9/operations#recovering-after-a-host-crash). ### One follower lags behind State-machine apply runs on the follower's owner core. If CPU is pinned, `last_applied_index` trails `committed_index`. Check: - Per-core mailbox depth (`mailbox_depths`). Sustained queueing means saturation. - Disk pressure on the WAL volume (`fsync` latency, in particular with `raft.wal.fsync = "always"`). - A pathological hot stream forcing constant cold flushes (`per_group_cold_hot_bytes`). ### Manually trigger a snapshot These are admin-plane endpoints (`server.admin_listen`, loopback `:4438` by default), reached over a tunnel (see [ursulactl's operation providers](/docs/0.9/cli#the-admin-plane)). The examples assume a forward from local `4438` to the node's admin plane. ```bash process_incarnation=$(curl --fail http://127.0.0.1:4438/__ursula/metrics | jq -er .process_incarnation) curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/{group_id}/snapshot curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/{group_id}/purge?upto={index}" ``` The snapshot request answers once the group has a snapshot of everything it had applied when the request arrived. It waits while a build of the group runs and while other snapshot builds hold all of the node's build permits, and a snapshot that a build begun since the request made answers it too. When no build of the group can start within 10 seconds, it answers `503` with `Retry-After: 1`. Send it again then. Unless the group's published snapshot already covers the request, it then waits up to 10 seconds for the group's running build to end and up to 10 more for the snapshot to show, and answers `500` when it does not. The purge request drops log entries through `upto`, at most through the group's published snapshot. While a snapshot of the group builds, OpenRaft has not published the one it built, or an earlier purge of the group has not landed, it answers `503` with `Retry-After: 1`. Send it again then. Purge only after the snapshot replicates. Otherwise a slow follower may need log entries you just dropped. ## Cold flush issues ### Writes succeed but data never reaches S3 - `storage.cold.backend` is `none`, or set to `memory` when you expected `s3`. - IAM permissions missing. Required: `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`, `s3:DeleteObject`. - `storage.cold.s3.endpoint` unreachable. - Cold flush failing or falling behind (`cold_flush_write_errors` or `cold_backpressure_events` climbing, `cold_hot_bytes` growing). If hot bytes grow unbounded, you'll eventually hit `storage.cold.max_hot_size_per_group` and writes start returning `503`. Fix the backend before that. ## SQLite VFS These entries cover the `sqlite-ursula-vfs` extension and `@tonbo/sqlite-ursula` 0.8. See [SQLite on Ursula](/docs/0.9/examples/sqlite-vfs). ### `synced()` does not resolve `synced()` waits until the stream holds every commit made before the call. Commits keep working meanwhile. `db.status().replication.condition` says why it waits: | `condition` | Cause | What to do | | ----------- | ----- | ---------- | | `retrying` | No definite answer yet: the stream is unreachable, answers `5xx`, `429` or `408`, or the connection dropped. The append is resent unchanged after backoff. A `429` with the code `UPLOADS_EXHAUSTED` or `UPLOAD_BYTES_EXHAUSTED` means the bucket's large bodies under way reached its `max_concurrent_uploads` or `max_upload_bytes` in the gateway's `--quota-policy` | Check that the stream URL's host resolves and that the gateway or the server behind it is up. Nothing is lost while it retries. For the upload refusals, raise those keys, or set `URSULA_VFS_MAX_REQUEST_BYTES` at or below the gateway's `--small-body-max-bytes` (1 MiB by default), whose bodies never count. Small commits alone do not stay below it, since an append packs commits up to 4 MiB | | `unauthorized` | Two `401`s in a row, the second after reading the token again | Fix the token passed to `open` or the file `URSULA_VFS_TOKEN_FILE` names. A changed file is read again at once | | `forbidden` | An authorizer in front of the stream answered `403` | Allow the write in the authorizer | | `payment_required` | A layer in front of the stream answered `402` | Settle what the layer asks for | | `request_too_large` | A layer in front of the stream allows request bodies below 4 KiB | Raise the layer's limit | | `redirected` | The stream URL answers with a redirect, which the extension never follows | Point the URL at a gateway or at a single node | | `wrong_cluster` | The stream URL now reaches another deployment (another `Ursula-Cluster-Id`) | Fix the routing. Replication resumes by itself within 30 s | | `rejected` | A status the extension does not expect, such as a layer that rewrites bodies | Look at the layers between the client and the gateway | A blocked condition is retried every 30 s, and commits continue until `commits.pending_bytes` reaches `maxPendingBytes`. A `synced()` with a `signal` rejects with the signal's reason and leaves the commits replicating. ### A write fails with error code 7178 `SQLITE_IOERR_AUTH` (`errcode` 7178 in `node:sqlite`, `UrsulaWriteRefusedError` from the package) means the write was refused before anything of it was committed. The database stays usable. - `backlog_full`: compressed commits waiting for the stream reached `maxPendingBytes` (64 MiB by default). Await `db.synced()` and run the transaction again. If it stays full, see [`synced()` does not resolve](#synced-does-not-resolve). - `unsupported`: the transaction ran in EXCLUSIVE locking mode, or its commit leaves WAL mode or 4096-byte pages. Change the transaction. Other transactions go on. - `too_large`: the commit's pages take more than `URSULA_VFS_MAX_COMMIT_BYTES` (128 MiB by default). The message names the commit's size and the limit. Split the transaction, or raise the limit (at most 1 GiB) on every host that may own the database, with memory for about three times the limit. A `VACUUM` of a large database hits it: see [Errors](/docs/0.9/examples/sqlite-vfs#errors) for the ways around it. ### Statements fail with error code 266 or 778 The database was reset. Every statement throws an `UrsulaResetError` (`errcode` 266 for a read, 778 for a write), and `db.status().reset` names the kind: - `fenced` (`stop`): another process opened the stream and owns it now. Find that owner before you open the stream again, since opening it takes the stream back. - `stale` (`reopen`): the stream moved on while this process could not see it, for example after it was idle past producer expiry or opened provisionally during an outage. Open the stream again. - `rewound` (`reopen`): a restore or an accepted loss removed commits the stream had acknowledged. Open the stream again: the open moves the cache aside into `salvage/` (see below) and builds from the stream. - `deleted` (`fix`): the stream was deleted. Open it with `recreate: true` to create it again from the cache's data, or open another stream. - `local_damage` (`reopen`): a local fsync failed, a WAL page read back other bytes, or SQLite ended a commit without publishing it. Open the stream again: the log replays every commit. Check the disk if it repeats. - `stream_state` (`fix`): another writer appended to the stream, the stream was closed, or the server lost acknowledged state. The stream no longer matches the cache. Look at who writes to the stream. - `internal` (`reopen`): a bug inside the extension. A `panic` line on stderr names it. Open the stream again, and report it. ### `open` fails `open` throws an `UrsulaSetupError` or, when its signal fires first, an `UrsulaUnavailableError`. Every refusal leaves the cache as it was and sends nothing to the stream. | `kind` | Cause | What to do | | ------ | ----- | ---------- | | `server_too_old` | The stream's `HEAD` lacks `Ursula-Protocol: 2` or `Ursula-Cluster-Id`: a 0.7 server, or a layer that answers `HEAD` itself, such as a gateway whose authorizer conceals the stream, whose `404` carries neither header | Run a 0.8 server, check the token, and send `HEAD` through to the server | | `stream_format` | The stream holds 0.7 data, or another content type | Move the database with an import, see [Moving from 0.7](/docs/0.9/examples/sqlite-vfs#moving-from-07) | | `wrong_cluster` | The cache was built on another deployment than the one the URL reaches now | Fix the routing, or use another `dir` for the other deployment | | `not_found` | The stream is missing and the cache holds data | Open with `recreate: true`, or open another stream | | `in_use` | Another process holds the cache | Stop that process, or use another `dir` | | `bound_elsewhere` | The stream is open in this process under another `dir` | Close that database, or pass its `dir` | | `stream_state` | The stream is closed, or an import into it is running or failed (`the stream up to N is the claim of an import`) | Open a closed stream read-only. Wait for a running import. After a failed import, delete or recreate the stream and import again | | `unreachable`, `unavailable` | The signal fired while the stream could not be reached or did not answer | Check the network and the cluster. With a cache that claimed the stream before, `open` resolves provisionally instead | ### A cache appears in `salvage/` `status().salvage` lists caches the extension moved aside, in `salvage/` in the stream's cache directory: after a history change that lost commits the cache had, after a history change while the cache held commits it had no acknowledgement for, when the db file was not the cache's own (`replaced`), and with `recreate` (`recreated`). Each holds a plain SQLite database `db`, its log and a `manifest.json` with the tokens and the count of unacknowledged commits. Open `db` with any SQLite, compare it with the stream's database, and write again what you need. The extension never deletes a salvage. ### `snapshot.error` is set `status().snapshot.error` says why snapshots are not published, and `snapshot.failures` counts the attempts. A database whose compressed snapshot is above the request limit (32 MiB) needs a cold backend on the server, which takes it in parts, up to 1 GiB. Until a snapshot publishes, the log grows and every fresh host reads all of it. Failed attempts wait from 100 ms up to five minutes, or an hour when the body is too large. ### `file is not a database` A cache's db file holds a marker in place of SQLite's header magic, and its WAL is `db-uwal`. SQLite outside the extension (the `sqlite3` shell, node:sqlite without it, a connection opened before it loaded, `?vfs=unix`) fails with `SQLITE_NOTADB` and changes nothing. Read the database through `open`, or export it with `VACUUM INTO` from a connection `open` returned. ### Closing a Pi session hangs during an outage On Pi Durable 1.3, `Session.close` waits for the storage calls in flight, and through `openUrsulaPiStorage` each waits for `synced()`, which waits for the stream. So `closeTimeoutMs` applies only once the session reaches the storage's `close`. Bound the shutdown in your host when it must end during an outage. The log keeps every commit for the next open. ## Still stuck? Open a [GitHub issue](https://github.com/tonbo-io/ursula/issues) with: - `/__ursula/metrics` from every node (with timestamps) - Config file and CLI overrides per node (redact credentials) - Last ~200 log lines at `RUST_LOG=ursula_runtime=debug,ursula_raft=debug,ursula=info` --- # Overview ## The problem Building a durable stream directly on S3 hits two walls: S3 has no append (every write either pays per-PUT or batches and eats the latency), and its conditional writes are optimistic compare-and-set, so concurrent writers retry-storm with exponential latency ([Chroma's wal3 post](https://www.trychroma.com/engineering/wal3) covers the mechanics). The market splits on which wall to lean against. [S2](https://s2.dev/blog/intro) writes through S3 Express to get under 50 ms but pays around 7x per-GB, while [WarpStream](https://www.warpstream.com/blog/minimizing-s3-api-costs-with-warpstream) batches to S3 Standard and accepts 250 ms+ p50. ## How Ursula works Ursula keeps S3 off the write path entirely. Writes go to a cluster of nodes that coordinates through [Raft consensus](https://en.wikipedia.org/wiki/Raft_(algorithm)). Once a majority of replicas has persisted a record, it is committed, with no compare-and-set races and no retry storms. S3 is the cold tier. Two structural choices shape the rest of the design: 1. **[Thread-per-core](https://seastar.io/shared-nothing/) shard ownership.** Every stream is statically hashed to one specific CPU core on each node. That core handles all reads and writes for its streams, in order, with no cross-core synchronization on the hot path. 2. **[Multi-Raft](https://tikv.org/deep-dive/scalability/multi-raft/).** Ursula runs hundreds to thousands of small Raft groups per cluster, not one log per node. Each stream hashes to a group, each group pins to a core. An unhealthy follower for one group does not stall traffic for another. Together, write throughput scales with healthy cores across the cluster instead of with any single leader's bandwidth, and failure domains stay per-group rather than per-node. ## The stack at a glance Ursula is async Rust. The pieces that show up in the diagrams below: - **axum** is the HTTP server library that terminates public client requests. - **tokio** is the async runtime. Each core runs its own single-threaded executor (`tokio::current_thread`) so work on that core never migrates. - **tonic** is the gRPC library used for peer-to-peer Raft traffic between nodes. - **OpenRaft** is the Raft consensus implementation Ursula plugs into. The Rust-named components in diagrams (`ShardRuntime`, `GroupActor`, `GroupEngine`, `StreamStateMachine`) are the structures inside one node, glossed in the sections below. ## Interface The API is the [Durable Streams protocol](https://durablestreams.com/), an open MIT-licensed spec published by [Electric](https://electric-sql.com/): create a stream, append, read from any offset, tail in real time over Server-Sent Events (SSE), with exactly-once write semantics. Plain HTTP plus SSE, with no custom binary protocol or mandatory client library. See the [API overview](/docs/0.9/api/overview) and the [extensions spec](/docs/0.9/specs/extensions) for what Ursula adds to it. ## Cluster topology ``` Clients (HTTP / SSE) │ ┌──────────────────┼──────────────────┐ v v v ┌──────────┐ ┌──────────┐ ┌──────────┐ │ Node A │ │ Node B │ │ Node C │ └────┬─────┘ └────┬─────┘ └────┬─────┘ cores 0..N cores 0..N cores 0..N │ │ │ └─── gRPC peer-to-peer Raft traffic ──┘ │ v ┌────────────────────────┐ │ Cold storage (S3) │ │ per-stream chunks │ └────────────────────────┘ ``` Every stream is replicated to all voting members of its Raft group. Hot data lives in memory on each replica. Cold data lives in a shared object store any replica can read from on demand. ## Inside a node ``` HTTP / SSE request │ v route(bucket_id, stream_id) │ v owner core │ v Raft group │ │ │ └── replicate to peer nodes │ ├── hot state and live watchers │ └── cold chunks in S3 ``` A request enters the HTTP server, routing maps the bucket and stream name to one owner core and one Raft group, and the rest of the request runs inside that group. The group owns its hot bytes, live-tail watchers, producer dedup state, and Raft state machine, so mutable stream state does not cross cores after routing. ## The replaceable group engine The `GroupEngine` trait is the seam between the runtime and a group's storage and replication strategy. Two engines ship today: - **In-memory, non-replicated.** Used in tests, as a performance baseline, and by the zero-config development node. - **Raft engine** over the disk WAL (`raft.wal.path = "DIR"`). OpenRaft groups on the same core share one checksummed, versioned journal under `DIR/raft-log`, and cross-group fsync batching avoids one writer per group. The journal is a run of segments: purge deletes whole segments once no group needs them, and the few entries quiet groups leave in old segments are copied forward. Each group keeps an index of where its entries are and a small cache of recent ones, and reads older entries from disk. Snapshot plus committed-tail replay rebuilds the in-memory state machine after restart. A single node without `raft.wal.path` keeps the journal in a temporary directory that is removed on clean shutdown. A factory selected at startup picks which engine each group opens, leaving the runtime code identical across modes. ## Write path ``` HTTP ---> Runtime ---> core mailbox ---> GroupActor │ ├─ leader? ──no─---> 307 redirect to leader │ v yes Raft.client_write │ v WAL append (fsync per policy), replicate to followers │ v commit StateMachine::apply │ v notify_read_watchers (same turn, no lock) │ v AppendResponse -> client ``` The leader appends the entry to its write-ahead log (WAL) and replicates it to followers in parallel. A replica acknowledges the entry once its journal holds it: in the page cache with the default `raft.wal.fsync = "never"`, after an `fsync` with `always` (see [Configuration](/docs/0.9/configuration#raft)). Once a quorum holds the entry, the state machine applies it and the response goes back to the client. Producer dedup headers ([exactly-once writes](/docs/0.9/concepts/exactly-once-writes)) are checked inside the state machine, so retries of an already-applied append return the original offset rather than appending again. `Stream-Seq` is checked in the same place, after producer dedup, as a per-stream compare-and-set ([conditional writes](/docs/0.9/concepts/conditional-writes)). ## Storage layout Each Raft group has two storage planes: a replicated log and a stream state machine. ``` Raft group │ ┌─────────┴─────────┐ v v replicated log state machine command order streams in this group quorum durability offsets / metadata / dedup │ ┌─────────┴─────────┐ v v hot ring memory cold chunk refs recent bytes S3 objects ``` The Raft log is the ordering and durability boundary for writes. A write is acknowledged only after a majority of that group's replicas hold the entry in their WAL. The entry carries the stream command and metadata needed to apply it deterministically. The stream data plane lives in the group's state machine. A group owns many streams, keyed by `(bucket_id, stream_id)`. For each stream it tracks metadata, offsets, producer dedup state, live snapshots, hot byte segments, and cold chunk references. Recent bytes stay in an in-memory hot ring on every replica. When a group's hot bytes exceed the configured cold-flush threshold, a background worker uploads older contiguous segments to the cold backend, usually S3. After upload succeeds, Ursula commits a metadata update through Raft that publishes the cold chunk reference and advances the stream's hot start offset. Reads assemble one logical stream from both tiers. The state machine first builds a read plan: which byte ranges are still hot and which ranges are cold chunk references. Hot bytes are copied from memory, and cold ranges are fetched with object-store range reads. Any replica that has applied the cold chunk metadata can serve those cold reads, as long as all nodes point at the same S3 bucket/prefix. ## Read path and cold offload ``` HTTP ---> Runtime ---> core mailbox ---> GroupActor │ ├─ leader? ──no─---> forward to leader │ v yes engine.read_stream_parts (under state-machine lock) │ v release lock GroupReadStreamParts { plan, cold_store } │ v tokio::spawn (off the actor turn) materialize: hot bytes + S3 range reads │ v ReadStreamResponse -> client ``` Under the state-machine lock the runtime computes only a read plan: which segments are still in the hot ring and which live as cold chunks in S3. Then it releases the lock. Materialization, including any S3 `GetObject` range reads, runs in a separately spawned task, so a slow cold fetch never blocks other commands waiting on the same group. Followers serve replicated historical catch-up bytes from their local applied state when the read does not need a leader-owned access mutation. Tail-sensitive responses such as `HEAD`, open-stream `Stream-Up-To-Date: true`, and live reads stay on the leader path (see [offsets](/docs/0.9/concepts/offsets)). ## SSE live tailing When a read finds no new data and the response carries `Stream-Up-To-Date: true`, the runtime registers a watcher inside the owning group actor. The watcher map is per-actor and mutated only inside that actor's turn, with no mutex and no awaiting while holding a lock. When a later append commits on the same group, the actor wakes the matching watchers in the same turn, deduplicates them by request shape, and dispatches materialization through the same fast path the read uses. For deployment topology (voting layout across availability zones, non-voting replicas, S3 prefix layout), see the [operations guide](/docs/0.9/operations). --- # Comparisons Ursula is for streams that diverse clients reach over the public internet with plain HTTP. One durable timeline per resource, and many small streams rather than a few big ones. Most streaming systems are built for a different shape. They assume an SDK client inside your network and a few high-throughput topics. That assumption decides most of the comparison below, so start there. ## Start with the access model Two questions decide whether Ursula fits. **Where do clients run, and how do they connect?** Ursula serves plain HTTP and SSE. A browser, a mobile app, a serverless function, or a third party can read, write, and tail a stream over the public internet with no SDK. Broker-style systems expect a client library inside the same network. **What is one stream?** In Ursula one stream maps to one resource: a document, a session, a task, an agent run. You address it by URL and keep thousands of them. Broker-style systems multiplex many resources into a few topics and partitions, and you rebuild per-resource state downstream. If your clients are in-network services and your data is a few high-throughput pipelines, Ursula is the wrong shape. If your clients are on the open internet and your data is many small per-resource timelines, the rest follows. ## Kafka / Redpanda Kafka and Redpanda are built for high-throughput pipelines. Topics, partitions, consumer groups, and SDK clients inside the cluster network. They are mature and fast at that job. They are not built for public-internet HTTP access or for thousands of small, independently addressable streams. There is no plain-HTTP read, write, and tail for an arbitrary client, and the unit of work is the topic, not the resource. Pick Kafka or Redpanda for a few high-throughput topics with in-network consumers. Pick Ursula for many small streams with open-internet clients. ## S2 / S2 Lite S2 is an object-storage-backed stream API and the closest alternative to Ursula. Acknowledged writes go through S3 Express One Zone, and the cold tier stays on Express-class storage. Self-hosted S2 Lite is a single serving process. The managed product uses its own API, not Durable Streams. S3 Express looks fine on its own. The point is that you do not need it. Ursula gets its write latency from an in-memory hot ring replicated by Raft quorum, which has nothing to do with the storage tier, and it keeps data on plain S3 Standard. In our 3-node benchmark Ursula commits appends faster than S2 Lite at every concurrency, since S2's per-append latency is bound by the S3 PUT round-trip. So S2 pays for a premium storage tier and still writes slower. | | Ursula | S2 | | --- | --- | --- | | Protocol | Durable Streams over plain HTTP and SSE, portable clients | Custom API, locked to one vendor | | Write latency | Single-digit to tens of ms p99 append | Hundreds of ms p99, bound by the S3 PUT | | Storage tier | Plain S3 Standard, commodity per-GB price | S3 Express, around 7× the per-GB storage bill | | Self-host | Raft cluster with quorum HA | Single process (Lite), no failover | On a workload that accumulates terabytes in the cold tier, the storage gap is the larger line on your bill, and it grows every month. Pick S2 for a managed service when you can adopt its API and pay for Express-class storage. Pick Ursula for lower-latency writes on an open protocol with a commodity S3 bill. ## S3 directly S3 is great for durable immutable blobs. It is a poor append log. Every append is a new object. Conditional writes are optimistic and retry-storm under load. Live readers have to poll. Ursula uses S3 for the cold tier and snapshots, never the write path. Appends commit at Raft quorum in an in-memory hot ring, and a background flush moves older data to S3. One read serves hot and cold transparently. Use S3 directly for immutable blobs with no append or tail. Use Ursula when you need an append log with live readers. ## etcd etcd is strongly consistent key-value over a single Raft log. It is excellent for small consistent state. Throughput is capped at one leader's serialization. Ursula runs many small Raft groups across cores and nodes, so throughput scales with the cluster. There is no single-log order across the whole namespace, but each stream stays linearizable. Use etcd for small consistent cluster-wide state. Use Ursula for many independent durable streams. ## When to pick which - Many small per-resource streams, open-internet HTTP clients, replay and live tail in one primitive: **Ursula**. - A few high-throughput topics with consumer groups: **Kafka / Redpanda**. - Managed service, willing to change API: **S2**. - Durable immutable blobs, no append or tail: **S3 directly**. - Small consistent key-value across the cluster: **etcd**. --- # Durable Streams Protocol > [!NOTE] > This page is a verbatim mirror of the upstream Durable Streams Protocol specification, authored by ElectricSQL. The authoritative source is [github.com/durable-streams/durable-streams](https://github.com/durable-streams/durable-streams/blob/main/PROTOCOL.md). Please open issues and pull requests against that repository, not Ursula's. Ursula's own [extensions](/docs/0.9/specs/extensions) are documented separately. **Document:** Durable Streams Protocol **Version:** 1.0 **Author:** ElectricSQL --- ## Abstract This document specifies the Durable Streams Protocol, an HTTP-based protocol for creating, appending to, and reading from durable, append-only byte streams. The protocol provides a simple, web-native primitive for applications requiring ordered, replayable data streams with support for catch-up reads, live tailing, and explicit stream closure (EOF). It is designed to be a foundation for higher-level abstractions such as event sourcing, database synchronization, collaborative editing, AI conversation histories, and finite response streaming. ## Copyright Notice Copyright (c) 2025 ElectricSQL ## Table of Contents 1. [Introduction](#1-introduction) 2. [Terminology](#2-terminology) 3. [Protocol Overview](#3-protocol-overview) 4. [Stream Model](#4-stream-model) - 4.1. [Stream Closure](#41-stream-closure) 5. [HTTP Operations](#5-http-operations) - 5.1. [Create Stream](#51-create-stream) - 5.2. [Append to Stream](#52-append-to-stream) - 5.2.1. [Idempotent Producers](#521-idempotent-producers) - 5.3. [Close Stream](#53-close-stream) - 5.4. [Delete Stream](#54-delete-stream) - 5.5. [Stream Metadata](#55-stream-metadata) - 5.6. [Read Stream - Catch-up](#56-read-stream---catch-up) - 5.7. [Read Stream - Live (Long-poll)](#57-read-stream---live-long-poll) - 5.8. [Read Stream - Live (SSE)](#58-read-stream---live-sse) 6. [Offsets](#6-offsets) 7. [Content Types](#7-content-types) 8. [Caching and Collapsing](#8-caching-and-collapsing) 9. [Extensibility](#9-extensibility) 10. [Security Considerations](#10-security-considerations) 11. [IANA Considerations](#11-iana-considerations) 12. [References](#12-references) --- ## 1. Introduction Modern web and cloud applications frequently require ordered, durable sequences of data that can be replayed from arbitrary points and tailed in real time. Common use cases include: - Database synchronization and change feeds - Event-sourced architectures - Collaborative editing and CRDTs - AI conversation histories and token streaming - Workflow execution histories - Real-time application state updates - Finite response streaming (proxied HTTP responses, job outputs, file transfers) While these patterns are widespread, the web platform lacks a simple, first-class primitive for durable streams. Applications typically implement ad-hoc solutions using combinations of databases, queues, and polling mechanisms, each reinventing similar offset-based replay semantics. The Durable Streams Protocol provides a minimal HTTP-based interface for durable, append-only byte streams. It is intentionally low-level and byte-oriented, allowing higher-level abstractions to be built on top without protocol changes. ## 2. Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. **Stream**: A URL-addressable, append-only byte stream that can be read and written to. A stream is simply a URL. The protocol defines how to interact with that URL using HTTP methods, query parameters, and headers. Streams are durable and immutable by position. New data can only be appended. **Offset**: An opaque, lexicographically sortable token that identifies a position within a stream. Clients use offsets to resume reading from a specific previously reached point. **Content Type**: A MIME type set on stream creation that describes the format of the stream's bytes. The content type is returned on reads and may be used by clients to interpret message boundaries. **Tail Offset**: The offset immediately after the last byte in the stream. This is the position where new appends will be written. **Closed Stream**: A stream that has been explicitly closed by a writer. Once closed, a stream is in a terminal state: no further appends are permitted, and readers can observe the closure as an end-of-stream (EOF) signal. Closure is durable and monotonic - once closed, a stream remains closed. ## 3. Protocol Overview The Durable Streams Protocol is an HTTP-based protocol that operates on URLs. A stream is simply a URL. The protocol defines how to interact with that URL using standard HTTP methods, query parameters, and custom headers. The protocol defines operations to create, append to, read, close, delete, and query metadata for streams. Reads have three modes: catch-up, long-poll, and Server-Sent Events (SSE). The primary operations are: 1. **Create**: Establish a new stream at a URL with optional initial content (PUT) 2. **Append**: Add bytes to the end of an existing stream (POST) 3. **Close**: Transition a stream to closed state, optionally with a final append (POST with `Stream-Closed: true`) 4. **Read**: Retrieve bytes starting from a given offset, with support for catch-up and live modes (GET) 5. **Delete**: Remove a stream (DELETE) 6. **Head**: Query stream metadata without transferring data (HEAD) The protocol does not prescribe a specific URL structure. Servers may organize streams using any URL scheme they choose (e.g., `/v1/stream/{path}`, `/{id}`, or domain-specific paths). The protocol is defined by the HTTP methods, query parameters, and headers applied to any stream URL. Streams support arbitrary content types. The protocol operates at the byte level, leaving message framing and schema interpretation to clients. **Independent Read/Write Implementation**: Servers **MAY** implement the read and write paths independently. For example, a database synchronization server may only implement the read path and use its own injection system for writes, while a collaborative editing service might implement both paths. ## 4. Stream Model A stream is an append-only sequence of bytes with the following properties: - **Durability**: Once written and acknowledged, bytes persist until the stream is deleted or expired - **Immutability by Position**: Bytes at a given offset never change. New data is only appended - **Ordering**: Bytes are strictly ordered by offset - **Content Type**: Each stream has a MIME content type set at creation - **TTL/Expiry**: Streams may have a sliding time-to-live window (resets on each read or write) or an absolute expiry time - **Retention**: Servers **MAY** implement retention policies that drop data older than a certain age while the stream continues. If a stream is deleted a new stream **SHOULD NOT** be created at the same URL. - **Stream State**: A stream is either **open** (accepts appends) or **closed** (no further appends permitted). Streams start in the open state and transition to closed via an explicit close operation. This transition is **durable** (persisted) and **monotonic** (once closed, a stream cannot be reopened). Clients track their position in a stream using offsets. Offsets are opaque to clients but are lexicographically sortable, allowing clients to determine ordering and resume from any point. ### 4.1. Stream Closure Stream closure provides an explicit end-of-stream (EOF) signal that allows readers to distinguish between "no data yet" and "no more data ever." This is essential for finite streams where writers need to signal completion, such as: - Proxied HTTP responses that have finished streaming - Completed job outputs or workflow executions - Finalized conversation histories or document streams **Properties of stream closure:** - **Durable**: The closed state is persisted and survives server restarts - **Monotonic**: Once closed, a stream cannot be reopened - **Idempotent**: Closing an already-closed stream succeeds (or returns a stable "already closed" response) - **Observable**: Readers can detect closure and treat it as EOF - **Atomic with final append**: Writers can atomically append a final message and close in a single operation After closure, the stream's data remains fully readable. Only new appends are rejected. **Stream-Closed Header Value:** The `Stream-Closed` header uses the value `true` (case-insensitive) to indicate closure. Servers **MUST** treat the header as present only when its value is exactly `true` (case-insensitive comparison). Other values such as `false`, `yes`, `1`, or empty string **MUST** be treated as if the header were absent. Servers **SHOULD NOT** return error responses for non-`true` values. They simply ignore the header. ## 5. HTTP Operations The protocol defines operations that are applied to a stream URL. The examples in this section use `{stream-url}` to represent any stream URL. Servers may implement any URL structure they choose. The protocol is defined by the HTTP methods, query parameters, and headers. ### 5.1. Create Stream #### Request ``` PUT {stream-url} ``` Where `{stream-url}` is any URL that identifies the stream to be created. Creates a new stream. If the stream already exists at `{stream-url}`, the server **MUST** either: - return `200 OK` if the existing stream's configuration (content type, TTL/expiry, and closure status) matches the request, or - return `409 Conflict` if it does not. This provides idempotent "create or ensure exists" semantics aligned with HTTP PUT expectations. **Closure status matching:** When checking for idempotent success (200 OK), servers **MUST** compare the `Stream-Closed` header in the request against the stream's current closure status. For example: - `PUT /stream` (no `Stream-Closed`) to an **open** stream with matching config -> `200 OK` - `PUT /stream` (no `Stream-Closed`) to a **closed** stream -> `409 Conflict` (closure status mismatch) - `PUT /stream + Stream-Closed: true` to a **closed** stream with matching config -> `200 OK` - `PUT /stream + Stream-Closed: true` to an **open** stream -> `409 Conflict` (closure status mismatch) #### Request Headers (Optional) - `Content-Type: ` - Sets the stream's content type. If omitted, the server **MAY** default to `application/octet-stream`. - `Stream-TTL: ` - Sets a sliding time-to-live window in seconds. The stream expires after being idle (no reads or writes) for this duration. Each read or write operation resets the expiry countdown to this value. `HEAD` requests do **not** reset the countdown. The value **MUST** be a non-negative integer in decimal notation without leading zeros, plus signs, decimal points, or scientific notation (e.g., `3600` is valid; `+3600`, `03600`, `3600.0`, and `3.6e3` are not). - TTL resets are a server-side concern: only requests that reach the origin server reset the countdown. Reads served from intermediate caches (CDN catch-up reads with `Cache-Control: public, max-age=60`) do not reach the server and do not reset the TTL. For live read modes (long-poll, SSE), the TTL resets when the server begins processing the request, not when data is delivered or the response completes. This means a stream with active live readers will not expire, even if no new data is being produced. - `Stream-Expires-At: ` - Sets an absolute expiry time as an RFC 3339 timestamp. - If both `Stream-TTL` and `Stream-Expires-At` are supplied, servers **SHOULD** reject the request with `400 Bad Request`. Implementations **MAY** define a deterministic precedence rule, but **MUST** document it. - `Stream-Closed: true` (optional) - When present, the stream is created in the **closed** state. Any body provided becomes the complete and final content of the stream. - This enables atomic "create and close" semantics for single-message or empty streams that are immediately complete (e.g., cached responses, placeholder errors, pre-computed results). - **Examples:** - `PUT /stream + Stream-Closed: true` (empty body): Creates an empty, immediately-closed stream (useful for "completed with no output" or error placeholders). - `PUT /stream + Stream-Closed: true + body`: Creates a single-shot stream with the body as its complete content (useful for cached responses, pre-computed results). #### Request Body (Optional) - Initial stream bytes. If provided, these bytes form the first content of the stream. #### Response Codes - `201 Created`: Stream created successfully - `200 OK`: Stream already exists with matching configuration (idempotent success) - `409 Conflict`: Stream already exists with different configuration - `400 Bad Request`: Invalid headers or parameters (including conflicting TTL/expiry) - `429 Too Many Requests`: Rate limit exceeded #### Response Headers (on 201 or 200) - `Location: {stream-url}` (on 201): Servers **SHOULD** include a `Location` header equal to `{stream-url}` in `201 Created` responses. - `Content-Type: `: The stream's content type - `Stream-Next-Offset: `: The tail offset after any initial content - `Stream-Closed: true`: Present when the stream was created in the closed state ### 5.2. Append to Stream #### Request ``` POST {stream-url} ``` Where `{stream-url}` is the URL of an existing stream. Appends bytes to the end of an existing stream. Supports both full-body and streaming (chunked) append operations. Optionally closes the stream atomically with the append. Servers that do not support appends for a given stream **SHOULD** return `405 Method Not Allowed` or `501 Not Implemented` to `POST` requests on that URL. #### Request Headers - `Content-Type: ` - **MUST** match the stream's existing content type when a body is provided. Servers **MUST** return `409 Conflict` when the content type is valid but does not match the stream's configured type. - **MAY** be omitted when the request body is empty (i.e., close-only requests with `Stream-Closed: true`). When the request body is empty, servers **MUST NOT** reject based on `Content-Type` and **MAY** ignore it entirely. This ensures close-only requests remain robust even when clients/libraries attach default `Content-Type` headers. - `Transfer-Encoding: chunked` (optional) - Indicates a streaming body. Servers **SHOULD** support HTTP/1.1 chunked encoding and HTTP/2 streaming semantics. - `Stream-Seq: ` (optional) - A monotonic, lexicographic writer sequence number for coordination. - `Stream-Seq` values are opaque strings that **MUST** compare using simple byte-wise lexicographic ordering. Sequence numbers are scoped per authenticated writer identity (or per stream, depending on implementation). Servers **MUST** document the scope they enforce. - If provided and less than or equal to the last appended sequence (as determined by lexicographic comparison), the server **MUST** return `409 Conflict`. Sequence numbers **MUST** be strictly increasing. - `Stream-Closed: true` (optional) - When present with value `true`, the stream is **closed** after the append completes. This is an atomic operation: the body (if any) is appended as the final data, and the stream transitions to the closed state in the same step. - If the request body is empty (Content-Length: 0 or no body), the stream is closed without appending any data. This is the only case where an empty POST body is valid. - Once closed, the stream rejects all subsequent appends with `409 Conflict` (see below). - **Close-only requests are idempotent**: if the stream is already closed and the request includes `Stream-Closed: true` with an empty body, servers **SHOULD** return `204 No Content` with `Stream-Closed: true`. - **Append-and-close requests are NOT idempotent** (without idempotent producer headers): if the stream is already closed and the request includes a body but no idempotent producer headers, servers **MUST** return `409 Conflict` with `Stream-Closed: true`, since the body cannot be appended. However, if idempotent producer semantics apply and the request matches the `(producerId, epoch, seq)` tuple that performed the closing append, servers treat it as a deduplicated success (see Section 5.2.1). #### Request Body - Bytes to append to the stream. Servers **MUST** reject POST requests with an empty body (Content-Length: 0 or no body) with `400 Bad Request`, **unless** the `Stream-Closed: true` header is present (which allows closing without appending data). #### Response Codes - `204 No Content`: Append successful (or stream already closed when closing idempotently) - `400 Bad Request`: Malformed request (invalid header syntax, missing Content-Type, empty body without `Stream-Closed: true`) - `404 Not Found`: Stream does not exist - `405 Method Not Allowed` or `501 Not Implemented`: Append not supported for this stream - `409 Conflict`: Content type mismatch with stream's configured type, sequence regression (if `Stream-Seq` provided), or **stream is closed** (when attempting to append without `Stream-Closed: true`) - `413 Payload Too Large`: Request body exceeds server limits - `429 Too Many Requests`: Rate limit exceeded #### Response Headers (on success) - `Stream-Next-Offset: `: The new tail offset after the append - `Stream-Closed: true`: Present when the stream is now closed (either by this request or previously) #### Response Headers (on 409 Conflict due to closed stream) When a client attempts to append to a closed stream (without `Stream-Closed: true`), servers **MUST** return: - `409 Conflict` status code - `Stream-Closed: true` header - `Stream-Next-Offset: `: The final offset of the closed stream (useful for clients to know the stream's final position) This allows clients to detect and handle the "stream already closed" condition programmatically without parsing the response body. Servers **SHOULD** keep the response body empty or use a standardized error format. Clients **SHOULD NOT** rely on parsing the body to determine the reason for rejection. **Error Precedence:** When an append request would trigger multiple conflict conditions (e.g., stream is closed AND content type mismatches), servers **SHOULD** check the stream's closed status first. This ensures clients receive the `Stream-Closed: true` header, enabling correct error handling. The recommended precedence is: 1. Stream closed -> `409 Conflict` with `Stream-Closed: true` 2. Content type mismatch -> `409 Conflict` 3. Sequence regression -> `409 Conflict` ### 5.2.1. Idempotent Producers Durable Streams supports Kafka-style idempotent producers for exactly-once write semantics. This enables fire-and-forget writes with server-side deduplication, eliminating duplicates from client retries. #### Design - **Client-provided producer IDs**: Zero RTT overhead, no handshake required - **Client-declared epochs, server-validated fencing**: Client increments epoch on restart. Server validates monotonicity and fences stale epochs - **Per-batch sequence numbers**: Separate from `Stream-Seq`, used for retry safety - **Two-layer sequence design**: - Transport layer: `Producer-Id` + `Producer-Epoch` + `Producer-Seq` (retry safety) - Application layer: `Stream-Seq` (cross-restart ordering, lexicographic) #### Request Headers All three producer headers **MUST** be provided together or none at all. If only some headers are provided, servers **MUST** return `400 Bad Request`. - `Producer-Id: ` - Client-supplied stable identifier (e.g., "order-service-1", UUID) - **MUST** be a non-empty string. Empty values result in `400 Bad Request` - Identifies the logical producer across restarts - `Producer-Epoch: ` - Client-declared epoch, starting at 0 - Increment on producer restart to establish a new session - Server validates that epoch is monotonically non-decreasing - **MUST** be a non-negative integer ≤ 2^53-1 (for JavaScript interoperability) - `Producer-Seq: ` - Monotonically increasing sequence number per epoch - Starts at 0 for each new epoch - Applies per-batch (per HTTP request), not per-message - **MUST** be a non-negative integer ≤ 2^53-1 (for JavaScript interoperability) #### Response Headers - `Producer-Epoch: `: Echoed back on success (200/204), or current server epoch on stale epoch (403) - `Producer-Seq: `: On success (200/204), the highest accepted sequence number for this `(stream, producerId, epoch)` tuple. Enables clients to confirm pipelined requests and recover state after crashes. - `Producer-Expected-Seq: `: On 409 Conflict (sequence gap), the expected sequence - `Producer-Received-Seq: `: On 409 Conflict (sequence gap), the received sequence #### Validation Logic ``` # Epoch validation (client-declared, server-validated) if epoch < state.epoch: -> 403 Forbidden -> Headers: Producer-Epoch: if epoch > state.epoch: if seq != 0: -> 400 Bad Request (new epoch must start at seq=0) -> Accept: update state.epoch = epoch, state.lastSeq = 0 -> 200 OK (new epoch established) # Same epoch: sequence validation if seq <= state.lastSeq: -> 204 No Content (duplicate, idempotent success) -> Ursula returns the original byte range when seq is the newest accepted seq, and no byte range for an older one (extensions §5.3) if seq == state.lastSeq + 1: -> Accept, update state.lastSeq = seq -> 200 OK if seq > state.lastSeq + 1: -> 409 Conflict -> Headers: Producer-Expected-Seq: , Producer-Received-Seq: ``` #### Response Codes (with Producer Headers) - `200 OK`: Append successful (new data) - `204 No Content`: Duplicate append (idempotent success, data already exists) - `400 Bad Request`: Invalid producer headers (e.g., non-integer values, epoch increase with seq != 0) - `403 Forbidden`: Stale producer epoch (zombie fencing). Response includes `Producer-Epoch` header with current server epoch. - `409 Conflict`: Sequence gap detected. Response includes `Producer-Expected-Seq` and `Producer-Received-Seq` headers. > **Note (informative).** Ursula refuses producer headers on create, adds `Producer-Expected-Seq: 0` to the `400` of an epoch increase with a non-zero seq, types its producer refusals with `Producer-Outcome`, and refuses a write with `Stream-Incarnation` that could reach a producer epoch at or below a stream's epoch floor. A write without it keeps this section's rule that a producer whose state expired is treated as new (see [extensions Section 5](/docs/0.9/specs/extensions#5-write-keys-and-typed-answers)). #### Bootstrap and Restart Flow 1. **Initial start (epoch=0)**: - Producer sends `(epoch=0, seq=0)` - Server accepts, establishes producer state 2. **Producer restart**: - Producer increments local epoch (0 -> 1), resets seq to 0 - Sends `(epoch=1, seq=0)` - Server sees epoch > state.epoch, accepts, updates state 3. **Zombie fencing**: - Old producer (zombie) still sending `(epoch=0, seq=N)` gets 403 Forbidden - Response includes `Producer-Epoch: 1` header #### Auto-claim Flow (for ephemeral producers) For serverless or ephemeral producers without persisted epoch: 1. Producer starts fresh with `(epoch=0, seq=0)` 2. If server has `state.epoch=5`, returns 403 with `Producer-Epoch: 5` 3. Client can retry with `(epoch=6, seq=0)` to claim the producer ID This is opt-in client behavior and should be used with caution. #### Concurrency Requirements Servers **MUST** serialize validation + append operations per `(stream, producerId)` pair. HTTP requests can arrive out-of-order. Without serialization, seq=1 arriving before seq=0 would cause false sequence gaps. #### Atomicity Requirements For persistent storage, servers **SHOULD** commit producer state updates and log appends atomically (e.g., in a single database transaction). Non-atomic implementations have a crash window where: 1. Data is appended to the log 2. Crash occurs before producer state is updated 3. On recovery, a retry may be re-accepted, causing duplicate data **Recovery for non-atomic stores**: Clients can bump their epoch after a crash to establish a clean session. This trades "exactly once within epoch" for "at least once across crashes" which is acceptable for many use cases. Stores **SHOULD** document their atomicity guarantees clearly. #### Producer State Cleanup Servers **MAY** implement TTL-based cleanup for producer state: - **In-memory stores**: 7 days TTL recommended, clean up on stream access - **Persistent stores**: Retain as long as stream data exists (stronger guarantee) After state expiry, the producer is treated as new. A zombie alive past TTL expiry can write again, which is acceptable for testing but persistent stores should use longer retention. #### Stream Closure with Idempotent Producers Idempotent producers can close streams using the `Stream-Closed: true` header. The behavior is: - **Close with final append**: Include body, producer headers, and `Stream-Closed: true`. The append is deduplicated normally, and the stream closes atomically with the final append. - **Close without append**: Include `Stream-Closed: true` with empty body. Producer headers are optional but if provided, the close operation is still idempotent. - **Duplicate close**: If the stream was already closed by the same `(producerId, epoch, seq)` tuple, servers **SHOULD** return `204 No Content` with `Stream-Closed: true`. When a closed stream receives an append from an idempotent producer: - If the `(producerId, epoch, seq)` matches the request that closed the stream, return `204 No Content` (duplicate/idempotent success) with `Stream-Closed: true` - Otherwise, return `409 Conflict` with `Stream-Closed: true` (stream is closed, no further appends allowed) ### 5.3. Close Stream To close a stream without appending data, send a POST request with `Stream-Closed: true` and an empty body: #### Request ``` POST {stream-url} Stream-Closed: true ``` #### Response Codes - `204 No Content`: Stream closed successfully (or already closed - idempotent) - `404 Not Found`: Stream does not exist - `405 Method Not Allowed` or `501 Not Implemented`: Append/close not supported for this stream #### Response Headers - `Stream-Next-Offset: `: The tail offset (unchanged, since no data was appended) - `Stream-Closed: true`: Confirms the stream is now closed This is the canonical "close-only" operation. For atomic "append final message and close", include a request body as described in Section 5.2. ### 5.4. Delete Stream #### Request ``` DELETE {stream-url} ``` Where `{stream-url}` is the URL of the stream to delete. Deletes the stream and all its data. In-flight reads may terminate with a `404 Not Found` on subsequent requests after deletion. #### Response Codes - `204 No Content`: Stream deleted successfully - `404 Not Found`: Stream does not exist - `405 Method Not Allowed` or `501 Not Implemented`: Delete not supported for this stream ### 5.5. Stream Metadata #### Request ``` HEAD {stream-url} ``` Where `{stream-url}` is the URL of the stream. Checks stream existence and returns metadata without transferring a body. This is the canonical way to find the tail offset, TTL, expiry information, and **closure status**. #### Response Codes - `200 OK`: Stream exists - `404 Not Found`: Stream does not exist - `429 Too Many Requests`: Rate limit exceeded #### Response Headers (on 200) - `Content-Type: `: The stream's content type - `Stream-Next-Offset: `: The tail offset (next offset after the current end) - `Stream-TTL: ` (optional): The stream's time-to-live window. Each read or write resets the expiry countdown to this value. - `Stream-Expires-At: ` (optional): Absolute expiry time, if applicable - `Stream-Closed: true` (optional): Present when the stream has been closed. Absence indicates the stream is still open. - `Cache-Control`: See Section 8 #### Caching Guidance Servers **SHOULD** make `HEAD` responses effectively non-cacheable, for example by returning `Cache-Control: no-store`. Servers **MAY** use `Cache-Control: private, max-age=0, must-revalidate` as an alternative, but `no-store` is recommended to avoid stale tail offsets and closure status. ### 5.6. Read Stream - Catch-up #### Request ``` GET {stream-url}?offset= ``` Where `{stream-url}` is the URL of the stream. Returns bytes starting from the specified offset. This is used for catch-up reads when a client needs to replay stream content from a known position. #### Query Parameters - `offset` (optional) - Start offset token. If omitted, defaults to the stream start (offset -1). - `consistency` (optional) - `local` (default) permits immutable catch-up bytes from a follower's applied state. `leader` routes the read to the current Raft leader and is intended for recovery that must observe an acknowledged write before proceeding. `leader` cannot be combined with a live read mode. #### Response Codes - `200 OK`: Data available (or empty body if offset equals tail) - `400 Bad Request`: Malformed offset or invalid parameters - `404 Not Found`: Stream does not exist - `410 Gone`: Offset is before the earliest retained position (retention/compaction) - `429 Too Many Requests`: Rate limit exceeded For non-live reads without data beyond the requested offset, servers **SHOULD** return `200 OK` with an empty body and `Stream-Next-Offset` equal to the requested offset. If the stream is closed, this response **MUST** also include `Stream-Closed: true` to signal EOF. #### Response Headers (on 200) - `Cache-Control`: Derived from TTL/expiry (see Section 8) - `ETag: {internal_stream_id}:{start_offset}:{end_offset}` - Entity tag for cache validation - `Stream-Cursor: ` (optional for catch-up, required for live modes) - Cursor to echo on subsequent long-poll requests to improve CDN collapsing. Servers **MAY** include this on catch-up reads. It is **required** for live modes when the stream is open (see Sections 5.7, 5.8). Servers **MAY** omit it when `Stream-Closed` is true. Clients **MUST** tolerate its absence when `Stream-Closed` is present. - `Stream-Next-Offset: ` - The next offset to read from (for subsequent requests) - `Stream-Up-To-Date: true` - **MUST** be present and set to `true` when the response includes all data available in the stream at the time the response was generated (i.e., when the requested offset has reached the tail and no more data exists). - **SHOULD NOT** be present when returning partial data due to server-defined chunk size limits (when more data exists beyond what was returned). - Clients **MAY** use this header to determine when they have caught up and can transition to live tailing mode. - **Important:** `Stream-Up-To-Date: true` does **NOT** imply EOF. More data may be appended in the future. Only `Stream-Closed: true` indicates that no more data will ever arrive. - `Stream-Closed: true` - **MUST** be present when the stream is closed **and** the client has reached the final offset **at the time the response is generated**. This includes: - Responses that return the final chunk of data, when the stream is already closed at response generation time, or - Responses with an empty body when the requested offset equals the tail offset of a closed stream (the canonical EOF signal). - When present, clients can conclude that no more data will ever be appended and treat this as EOF. - **SHOULD NOT** be present when returning partial data from a closed stream (when more data exists between the response and the final offset). In this case, `Stream-Closed: true` will be returned on a subsequent request that reaches the final offset. - **Timing note:** If a stream is closed **after** the final chunk was served (or cached), that chunk will not include `Stream-Closed: true`. Clients discover closure by requesting the next offset (`Stream-Next-Offset` from the previous response), which returns an empty body with `Stream-Closed: true`. This is the expected flow when closure occurs between chunk responses or when serving cached chunks. - Clients that need to know closure status before reaching the tail **SHOULD** use `HEAD` (see Section 5.5). #### Response Body - Bytes from the stream starting at the specified offset, up to a server-defined maximum chunk size. ### 5.7. Read Stream - Live (Long-poll) #### Request ``` GET {stream-url}?offset=&live=long-poll[&cursor=] ``` Where `{stream-url}` is the URL of the stream. If no data is available at the specified offset, the server waits up to a timeout for new data to arrive. This enables efficient live tailing without constant polling. #### Query Parameters - `offset` (required) - The offset to read from. **MUST** be provided. - `live=long-poll` (required) - Indicates long-polling mode. - `cursor` (optional) - Echo of the last `Stream-Cursor` header value from a previous response. - Used for collapsing keys in CDN/proxy configurations. #### Response Codes - `200 OK`: Data became available within the timeout - `204 No Content`: Timeout expired with no new data - `400 Bad Request`: Invalid parameters - `404 Not Found`: Stream does not exist - `429 Too Many Requests`: Rate limit exceeded #### Response Headers (on 200) - Same as catch-up reads (Section 5.6), plus: - `Stream-Cursor: `: Servers **MUST** include this header. See Section 8.1. #### Response Headers (on 204) - `Stream-Next-Offset: `: Servers **MUST** include a `Stream-Next-Offset` header indicating the current tail offset. - `Stream-Up-To-Date: true`: Servers **MUST** include this header to indicate the client is caught up with all available data. - `Stream-Cursor: `: Servers **MUST** include this header when the stream is open. Servers **MAY** omit this header when `Stream-Closed` is true (cursor is unnecessary when no further polling is expected). Clients **MUST** tolerate its absence when `Stream-Closed` is present. See Section 8.1. - `Stream-Closed: true`: **MUST** be present when the stream is closed (see Section 5.6 for semantics). A `204 No Content` with `Stream-Closed: true` indicates EOF. **EOF Signaling Across Modes:** Clients should treat **either** of the following as EOF, depending on the mode used: - **Catch-up mode**: `200 OK` with empty body and `Stream-Closed: true` - **Long-poll mode**: `204 No Content` with `Stream-Closed: true` - **SSE mode**: `control` event with `streamClosed: true` In all cases, `Stream-Closed` / `streamClosed` is the definitive EOF signal. The presence of `Stream-Up-To-Date` / `upToDate` alone does **not** indicate EOF - it only means the client has caught up with currently available data, but more may arrive. #### Stream Closure Behavior in Long-poll Mode When the stream is closed and the client is already at the tail offset: - Servers **MUST NOT** wait for the long-poll timeout - Servers **MUST** immediately return `204 No Content` with `Stream-Closed: true` and `Stream-Up-To-Date: true` This ensures clients observing a closed stream do not have hanging connections waiting for data that will never arrive. #### Response Body (on 200) - New bytes that arrived during the long-poll period. #### Timeout Behavior The timeout for long-polling is implementation-defined. Servers **MAY** accept a `timeout` query parameter (in seconds) as a future extension, but this is not required by the base protocol. ### 5.8. Read Stream - Live (SSE) #### Request ``` GET {stream-url}?offset=&live=sse ``` Where `{stream-url}` is the URL of the stream. Returns data as a Server-Sent Events (SSE) stream. SSE mode supports all content types. Servers **MUST** include `stream-data-content-type` to identify the decoded `event: data` payload type. For `application/json` streams, this value is `application/x-ndjson`. For all other streams, this value is the stream's configured content type. For streams with `content-type: text/*` or `application/json`, data events carry UTF-8 text directly. For streams with any other `content-type` (binary streams), servers **MUST** automatically base64-encode data events and include the response header `stream-sse-data-encoding: base64`. SSE responses **MUST** use `Content-Type: text/event-stream` in the HTTP response headers. Clients **MUST** check `stream-sse-data-encoding` before interpreting `event: data`. If it is `base64`, clients decode the event data first, then interpret the decoded bytes according to `stream-data-content-type`. For `application/x-ndjson` SSE payloads, `data:` lines are transport lines, not record boundaries; clients MUST buffer until newline before parsing records. #### Query Parameters - `offset` (required) - The offset to start reading from. - `live=sse` (required) - Indicates SSE streaming mode. - `max_bytes` (optional) - Maximum bytes to read in one data batch before SSE framing. For non-base64 UTF-8 data, servers **MUST NOT** emit invalid UTF-8; if a read window ends inside a code point, servers shorten the emitted payload and set `streamNextOffset` to the offset just past the emitted data. For `application/x-ndjson` SSE payloads, servers **SHOULD** prefer ending at the last complete newline-delimited record when one is available, but **MAY** emit a partial record to make progress. #### Response Codes - `200 OK`: Streaming body (SSE format) - `400 Bad Request`: Invalid parameters - `404 Not Found`: Stream does not exist - `429 Too Many Requests`: Rate limit exceeded #### Response Format Data is emitted in [Server-Sent Events format](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). **Events:** - `data`: Emitted for each batch of data - Each line prefixed with `data:` - The decoded data payload type is identified by the response header `stream-data-content-type`. - `data:` lines are transport framing and are not guaranteed to align with application message boundaries. - For binary streams (where `stream-sse-data-encoding: base64` is present), the `data` event payload represents bytes encoded using standard base64 per [RFC 4648](https://www.rfc-editor.org/rfc/rfc4648) (alphabet: A-Z, a-z, 0-9, +, /). - Servers **MAY** split the base64 text across multiple `data:` lines within the same SSE `data` event. - Clients **MUST** concatenate the `data:` lines for the event (per SSE rules) and **MUST** remove all `\n` and `\r` characters inserted between lines before base64-decoding. - The resulting string (after removing `\n` and `\r`) **MUST** be valid base64 text with length that is a multiple of 4 (or empty). - If a `data` event's byte payload length is 0, the base64 text **MUST** be the empty string. - Base64 encoding affects only `event: data` payloads. `event: control` events remain JSON as specified and are not encoded. - When the stream content type is `application/json`, implementations **MAY** batch multiple logical messages into a single SSE `data` event by streaming newline-delimited JSON across multiple `data:` lines, as in the example below. - `control`: Emitted after every data event - **MUST** include `streamNextOffset`. See Section 8.1. - **MUST** include `streamCursor` when the stream is open. Servers **MAY** omit `streamCursor` when `streamClosed` is true (cursor is unnecessary when no reconnection is expected). - **MUST** include `upToDate: true` when the client is caught up with all available data. Note: `streamClosed: true` implies `upToDate: true` (a closed stream at the final offset is by definition up-to-date), so `upToDate` **MAY** be omitted when `streamClosed` is true. - **MUST** include `streamClosed: true` when the stream is closed and all data up to the final offset has been sent. - Format: JSON object with offset, cursor (when applicable), up-to-date status, and optionally closed status. Field names use camelCase: `streamNextOffset`, `streamCursor`, `upToDate`, and `streamClosed`. **Example (normal data):** ``` event: data data: {"k":"v"} data: {"k":"w"} event: control data: {"streamNextOffset":"123456_789","streamCursor":"abc"} ``` **Example (final data with stream closure):** ``` event: data data: {"k":"final"} event: control data: {"streamNextOffset":"123456_999","streamClosed":true} ``` Note: `streamCursor` is omitted when `streamClosed` is true, since clients must not reconnect after receiving a closed signal. **Client Compatibility:** Clients **MUST** tolerate the absence of `streamCursor` (in SSE) and `Stream-Cursor` (in HTTP headers) when `streamClosed` / `Stream-Closed` is present. Implementations that assume cursor is always present will break when processing closed stream responses. #### Stream Closure Behavior in SSE Mode When the stream is closed: - The final `control` event **MUST** include `streamClosed: true` - After emitting the final control event, servers **MUST** close the SSE connection - Clients receiving `streamClosed: true` **MUST NOT** attempt to reconnect, as no more data will arrive If the stream is already closed when an SSE connection is established and the client's offset is at the tail: - Servers **MUST** immediately emit a `control` event with `streamClosed: true` and `upToDate: true` - Servers **MUST** then close the connection **Example (binary stream with automatic base64 encoding):** ``` event: data data: AQIDBAUG data: BwgJCg== event: control data: {"streamNextOffset":"123456_789","streamCursor":"abc"} ``` #### Connection Lifecycle - Server **SHOULD** close connections roughly every ~60 seconds to enable CDN collapsing - Client **MUST** reconnect using the last received `streamNextOffset` value from the control event - Client **MUST NOT** reconnect if the last control event included `streamClosed: true` ## 6. Offsets Offsets are opaque tokens that identify positions within a stream. They have the following properties: 1. **Opaque**: Clients **MUST NOT** interpret offset structure or meaning 2. **Lexicographically Sortable**: For any two valid offsets for the same stream, a lexicographic comparison determines their relative position in the stream. Clients **MAY** compare offsets lexicographically to determine ordering. 3. **Persistent**: Offsets remain valid for the lifetime of the stream (until deletion or expiration) 4. **Unique**: Each offset identifies exactly one position in the stream. No two positions **MAY** share the same offset. 5. **Strictly Increasing**: Offsets assigned to appended data **MUST** be lexicographically greater than all previously assigned offsets. Server implementations **MUST NOT** use schemes (such as raw UTC timestamps) that can produce duplicate or non-monotonic offsets. Time-based identifiers like ULIDs, which combine timestamps with random components to guarantee uniqueness and monotonicity, are acceptable. **Format**: Offset tokens are opaque, case-sensitive strings. Their internal structure is implementation-defined. Offsets are single tokens and **MUST NOT** contain `,`, `&`, `=`, `?`, or `/` (to avoid conflict with URL query parameter syntax). Servers **SHOULD** use URL-safe characters to avoid encoding issues, but clients **MUST** properly URL-encode offset values when including them in query parameters. Servers **SHOULD** keep offsets reasonably short (under 256 characters) since they appear in every request URL. **Sentinel Values**: The protocol defines two special offset sentinel values: - **`-1` (Stream Beginning)**: The special offset value `-1` represents the beginning of the stream. Clients **MAY** use `offset=-1` as an explicit way to request data from the start. This is semantically equivalent to omitting the offset parameter. Servers **MUST** recognize `-1` as a valid offset that returns data from the beginning of the stream. - **`now` (Current Tail Position)**: The special offset value `now` allows clients to skip all existing data and begin reading from the current tail position. This is useful for applications that only care about future data (e.g., presence tracking, live monitoring, late joiners to a conversation). The behavior varies by read mode: **Catch-up mode** (`offset=now` without `live` parameter): - Servers **MUST** return `200 OK` with an empty response body. - Servers **MUST** include a `Stream-Next-Offset` header set to the current tail position - Servers **MUST** include `Stream-Up-To-Date: true` header - Servers **SHOULD** return `Cache-Control: no-store` to prevent caching of the tail offset - The response **MUST** contain no data messages, regardless of stream content **Long-poll mode** (`offset=now&live=long-poll`): - Servers **MUST** immediately begin waiting for new data (no initial empty response) - This eliminates a round-trip: clients can subscribe to future data in a single request - If new data arrives during the wait, servers return `200 OK` with the new data - If the timeout expires, servers return `204 No Content` with `Stream-Up-To-Date: true` - The `Stream-Next-Offset` header **MUST** be set to the tail position **SSE mode** (`offset=now&live=sse`): - Servers **MUST** immediately begin the SSE stream from the tail position - The first control event **MUST** include the tail offset in `streamNextOffset` - If no data has arrived, the first control event **MUST** include `upToDate: true` - If data arrives before the first control event, `upToDate` reflects the current state - No historical data is sent. Only future data events are streamed **Closed streams** (`offset=now` on a closed stream): - Regardless of the `live` parameter, servers **MUST** return immediately with the closure signal - The response **MUST** include `Stream-Closed: true` and `Stream-Up-To-Date: true` headers - The `Stream-Next-Offset` header **MUST** be set to the stream's final (tail) offset - For catch-up mode: `200 OK` with empty body - For long-poll mode: `204 No Content` (no waiting, immediate return) - For SSE mode: The first (and only) control event includes `streamClosed: true` and `upToDate: true`, then the connection closes - This ensures clients using `offset=now` can immediately discover that a stream has no future data **Reserved Values**: The sentinel values `-1` and `now` are reserved by the protocol. Server implementations **MUST NOT** generate these strings as actual stream offsets (in `Stream-Next-Offset` headers or SSE control events). This ensures clients can always distinguish between sentinel requests and real offset values. The opaque nature of offsets enables important server-side optimizations. For example, offsets may encode chunk file identifiers, allowing catch-up requests to be served directly from object storage without touching the main database. Clients **MUST** use the `Stream-Next-Offset` value returned in responses for subsequent read requests. They **SHOULD** persist offsets locally (e.g., in browser local storage or a database) to enable resumability after disconnection or restart. ## 7. Content Types The protocol supports arbitrary MIME content types. Most content types operate at the byte level, leaving message framing and interpretation to clients. The `application/json` content type has special semantics defined below. **SSE Encoding:** - SSE mode (Section 5.8) supports all content types. For streams with `content-type: text/*` or `application/json`, data events carry UTF-8 text natively. For all other content types, servers automatically base64-encode data events (see Section 5.8). Clients **MAY** use any content type for their streams, including: - `application/json` for JSON mode with message boundary preservation - `application/ndjson` for newline-delimited JSON - `application/x-protobuf` for Protocol Buffer messages - `text/plain` for plain text - Custom types for application-specific formats ### 7.1. JSON Mode Streams created with `Content-Type: application/json` have special semantics for message boundaries and batch operations. #### Message Boundaries For `application/json` streams, servers **MUST** preserve message boundaries by storing each normalized message as one newline-delimited JSON record. `HEAD` responses report the configured stream content type (`application/json`). GET responses **MUST** return those records as newline-delimited JSON with `Content-Type: application/x-ndjson`, because that is the read representation. #### Array Flattening for Batch Operations When a POST request body contains a JSON array, servers **MUST** flatten exactly one level of the array, treating each element as a separate message. This enables clients to batch multiple messages in a single HTTP request while preserving individual message semantics. **Examples (direct POST to server):** - POST body `{"event": "created"}` stores one message: `{"event": "created"}` - POST body `[{"event": "a"}, {"event": "b"}]` stores two messages: `{"event": "a"}`, `{"event": "b"}` - POST body `[[1,2], [3,4]]` stores two messages: `[1,2]`, `[3,4]` - POST body `[[[1,2,3]]]` stores one message: `[[1,2,3]]` **Note:** Client libraries **MAY** automatically wrap individual values in arrays for batching. For example, a client calling `append({"x": 1})` might send POST body `[{"x": 1}]` to the server, which flattens it to store one message: `{"x": 1}`. #### Empty Arrays Servers **MUST** reject POST requests containing empty JSON arrays (`[]`) with `400 Bad Request`. Empty arrays in append operations represent no-op operations with no semantic meaning and likely indicate a client bug. PUT requests with an empty array body (`[]`) are valid and create an empty stream. The empty array simply means no initial messages are being written. #### JSON Validation Servers **MUST** validate that appended data is valid JSON. If validation fails, servers **MUST** return `400 Bad Request` with an appropriate error message. #### Response Format GET responses for `application/json` streams **MUST** return `Content-Type: application/x-ndjson` with a body containing newline-delimited JSON messages from the requested offset range. `max_bytes` applies to the encoded byte stream and MAY end a response mid-record; clients MUST resume from `Stream-Next-Offset` and buffer incomplete trailing records before parsing. ```http HTTP/1.1 200 OK Content-Type: application/x-ndjson {"event":"created"} {"event":"updated"} ``` If no messages exist in the range, servers **MUST** return an empty body. ## 8. Caching and Collapsing ### 8.1. Catch-up and Long-poll Reads For **shared, non-user-specific streams**, servers **SHOULD** return: ``` Cache-Control: public, max-age=60, stale-while-revalidate=300 ``` For **streams that may contain user-specific or confidential data**, servers **SHOULD** use `private` instead of `public` and rely on CDN configurations that respect `Authorization` or other cache keys: ``` Cache-Control: private, max-age=60, stale-while-revalidate=300 ``` This enables CDN/proxy caching while allowing stale content to be served during revalidation. **Caching and Stream Closure:** Catch-up chunks remain fully cacheable, including chunks at the tail of the stream. When a chunk is returned, it may or may not be the final chunk - this is unknown until the client requests the next offset. The closure signal is discovered when the client requests the offset **after** the final data: 1. Client reads data and receives `Stream-Next-Offset: X` (the tail offset) 2. Client requests offset `X` 3. If stream is closed: server returns `200 OK` with **empty body** and `Stream-Closed: true` 4. If stream is open: server returns `200 OK` with empty body and `Stream-Up-To-Date: true` (or long-poll/SSE waits for data) This design ensures: - All data chunks are cacheable (a chunk that later becomes "final" was still valid data) - The closure signal is a distinct request/response at the tail offset - Cached chunks never become "stale" due to closure - clients simply make one more request to discover EOF **ETag Usage:** Servers **MUST** generate `ETag` headers for GET responses, except for `offset=now` responses. Clients **MAY** use `If-None-Match` with the `ETag` value on repeat catch-up requests. When a client provides a valid `If-None-Match` header that matches the current ETag, servers **MUST** respond with `304 Not Modified` (with no body) instead of re-sending the same data. This is essential for fast loading and efficient bandwidth usage. **ETag and Stream Closure:** ETags **MUST** vary with the stream's closure status. When a stream is closed (without new data being appended), the ETag **MUST** change to ensure clients do not receive `304 Not Modified` responses that would hide the closure signal. Implementations **SHOULD** include a closure indicator in the ETag format (e.g., appending `:c` to the ETag when the stream is closed). **Query Parameter Ordering:** For optimal cache behavior, clients **SHOULD** order query parameters lexicographically by key name. This ensures consistent URL serialization across implementations and improves CDN cache hit rates. **Collapsing:** Clients **SHOULD** echo the `Stream-Cursor` value as `cursor=` in subsequent long-poll requests. This, along with the appropriate `Cache-Control` header, enables CDNs and proxies to collapse multiple clients waiting for the same data into a single upstream request. **Server-Generated Cursors:** To prevent infinite CDN cache loops (where clients receive the same cached empty response indefinitely), servers **MUST** generate cursors on all live mode responses: - **Long-poll**: `Stream-Cursor` response header - **SSE**: `streamCursor` field in `control` events The cursor mechanism works as follows: 1. **Interval-based Calculation**: Servers divide time into fixed intervals (default: 20 seconds) counted from an epoch (default: October 9, 2024 00:00:00 UTC). The cursor value is the interval number as a decimal string. 2. **Cursor Generation**: For each live response, the server calculates the current interval number and returns it as the cursor value. 3. **Monotonic Progression**: Servers **MUST** ensure cursors never go backwards. When a client provides a `cursor` query parameter that is greater than or equal to the current interval number, the server **MUST** return a cursor strictly greater than the client's cursor (by adding random jitter of 1-3600 seconds). This guarantees monotonic progression and prevents cache cycles. 4. **Client Behavior**: Clients **MUST** include the received cursor value as the `cursor` query parameter in subsequent requests. This creates different cache keys as time progresses, ensuring CDN caches eventually expire. **Example Cursor Flow:** ``` # Client makes initial long-poll request GET /stream?offset=123&live=long-poll # Server returns cursor based on current interval (e.g., interval 1000) < Stream-Cursor: 1000 # Client echoes cursor on next request GET /stream?offset=123&live=long-poll&cursor=1000 # If still in same interval, server adds jitter and returns advanced cursor < Stream-Cursor: 1050 ``` **Long-poll Caching:** CDNs and proxies **SHOULD NOT** cache `204 No Content` responses from long-poll requests in most cases. Long-poll `200 OK` responses are safe to cache when keyed by `offset`, `cursor`, and authentication credentials. ### 8.2. SSE SSE connections **SHOULD** be closed by the server approximately every 60 seconds. This enables new clients to collapse onto edge requests rather than maintaining long-lived connections to origin servers. ## 9. Extensibility The Durable Streams Protocol is designed to be extended for specific use cases and implementations. Extensions **SHOULD** be pure supersets of the base protocol, ensuring compatibility with any client that implements the base protocol. ### 9.1. Protocol Extensions Implementations **MAY** extend the protocol with additional query parameters, headers, or response fields to support domain-specific semantics. For example, a database synchronization implementation might add query parameters to filter by table or schema, or include additional metadata in response headers. Extensions **SHOULD** follow these principles: - **Backward Compatibility**: Extensions **MUST NOT** break base protocol semantics. Clients that do not understand extension parameters or headers **MUST** be able to operate using only base protocol features. - **Pure Superset**: Extensions **SHOULD** be additive only. New parameters and headers **SHOULD** be optional, and servers **SHOULD** provide sensible defaults or fallback behavior when extensions are not used. - **Version Independence**: Extensions **SHOULD** work with any version of a client that implements the base protocol. Extension negotiation **MAY** be handled through headers or query parameters, but base protocol operations **MUST** remain functional without extension support. ### 9.2. Authentication Extensions See Section 10.1 for authentication and authorization details. Implementations **MAY** extend the protocol with authentication-related query parameters or headers (e.g., API keys, OAuth tokens, custom authentication headers). ## 10. Security Considerations ### 10.1. Authentication and Authorization Authentication and authorization are explicitly out of scope for this protocol specification. Clients **SHOULD** implement all standard HTTP authentication primitives (e.g., Basic Authentication [RFC7617], Bearer tokens [RFC6750], Digest Authentication [RFC7616]). Implementations **MUST** provide appropriate access controls to prevent unauthorized stream creation, modification, or deletion, but may do so using any mechanism they choose, including extending the protocol with authentication-related parameters or headers as described in Section 9.2. ### 10.2. Multi-tenant Safety If stream URLs are guessable, servers **MUST** enforce access controls even when using shared caches. Servers **SHOULD** validate and sanitize stream URLs to prevent path traversal attacks and ensure URL components are within acceptable limits. ### 10.3. Untrusted Content Clients **MUST** treat stream contents as untrusted input and **MUST NOT** evaluate or execute stream data without appropriate validation. This is particularly important for append-only streams used as logs, where log injection attacks are a concern. ### 10.4. Content Type Validation Servers **MUST** validate that appended content types match the stream's declared content type to prevent type confusion attacks. ### 10.5. Rate Limiting Servers **SHOULD** implement rate limiting to prevent abuse. The `429 Too Many Requests` response code indicates rate limit exhaustion. ### 10.6. Sequence Validation The optional `Stream-Seq` header provides protection against out-of-order writes in multi-writer scenarios. Servers **MUST** reject sequence regressions to maintain stream integrity. ### 10.7. Browser Security Headers When serving streams to browser clients, servers **SHOULD** include the following headers to prevent MIME-sniffing attacks, cross-origin embedding exploits, and cache-related vulnerabilities: - `X-Content-Type-Options: nosniff` - Servers **SHOULD** include this header on all responses. This prevents browsers from MIME-sniffing the response content and potentially executing it as a different content type (e.g., interpreting binary data as HTML/JavaScript). - `Cross-Origin-Resource-Policy: cross-origin` (or `same-origin`/`same-site`) - Servers **SHOULD** include this header to explicitly control cross-origin embedding. Use `cross-origin` to allow cross-origin access via `fetch()`, `same-site` to restrict to the same registrable domain, or `same-origin` for strict same-origin only. This prevents Cross-Origin Read Blocking (CORB) issues and protects against Spectre-like side-channel attacks. - `Cache-Control: no-store` - Servers **SHOULD** include this header on HEAD responses and on responses containing sensitive or user-specific stream data. This prevents intermediate proxies and CDNs from caching potentially sensitive content. For public, non-sensitive historical reads, servers **MAY** use `Cache-Control: public, max-age=60, stale-while-revalidate=300` as described in Section 8. - `Content-Disposition: attachment` (optional) - Servers **MAY** include this header for `application/octet-stream` responses to prevent inline rendering if a user navigates directly to the stream URL. These headers provide defense-in-depth for scenarios where stream URLs might be accessed outside the intended programmatic fetch context (e.g., direct navigation, malicious cross-origin embedding via `