Release notes

0.7.6

0.7.6 is a drop-in patch for 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4 and 0.7.5: format epoch 3 is unchanged, with no storage, snapshot, WAL or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5 and 0.7.6 voters run side by side during the upgrade. 0.7.6 changes only the gateway. With unchanged values, the 0.7.6 chart renders the same server and gateway configuration as the 0.7.5 chart. From 0.7.0, a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.6 together with the chart, as for 0.7.1. From 0.7.4 or earlier, the upgrade notes of 0.7.5 apply too, from 0.7.3 or earlier those of 0.7.4, and from 0.7.2 or earlier those of 0.7.3.

Before the upgrade, check the gateway's memory. The gateway now bounds the request bodies it buffers at once with a body budget, 256 MiB by default, and may buffer that much. If you raised --max-request-body-bytes (the chart's gateway.maxRequestBodyBytes), an unset budget grows to that limit plus the 32 MiB small-body headroom, and the gateway warns at startup. Set --body-budget-bytes (gateway.bodyBudgetBytes) to at least that grown value, and give the gateway about twice its budget in memory. A budget set below the limit plus 32 MiB makes the gateway exit at startup, and the chart refuses it. If the pod cannot hold twice that, lower --max-request-body-bytes or give the pod more memory. A gateway before 0.7.6 exits on --body-budget-bytes and on the other new flags, so unset gateway.bodyBudgetBytes and remove those flags from gateway.extraArgs before you roll a gateway back.

Clients can now get three new refusals from the gateway, each marked as one that applied nothing: 503 UploadBudgetExhausted with Retry-After: 1 when the budget stays spent, 408 RequestBodyTooSlow for a body its client abandoned, and 429 TenantUploads or TenantUploadBytes with Retry-After: 1 where a quota policy sets the new per-bucket keys max_concurrent_uploads or max_upload_bytes. The SQLite extension takes each as a refusal that applied nothing, so none of them poisons a file, and SQLite VFS owners need no upgrade: 0.7.6 changes neither the extension nor @tonbo/sqlite-ursula. These changes are described below.

• Rust library APIs changed. In ursula-gateway, GatewayConfig::max_request_body_bytes moved into the new required field uploads, an uploads::UploadLimits that only UploadLimits::try_from(UploadLimitsSpec) builds, failing with UploadLimitsError. admission::TenantLimits gains max_concurrent_uploads and max_upload_bytes, so code that builds it with a struct literal must set them, AdmissionRejection gains UploadsExhausted and UploadBytesExhausted, and Admission::admit_upload and UploadGuard are new. None of these changes touches the wire or the disk.

Gateway

• The gateway bounds the request bodies it buffers across requests. It buffered each body up to --max-request-body-bytes, with no bound across requests, so concurrent or slow uploads could exhaust its memory. Before it reads a body, a request now reserves the body's Content-Length from a node-wide budget, --body-budget-bytes (256 MiB by default), and holds it until the gateway has forwarded the body. The gateway copies each body into one buffer of that size as it arrives, so the memory a body holds stays within its share however small the pieces it arrives in. A body without a declared length reserves --small-body-max-bytes (1 MiB by default) at first, and more as it arrives. --small-body-headroom-bytes (32 MiB by default) of the budget is kept for bodies of at most --small-body-max-bytes, so SQLite commits do not wait behind snapshot uploads. A request that finds the budget spent waits --body-budget-wait-ms (1000 by default) and then gets 503 with Retry-After: 1, marked UploadBudgetExhausted. The SQLite extension retries it as a refusal that applied nothing.
• The body budget follows a raised --max-request-body-bytes. With --body-budget-bytes unset, the budget is 256 MiB or the limit plus the 32 MiB small-body headroom, whichever is larger, so a gateway whose limit was raised (for example to 1 GiB for large snapshots) keeps starting, with a warning, and may buffer that much at once. Give the gateway about twice its budget in memory. The budget cannot be set below the limit plus the headroom, so if that is more than the pod's memory can hold, lower --max-request-body-bytes (the chart's gateway.maxRequestBodyBytes) and set --body-budget-bytes (gateway.bodyBudgetBytes) with it, or give the pod more memory. At startup the gateway also warns when its budget, set or grown, exceeds half of its cgroup memory limit, which it looks up along its own cgroup path. Once grown, the budget holds one body at the limit beside the headroom, so a request near the limit that waits for the whole budget takes every share released meanwhile, and large bodies behind it can get 503 too. A budget set explicitly below the limit plus the headroom makes the gateway exit at startup with LimitAboveSharedBudget, and the chart refuses such a gateway.bodyBudgetBytes. The chart renders --body-budget-bytes only when the value is set, and a gateway before 0.7.6 exits on that flag, so unset the value before you roll a gateway back. The other new flags (--small-body-headroom-bytes, --small-body-max-bytes, --body-budget-wait-ms, --body-stall-timeout, --body-min-bytes-per-sec) have no chart value. Add them to gateway.extraArgs only once every gateway runs 0.7.6, and remove them before a rollback, for the same reason.
• The gateway cuts bodies whose client abandoned them. It put no time limit on the body of a request it forwards. It now cuts a body once no byte of it arrived for --body-stall-timeout (30 s by default), or once less than --body-min-bytes-per-sec (1 KiB/s by default) arrived over one such window, and answers 408, marked RequestBodyTooSlow, with nothing forwarded. Nothing bounds a reading in all, so a slow upload that keeps arriving is never cut, and with the defaults the gateway never cuts a commit or a snapshot the SQLite extension still waits for. Over HTTP/1.1 the connection closes after the 408. Keep --body-stall-timeout above the extension's 10 s commit timeout. The gateway warns at startup otherwise.
• A quota policy can bound each bucket's uploads. max_concurrent_uploads and max_upload_bytes, under [default] or a [[bucket]] of --quota-policy, bound the bucket's bodies over --small-body-max-bytes while the gateway reads them. A request past either gets 429 with Retry-After: 1, marked TenantUploads or TenantUploadBytes. A body stops counting once the gateway has buffered it, and smaller bodies never count. A policy that sets neither limit, and a gateway without access control or a quota policy, answer no new 429.
• The refusals above reach a client that sends its body first. Like the refusals the gateway decides from the head, each is sent once the gateway has read the body to its end and dropped it, here at the pace of a forwarded body. A body declared over its bucket's own max_request_body_bytes gets its 413 before it reserves any of the budget. Only a body past --max-request-body-bytes still gets its 413 with the rest unread.
• New gateway metrics. /__ursula/gateway/metrics reports upload_refusals_budget, upload_refusals_tenant_uploads, upload_refusals_tenant_bytes and upload_refusals_too_slow, the refusals above by reason, upload_budget_bytes, the budget in effect, upload_reserved_bytes, the budget reserved now (each body's share in whole KiB), upload_tenants, the buckets with a large body under way under an upload limit, and body_bytes_buffered, the bytes of bodies it read to forward them. Its refusal_drains and refusal_drains_cut count the refusals above too, and the new refusal_drain_bytes counts the bytes of refused bodies it read and dropped.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests show that a failed write of the poisoned status fails closed.
• #452: the HTTP clients of the experimental event-time indexer have no connect or request timeouts, so an indexer call to a node that loses power can hang for minutes, until the kernel gives up on the connection.
• #461: a voter restarted without a graceful drain wins elections for dozens of groups within seconds of starting. For about 5 s every node then answers /__ursula/ready with 503 raft_maintenance_unready, and clients get 503s. Graceful rollouts avoid it, because the new pod starts drained, except a reserved rollout (server.gracefulRollout.maintenanceReservation) without server.startupOwnership, whose new pod starts undrained.
• #465: replaying the WAL at startup takes longer as the WAL grows, 17 s for 1.2 GB in one test, so in a test of repeated pod deletions over three hours, the recovery after a deletion grew from about 25 s to between 40 and 74 s.
• #484: when the replacement voter of a host recovery dies after its group leaders started rebuilding it, they keep it as a learner in those groups, so ursulactl verify-survivors refuses for good and the host restaging cannot proceed. No acknowledged write is at risk, since the survivors refuse. There is no automatic host-recovery consumer yet, and the chart keeps server.startupOwnership off by default.
• #490: a pass that prunes retired S3 snapshots prunes nothing when a pin it listed is released before it reads it, and logs failed to prune retired OpenRaft snapshots with NotFound, so retired snapshot objects can stay in S3 past their hour of grace. No data is at risk.
• #492: after a peer loses power, the surviving nodes keep its dead inbound connections until they restart, because the server sets no TCP or HTTP/2 keepalive on the connections it accepts.
• #494: a group prunes its retired S3 snapshots only when it builds a snapshot, so a group that stops taking writes keeps the snapshots it retired in its last busy hour until it builds again.
• #496: under moderate load, leaders log raft append stream exceeded the OpenRaft hard TTL for replication and heartbeat calls to followers. No commit failed where this was seen, but under a heavier load a busy group could elect a new leader.
• #497: a SQLite VFS attach that installs a snapshot is slower than one that replays as many bytes of log, about twice as long for a 1 GB database in one test.
• #517: a group sends each follower one Append call at a time, and since 0.7.2 a call carries at most 4 MiB of entries, so one group replicates to a follower at most about 4 MiB per round trip. That can slow the catch-up of a very busy group over a link with a long round trip.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described under 0.7.2.
• #539: a node answers every leadership transfer it refuses with the gRPC status FAILED_PRECONDITION, also when its Raft core has stopped, so the sending leader cannot tell a stopped core from a refusal by policy.
• #550: a SQLite VFS snapshot whose body is over a request size limit in front of the stream, such as the gateway's --max-request-body-bytes, is still sent. It usually meets a closed connection rather than the 413, is reported as a failed write and backs off. Compare snapshot_bytes with the limit. Commits stay below such a limit with URSULA_VFS_MAX_FRAME_BYTES.
• #561: the snapshot driver's gauges are reported only at /__ursula/metrics, not over OTLP. In the validation runs of 0.7.4 with two builds, /__ursula/ready answered 503 for apply_lag in one or two 5 s samples per run, too briefly to fail a probe that needs three failures in a row.
• #566: a SQLite VFS commit's append must be sent and answered within 10 s, its upload included, so a commit frame near the 32 MiB default of URSULA_VFS_MAX_FRAME_BYTES needs about 27 Mbit/s of upload. On a slower uplink a large commit times out with part of its body sent, which leaves its outcome unknown, so it poisons the database once the retry budget runs out, whether or not the stream would have taken it. At 10 Mbit/s up that happens from about 12 MiB. On a slow uplink, set URSULA_VFS_MAX_FRAME_BYTES well below what the uplink sends in 10 s, so that a larger transaction fails alone, with refused_kind too_large, instead. Snapshots have 120 s and are not affected.
• A node shed for failed writes can shed again every few minutes. A node shed because only its own cold-flush writes fail, while its probes answer, hands off its leaderships and then flushes nothing, so it heals about a minute after its last failed write even if its writes still fail. It sheds again once the leadership balancer hands leaderships back and a flush fails, for as long as such an outage lasts.
• A few slow clients can hold the gateway's body budget. A request reserves its declared length before it sends the body, and the gateway gives the reservation back only when the body stalls or arrives below --body-min-bytes-per-sec. With the defaults, about seven connections that each send 1 KiB/s can hold a 256 MiB budget, and other large bodies then get 503. That is the cost of never cutting a slow upload that keeps arriving. The per-bucket upload limits are keyed by bucket only, with no limit per principal, and without access control no limit per client address.
• One bucket's small bodies can take the whole small-body headroom. The headroom is shared by every bucket, so it keeps small writes from waiting behind large uploads, not one bucket's small writes from another's.

0.7.5

0.7.5 is a drop-in patch for 0.7.0, 0.7.1, 0.7.2, 0.7.3 and 0.7.4: format epoch 3 is unchanged, with no storage, snapshot, WAL or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4 and 0.7.5 voters run side by side during the upgrade. With unchanged values, the 0.7.5 chart renders the same server and gateway configuration as the 0.7.4 chart. From 0.7.0, a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.5 together with the chart, as for 0.7.1. From 0.7.3 or earlier, the upgrade notes of 0.7.4 apply too, and from 0.7.2 or earlier those of 0.7.3.

Before the upgrade, check raft.snapshot_build_max_concurrency (#561). A configuration file that sets it to 0 or to more than 16 no longer loads, so set it to a value between 1 and 16 or remove it. The 0.7.5 chart's schema refuses such a raft.snapshotBuildMaxConcurrency, so helm upgrade fails before any pod restarts.

SQLite VFS owners upgrade one at a time, as within any minor: the 0.7.x extensions take over each other's streams. Upgrade @tonbo/sqlite-ursula together with the extension. 0.7.5 changes no format. Pi users install @earendil-works/pi-durable themselves, in the range >=1.0.0 <1.2.0 (npm refuses a version outside it with ERESOLVE), and import openUrsulaPiStorage from @tonbo/sqlite-ursula/pi. The extension now sends User-Agent: sqlite-ursula-vfs/0.7.5, so update any proxy, WAF or log rule keyed on the User-Agent before upgrading owners. Through openUrsulaPiStorage, a fenced commit or read now rejects with an UrsulaFencedError and an unknown outcome with an UrsulaReattachError, both subclasses of UrsulaReplicationError whose name is their own, so code that matches the name or the class a logger prints must match with instanceof UrsulaReplicationError instead. On Linux and macOS, attach now refuses (in_use) while the process has the file, its -wal, -shm or -journal open outside the extension, so close every such connection, such as one opened before the extension was loaded, before the attach. An attach that cannot reach the endpoint before its claim throws UrsulaWriteRefusedError where it threw SQLite's error, and attach refuses a stream URL with a query or a fragment. These changes are described below.

• Rust library APIs changed. In ursula-raft, SnapshotBuildCoordinator::start_snapshot takes the log the snapshot is asked for, a check of the snapshot OpenRaft published where it took covered, and a future that ends once a published snapshot covers the request. SnapshotBuildCoordinator::acquire and acquire_below fail with the new SnapshotPermitError where they failed with GroupEngineError, and SnapshotStartError::Permit carries it too. SnapshotBuildCoordinator gains busy, publishing and add_publishing, build_ended is now public, and hand_off_and_trigger also answers Handoff::Busy while a build of the group holds a permit. ursula_config::validate::ValidationError gains SnapshotBuildConcurrency, which is not #[non_exhaustive], and ursula_config::RaftConfig::MAX_SNAPSHOT_BUILD_CONCURRENCY is new. ursula_observability::serve::serve_until_shutdown takes any axum::serve::Listener of TcpStreams, where it took a tokio::net::TcpListener. None of these changes touches the wire or the disk.

Snapshot builds

• raft.snapshot_build_max_concurrency must be between 1 and 16 (#561). 0.7.4 took 0 as 1 and set no maximum. From 0.7.5 a configuration file that sets 0 or more than 16 no longer loads, and the node exits at startup with an error such as raft.snapshot_build_max_concurrency (17) must be between 1 and 16. Before the upgrade, set it to a value between 1 and 16 or remove it. The 0.7.5 chart's schema refuses such a raft.snapshotBuildMaxConcurrency, so helm upgrade fails before any pod restarts. See Cold storage.
• The snapshot driver and an operator's snapshot see a build that has just ended (#561). A group whose trigger went unclaimed is now held out of the driver's passes for 0.5 s, doubling for each such trigger in a row up to 16 s, where it sat out 2 to 32 passes. A group whose build has just ended, with a snapshot OpenRaft has not published yet, is left out of the passes for up to 5 s and counts in the pass log's building. raft_snapshot_pressure_passes no longer counts a pass that only carries the same held groups as the pass before it, so it counts the pressure passes that planned a build, ended because the log outgrew them, or changed the groups held out. The pass log raft snapshot pressure pass completed gains busy, the groups skipped because a build of them runs or a permit was just handed to them. Such a group is skipped at once and not held. 0.7.4 counted it in not_started when a permit was handed to it. When a build of it ran, 0.7.4 triggered it all the same and OpenRaft dropped the trigger: the pass waited 5 s, logged did not claim its build permit within 5s, then counted the group in unclaimed and held it. POST /__ursula/raft/{group}/snapshot no longer starts a second build when a build of the group that has just ended covers the request, and in that case no longer waits 5 s on a trigger OpenRaft dropped or answers 503 although the covering snapshot exists. 0.7.4 lists both as a known issue. Unless the snapshot OpenRaft has published already covers the request, it now waits up to 10 s for the group's running build to end before its 10 s wait for the snapshot, so a request can take up to about 30 s where it took up to about 20 s. It still answers 500 when the snapshot does not show in time. See Manually trigger a snapshot.

SQLite VFS

• openUrsulaPiStorage moved to @tonbo/sqlite-ursula/pi, and the package has no runtime dependencies. @tonbo/sqlite-ursula used to depend on @earendil-works/pi-durable, so an application that only attaches files for node:sqlite installed Pi Durable and the model SDKs it pulls in, over 100 MB. The root entry now imports nothing outside Node.js, and Pi Durable is an optional peer dependency in the range >=1.0.0 <1.2.0, which npm installs only when the application depends on it. npm refuses the install with ERESOLVE when the application's version is outside that range, where 0.7.4 installed a second copy for the package without a warning, so move it into the range before or with the upgrade. This is a breaking change for Pi users: install @earendil-works/pi-durable next to the package, and import openUrsulaPiStorage and UrsulaPiStorageOptions from @tonbo/sqlite-ursula/pi. Every other export, the error classes included, stays in @tonbo/sqlite-ursula. See Pi Durable.
• The extension identifies itself. Every request it sends, including the CONNECT to a proxy, carries User-Agent: sqlite-ursula-vfs/0.7.5, the extension's name and version, so a gateway or proxy can attribute traffic and require a minimum version. Extensions up to 0.7.4 send ureq's default, ureq/3.4.2. Update any proxy, WAF or log rule keyed on the User-Agent before upgrading owners.
• Attach refuses a file open outside the extension, and a failed attach locks the file out of the process (#570). Attach refuses (in_use), changing nothing, while the process has the file, its -wal, -shm or -journal open outside the extension, read-only or not, such as a connection opened before the extension was loaded. It checks where it can read the process's open files: on Linux with /proc mounted, and on macOS. After a failed attach the file opens in the process only once an attach succeeds, with or without a sidecar, and a failed attach that wrote nothing into a new file removes the sidecar it created, which never touches the db file, -wal or -shm. As before, a failed attach may already have discarded local files it did not trust or folded their WAL into the db file, and the next attach resumes the file when its sidecar names it and rebuilds it otherwise. An attach refused because of the file itself before it changed anything (in_use, unreplicated_file, other_stream) is no failed attach and changes nothing, so a database that holds data but was never replicated still opens to copy its rows out. Writes to an attached file outside the extension are still not detected, as in 0.7.4.
• Fenced owners stop serving reads. Once an owner knows it is fenced, its new read transactions fail too (errcode 266), and through openUrsulaPiStorage they reject with UrsulaFencedError, an UrsulaReplicationError (fenced: true, a message that starts with read refused). A read transaction already open keeps its snapshot until it ends.
• Errors name a stable kind. Every error of the extension's SQL functions starts with <function>: <kind>: , and status(file).error reports kind, action (retry, reattach, stop or fix) and message while writes fail. An attach refused with a 4xx other than 404 carries the server's reason, and a token refused when the retries run out keeps the layer's explanation. A database without a sidecar is refused as unreplicated_file with a message that says how to copy its rows out. An attach that fails after its claim reports outcome_unknown, since the claim may have fenced the earlier owner, and a local file the setup refuses (no permission, a read-only file system, no space left) reports local_setup, whose action is fix. See Health and alerts and the design doc for the kinds.
• open(streamUrl) opens a replicated database in one call, and its errors say what to do. open loads the extension, attaches a local file named after the stream (or the path it is given, made absolute), opens it, and returns a node:sqlite DatabaseSync. A failure of the extension in it throws an UrsulaError, whose action is retry, reattach, stop or fix and whose cause is the error it surfaced as, and every other error, such as a constraint violation, is SQLite's own, unchanged. The ROLLBACK that a transaction helper (drizzle's, say) runs after such a failure, which SQLite has rolled back already, does not replace it. A read-only open keeps a file of its own, so it opens next to the owner on its host. The new classes are UrsulaError, UrsulaFencedError (stop), UrsulaReattachError (reattach) and UrsulaSetupError (fix). UrsulaReplicationError, UrsulaWriteRefusedError and UrsulaPaymentRequiredError are now subclasses of UrsulaError. attach, status, setToken and drainStats throw UrsulaErrors too, with the extension's message, and with SQLite's code, errcode and errstr copied, so code that reads them keeps working. openUrsulaPiStorage reports through the same mapping: a fenced commit or read is an UrsulaFencedError and an unknown outcome an UrsulaReattachError, both UrsulaReplicationErrors as before. The docs no longer ask for PRAGMA journal_mode=WAL: attach leaves the file in WAL mode. What a caller can see change, see SQLite on Ursula:
  • An attach whose claim is refused other than with a 402 has kind refused where it had null.
  • An attach that fails before it sends its claim because the endpoint cannot be reached, puts it off or refuses the credentials throws UrsulaWriteRefusedError (unreachable, unavailable or unauthorized) where it threw SQLite's error.
  • attach and openUrsulaPiStorage refuse a stream URL with a query or a fragment (invalid_url) before sending anything, where the attach created the stream, appended a claim and failed. openUrsulaPiStorage refuses :memory:, "" and file: URIs as its file (invalid_argument), and makes a relative one absolute.
  • The errors openUrsulaPiStorage builds for a failed commit or read have messages that end with one sentence that says what to do, after commit not replicated: <reason>, read refused, the file is fenced: <reason> and commit refused ....
  • Those errors for a fenced commit or read, and for an unknown outcome, have the name UrsulaFencedError or UrsulaReattachError, where it was UrsulaReplicationError, which shows in String(error), the stack and loggers that print the class. instanceof UrsulaReplicationError holds as before: match with it.
  • A statement of Pi's storage on a poisoned file that fails with an error that is not an I/O error, such as a constraint violation, rejects with that error where it rejected with UrsulaReplicationError. A failure of the extension that Pi wraps in its StorageRejected still rejects with the Ursula error.

Event-time indexer

• The indexer sets TCP_NODELAY on the connections it accepts, as the server and the gateway already did. Without it, a reply of ursula indexer (the experimental event-time indexer) that followed another on the same connection could wait for the client's delayed acknowledgement of the first, 40 ms or more on Linux.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests show that a failed write of the poisoned status fails closed.
• #452: the HTTP clients of the experimental event-time indexer have no connect or request timeouts, so an indexer call to a node that loses power can hang for minutes, until the kernel gives up on the connection.
• #461: a voter restarted without a graceful drain wins elections for dozens of groups within seconds of starting. For about 5 s every node then answers /__ursula/ready with 503 raft_maintenance_unready, and clients get 503s. Graceful rollouts avoid it, because the new pod starts drained, except a reserved rollout (server.gracefulRollout.maintenanceReservation) without server.startupOwnership, whose new pod starts undrained.
• #465: replaying the WAL at startup takes longer as the WAL grows, 17 s for 1.2 GB in one test, so in a test of repeated pod deletions over three hours, the recovery after a deletion grew from about 25 s to between 40 and 74 s.
• #484: when the replacement voter of a host recovery dies after its group leaders started rebuilding it, they keep it as a learner in those groups, so ursulactl verify-survivors refuses for good and the host restaging cannot proceed. No acknowledged write is at risk, since the survivors refuse. There is no automatic host-recovery consumer yet, and the chart keeps server.startupOwnership off by default.
• #490: a pass that prunes retired S3 snapshots prunes nothing when a pin it listed is released before it reads it, and logs failed to prune retired OpenRaft snapshots with NotFound, so retired snapshot objects can stay in S3 past their hour of grace. No data is at risk.
• #492: after a peer loses power, the surviving nodes keep its dead inbound connections until they restart, because the server sets no TCP or HTTP/2 keepalive on the connections it accepts.
• #494: a group prunes its retired S3 snapshots only when it builds a snapshot, so a group that stops taking writes keeps the snapshots it retired in its last busy hour until it builds again.
• #496: under moderate load, leaders log raft append stream exceeded the OpenRaft hard TTL for replication and heartbeat calls to followers. No commit failed where this was seen, but under a heavier load a busy group could elect a new leader.
• #497: a SQLite VFS attach that installs a snapshot is slower than one that replays as many bytes of log, about twice as long for a 1 GB database in one test.
• #517: a group sends each follower one Append call at a time, and since 0.7.2 a call carries at most 4 MiB of entries, so one group replicates to a follower at most about 4 MiB per round trip. That can slow the catch-up of a very busy group over a link with a long round trip.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described under 0.7.2.
• #539: a node answers every leadership transfer it refuses with the gRPC status FAILED_PRECONDITION, also when its Raft core has stopped, so the sending leader cannot tell a stopped core from a refusal by policy.
• #550: a SQLite VFS snapshot whose body is over a request size limit in front of the stream, such as the gateway's --max-request-body-bytes, is still sent. It usually meets a closed connection rather than the 413, is reported as a failed write and backs off. Compare snapshot_bytes with the limit. Commits stay below such a limit with URSULA_VFS_MAX_FRAME_BYTES.
• #561: the snapshot driver's gauges are reported only at /__ursula/metrics, not over OTLP. In the validation runs of 0.7.4 with two builds, /__ursula/ready answered 503 for apply_lag in one or two 5 s samples per run, too briefly to fail a probe that needs three failures in a row.
• #566: a SQLite VFS commit's append must be sent and answered within 10 s, its upload included, so a commit frame near the 32 MiB default of URSULA_VFS_MAX_FRAME_BYTES needs about 27 Mbit/s of upload. On a slower uplink a large commit times out with part of its body sent, which leaves its outcome unknown, so it poisons the database once the retry budget runs out, whether or not the stream would have taken it. At 10 Mbit/s up that happens from about 12 MiB. On a slow uplink, set URSULA_VFS_MAX_FRAME_BYTES well below what the uplink sends in 10 s, so that a larger transaction fails alone, with refused_kind too_large, instead. Snapshots have 120 s and are not affected.
• A node shed for failed writes can shed again every few minutes. A node shed because only its own cold-flush writes fail, while its probes answer, hands off its leaderships and then flushes nothing, so it heals about a minute after its last failed write even if its writes still fail. It sheds again once the leadership balancer hands leaderships back and a flush fails, for as long as such an outage lasts.

0.7.4

0.7.4 is a drop-in patch for 0.7.0, 0.7.1, 0.7.2 and 0.7.3: format epoch 3 is unchanged, with no storage, snapshot, WAL or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1, 0.7.2, 0.7.3 and 0.7.4 voters run side by side during the upgrade. With unchanged values, the 0.7.4 chart renders the same server and gateway configuration as the 0.7.3 chart. From 0.7.0, a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.4 together with the chart, as for 0.7.1. From 0.7.2 or earlier, the upgrade notes of 0.7.3 apply too.

A gateway with access control (--auth-issuer) now answers 404 on its metrics path, /__ursula/gateway/metrics (#548). If something scrapes that path, add --auth-public-metrics to gateway.extraArgs in the same helm upgrade that moves the gateways to 0.7.4. A gateway before 0.7.4 exits at startup on the unknown flag, so do not add it before that upgrade, and remove it before rolling a gateway back to an earlier release. The flag makes the counters public: the gateway then serves them to every caller, with or without a token.

The SQLite VFS is no longer experimental (#468). Within a minor release (0.7.x), the formats the extension writes to the stream and to the local files do not change, and the 0.7.x extensions take over each other's streams, so owners upgrade one at a time. The API of the extension and of @tonbo/sqlite-ursula can still change in a patch release, as it did in 0.7.3, so upgrade the package together with the extension. A new minor release may also change the formats. Either way, the release notes say how to upgrade. 0.7.4 changes no format. Its API changes are the refusal kind unreachable and an attach that fails as refused when nothing of it went out, both described below. The @tonbo/sqlite-ursula README on npm drops the experimental label with the 0.7.4 publish. See SQLite on Ursula.

• Rust library APIs changed. In ursula-raft, SnapshotBuildCoordinator no longer exposes reclaimable_log_progress, try_acquire, hand_off, handoff_pending or reclaim_handoff, and gains max_concurrency, node_log_bytes, builds_in_flight, running_builds, acquire_below, hand_off_and_trigger and start_snapshot, with the new Handoff, SnapshotStartError and SNAPSHOT_HANDOFF_WAIT. hand_off_and_trigger takes one hand-off per group at a time (Handoff::Busy otherwise) and gives its permit back when its future is dropped. In ursula_raft::snapshot_cadence, SnapshotPlan::pressure is a LogPressure instead of a bool, and SnapshotCadence::plan_with_lagging gives way to plan_pass, which takes GroupDriveStates within DriveLimits, next to the new pressure, pass_step (with PassStep), hard_limit_bytes and resume_bytes. In ursula-runtime, RuntimeMetrics::record_raft_snapshot_pressure takes the pass's time as a second argument, RuntimeMetrics::record_raft_log_pressure and RaftLogPressureSample are new, and RuntimeMetricsSnapshot has seven new public fields, so code that builds it with a struct literal must set them. ursula_gateway::auth::AccessControl::with_public_metrics is new, and a gateway built with Gateway::with_access_control answers 404 on its metrics path unless it is set. ursula_config::RaftConfig now defaults snapshot_build_max_concurrency to 2. None of these changes touches the wire or the disk.

Snapshot builds

• Snapshot builds escalate before the Raft log reaches its hard limit (#519). Under a sustained write load, the unsnapshotted Raft log of every node could reach its hard limit, twice raft.snapshot_log_budget, at about the same time, and the nodes then refused writes with 503 RaftLogPressure for about 20 seconds until snapshots caught up. By default the snapshot driver ran one build at a time, and its pressure passes took the groups that free the most log per snapshot byte first, which put near-empty groups first. They now snapshot the groups that hold the most log first, and at or past the budget they run more builds at once the further the log is past it, up to raft.snapshot_build_max_concurrency, whose default rises from 1 to 2. With 4, a node runs one build at the budget, two just past it, three past 1.33 budgets and four past 1.67 budgets. As the builds bring the log back down the driver runs fewer of them, and below the budget it starts a build only when no other runs. At or past the budget a pass starts up to raft.snapshot_pressure_max_groups_per_tick groups for each build it keeps in flight. The hard limit and the point where writes resume, 1.5 budgets, are unchanged. On a 3-voter test cluster with real S3 and 128 SQLite VFS owners writing flat out, 0.7.3, and 0.7.4 set to one build, crossed the hard limit on every node and refused writes for 11 to 63 seconds at a time, while 0.7.4 with the default of 2 stayed well below it.
• What the new default costs. A build holds its group's captured state and, while it uploads, its compressed snapshot, so two builds hold twice what one did. With snapshots of 2.8 MB on average and 67 MB at most, as under the load of #519, two builds hold about 11 MB in the common case, about 270 MB when both build the largest groups, and about 400 MB if both uploads fail and fall back to an inline snapshot. At or past the budget a node also sends up to two builds' requests to the snapshot store at once. To keep one build at a time, set raft.snapshot_build_max_concurrency = 1 (the chart's raft.snapshotBuildMaxConcurrency: 1), which is safe at any time. A node with memory to spare can set 4. Set a value above 1 only after every node runs 0.7.4: servers before 0.7.4 read the setting too, and a helm upgrade changes the chart's ConfigMap in place, so a value set in the same upgrade reaches an older pod that restarts before the rollout replaces it. That pod's earlier driver then runs up to that many builds at once at any log level, which was never validated. The chart renders the setting only when raft.snapshotBuildMaxConcurrency is set, so with it unset each server keeps its own default, which is 1 before 0.7.4. With the snapshot driver off (storage.snapshot.drive_interval = 0), OpenRaft's own policy may now run two builds at once too. See Cold storage.
• Metrics for the snapshot driver. /__ursula/metrics reports the node's unsnapshotted Raft log and the most it held (raft_log_bytes, raft_log_bytes_high_watermark), the driver's pressure level (raft_snapshot_pressure_level: 0 below three quarters of the budget, 1 below the budget, 2 at or past it), the builds a pass keeps in flight at that level (raft_snapshot_pressure_builds), the builds that run now and the most that ran at once (raft_snapshot_builds_in_flight, raft_snapshot_builds_in_flight_max), and the time of its pressure passes (raft_snapshot_pressure_pass_ns). Only raft_snapshot_pressure_passes changed meaning. It counts the pressure passes that planned a build or held out groups whose last trigger went unclaimed, where it counted every pressure pass, so the mean pass time, raft_snapshot_pressure_pass_ns over raft_snapshot_pressure_passes, covers those passes only. raft_snapshot_pressure_groups did not change. The driver logs each pass it counts as raft snapshot pressure pass completed. See Metrics.
• An operator's snapshot waits for a build permit (#529). POST /__ursula/raft/{group}/snapshot now waits while a build of the group runs and while other builds hold every snapshot build permit. It answers 200 when a snapshot covers what the group had applied when the request arrived, and 503 with Retry-After: 1 when no build of the group can start within 10 s, where it answered 500 after about 10 s. A request can now take up to about 20 s, where it took up to about 10 s: up to 10 s for a build of the group to start, then up to 10 s for the snapshot, after which it answers 500 as before. A build of the group that has just ended can go unseen for a moment, so the request may then start another build (#561). See Manually trigger a snapshot.

Gateway

• The gateway's metrics follow its access control (#548). A gateway with access control (--auth-issuer) no longer serves /__ursula/gateway/metrics. It answers 404 to every caller, with a token or without, like every other /__ursula/* path, unless --auth-public-metrics is set (with the Helm chart, in gateway.extraArgs). The flag requires --auth-issuer. Public metrics are served without a token check, so an invalid or expired token gets them too. The counters hold no tenant data but show traffic volume, so set the flag only where untrusted clients cannot reach the gateway, or where a proxy in front of it denies /__ursula/*. A gateway without access control serves its metrics as before. While the gateways roll, a scraper gets both answers. See Gateway access control.
• The gateway reads the body before a refusal it decided from the request head (#540). ursula gateway decides some refusals from a request's head alone: the access-control refusals (401, 403, 404, and the 503s while the token issuer or the authorizer is unavailable) and a tenant's 429. It used to send them with the body unread and close the connection, so a client that sends its whole body before it reads the answer, as the SQLite VFS does, often met a broken pipe or a reset instead of the refusal. The gateway now reads the body and discards it before it sends such a refusal, with the same status and headers, and stops after 5 s without new data or 30 s in all. A SQLite VFS commit larger than one read that the gateway refuses this way now fails alone with the refusal's kind, as a small one already did, instead of poisoning the database once the retry budget runs out. Only the gateway needs 0.7.4 for this: a 0.7.3 extension behind it gets the refusal too.
  • The gateway reads at most --max-request-body-bytes of such a body. A body that goes past the limit, stops arriving for 5 s or is still arriving after 30 s gets the refusal at once, and the connection closes. A refused request whose Content-Length is over the limit gets its refusal at once, with its body unread, and its connection closes. A request the gateway would forward still gets its 413 for a body over the limit, as before: the gateway stops reading at the limit and closes the connection, so a client still sending the body may meet the closed connection instead. Keep commits below the limit with URSULA_VFS_MAX_FRAME_BYTES.
  • A refused upload costs the gateway the body's ingress, but no memory, and once the body was read to its end the connection stays open for the client's next request. Over HTTP/2 a refused request also uploads its body before it gets the refusal, and a reading cut short resets only that request's stream. A client that sends Expect: 100-continue (curl does for uploads over 1 MiB) now gets 100 Continue and uploads its body before the refusal, unless its Content-Length is over the limit. Programs that embed the gateway get the same from Gateway::handle.
  • /__ursula/gateway/metrics reports the refused requests with a body as refusal_drains, and those whose body the gateway did not read to its end as refusal_drains_cut. Only a gateway with access control refuses requests from their head, and it serves these counters only with --auth-public-metrics, which makes them public.
  • A SQLite VFS snapshot over a limit in front of the stream is still sent, and still usually meets a closed connection (#550).

SQLite VFS

• A commit whose attempts never went out fails alone. An attempt that failed before any byte of it went out, because the stream's host did not resolve, no connection was made or the TLS handshake failed, now proves that nothing of it was applied, like the refusals of 0.7.3. A commit whose every attempt failed this way or was refused fails alone with SQLITE_IOERR_AUTH (errcode 7178) once the retry budget runs out, instead of poisoning the database. refused_kind is unreachable when the last attempt never went out, and refused_reason gives its failure, such as timeout: resolve. The commit's producer sequence stays free, and the next commit succeeds once the endpoint is reachable, with no new attach. A failure after a connection was made, on a pooled connection too, still leaves the commit's outcome unknown, and poisons once the retry budget runs out. An attach, and an owner's later claim of its producer, whose every attempt failed before it went out now fail as refused, with nothing claimed: ursula_attach fails with SQLITE_IOERR_AUTH and attach throws UrsulaWriteRefusedError, where both failed with an unknown outcome. An attach usually meets an unreachable endpoint at its first HEAD, which fails as before. TypeScript's WriteRefusedKind gains unreachable, so a switch that handles every kind needs a case for it. Troubleshooting says what to check, with the TLS failures as rustls reports them.
• Host addresses are cached. Every request used to look its host up, also on a pooled connection. In Kubernetes that could exhaust the connection tracking of the node that runs CoreDNS, and a DNS outage of 30 s poisoned every owner whose commits met it. The extension now looks a host up at most once every 30 s, one request at a time, while the host's other requests go on with its last addresses. The request that looks the host up waits for the lookup, up to the 5 s resolve timeout, even with a pooled connection ready. While lookups fail, the last addresses serve requests for up to 10 minutes after the last successful lookup, the host is looked up again 5 s after each failed lookup, and the extension logs dns_stale. A host with no addresses to use fails its requests as never sent until a lookup succeeds, with one lookup every 5 s. A failed connection makes the host's next lookup due, no sooner than 5 s after the last one, so an endpoint that moved is found at the next attempt.
• A reset that macOS reports as Invalid argument is retried (#542). The extension ships prebuilt for macOS on Apple silicon. macOS reports a connection reset as Invalid argument (os error 22) when it lands between two of the extension's calls on the socket, and as a reset otherwise. The extension took that error as final, so a read, HEAD, create or snapshot GET whose pooled connection the server had just closed failed the attach or the snapshot pass. Such a request is now sent again on a new connection within URSULA_VFS_RETRY_MS, as on Linux, where nothing changes. This fixes the known issue listed under 0.7.3.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests show that a failed write of the poisoned status fails closed.
• #452: the HTTP clients of the experimental event-time indexer have no connect or request timeouts, so an indexer call to a node that loses power can hang for minutes, until the kernel gives up on the connection.
• #461: a voter restarted without a graceful drain wins elections for dozens of groups within seconds of starting. For about 5 s every node then answers /__ursula/ready with 503 raft_maintenance_unready, and clients get 503s. Graceful rollouts avoid it, because the new pod starts drained.
• #465: replaying the WAL at startup takes longer as the WAL grows, 17 s for 1.2 GB in one test, so in a test of repeated pod deletions over three hours, the recovery after a deletion grew from about 25 s to between 40 and 74 s.
• #484: when the replacement voter of a host recovery dies after its group leaders started rebuilding it, they keep it as a learner in those groups, so ursulactl verify-survivors refuses for good and the host restaging cannot proceed. No acknowledged write is at risk, since the survivors refuse. There is no automatic host-recovery consumer yet, and the chart keeps server.startupOwnership off by default.
• #490: a pass that prunes retired S3 snapshots prunes nothing when a pin it listed is released before it reads it, and logs failed to prune retired OpenRaft snapshots with NotFound, so retired snapshot objects can stay in S3 past their hour of grace. No data is at risk.
• #492: after a peer loses power, the surviving nodes keep its dead inbound connections until they restart, because the server sets no TCP or HTTP/2 keepalive on the connections it accepts.
• #494: a group prunes its retired S3 snapshots only when it builds a snapshot, so a group that stops taking writes keeps the snapshots it retired in its last busy hour until it builds again.
• #496: under moderate load, leaders log raft append stream exceeded the OpenRaft hard TTL for replication and heartbeat calls to followers. No commit failed where this was seen, but under a heavier load a busy group could elect a new leader.
• #497: a SQLite VFS attach that installs a snapshot is slower than one that replays as many bytes of log, about twice as long for a 1 GB database in one test.
• #517: a group sends each follower one Append call at a time, and since 0.7.2 a call carries at most 4 MiB of entries, so one group replicates to a follower at most about 4 MiB per round trip. That can slow the catch-up of a very busy group over a link with a long round trip.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described under 0.7.2.
• #539: a node answers every leadership transfer it refuses with the gRPC status FAILED_PRECONDITION, also when its Raft core has stopped, so the sending leader cannot tell a stopped core from a refusal by policy.
• #550: a SQLite VFS snapshot whose body is over a request size limit in front of the stream, such as the gateway's --max-request-body-bytes, is still sent. It usually meets a closed connection rather than the 413, is reported as a failed write and backs off. Compare snapshot_bytes with the limit. Commits stay below such a limit with URSULA_VFS_MAX_FRAME_BYTES.
• #561: the snapshot driver's new gauges are reported only at /__ursula/metrics, not over OTLP, and raft.snapshot_build_max_concurrency = 0 is taken as 1 rather than refused. In the validation runs with two builds, /__ursula/ready answered 503 for apply_lag in one or two 5 s samples per run, too briefly to fail a probe that needs three failures in a row. An operator's snapshot can miss a build of its group that has just ended and start another one. When that build covered the request, the request can wait 5 s and answer 503 although a covering snapshot exists, and a retry then answers 200.
• A node shed for failed writes can shed again every few minutes. A node shed because only its own cold-flush writes fail, while its probes answer, hands off its leaderships and then flushes nothing, so it heals about a minute after its last failed write even if its writes still fail. It sheds again once the leadership balancer hands leaderships back and a flush fails, for as long as such an outage lasts.

0.7.3

0.7.3 is a drop-in patch for 0.7.0, 0.7.1 and 0.7.2: format epoch 3 is unchanged, with no storage, snapshot or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1, 0.7.2 and 0.7.3 voters run side by side during the upgrade, and the 0.7.3 chart passes the same flags as the 0.7.2 chart. From 0.7.0, a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.3 together with the chart, as for 0.7.1. A configuration file that sets storage.cold.s3.probe_timeout, storage.cold.s3.unhealthy_ticks, storage.cold.s3.heal_ticks or governance.leadership_balance.peer_timeout to 0 no longer loads, so remove such a setting before the upgrade.

Upgrading gateways before voters, and rolling back voters before gateways, is recommended, not required. Clients trust the new x-ursula-not-applied marker only together with x-ursula-not-applied-checked, which only gateways from 0.7.3 on send, so no order in which a helm upgrade or helm rollback replaces gateways and voters can make a client count a committed write as never applied. Upgrading gateways first keeps the marker useful during the rollout. Before relying on the marker, check the layers around the gateway. A proxy or service mesh in front of it must not retry writes or follow redirects for them, and may answer 401, 402 or 429 only on a request it did not pass on. Nothing between the gateway and the nodes may resend requests or answer 401, 402 or 429, and a 0.7.3 gateway must not sit in front of an older gateway. See redirects and retries.

SQLite VFS owners upgrade one at a time, as within any minor: the 0.7.x extensions take over each other's streams. Upgrade @tonbo/sqlite-ursula together with the extension, because a 0.7.2 package over a 0.7.3 extension reports every refused commit as UrsulaPaymentRequiredError. The SQLite VFS stays experimental (#468).

• Rust library APIs changed. ursula_gateway::GatewayError has two new variants, LeaderUnreachable { upstream, source, applied_nothing } and LeaderNotFound { followed, applied_nothing }, and ursula-gateway now depends on ursula-proto. In ursula-raft, RaftGroupHandleRegistry::handle_transfer_leader returns the new IncomingTransferError instead of GroupEngineError, OwnerRaftHandle::handle_transfer_leader is no longer public, and RaftGroupHandleRegistry::request_self_election, SelfElectionError and OwnerRaftHandle::handoff_settle_time are new. ursula::SpawnRuntimeError gains PeerStatusClient, and ursula_config::validate::ValidationError gains ZeroS3HealthSetting (with the new S3HealthSetting) and ZeroPeerTimeout. None of these enums is #[non_exhaustive], so an exhaustive match on one outside its crate needs the new variants. The rest are additions: ursula_proto::headers with NOT_APPLIED_HEADER and NOT_APPLIED_CHECKED_HEADER, ursula_runtime::GroupEngineError::refused_before_proposal with ursula_runtime::RefusedBeforeProposal, and ursula_runtime::s3_operation_budget. None of these changes touches the wire or the disk.

Refusals that applied nothing

• Responses can prove that a write attempt applied nothing. A response that carries both x-ursula-not-applied and x-ursula-not-applied-checked answers an attempt that was refused before anything of it was proposed, so nothing of it can apply later, whatever the status. The first header, the marker, names the refusal with a token such as RaftLogPressure. The server marks the refusals of its write admission: 503 with WalDiskPressure, RaftLogPressure or IngressBodyBytesLimitReached, and 413 with RequestBodyTooLarge for a Content-Length over the route's limit. The gateway marks the requests no node received: its own refusals before forwarding, such as a 503 while its token issuer is down or a 403 from its authorizer, and the 503 it answers when it cannot connect to the first upstream it picked (UpstreamUnreachable). The second header is the gateway's vouch. A gateway from 0.7.3 on sets it on every response it returns, after it dropped any copy from an upstream and any node marker that a leader redirect may have made stale. Nodes never set it. So a client can rely on the marker only behind a 0.7.3 or newer gateway. It keeps counting 401, 402 and 429 as never applied by their status, and any other 5xx as an unknown outcome. The headers speak for one attempt only. The chart's TLS ingress example now sets nginx.ingress.kubernetes.io/proxy-next-upstream: "off", so ingress-nginx never resends a request to another gateway pod. In a service mesh, turn off retries for the stream routes on both sides of the gateway. Istio before 1.24, for example, resends a request up to twice on a 503 by default. Istio 1.24 to 1.26 does too when istiod runs with compatibilityVersion=1.23. See refusals that applied nothing.
• Refusals raised before a write was proposed are marked, redirects included. A node now marks a create, append, close or delete that it refuses before it proposes the write, when it can tell that it did: a 307 to the leader, or a 503 while no leader is known, from its own leadership check (NotLeader), a node that does not host the stream's Raft group (GroupNotHosted, a 307 to a node that does or a 503), and the 503 of uncommitted-bytes or cold-write backpressure (RaftUncommittedBackpressure, ColdBackpressure). NotLeader covers appends and closes, and creates unless the body is staged in object storage or storage.cold.max_hot_size_per_group is 0. A redirect or 503 that can follow a proposal stays unmarked, and so does a follower's redirect for a delete or for one of those creates. Snapshot and retention PUTs stay unmarked, because repeating them is harmless. Through a 0.7.3 gateway with several upstreams, a write that reaches a leader refusing at admission, such as 503 RaftLogPressure under sustained write load, through a follower's redirect now keeps the leader's marker, with the vouch.

Gateway redirects

• The gateway follows leader redirects until a node answers (#460). During leader changes, clients behind ursula gateway no longer get 307 redirects. The gateway follows Raft leader redirects itself, up to three hops after the first, none started later than 5 s after the first redirect. When a redirect names an address that is none of its upstreams, as happens when --upstream names the nodes otherwise than their peer URLs, or one the request already went to, it tries the upstreams it has not tried yet, in the configured order from the one after the upstream that redirected, and skips ejected ones. When it finds no leader within those limits, it answers 503 with Retry-After: 1, which Durable Streams clients retry. Before, it followed one redirect to a configured upstream and passed any other on with a path-only Location, which redirect-following clients looped on. Other 307s, which name no Raft leader, still reach the client with a path-only Location.
  • The final answer keeps its x-ursula-not-applied only when every redirect on the way carried one. Only then are the gateway's own 503s after redirects marked: LeaderNotFound when it stops following, and UpstreamUnreachable for a node it then could not connect to. A hop that was sent and then failed ends the request with an unmarked 502 or 504.
  • Each hop sends the request again. A plain append, without a producer or Stream-Seq, can apply once for each node that proposed it and then redirected, up to four times, where a gateway before 0.7.3 applied it at most twice. Use a producer for exactly-once writes.
  • The gateway learns a leader route only from a 2xx after redirects, not when it follows a redirect. /__ursula/gateway/metrics gains leader_searches and leader_not_found, and leader_redirects now counts each hop that follows a redirect.

Leadership shedding

• One failed S3 health probe no longer moves every leadership (#508). The snapshot-store health gate used to shed after one bad tick: a probe that missed its 2s deadline while the S3 client was still retrying it moved every leadership on the node to one peer within a second, and commits on several groups stalled. The gate now sheds only after three bad ticks with no good tick between them. A tick is bad when the probe fails or misses its deadline, or when a cold-flush write failed and no write has succeeded since, for as long as the S3 client can take to give up on a write (55 seconds with the defaults). So a store that refuses writes sheds the node even while the probe answers. With the defaults a node sheds about 10 to 20 seconds after its probes start failing, about 10 to 30 seconds after its writes start failing (sooner when the store refuses them outright), and about a minute after its write attempts start hanging until their timeout. While the gate judges the store unhealthy, routine snapshots are held back. A node shed for failed writes may lead again about two ticks after a write succeeds, and a node that has handed off every leadership waits until its last failed write is about a minute old.
• A shedding node moves its leaderships a few at a time, and only to peers that may campaign. The store gate and the cold-health gate each hand off at most four leaderships per tick, the groups with the most unflushed data first, and keep a handoff pending until another node leads the group. A node with N leaderships evacuates in about ceil(N / 4) ticks. Both gates pass over a peer that is draining, behind its recovery gate or shedding for its own store, cluster egress or WAL disk, and a peer that does not report its leadership-shed state within governance.leadership_balance.peer_timeout (500ms by default). The cluster-egress gate still moves every leadership at once, and now also reads its peers' leadership-shed state. It passes over a peer that answered that it may not campaign, tries a peer that did not answer at most once per leadership in each shed, and skips the peers that did not answer on a tick when another peer answered that it may not campaign.
• S3 health settings. storage.cold.s3.probe_timeout (default now 5s, was 2s), storage.cold.s3.unhealthy_ticks (now 3, was 1) and storage.cold.s3.heal_ticks (2) are now documented under Cold storage. A configuration that sets them keeps its values, and the flush rule and the pace of four handoffs per tick apply either way. 0 is rejected for each of them and for governance.leadership_balance.peer_timeout.

Raft

• Self-election on the leader is refused (#516). POST /__ursula/raft/{group}/self-election on a node that already leads the group now answers 409 Conflict and leaves the term unchanged. Before, it started a new election that could trip an OpenRaft invariant, the leader's vote falling behind the node's vote. In debug builds that stopped the core thread, and in release builds the invariant was silently violated. The endpoint also answers 503 Service Unavailable instead of 409 when the replica's Raft core or apply loop has stopped. A follower's self-election is unchanged, and no ursulactl command calls the endpoint.

SQLite VFS (experimental)

• A commit refused before it reaches the stream no longer poisons the database. When every attempt of a commit is refused in a way that proves nothing was applied, the commit fails alone with SQLITE_IOERR_AUTH (errcode 7178 in node:sqlite), as a 402 already did. That covers a token refused until URSULA_VFS_RETRY_MS runs out, a 429, and a 4xx or 5xx with both x-ursula-not-applied and the vouch of a 0.7.3 or newer gateway, such as the 503 RaftLogPressure of #519. Without a gateway, or behind an older one, the marker counts for nothing and only a 401, 402 or 429 fails a commit alone. A 403 without both headers still poisons, because a filtering proxy can send one after the stream applied the write. Reads keep working, and the next commit succeeds once the refusals stop, with no new attach. A refused commit with any attempt whose outcome is unknown, such as no answer, a 307 or a 5xx without both headers, still poisons, as before. status(file) reports writes_refused, refused_kind and refused_reason, the extension logs writes_refused, and instrument() adds the gauge sqlite_ursula.writes_refused. openUrsulaPiStorage rejects such a commit with the new UrsulaWriteRefusedError, and UrsulaPaymentRequiredError is its subclass for a 402. An attach whose claim is refused this way throws UrsulaWriteRefusedError, and ursula_attach now fails with SQLITE_IOERR_AUTH instead of SQLITE_ERROR for such a claim, while a 402 keeps SQLITE_AUTH. See Writes refused before they reach the stream.
• The commit after a refused one checks the refusal. If a layer in front of Ursula refuses a commit that the stream applied, the next commit now usually poisons the database, with a reason that says a commit refused as never applied may have been applied after all, instead of being acknowledged without its pages. Attaching again replays the commit the stream holds. The same poison can follow a commit that a gateway sent again to a new leader right after a refused commit, which costs an attach and loses nothing. When an attempt of that next commit got no answer first, the commit is still acknowledged, and the extension logs refused_commit_unchecked. Turn off retries and redirect following for writes in any proxy or service mesh in front of Ursula, because such a proxy can relay a refusal for a write that an earlier exchange applied.
• A transaction too large for one request fails alone. The extension never sends a commit frame over URSULA_VFS_MAX_FRAME_BYTES, 32 MiB after compression by default, the server's request body limit. That is about 8,000 changed 4 KiB pages of incompressible data, and about three times as many of typical text. A larger transaction fails alone with SQLITE_IOERR_AUTH, status() reports refused_kind: "too_large" with the frame's size and the limit, and later, smaller commits succeed without a new attach. A 413 with both headers counts the same. Before, the server's 413, or a connection it closed, poisoned the database.
  • Split bulk imports into smaller transactions. CREATE INDEX, REINDEX, a rebuilding ALTER TABLE and VACUUM are one transaction too. To VACUUM a database above the limit, first delete rows so that what it writes compresses below the limit, or copy the rows into a new database in transactions below the limit.
  • Behind a proxy or gateway with a lower body limit, set URSULA_VFS_MAX_FRAME_BYTES at or below it. It takes 64 KiB to 32 MiB. Otherwise a large commit usually meets a closed connection rather than a 413 (#540), and still poisons once the retry budget runs out.
• The extension never follows redirects. During a leadership change behind a gateway with access control, a redirected request used to go out without its bearer token, so an attach could fail as if the stream were missing, a snapshot read-back could fail, and a stream create could fail with "redirect failed". A 307 is now sent again to the same URL with the token, like a 503, within URSULA_VFS_RETRY_MS. Any other redirect, a 308 from a proxy included, fails the request at once with an error that names where it pointed. Point the stream URL at a gateway or at a single-node server, at the address a proxy would redirect to. A URL that names one node of a multi-node cluster works only while that node leads the stream, and a URL that a proxy redirects (from http:// to https://, say) fails. Snapshot publishes and retention moves now wait for the Retry-After of an answer they retry.
• Connections are kept for 90 s between requests, up from 15 s (URSULA_VFS_IDLE_MS, in milliseconds, 0 turns reuse off). A database that commits every 15 to 90 s reuses its connection instead of opening a new TCP and TLS connection for almost every commit. A request that gets no answer, such as one sent on a connection a proxy closed at that moment, is now sent again on a new connection: reads and HEADs within URSULA_VFS_RETRY_MS, where they used to fail the attach at once. Behind a NAT, a load balancer or a firewall that drops idle connections sooner than 90 s without telling either end, set URSULA_VFS_IDLE_MS below its limit. Resolving the stream's host and connecting to it, the TLS handshake included, now time out after 5 s each, so a host that does not answer no longer holds a request for its whole timeout.
• A snapshot that keeps failing backs off. It used to be read, compressed and sent again every 30 seconds, for example while it was larger than the gateway's --max-request-body-bytes. The next attempt now waits 100 ms after a failure and twice as long after each further one, up to an hour while the body is too large and up to five minutes otherwise, give or take a fifth. It starts at once after the database shrinks by more than a quarter, as after a VACUUM.
  • A snapshot over 1 MiB, and larger than any this owner published or installed, whose publish fails while the stream still answers counts as too large, and one that fails while the stream does not answer as an outage. A publish the server puts off, with a 503 that carries Retry-After or the marker, is an outage too, and the next attempt waits at least as long as it asked. After an outage, the first commit that succeeds brings the next attempt forward once, to within 30 seconds.
  • A snapshot publish over 1 MiB is sent twice at most per attempt, so a short outage during one now counts in snapshot_failures. A failed retention move is logged as retention_failed and tried again after the next snapshot, without counting as a failed snapshot. snapshot_failed gains retry_ms and kind (size, outage or other).
  • status(file) reports the latest snapshot body's compressed size as snapshot_bytes, and instrument() as the gauge sqlite_ursula.snapshot.bytes. Alert when it nears the gateway's limit. Behind Ursula's gateway, a snapshot a little over the limit is reported with a 413 and its size, and one far over it as a failed write (#540).
• Measurements are bounded, and instrument() drains them every second. An attached database no longer grows by up to 64 MB when nothing drains its measurements. Between drains it keeps the newest 8,192 commit, checkpoint and snapshot measurements, 512 KiB of commits. drainStats reports what it dropped in dropped. instrument() still counts every commit's bytes and retries, and counts the commits whose durations were dropped in the new counter sqlite_ursula.commit.unmeasured. It now drains every second by default instead of every 10 seconds, so its duration histograms see every commit up to about 8,000 commits a second. A file that commits faster needs a shorter intervalMs, and intervalMs: 10000 keeps the old cadence.
• A panic inside the extension no longer aborts the application that loaded it. The call fails instead: a database operation with an I/O error, after which the database must be attached again, and a SQL function with an error. The extension logs event=panic at level=error, with the entry point and the panic's message.
• @tonbo/sqlite-ursula declares the Pi Durable versions it supports: @earendil-works/pi-durable >=1.0.0 <1.2.0, where it was ^1.0.0. openUrsulaPiStorage passes Pi's storage conformance suite through the extension on 1.0.0 and on 1.1.0, including the scan order that 1.1.0 added to the Storage contract, so either version works without a code change. Keep your application's @earendil-works/pi-durable in the same range, so that npm installs one copy for both. With a version outside it, npm installs a second copy for @tonbo/sqlite-ursula without a warning, and openUrsulaPiStorage returns that copy's SqliteStorage. It implements the Storage contract of its own version, so a harness from another version can get wrong results from it. npm ls @earendil-works/pi-durable lists the copies.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests show that a failed write of the poisoned status fails closed.
• #517: a group sends each follower one Append call at a time, and since 0.7.2 a call carries at most 4 MiB of entries, so one group replicates to a follower at most about 4 MiB per round trip. That can slow the catch-up of a very busy group over a link with a long round trip.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described under 0.7.2.
• #519: under a sustained write load, the unsnapshotted Raft log of every node can reach its hard limit, twice raft.snapshot_log_budget, at about the same time, and the nodes then refuse client writes for about 20 seconds until snapshots catch up. These refusals now carry the marker RaftLogPressure, so behind a 0.7.3 gateway a SQLite VFS commit refused this way fails alone instead of poisoning the database.
• #529: POST /__ursula/raft/{group}/snapshot answers 500 after about 10 seconds when another group's snapshot build holds the node's build permit, because the deferred build is not retried for the request. Send the request again.
• #539: a node answers every leadership transfer it refuses with the gRPC status FAILED_PRECONDITION, also when its Raft core has stopped, so the sending leader cannot tell a stopped core from a refusal by policy.
• #540: the gateway answers some refusals before it reads the request body, such as an access-control refusal or a 413 for a body over --max-request-body-bytes, and then closes the connection. A client that sends the whole body before it reads the answer often sees a reset instead of the refusal. A SQLite VFS commit larger than one read that is refused this way is retried as an unknown outcome and poisons the database once the retry budget runs out, as in 0.7.2, and such a snapshot is reported as a failed write rather than a 413.
• #542: on macOS, a write to a socket that the peer already reset can fail with EINVAL, which the SQLite VFS does not retry as a request that got no answer. A read, HEAD, create or snapshot GET sent on a pooled connection that the server had just closed then fails the attach or the snapshot pass, where Linux retries it. Appends are not affected.
• A node shed for failed writes can shed again every few minutes. A node shed because only its own cold-flush writes fail, while its probes answer, hands off its leaderships and then flushes nothing, so it heals about a minute after its last failed write even if its writes still fail. It sheds again once the leadership balancer hands leaderships back and a flush fails, for as long as such an outage lasts.

0.7.2

0.7.2 is a drop-in patch for 0.7.0 and 0.7.1: format epoch 3 is unchanged, with no storage, snapshot or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1 and 0.7.2 voters run side by side during the upgrade.

• Rust library API changed. ursula_raft::RaftGrpcMetricsSnapshot has a new field, raft_grpc_append_stream_frame_bytes_max.

Fixed

• Server memory no longer grows with uptime (#507). Each voter keeps one long-lived gRPC stream per peer and core for Raft replication, and the gRPC library keeps a stream's encode and decode buffers at the largest message the stream ever carried. A replication message could batch 32 calls of up to 300 entries each, so one burst of large writes, or one follower catching up, left every stream holding buffers of up to hundreds of MiB until a peer restarted. In the 0.7.0 soak, memory grew by about 0.2 GB per hour on nodes whose peers stayed up, and it stayed up after the load stopped. A replication read now returns at most 4 MiB of entries, and a message stops taking calls once it holds 4 MiB, so it stays under about 8 MiB. After the same burst of large writes, the replication buffers a node kept fell from between 1.2 and 1.7 GB to about 0.4 GB, and its memory from about 2.7 GB to between 1.2 and 1.8 GB. /__ursula/metrics reports the largest message sent as raft_grpc_append_stream_frame_bytes_max. A single entry larger than a read still travels alone and leaves its stream's buffers at its size (#518). With a cold store, writes of at least runtime.external_payload_min_size (1 MiB by default) are staged in object storage, so such an entry comes only from a restore or from a JSON append of several hundred thousand messages, whose message offsets stay in the entry. Without a cold store, request bodies of up to 32 MiB go through Raft, so leave memory headroom on such nodes.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests now show that a failed write of the poisoned status fails closed.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described above.
• #519: under a sustained write load, the unsnapshotted Raft log of every node can reach its 2 GiB hard limit at about the same time, and the nodes then refuse client writes for about 20 seconds until snapshots catch up. It happened once in two runs of 128 SQLite owners writing flat out on three voters.

0.7.1

0.7.1 is a drop-in patch for 0.7.0: format epoch 3 is unchanged, with no storage, snapshot or Raft protocol change. Upgrade from 0.7.0 in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0 and 0.7.1 voters run side by side during the upgrade. The 0.7.1 chart passes the gateway the new --upstream-tcp-user-timeout flag, which a 0.7.0 gateway refuses at startup, so a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.1 together with the chart. SQLite VFS owners upgrade one at a time, as within any minor: the 0.7.0 and 0.7.1 extensions take over each other's streams.

• Rust library APIs changed. The ursula-control, ursula-raft (MetaRaftHandle migration methods, and RaftWal::open, which now returns RaftWalError with new variants), ursula-runtime (GroupInfraError::WalOpen), ursula-proto (ProcessIncarnation::try_from error type) and ursula-gateway Rust APIs changed, with no wire or on-disk change. GatewayConfig gains the public field upstream_tcp_user_timeout_secs (NonZeroU64), so code that builds it with a struct literal must set it.

Changed

• SQLite VFS: a 402 no longer poisons the file, and files can attach read-only. A layer in front of the stream, such as an authorizer or a quota or billing service, may refuse writes with 402 Payment Required. A commit it refuses now fails alone with SQLITE_IOERR_AUTH (UrsulaPaymentRequiredError through openUrsulaPiStorage). The file is not poisoned, reads keep working, and the next commit succeeds once the layer accepts writes again, without a new attach. status(file) reports payment_required and payment_reason. An attach it refuses fails with SQLITE_AUTH (UrsulaPaymentRequiredError from attach). attach(file, url, { readOnly: true }) (ursula_attach(path, url, 'read_only')) catches a file up without claiming the stream. It never writes the stream and fences no owner, and every write to the file fails with SQLITE_READONLY. See SQLite VFS.
• Release images carry their provenance and SBOM as GitHub attestations. The ursula and ursula-chaos-agent image indexes on GHCR hold only their linux/amd64 and linux/arm64 images, with no attestation manifests. Build provenance and an SPDX 2.3 SBOM are published as GitHub artifact attestations instead. Verify them with gh attestation verify oci://ghcr.io/tonbo-io/ursula:<version> --repo tonbo-io/ursula, and add --predicate-type https://spdx.dev/Document/v2.3 for the SBOM. GHCR receives stable releases only, with no 0.0.0-* candidate images or charts, and latest moves only to the highest stable release.

Fixed

• The gateway gives up on a connection to a vanished node within seconds (#495). A voter pod deleted with --force, or a host that loses power, leaves the gateway's pooled connections to it open without an answer. A request sent on one waited up to 30 s, reqwest's default TCP_USER_TIMEOUT, which is longer than the SQLite VFS's 10 s request timeout, so owners stalled for 10 to 22 s in failover tests. On Linux, upstream connections now give up after 5 s of unacknowledged data, set by the new --upstream-tcp-user-timeout flag (Helm gateway.upstreamTcpUserTimeoutSeconds). The request fails with 502, and the client decides whether to retry it. TCP keepalive follows the flag: an idle connection is probed after 2/5 of it and then every 1/5, each at least 1 s. With the default, three probes go out at 2, 3 and 4 s and a dead connection closes at about 5 s, so one lost probe does not close a healthy long-poll or SSE read. The flag takes at least 1 s.
• The gateway stops picking a node that does not answer (#454). After a transport failure it dropped the cached leader route but picked the next upstream at random, often the dead one again, until that node came back. After a host power-off, one stream failed through the gateway for about 60 s. An upstream that refuses or times out a connection, or breaks an exchange, is now skipped for 10 s, and each failure after a window doubles the next one, up to 60 s. A missing response within --response-header-timeout does not count, because one slow Raft group on a live node causes it too. When a window expires, one request probes the upstream while the others keep skipping it, for at most --connect-timeout plus --upstream-tcp-user-timeout. Any response puts it back. If every upstream is skipped, requests still go out, along their cached leader route first. /__ursula/gateway/metrics reports upstream_ejections and upstream_ejected.
• A node that lost its WAL run state recovers above the epochs its cores verified (#411). Without run-state.bin, 0.7.0 restarted the recovery epoch at 1, so a core that had already verified a later epoch took its journal as verified and skipped the recovery rewrite of its durable prefix. The next epoch now starts above the highest one any core verified. Empty journals whose core metadata records a verified epoch count as earlier history, so the node recovers behind the recovery gate instead of starting as a new disk.
• Snapshot pointers survive power loss and failed writes (#411). When a group's snapshot pointer write failed after its rename, for example in the directory fsync, the node could delete the snapshot object that the new pointer names, and a restart that found that pointer could not restore the snapshot. That object now stays in object storage. Later pointer writes of that group fail with snapshot metadata publication stopped after an earlier failure until the node restarts, and a restart fsyncs the pointer's directory before it releases older snapshots. A directory created for a pointer is now fsynced before the pointer counts as published.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests now show that a failed write of the poisoned status fails closed.
• #507: in a soak with a steady write load, server memory grew by about 0.2 GB per hour of uptime and stayed up after the load stopped, while a restarted node used less. Leave headroom in the servers' memory limit and watch their memory. Fixed in 0.7.2.

0.7.0

0.7.0 is a breaking release. It changes the Raft WAL format (format epoch 3) and removes the memory WAL, so the per-core disk journal is the only Raft WAL. 0.7 cannot read 0.6 data: a node refuses 0.6 Raft WAL directories and object-storage prefixes at startup, ursulactl restore refuses 0.6 backups, and a node will not join a cluster with 0.6 nodes. There is no in-place upgrade, no automatic migration and no rollback.

• If you run 0.6 with data you need to keep, stay on 0.6.2. 0.7 provides no migration path from 0.6.
• Deploy 0.7 fresh: empty PVCs and a never-used object-storage prefix.
• raft.wal.fsync now defaults to never. The 0.6 disk WAL fsynced every batch. Set always to keep that behavior.
• helm upgrade from 0.6 stops before a second voter is replaced. Recover with helm rollback.

This release adds an fsync policy and a recovery gate to the Raft WAL, splits each core's journal into segments with bounded memory, binds the WAL to the routing configuration, and removes the memory-WAL restart tooling.

Installing 0.7.0

See Upgrading to 0.7.

1. Install fresh. Install with a new s3.prefix and empty PVCs (with cold storage off and S3 snapshots, also a new snapshotStore.prefix). With the event-time indexer, use a new or emptied indexer.s3.prefix and register the indexes again.
2. Never reuse a prefix that an earlier release, or a failed upgrade attempt, wrote to.
3. Update the configuration. Remove the keys and chart values listed under Removed in 0.7.0. Set raft.wal.path on every node of a multi-node cluster: 0.6 defaulted to the memory WAL, and a 0.7 node with raft.peers refuses to start without a WAL path. Choose raft.wal.fsync (Helm raft.walFsync) as described under WAL fsync policy.
4. Update runbooks and tooling. Take the removed ursulactl verbs out of scripts. Custom tooling that calls admin mutations must send x-ursula-process-incarnation (Changed).

A values file that still sets a removed chart value fails validation, and nothing changes. Otherwise helm upgrade from 0.6 replaces the first pod, which finds the 0.6 data on its volume and exits at startup, so no second voter is replaced. Recover with helm rollback. With OnDelete, also delete the refused pod after the rollback so that it starts from the 0.6 template again. The refused pod wrote nothing. A 0.7 node that contacts 0.6 peers makes each of them answer 503 with format_epoch_mismatch on /__ursula/ready until it restarts, so keep the peer lists of 0.6 and 0.7 nodes apart.

WAL fsync policy

raft.wal.fsync (Helm raft.walFsync) chooses when journal appends reach stable storage. Votes, each group's log state and the node's run state are always written with fsync.

• never, the new default. A replica acknowledges an ordinary data append after writing it to the page cache, without fsync. A data write is acknowledged once a quorum of voters holds it in the page cache. Membership batches are always fsynced before acknowledgment, including the bootstrap membership. A process crash loses nothing, and a graceful shutdown fsyncs every journal. A host crash or power loss can cost that replica the unsynced tail of its journals. While at most a minority of a group's voters lose their tail, the other voters hold every acknowledged write. If a majority of a group's voters crash at once, writes acknowledged after their last fsync can be lost, and the group stops until an operator accepts that loss.
• always. A replica acknowledges a batch only after fsync, as the 0.6 disk WAL did. A host crash loses no acknowledged write. Each batch waits for an fsync, which costs write latency.

See Configuration and Recovering after a host crash.

Recovery gate

A replica that may have lost acknowledged writes recovers behind a gate for each of its groups. That covers a host crash under never, a run stopped by a journal I/O error, a wiped or replaced WAL directory, a missing run state next to a non-empty journal, and a damaged journal tail.

• A gated replica neither votes nor campaigns until it has applied a fresh ReadIndex barrier from the group's leader. A replica that led the group restarts as a follower. Meanwhile /__ursula/ready answers 503 with reason recovery_gate_closed, and recovery_gates in /__ursula/metrics lists the gated groups.
• The leader rebuilds a replica that lost entries it had acknowledged. It removes the voter, adds it back as a learner and promotes it once it has caught up. While a majority of the group's voters are intact, no operator step is needed.
• A replica whose WAL directory was wiped also lost its vote. It refuses replication until the group's leader proves a fresh quorum without it. In a 3-voter group that needs both other voters up.
• If a majority of a group's voters are gated, the group has no leader and refuses writes. After 30 seconds the gated replicas report the group stalled (readiness reason recovery_stalled). An operator then runs POST /__ursula/raft/{group}/recovery/accept-unsynced-loss on the gated replicas with the most up-to-date last log id (highest last_log_term, then last_log_index) until the open replicas form a majority. The request names the replica's last_log_index and current_term. It answers 409 and changes nothing when the gate is not stalled, when the replica no longer holds that log, or when the replica lost its vote and has no durable vote floor.
• Under always, a host crash can still leave a complete invalid frame after the last fsync in a journal's newest segment, for example a torn sector inside an unacknowledged batch. Recovery cannot tell it from media corruption, so it gates every group on that core. A single-voter group, or a group gated on a majority of its voters, then needs accept-unsynced-loss although no acknowledged write was lost.
• A single-voter group has no other replica to recover from. Under never it needs accept-unsynced-loss after every host crash, once its gate reports the group stalled.

See Recovering after a host crash and Rejoining a wiped node.

New

• Segmented journal with bounded memory. Each core's journal is a run of segments (raft.wal.segment_size, default 64 MiB, Helm raft.walSegmentBytes). Purge deletes whole segments, and the few entries quiet groups leave in old segments are copied forward. Each group keeps an index of its entries and a cache of recent ones (raft.wal.cache_size, default 256 MiB per node, Helm raft.walCacheBytes). Older entries, such as a lagging follower's catch-up, are read from disk. /__ursula/metrics reports the new work under wal_segments, wal_rotations, wal_rewritten_bytes, wal_pinned_segments, wal_lagging_groups and the wal_cache_* and wal_disk_read* counters.
• WAL layout guard. {raft.wal.path}/raft-log/topology.bin records runtime.core_count and raft.group_count. A node refuses to start when either differs from its configuration, and when an existing WAL has no record or a corrupt one. Before, a changed core count moved groups to other journals and a changed group count remapped streams. Keep both values explicit. See WAL routing configuration.
• Graceful voter shutdown. On the first SIGTERM or Ctrl-C a voter marks itself drained and hands each leadership to a caught-up voter for up to 5 seconds. It then drains its listeners for up to 10 seconds, stops its groups, fsyncs every journal and records a clean shutdown. The whole shutdown has a deadline of 20 seconds from the signal. A shutdown that has not finished by then exits without recording a clean shutdown, and the next start treats the run as a crash. The chart's new server.terminationGracePeriodSeconds defaults to 30 to fit. See Graceful voter shutdown.
• Raft work stays on the owning core. Raft RPCs from peers run on the core that owns the group, and each core keeps its own peer connections. A group's journal append returns once the batch is queued, and Raft counts the entries once the fsync policy is met, so a group no longer waits for one journal write before it queues the next. raft.append_transport_budget_bytes (default 128 MiB) caps the append bytes queued for Raft transport in each direction across the node.
• CPU pinning. On Linux, each core's worker thread is pinned to a CPU when the process may use exactly runtime.core_count CPUs. runtime.cpu_affinity = true forces pinning and false turns it off.
• Fresh-quorum checks. ursulactl verify-quorum confirms each group's leader with a fresh ReadIndex round and waits until every replica has applied that prefix (admin route GET /__ursula/raft/{group}/quorum). ursulactl verify-survivors runs the same check on the two remaining voters when one is down. See ursulactl.
• Maintenance fencing for rollout tooling. ursulactl pin-incarnations, activate-maintenance-fence, retire-maintenance-fence, startup-admit and the reservation-* verbs, with the chart values server.gracefulRollout.maintenanceReservation and server.startupOwnership, serialize planned voter replacements through a shared reservation and bind them to process and host identity. Both chart values default to false. Reservation stores and verify-quorum proofs each have one schema version, and ursulactl refuses any other. See Maintenance executor admission.
• Restore checks cold objects first. ursulactl restore asks the target cluster to check every group's cold references against its cold store (POST /__ursula/backup/group/{group}/cold-check). If any object is missing, it imports nothing and names the first missing keys. See Backup and disaster recovery.
• SQLite VFS on npm (experimental). The TypeScript helpers are published as @tonbo/sqlite-ursula, with the extension prebuilt for linux-x64-gnu, linux-arm64-gnu, linux-x64-musl and darwin-arm64. The VFS is experimental: its formats and API may change in a minor release (#468). It speaks https:// and sends a bearer token to a gateway with access control (URSULA_VFS_TOKEN_FILE or setToken). Attach refuses a file whose stream is missing, which may hold the only copy of its data, instead of starting it over. A WAL write-back that the local file lost poisons the database before a commit builds on it, and on Linux before a published snapshot does. status(file), instrument(file, meter) and logfmt lines on stderr report the owner's health. See SQLite VFS.

Changed

• Admin mutations are bound to one server process. Every mutating admin request needs x-ursula-process-incarnation, the process_incarnation that GET /__ursula/metrics reported before the operation. Without it a node answers 428. A node that restarted since then answers 412 and changes nothing. ursulactl sends the header. Custom tooling must add it, bucket purge included. See Admin endpoints.
• Format epoch 3. The Raft WAL journal and state files, the WAL directory and object-storage markers, group and stream snapshots, snapshot references, backups (format_version 3) and the Raft gRPC protocol carry format epoch 3. A node refuses data, backups and peers of another epoch.
• A journal write or fsync error stops the process. The journal writer retries nothing and aborts the process. The next start keeps the verified prefix of the journals and puts the node's groups behind the recovery gate. Journal frames now carry checksums over their header, payload and journal sequence, so a damaged length is reported as corruption. Recovery refuses damage in sealed segments and repairs only the tail of the newest one.
• Bootstrap. A replica that ever held a group never initializes it again. A replica that holds nothing initializes the group only when every configured voter reports an empty group and no leader. This replaces the 0.6.2 restart guard, and 0.7 neither writes nor reads its URSULA_GROUP_INITIALIZED/ markers in object storage.
• Raft without a WAL path. 0.6 defaulted to the memory WAL. Every preset other than default, and a config file loaded without --preset default, now runs Raft on the disk WAL. A single node without raft.wal.path keeps its WAL in a temporary directory that it logs at startup and removes on a clean shutdown. The Docker image's default configuration runs this way.
• Chart rollouts. The graceful-rollout hook runs drain, delete, wait and undrain for one voter at a time. An external OnDelete controller runs the same sequence.
• Leadership transfer errors. POST /__ursula/raft/{group}/leader/transfer/{node} answers 409 for a changed leader or a recovering target, 400 for an invalid target, 404 for an unregistered group and 500 for a stopped Raft. ursulactl drain observes again and plans again after a 409.
• Leader forwarding is bounded. A follower that forwards a read, HEAD or bucket purge to a leader it cannot reach answers 503 with Retry-After within about 3 seconds, or 10 seconds when the leader accepts the call and never answers. Before, it answered 500 once the kernel gave up on the connection, about 16 minutes after the leader's host lost power, and the group's later requests waited behind it. Connections between nodes now send HTTP/2 PINGs to detect a silent peer.
• Gateway upstream failures. ursulagw forgets a cached leader when its upstream fails at the transport. An upstream it cannot connect to answers 503 with Retry-After, and one that sends no response headers in time answers 504. Both used to answer an empty 502.
• Readiness checks the node's own Raft replicas. /__ursula/ready answers 503 with the reason raft_replica_unready while one of the node's replicas is not a healthy voter, for example a wiped replica that its leaders rebuild as a learner. It also adds the reasons recovery_gate_closed and recovery_stalled and the list recovery_stalled_groups, and the admin plane now serves it next to /__ursula/metrics. Readiness does not report lost redundancy: healthy surviving voters stay ready while a replica rebuilds. Alert on incomplete_voter_set in raft_maintenance.group_issues instead (see Observability).
• Metrics. /__ursula/metrics adds process_incarnation, raft_maintenance, wal_recovery, recovery_gates and each group's last_log_term, and OpenTelemetry adds the gauges ursula.wal.recovering and ursula.raft.recovery_gates. memory_wal_full_restart_groups and the wal_backend metric are gone.
• ursulactl wait-ready waits for full redundancy. Besides a leader for every group, it requires each node's raft_maintenance report to show the complete voter set, no uncertain maintenance fence, and 200 from /__ursula/ready. It retries connection failures until its timeout.
• Retention reclaims external payloads it drops from state. An append of at least runtime.external_payload_min_size keeps its object's locator in replicated state until the leader moves it into a cold-index page. Retention that passed such an object before that dropped the locator without queueing the object, which then stayed in object storage until the cold orphan sweep deleted it, a day later. Retention now queues it for the cold GC with the same 5-minute grace as dropped pack slices (#475).

Removed in 0.7.0

• The memory WAL and its configuration keys: raft.wal.backend, raft.wal.allow_volatile_multi_peer and raft.memory_bootstrap_marker_dir, and also raft.rejoin_probe. A config file that still sets one fails to load.
• Chart values raft.storageMode, raft.allowVolatileMultiPeer and persistence.enabled. Every server pod gets a raft-data PVC, and raft.logDir must be set. The values schema refuses unknown keys under raft and persistence.
• ursulactl verbs prepare-restart, restart-quiesce-capability, finish-prepared-restart, abort-prepared-restart, prepare-amnesiac-restart, repair-restarted-voter, prepare-recovery-handoff and classify-amnesiac. A planned restart is drain, restart, wait and undrain (see Restarting a voter). A voter that lost entries needs no verb: its group leaders rebuild it.
• Admin endpoints POST /__ursula/raft/quiesce-for-restart, POST /__ursula/raft/{group}/rejoin/adopt-survivor/{node} and POST /__ursula/raft/{group}/rejoin/reinitialize. accept-unsynced-loss replaces the last two.

Known issues

• #411: an adversarial review of recovery left residual risks open, mostly liveness. For example, a node that loses its run state file can restart a recovery epoch that its cores already verified, and a failure to record the poisoned status is not yet covered by tests. 0.7.1 fixes the run state case.
• #454: after a voter host loses power, ursulagw keeps choosing the dead upstream at random until its pod returns. Each such attempt waits for the 5 second connect timeout, so a stream can fail through the gateway for about a minute although the cluster elected new leaders within seconds. Clients that connect to the servers directly are not affected. Fixed in 0.7.1.

0.6.2

0.6.2 is a drop-in patch for 0.6.x: format epoch 2 is unchanged, with no change to existing storage records, snapshots or the Raft protocol. It adds one small marker object per memory-WAL group in object storage, which 0.6.0 and 0.6.1 ignore. Upgrade from 0.6.0 or 0.6.1 in place with an ordinary rolling restart (helm upgrade); no fresh install or data move is needed.

0.6.0 or 0.6.1 voters and 0.6.2 voters run side by side safely during the upgrade; a group whose leader still runs 0.6.0 or 0.6.1 is rebuilt by the graceful-rollout hook as before, which runs the ursulactl of the chart's image.

Fixed

• A memory-WAL node restarted alone heals itself. It no longer initializes the groups it bootstraps again (it does so only when every configured voter reports an empty group and no leader), and the leaders rebuild it with no operator: a leader that sees a follower lose entries it had acknowledged commits a voter set without it, adds it back as a learner, and promotes it once it has caught up, the same sequence as ursulactl repair-restarted-voter. The steps are read off the current membership, so a leader change, a second restart or repair-restarted-voter running at the same time converge. Leadership handoffs skip such a follower until it is rebuilt.
• An empty memory-WAL replica never votes before it has caught up. Until it holds the log its leader had committed when it first heard from it, it refuses every candidate with a non-empty log, so it can no longer help elect a leader that lacks entries committed with its lost copy. A fresh group's first election is unaffected.
• A majority of a memory-WAL group restarted at once stops the group instead of losing writes. No leader is elected from the incomplete logs and writes are refused (503 while no leader exists) until an operator adopts the survivor's log with POST /__ursula/raft/{group}/rejoin/adopt-survivor/{node} on every node of the group. See Restarting a memory-WAL node.
• A restart of every memory-WAL node stops the groups that held writes instead of re-initializing them empty. Each memory-WAL group keeps an "initialized" marker in object storage, URSULA_GROUP_INITIALIZED/group-{id} beside the format-epoch marker; a leader writes it before the group's first write and acknowledges no write without it, and a replica of a group that already holds writes (created by 0.6.1) writes it as well. When every voter reports an empty log and the marker exists, the group stays without a leader, refuses writes (503), and is listed under memory_wal_full_restart_groups in GET /__ursula/ready on its initializer, until an operator accepts the loss with POST /__ursula/raft/{group}/rejoin/reinitialize?accept_data_loss=true on that node. Without a marker (a fresh install, or every node crashed before the first write) the group bootstraps as before; if the marker cannot be read, it retries and never initializes blind. Without object storage a full restart cannot be detected and the groups still initialize again, empty. 0.6.1 ignores the marker, so rolling upgrade and downgrade stay safe; an upgraded cluster is protected once every pod runs 0.6.2.
• ursulactl repair-restarted-voter accepts a group the leader already rebuilt.

0.6.1

0.6.1 is a drop-in patch for 0.6.0: format epoch 2 is unchanged, with no storage, snapshot or Raft protocol change. Upgrade from 0.6.0 in place with an ordinary rolling restart (helm upgrade); no fresh install or data move is needed.

The chart's graceful-rollout hook runs the ursulactl of the chart's image, so the upgrade to 0.6.1 already uses the fixed readiness check below.

Fixed

• A memory-WAL voter restarted empty no longer passes readiness. On restart it rejoins as a voter with an empty log for the groups it bootstraps, and an idle leader never backfills it. ursulactl counted such a voter ready whenever the peers' commit index was within --lag-tolerance, so repair-restarted-voter skipped those groups and a rollout continued with two real copies. A voter that has applied nothing while its peers have committed entries is now never ready, and repair-restarted-voter rebuilds it through the learner catch-up path; classify-amnesiac counts such groups as missing for amnesiac recovery, so an empty voter outside --lag-tolerance is now recoverable instead of refused.
• The leadership balancer hands a group only to a caught-up peer. It used to hand leadership to a voter that had not replicated the leader's log; the transfer then left the leader muted while the target could not win, and the next restart fence timed out with "leader … is stale". The balancer now waits until the leader's replication metrics show the target matched its last log entry.
• The restart fence waits out an in-flight leadership transfer. The restart fence (prepare-restart, prepare-amnesiac-restart, repair-restarted-voter) now waits 1.5 s (the Raft minimum election timeout) after fencing the surviving voters before it checks leadership, so a transfer accepted just before the fence settles first.

Known issue (fixed in 0.6.2): a restarted memory-WAL node rejoins as an empty voter

A node on raft.wal.backend = "memory" restarts with an empty Raft log, but the group leaders still count it as a voter, do not resend what it lost, and it grants its vote to any candidate. Until it is rebuilt, each of its groups holds two real copies while reporting three, and one more failure can lose acknowledged writes. A prepared rollout rebuilds it (repair-restarted-voter, fixed above); a crash, OOM kill, eviction or node loss does not. Production clusters must use the disk WAL. On a memory-WAL cluster, after any unplanned restart run ursulactl verify-cluster and, if it reports the node unready, ursulactl repair-restarted-voter --node N. See Restarting a memory-WAL node.

0.6.0

0.6.0 is a breaking release. It changes the storage format (format epoch 2) and removes several protocol features (listed in the release notes). It refuses to start on data written by 0.5.x (Raft WAL directories, object-storage prefixes, snapshots and backups) and will not join a cluster with 0.5.x nodes. There is no in-place upgrade, no automatic migration and no rollback.

• If you run 0.5.x with data you need to keep, do not upgrade yet. Stay on 0.5.1 and open an issue so we can work out a migration path with you.
• New deployments install on fresh storage: empty PVCs and a never-used object-storage prefix.
• helm upgrade from 0.5.x stops at the first pod. Recover with helm rollback.

This release changes the storage format to epoch 2, removes the protocol features listed under Removed, stores JSON messages exactly as written, and stops per-stream replicated memory from growing with message count or cold history, without retention (see Memory without retention).

Installing 0.6.0

See Upgrading to 0.6.

1. Install fresh. Install with a new s3.prefix and empty PVCs (with cold storage off and S3 snapshots, also a new snapshotStore.prefix). With the event-time indexer, use a new or emptied indexer.s3.prefix and register the indexes again (Event-time indexer).
2. Never reuse a prefix that an earlier release, or a failed upgrade attempt, wrote to.
3. helm upgrade from 0.5.x stops at the first pod. The new pod exits at startup or reports not ready (/__ursula/ready answers 503 with format_epoch_mismatch), so no second voter is replaced and the refused pod writes nothing. Recover with helm rollback.
4. SQLite VFS. Stop every owner of a stream before upgrading the extension, then delete <db>, <db>-wal and <db>-ursula or attach under a new stream URL (SQLite VFS).

Always on

Every group runs one behaviour from its first command; there are no feature levels to raise. Always on: bounded producer receipts (1,024 receipt items per stream plus each producer's newest; older duplicates are answered 204 without ranges), 7-day idle-producer expiry, the 4,096-producer cap, unique incarnations, external locators committed before indexing, the compact hot window (no per-message bookkeeping), and cold-tier snapshot bodies up to 1 GiB.

Observable changes for JSON streams

Every application/json stream now follows JSON message text: Ursula validates the body and stores each message as the writer's text with insignificant whitespace removed, followed by one LF. Compared with earlier releases, which re-serialized every message:

Input	Before	Now
{"b":1,"a":2}	{"a":2,"b":1} (members sorted)	{"b":1,"a":2} (writer's order)
{"a":1,"a":2}	one a kept	both members kept
1.50e3	1500.0	1.50e3
1e400	400 Bad Request	stored as written
"\ud800" (unpaired surrogate escape)	400 Bad Request	stored as written
[{"a":[…127 levels…]}]	depth counted including the outer array	depth counted per message: a message inside an array body may be as deep as a bare one (127)

• Every read surface returns the stored text: NDJSON reads, SSE and bootstrap parts.
• Offsets, ETags and byte counts refer to the stored text, so a message's size can differ from what the previous release would have stored for the same input.
• Clients that parse records must accept unpaired surrogate escapes and numbers outside the double range. Ursula's event-time indexer does.

New

• Stream-Incarnation as a request precondition. Every request that addresses a stream may carry Stream-Incarnation; the server compares it with the stream's incarnation atomically with the request (a write when it is applied) and answers 412 Precondition Failed (a refused write applies nothing) when the stream was deleted and recreated, including by TTL expiry and a later create. A PUT with it for a missing stream answers 412 and creates nothing; a 404, 410 or 416 stays as it is; a long-poll or SSE read with it ends when its stream is recreated. Without the header nothing changes. A read refused by the precondition returns no data but, like any request that reaches a stream, may still renew the sliding TTL of the stream now at that path. Every successful stream response except a DELETE now carries Stream-Incarnation, not only HEAD. The SQLite VFS sends it on every request after attach, so an owner of a deleted stream can no longer reach the stream recreated at its path, and the event-time indexer sends it on reads. See Stream incarnation.
• Metrics. /__ursula/metrics adds group_state_gauges (per-group state sizes).
• External payload locators. External appends commit their locator in replicated state first and the leader indexes it afterwards, so runtime.external_payload_min_size may be lowered.
• Compact hot window. Replicated state keeps nothing per message, and cold admission and flush thresholds charge hot payload only.
• Snapshot bodies in the cold tier. With a cold backend, snapshot bodies of at least runtime.external_payload_min_size are streamed into the cold tier instead of replicated state, and bodies of up to 1 GiB are accepted (32 MiB inline). Snapshot reads and /bootstrap stream them back and send Content-Length. Superseded bodies are deleted five minutes after the publish that replaced them.

Changed

• Bootstrap is an honest partial. /bootstrap never reads cold storage and never skips a byte. When updates after the snapshot are already cold, it returns the snapshot part alone with Stream-Next-Offset at the snapshot point and no Stream-Up-To-Date; when updates exceed 8 MiB, a JSON stream stops at the last message boundary within the cap and any other stream answers the snapshot part alone. Clients continue from Stream-Next-Offset with ordinary reads, as the specification already required. On a JSON stream each update part holds exactly one message, split on LF, so a JSON array append now appears as one part per element.
• Producer receipts are bounded. Each stream keeps exact receipts for 1,024 items plus each producer's newest one. An older duplicate is still deduplicated, but answered 204 without range headers. A producer idle on a stream for 7 days is forgotten and must restart at sequence 0 or bump its epoch. See Exactly-once writes.
• Producer-Id and Stream-Seq are capped at 256 bytes on every write path, with 400 above it.
• Unique incarnations. A stream's created_at_ms strictly increases within a Raft group, so a stream deleted and recreated at the same path is always a distinguishable incarnation, and stream GC only removes the deleted incarnation's objects.
• Snapshots are written in a compact MessagePack envelope that earlier releases cannot read.
• Producer cap. A stream holds at most 4,096 producers; a new producer evicts producers idle for at least an hour, and when none is, the write answers 429 with a body starting producer_limit.
• Reads are capped at 8 MiB. A larger max_bytes is clamped. A capped offset read may end inside a message, on a JSON stream too: an offset read without max_bytes, or with max_bytes above 8 MiB, no longer ends at a message boundary. Continue from Stream-Next-Offset.
• Binary bootstrap returns one part. On a stream that is not application/json, /bootstrap returns the hot updates after the snapshot as one part, which may contain several messages; a part boundary is not a message boundary. When those updates exceed 8 MiB, or an external append among them is not hot, the response is the snapshot part alone.
• Non-JSON snapshot and retention offsets need no message alignment. Any offset between the retained offset and the tail is accepted; an offset inside a message used to answer 400 or 204 depending on whether it had been flushed.
• JSON snapshot and retention offsets must follow an LF, and the check fails closed. On an application/json stream, PUT …/snapshot/{offset} and PUT …/retention/{offset} answer 400 for an offset inside a message, whether its bytes are hot or already flushed (an offset below the flushed point used to be accepted), and 503 when the server cannot read the byte before the offset to check it (it used to skip the check). Retention is exact: it no longer lands below its target. Snapshot publishes and retention advances no longer cost a linearizable HEAD on every stream; a flushed JSON boundary costs one leader read.
• Hot tails are flushed after storage.cold.flush_max_hot_age (default 5 minutes, 0 disables), even when the group is below its flush threshold.
• Defaults. storage.cold.compaction_enabled now defaults to true, and storage.snapshot.backend defaults to auto (S3 when the cold backend is S3, inline otherwise). Raft snapshots are taken by log bytes against a node-wide raft.snapshot_log_budget (default 1 GiB) with a raft.snapshot_backstop_logs entry backstop.
• Format epoch 2. The Raft WAL directory and the object-storage prefix carry a format-epoch marker, every snapshot starts with a format-epoch frame, snapshot pointers are MessagePack only, the WAL journal header, the Raft gRPC protocol (now also on forwarded reads and writes), S3 snapshot references and backups are version 2, and ursulactl checks the target cluster's backup format before the first import. A node checks its WAL directory, its object-storage prefix and its peers before it writes anything, and /__ursula/ready answers 503 with format_epoch_mismatch after any later Raft protocol mismatch.
• Reserved bucket IDs. snapshots (the default raft-snapshot namespace in the object store) and IDs starting with __ answer 400.
• PUT /{bucket} validates the bucket ID. It answers 400 for an ID that does not match [a-z0-9_-]{4,64} and 201 otherwise, and still changes no state: buckets are implicit namespaces. See Buckets.
• Helm chart defaults. The chart now renders coldStorage.compaction.enabled: true and snapshotStore.backend: auto (previously false and inline), so a release with cold storage gets compaction and S3 snapshots unless its values set the old ones.
• TTL renewal is monotonic. A read or write never moves a stream's expiry earlier. Before, a request stamped by a node whose clock lagged the node that last renewed the stream could shorten the stream's life by that lag.
• consistency=leader reads are linearizable. Before it answers a consistency=leader read, HEAD, /bootstrap, a snapshot read or a long-poll/SSE registration, the leader confirms with a majority of voters that it still leads (Raft ReadIndex) and waits until it has applied everything committed up to that point. Each such request costs one round trip to the voters, shared by concurrent requests to the same Raft group. On the leader, the confirmation runs before the request reaches the group, so appends and other reads to that group do not queue behind it. offset=now without consistency=leader takes no confirmation and may resolve to a lagging tail right after a leader change. A long-poll or SSE read takes its offset=now position, existence, content type and incarnation from its registration, and its first read is served only by the leader that registered it. A long-poll also answers 416 at registration for an offset beyond the tail (an SSE session reports it as an error event from its first read), ends when the stream it registered against is recreated even without Stream-Incarnation, and on timeout answers 204 at the requested offset. A leader cut off from the majority, or one that cannot confirm in time, answers 503 with Retry-After instead of serving state that may miss appends a newer leader acknowledged. Before, a deposed leader could serve these reads stale, and a follower dropped consistency=leader when it forwarded the read.
• SQLite VFS: opaque offsets. ursula_attach, ursula_status, ursula_stats and @tonbo-io/sqlite-ursula (attach, status, drainStats) return stream offsets as the server's strings ("-1" for none) instead of numbers; compare them as strings. The extension never computes an offset: it verifies its claim by reading from the tail it caught up to, and every commit carries a Stream-Seq, so a foreign append with a higher one fences the owner. The sidecar (<db>-ursula, format 2) and the snapshot body (USS2) changed: the first attach after the upgrade rebuilds the local file, and a stream whose latest snapshot is an older body (USS1) cannot be attached: attach under a new stream URL (deleting the local files is not enough).
• Bucket purge is admin-plane only. DELETE /__ursula/purge/{bucket} is served only on the admin listener (server.admin_listen, loopback by default); the client listener answers 404. A Raft group whose leader is unknown or changing answers 503 with Retry-After instead of redirecting (307) to a peer's client URL; purge is idempotent, so retry it. /__ursula/usage stays on both listeners. See bucket purge.

SQLite VFS

• Trusted network only. The extension speaks plain HTTP to Ursula, with no TLS and no credentials. Run it, and the Ursula or ursulagw endpoint it talks to, on a network you trust.
• The local file is a cache. A commit is durable once Ursula acknowledges it; the extension no longer fsyncs the WAL or the sidecar on every commit. After every reboot (a new kernel boot id), and whenever the local files cannot be verified, attach rebuilds the database from its latest snapshot plus the tail since it. In the 0.6 benchmarks a fresh host took 16 to 27 s to attach a 1 GB database, so a fleet-wide rolling reboot is a burst of snapshot reads, and a stream whose snapshots cannot be published replays from the last one it has. See the SQLite VFS design.
• Owners are bound to a stream incarnation. The extension's Producer-Id is sqlite-ursula-vfs/<Stream-Incarnation> and every request after attach carries Stream-Incarnation, so an owner of a deleted stream is fenced instead of writing into the stream recreated at its path. Versions with the old Producer-Id and this one do not fence each other: stop every owner of a stream before upgrading (or downgrading) the extension.

Event-time indexer (experimental)

• API break. Query and status return (offset, len) locators with opaque next and through tokens instead of record ordinals. The 0.5.1 query parameters after_captured_at_ms, after_record and through_record answer 400, and unknown registration fields answer 422. Registration takes extract (JSON pointers, with fallbacks and units) and start; the legacy timestamp_field is still accepted. Re-register every index under a new or emptied indexer.s3.prefix.
• NDJSON sources (application/x-ndjson) are indexed, and nested or unit-ful timestamps (OTLP JSON) are supported. Messages without a usable event time are skipped and counted instead of blocking the index.
• Chart values. indexer.workers.segmentBytes replaces readBatchRecords and segmentRecords.
• Polling cost. Every indexer pod sends a HEAD (always confirmed by a ReadIndex) for every registration on every poll (indexer.ingest.pollIntervalMs, default 250 ms), and each one costs the stream's leader a ReadIndex round trip to its voters, even when the stream is idle. Keep registrations few, or raise the poll interval.

Removed

• Feature levels. /__ursula/feature-level has no route any more (the admin listener answers 404), the 409 with feature_not_enabled is gone, the per-group feature_level gauge is gone from /__ursula/metrics, and ursulactl cluster feature-level and cluster enable-feature are gone.
• Keyed streams and clients/pi-durable-ursula. profile=keyed-batch-v1 is an inert media-type parameter, {stream}/keyed-state answers 404, and [keyed_state], server.keyed_state_upstream and the indexer's keyed mode are gone. Pi Durable runs on clients/sqlite-vfs and clients/sqlite-ursula.
• Data-plane bucket quotas. PUT /__ursula/quota/{bucket} is gone from both listeners, and creates and appends no longer answer 429 for a bucket's stream count or retained bytes. The per-group limits were a backstop worth limit x group_count cluster-wide. Nothing in Ursula replaces them: the gateway's --quota-policy limits request rate, live reads and body size only (Security). Cap a tenant's streams or bytes in your own control plane, using /__ursula/usage. The 429 with a producer_limit body is unchanged.
• Bucket lifecycle rules in the extensions specification. Explicit bucket creation, bucket metadata, stream listing and bucket delete (Sections 1.2 to 1.5) were never implemented and are withdrawn; see Buckets. Removing a bucket stays an operator action (bucket purge), which permanently fences the bucket ID: later stream creates in it answer 404.
• Stream attributes. GET and PUT /{bucket}/{stream}/attrs are gone, a Stream-Attrs header on create is ignored, and create idempotency no longer compares attributes. Keep application metadata about a stream in your own store.
• Snapshot delete, the latest-snapshot redirect and Stream-Snapshot-Match. DELETE /{bucket}/{stream}/snapshot/{offset}, which never succeeded, and GET /{bucket}/{stream}/snapshot, which redirected (307) to the latest snapshot, now answer 405 from a node (an access-controlled ursulagw answers 404 for the DELETE, as for any unrecognized route). To read the latest snapshot, take Stream-Snapshot-Offset from HEAD and GET /{bucket}/{stream}/snapshot/{offset}. A snapshot publish carrying Stream-Snapshot-Match answers 400 rather than publishing unconditionally.
• Append Batch and Prefer: return=minimal. POST /{bucket}/{stream}/append-batch and its affinity form are gone, and Prefer: return=minimal is ignored. To send several messages in one request to a JSON stream, POST a JSON array: it is committed atomically under one producer sequence. /__ursula/metrics no longer reports the 18 raft_write_many_* series.
• Path affinity and $transaction. Three-segment stream paths (/{bucket}/{affinity}/{stream} and their subresources), POST /{bucket}/{affinity}/$transaction, the admin route /__ursula/flush-cold/{bucket}/{affinity}/{stream} and the path-affinity-v1 and group-append-transaction-v1 extension tokens are gone; such paths answer 404. Streams created under the three-segment form are not carried over. Address every stream as /{bucket}/{stream}; there is no atomic append across streams, but a JSON array POST is atomic within one stream. A stream ID may no longer start with $ (400); that prefix is reserved for future bucket-level subresources. $transaction, append-batch, attrs, bootstrap, retention and snapshot stay reserved as stream subresource names.
• JSON record coordinates. Record ordinals and everything built on them are gone: the read parameters record= (including record=now), tail_records, max_records and record_view (with the application/vnd.durable-stream-records+ndjson envelope and its SSE form) now answer 400 on catch-up, long-poll and SSE reads; Stream-Record-Match on an append answers 400; the Stream-Record-First, Stream-Record-Start and Stream-Record-Next headers, the record fields of SSE control events and the Stream-Extensions header are no longer sent; PUT …/snapshot?record= and PUT …/retention?record= are gone (405 and 404 from a node; an access-controlled ursulagw answers 404 for both); and record_coordinate_corruptions and the record gauges (record_marks, dense_record_entries, hot_records, hot_real_bytes) left /__ursula/metrics. Resume from the opaque offsets in Stream-Next-Offset and SSE streamNextOffset; append conditionally with Stream-Seq (conditional writes). JSON messages are still validated, stored one per line, and counted in usage committed_records.
• Compatibility-only configuration keys. raft.snapshot_pressure_unpurged_logs (chart value raft.snapshotPressureUnpurgedLogs) and storage.snapshot.drive_flush_concurrency had no effect and are gone. A config file that still sets either now fails at startup; the Helm chart ignores raft.snapshotPressureUnpurgedLogs if a values file still sets it.
• Bucket purge's legacy-pack pass. Purge no longer rewrites slices of packs written before buckets were erasure domains, and its response no longer carries legacy_shared_chunks_pending: every pack of this release lives under its bucket's prefix.
• Stream-Integrity-* headers. HEAD no longer returns the seven undocumented Stream-Integrity-* headers. They exposed a per-stream setsum tally that snapshots carried forward without recomputing it from stored bytes, so it could not detect cold-tier or read-path corruption, and it cost a SHA3-256 pass over every inline-appended byte on every replica. HEAD still returns Stream-Cold-Hot-Start-Offset.

Fixed

• /bootstrap silently dropped bytes after a checkpoint that retention had not followed.
• The /bootstrap multipart boundary was a hash of the response's shape, so a writer could predict it and forge part breaks inside an update. Each response now uses a fresh 128-bit random boundary.
• After a stream with small hot bytes received an append of 1 MiB or more and then flushed, the large append could become unreadable and snapshots holding the stream could not be restored.
• Cold-index pages could keep entries of rejected large appends and later serve wrong bytes; existing pages are clipped and repaired.
• Stream GC after a delete and recreate could delete objects of the new incarnation.
• Staged external payloads were deleted after ambiguous failures, even when the append might have committed.
• Several node-local structures grew without bound: the cold-index page cache's LRU, cold-read readahead readers, per-stream append counts and the TTL index.

Documentation

• Read path and HEAD are specified. The Read path section of the extensions specification now defines consistency=leader, the 8 MiB read cap with max_bytes, JSON stream reads and binary SSE, next to timeout_ms. Snapshot metadata on HEAD adds Stream-Retained-Offset and requires Stream-Snapshot-Offset and Stream-Snapshot-Digest whenever a snapshot exists. HEAD also lists Stream-Cold-Hot-Start-Offset, as a diagnostic header that may change in any release. The HEAD, read, bootstrap and snapshot-read pages list the 307 and 503 answers of leader-confirmed requests.
• Stream-Seq is documented as the compare-and-set it is. Conditional writes describes the per-stream, bytewise check, the prefix-count pattern for several writers, and publishing committed facts with producer headers. Ursula adds no If-Match tail precondition.
• Corrected statements. A creating PUT stores Stream-Seq without checking it. POST never creates a stream. ?cursor= is an echo of Stream-Cursor, not an alternative to offset, and ?stream-cursor= is not an alias for it. A catch-up read at the tail answers 200 with an empty body, not 204. SSE sends no heartbeat comments. A stale Producer-Epoch answers 403, not 409. offset=-1 answers 410 once retention has trimmed the start of the stream. The 122-byte stream-ID limit is enforced. Stream-Up-To-Date is only ever sent as true; a read response without it is partial.