Release notes

0.11.0

0.11.0 is a breaking release. It counts usage per stream version and moves the storage and protocol format to format epoch 7. A 0.11 node refuses epoch-6 Raft WAL directories and object-storage prefixes at startup and refuses 0.10 peers, and a 0.10 node refuses 0.11 peers. The backup format stays 5, so ursulactl restore of 0.11 restores a 0.10 backup. There is no in-place upgrade, no rolling upgrade and no rollback.

• If you cannot move your data yet, stay on 0.10.1.
• Deploy 0.11 fresh: empty PVCs and never-used object-storage prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.10, or a failed upgrade attempt, wrote to.
• Move your data with a backup, and keep the 0.10 server.cluster_id and raft.group_count. Stop client writes and every SQLite session, back up 0.10 with ursulactl backup-create and backup-verify of 0.10, stop the 0.10 nodes, copy their cold objects into the 0.11 cold root, skipping URSULA_FORMAT_EPOCH and snapshots/, and run ursulactl restore of 0.11, which checks the copy first. Every restored stream answers with a new Stream-Incarnation, as after any restore.
• What a restore keeps of usage. /__ursula/usage continues from the backup: each bucket's committed counters are the backup's, and its gauges come from the restored streams. Per-stream usage starts at the restore. Every restored stream version has zero counters, no last write until it writes again, and a new counted_since. Streams removed before the backup have no end record, so their usage stays only in their buckets' counters. A metering consumer starts from cursor start in every group once ursulactl restore has finished.
• helm upgrade from 0.10 stops at the first pod. The new pod finds the 0.10 data on its volume and exits at startup, so no second voter is replaced and the refused pod writes nothing. Recover with helm rollback.
• Keep the peer lists of 0.10 and 0.11 nodes apart. A node that exchanges a Raft RPC with a node of the other release answers 503 with format_epoch_mismatch on /__ursula/ready until it restarts.
• raft.deleted_data_max_age has a higher floor. It must also be long enough for an end record to leave within it (see Deleted data below). With the default install timeout of 120 seconds, a value set below 980 seconds with an S3 snapshot store, or 964 seconds inline, refuses to start with DeletedDataMaxAgeTooShort, where 0.10 accepted 960. The default of one hour is unchanged.
• Upgrade the SQLite extension with the server. Ursula-Protocol now carries the server's release line, 11, where 0.8 to 0.10 sent 2. The 0.11 extension refuses a server of another line with protocol_mismatch at open and at import, and an answer without Ursula-Protocol and Ursula-Cluster-Id with no_server_identity. Both replace server_too_old. The 0.8 to 0.10 extensions refuse a 0.11 server at once with server_too_old. A running session checks the line only at its next HEAD, and a running 0.10 session checks only the cluster id, so neither notices the move by itself. Stop every SQLite session before you move a cluster to another release line.

See Upgrading to 0.11.

Formats

• Format epoch 7. The Raft WAL journals and state files, the meta journal, the WAL directory and object-storage markers, group snapshots and the Raft gRPC protocol carry format epoch 7. A node refuses data and peers of another epoch.
• Per-stream usage in group snapshots. A group snapshot holds each live stream's usage in its stream frame, the group's end records as frames of their own, and the group's usage sequence, its fold watermark, each bucket's retired usage and its genesis history in the header. The per-stream append count is one of the usage counters, so its snapshot frame is gone. An install requires every bucket's counters to equal its retired usage plus its live stream versions plus its end records, so a replica that installs a snapshot keeps every stream version's usage and every end record.
• Backup format 5. Unchanged: a backup holds each group's stream snapshot with its buckets' counters and no per-stream usage, and a 0.10 backup restores on 0.11. A restore refuses a backup that names one bucket's counters twice, or holds a stream but no history event, which no release since 0.8 writes.

Admin API

• Per-stream usage export. POST /__ursula/usage/streams lists, per Raft group, the usage of each stream version that changed after a cursor: its counters, appends, last write and counters at the group's latest history event, its retained and snapshot bytes while it lives, and how and when it ended. It relists a group with each bucket's retired usage and totals when the cursor needs it or the caller asks. It is served on the admin listener only, from local applied state, and a group that cannot be read answers unavailable without failing the answer. The answer's end_record_retention_ms is half of the answering node's raft.deleted_data_max_age, how long at least an end record stays listed. See Per-stream usage.
• History ids are hex. history in the answer of POST /__ursula/backup/group/{group}/import is 16 lowercase hex digits, as in the usage export, where it was a JSON number. ursulactl restore prints it the same way.
• Purge report. DELETE /__ursula/purge/{bucket} reports in groups each group's entry that purged the bucket: its commit_index and the history it applied in. A metering consumer holds the purge complete once it read each group in that history at or past that index.
• Usage report. /__ursula/usage keeps version 2 and its values. Each stream version counts its committed bytes, records, write units and appends in replicated state, where the bucket counter counted them, and a bucket's counters are its retired usage plus its live stream versions plus its end records, derived when usage is read. A removed stream's usage stays in its end record until the end record is folded into its bucket's retired usage. A restore retires each bucket's counters from the backup and starts every restored stream version at zero.
• Gauges. group_state_gauges in /__ursula/metrics gains end_records, end_record_bytes and oldest_end_record_at_ms. /__ursula/metrics gains usage_end_records_retired, the end records the group sweep folded, and usage_end_records_folded_early, the end records folded at once because their group held 65,536. Every replica counts the folds it applies, a single node included.

Gateway

• A request gets time for its size (#891). The gateway waits for a node's answer for --response-header-timeout (gateway.responseHeaderTimeoutSeconds) plus the time the node may take to store or copy the request's bytes before it answers, at 8 MiB per second. A request counts its body, which a node with a cold store writes to object storage whole when it is large, and a snapshot parts commit counts its parts at the 32 MiB request limit each, up to the 1 GiB snapshot limit. A 32 MiB body waits 4 s longer, and a commit of 32 parts or more 128 s longer. A large append, create, snapshot or commit no longer answers 504 while the node still works on it, so the base timeout can stay just above the leader lease. An object store slower than 8 MiB per second still needs a higher timeout. A node that stops answering is still cut off at the end of the longer wait, and the probes eject it within 1.75 s as before.
• A gateway drain can cut a large commit's answer. A commit waits up to the base plus 128 s, longer than the chart's 45 s gateway.gracefulShutdownTimeoutSeconds. A commit in flight when the drain ends loses its answer, and repeating it gets the answer from its key. Every other write waits at most the base plus 4 s.

Server metadata

• Ursula-Protocol is the server's release line. It is one unsigned decimal, major × 1000 + minor of the server's version: 11 for every 0.11.x. Patches of one line interoperate, and lines do not. See Server metadata.

SQLite VFS

• The VFS waits for an answer as the gateway does. Its 15 s wait for an answer's head grows by the same 8 MiB per second allowance, for its body or for the parts a commit copies, counted as the gateway counts them. It waited 16 MiB per second for a commit's actual bytes, which gave up before a gateway in front of the node for a commit of more than five full parts.
• protocol_mismatch and no_server_identity replace server_too_old as open and import kinds, both with action fix. A bound database whose HEAD meets another line or no identity blocks with the same name as replication.condition and retries every 30 s. It sends that HEAD to confirm an answer such as a 412 or a 404, not on every write. A snapshot publish checks the HEAD the same way.

Deleted data

• A bucket purge removes its streams in stream-id order. The replicas of a group removed a purged bucket's streams in the order of their own in-memory hash map, and each removal numbers the cold-object cleanup entries it queues. Replicas could therefore give one number to different streams. The leader's acknowledgement of reclaimed objects then drained other entries on a follower, so after a leader change cold GC could skip some of a purged stream's objects. Those objects stayed in object storage until a purge request reported cold_gc_complete=true, whose erase of the {bucket}/ prefix removes them, so a purge that reported cold_gc_complete=true left nothing behind. Every replica now removes the streams by bucket and stream id and numbers the entries alike, and cold GC reclaims every purged stream's objects after a leader change. 0.10.1 has the same issue and does not carry the fix: on 0.10, repeat a purge until it reports cold_gc_complete=true (Tenant offboarding).
• End records. A deleted, expired or purged stream leaves an end record in its group's state with its id, how and when it ended, and its final usage. A bucket purge stamps one time for every group, and every purged stream ends at it. Each group's leader folds an end record into its bucket's counters half of raft.deleted_data_max_age after the removal, by its own monotonic clock, in the background pass that also removes expired streams, and the fold counts as a removal for the snapshot driver. A group holds at most 65,536 end records, and a removal past that folds the oldest one into its bucket's counters first. At the defaults a removed stream's id leaves a replica's snapshot about 49 minutes after the removal. A node that restarts or installs a snapshot dates the end records it holds then by the removal time each carries, so they keep their age when the nodes' clocks agree. Keep the nodes' clocks in sync.
• The background pass runs more often below 16 minutes. It runs every minute, or every sixteenth of raft.deleted_data_max_age if that is shorter, where 0.10 used a quarter. Its Sweep Raft entries replace ExpireStreams: each removes up to 256 expired streams and then folds up to 256 end records that are due. ttl_expiry_sweeps counts the entries that only fold end records too, and ttl_expiry_sweep_due_groups counts a group left with due end records.

Rust library APIs

• ursula_headers::MAX_SNAPSHOT_BYTES is new: the 1 GiB snapshot limit a node with a cold store advertises in Ursula-Max-Snapshot-Bytes. ursula_stream::MAX_COLD_SNAPSHOT_BYTES and its ursula_runtime re-export are removed. Use ursula_headers::MAX_SNAPSHOT_BYTES.
• ursula_headers::answer_allowance, STORE_FLOOR_BYTES_PER_SECOND and PartCount::copy_bytes are new: the time beyond a base wait a node's answer may take for the bytes it stores or copies first, which the gateway and the VFS both add.
• Per-stream usage. ursula_stream gains CommittedUsage, VersionCounters, VersionUsage and LastWrite, and StreamStateMachine gains version_usage, usage_seq, usage_genesis and with_genesis and loses stream_append_count and add_stream_append_count. A StreamStateMachine refuses a create before its genesis history event with HistoryPending, as both engines already did. StreamSnapshotError gains DuplicateBucketUsage and StreamWithoutHistory. ursula_runtime::AppendResponse loses stream_append_count: a version's applied appends are counted by apply in its usage.
• Usage in group snapshots. ursula_stream gains UsageSnapshot, BucketRetired, UsageSnapshotError and UsageCounter, StreamSnapshotError gains Usage, CommittedUsage gains first_difference, and StreamStateMachine gains restore_group and usage_snapshot and loses set_stream_append_count and stream_append_counts. GroupStateGauges gains end_records, end_record_bytes and oldest_end_record_at_ms. ursula_runtime::GroupSnapshot replaces stream_append_counts with usage and has no serde derive, StreamAppendCount is gone, SnapshotStoreError gains UsageFieldZero, StreamEndUnknown and UsageMismatch, and InMemoryGroupEngine gains set_metrics, through which it counts the end records the entries it applies fold. ursula_proto gains the StreamUsageV1, EndRecordV1, BucketRetiredV1, VersionCountersV1 and CommittedUsageV1 messages and the StreamEndV1 enum, and loses StreamAppendCountV1.
• End records. ursula_stream gains EndRecord, StreamEnd and MAX_END_RECORDS, StreamStateMachine gains end_records and usage_retired_through, and RemovedState gains end_records and end_records_early. StreamCommand::PurgeBucket gains now_ms. ursula_runtime gains PurgeBucketRequest, which GroupEngine::purge_bucket takes in place of the bucket id, and ShardRuntime::purge_bucket_all_groups and RaftGroupHandleRegistry::purge_bucket_through_leader take it by reference.
• The group sweep. StreamCommand::Sweep { now_ms, limit, retire_through } and StreamResponse::Swept { expired, retired } replace ExpireStreams and StreamsExpired, and StreamStateMachine gains sweep_due and end_records_ended_by. In ursula_runtime, SweepResponse, GroupWriteResponse::Sweep and ResponseKind::Sweep replace their ExpireStreams forms, TidyStreamsRequest gains deleted_data_max_age, TidyStreamsResponse gains end_records_retired, expiry_sweep::RetirementClock is new, and InMemoryGroupEngine::sweep_due and sample_retirement replace expiry_due.
• The bound on deleted data. UrsulaConfig::min_deleted_data_max_age and effective_deleted_data_max_age replace the RaftConfig methods, since the floor depends on storage.snapshot.drive_interval, and UrsulaConfig::end_record_retention, RaftConfig::end_record_retention, RaftConfig::sweep_interval, RaftSnapshotConfig::effective_drive_interval and ursula_config::end_record_residence_floor are new.
• Per-stream usage export. ursula_stream gains StreamUsageRead, UsageCursor, UsagePosition, UsageLogId, GroupUsageView, UsageListing, RelistReason, BucketUsageRow, UsagePage, UsageEntry and Lifecycle, StreamStateMachine gains stream_usage, and StreamResponse::BucketPurged gains history. ursula_runtime::GroupEngine gains the required stream_usage, which returns the new ursula_runtime::GroupStreamUsageFuture, ursula_runtime::ShardRuntime gains stream_usage and stream_usage_groups, which answers the new ursula_runtime::StreamUsageGroups, and ursula_runtime::InMemoryGroupEngine gains stream_usage_view. ursula_runtime::PurgeBucketResponse gains history, and ursula_runtime::PurgeBucketReport gains purges, one new ursula_runtime::GroupPurge per group. ursula_proto::admin gains the export's request and answer types, UsageNext among them, HistoryId, MAX_STREAM_USAGE_GROUP_ENTRIES and MAX_STREAM_USAGE_ANSWER_ITEMS, ursula_proto::admin::BucketPurgeReport gains groups, and ursula_proto::admin::BackupImportResponse::history and ursula_ctl::backup::RestoredGroup::history are a HistoryId. ursula::HttpState gains with_end_record_retention.
• ursula_headers::PROTOCOL_LINE replaces PROTOCOL_VERSION. ServerIdentity::current sends it, and ReleaseLine shows a line as its releases (0.11, or 0.8 to 0.10 for 2).

0.10.1

0.10.1 is a patch of 0.10 and interoperates with 0.10.0: format epoch 6, backup format 5 and every storage, wire and protocol format are unchanged, and the usage report only gains an optional field. 0.10.0 and 0.10.1 nodes run in one cluster during a rolling upgrade, and 0.10.0 and 0.10.1 owners and readers of one stream work side by side. Upgrade in place with an ordinary rolling restart (helm upgrade), with no fresh install or data move. Rolling back to 0.10.0 is safe.

The DST Nightly gate (deterministic simulation) was waived for this release.

Admin API

• /__ursula/usage gains snapshot_bytes. Each bucket reports the summed body length of its live streams' latest snapshots, as uploaded: the image each stream keeps, derived from applied state when usage is read (Usage export). A 0.10.1 node reports it for every bucket, zero included. A 0.10.0 node sends reports without the field, and a missing field means unknown, not zero: it decodes as None, and a sum with an unknown part is None, so during a rolling upgrade a collector that keeps a 0.10.0 report for a group gets an unknown total for that group's buckets and must not bill it as 0. An unknown value is left out of the JSON, never written as null. No admin usage type rejects unknown fields, so a collector built on the 0.10.0 ursula_proto reads a 0.10.1 report. No wire, Raft or on-disk format changes.

@tonbo/sqlite-ursula

• gate(db, work, { signal }) applies the one rule for you. It runs work and settles with its outcome, a result or an error, only once a synced() call made after work settled resolved on every database given. A returned Response comes back with the same status and headers, and its body passes each chunk on only after a synced() call made after the chunk was read, one call per batch of chunks. An UrsulaError thrown by work is rethrown at once, a reset during the wait rejects with the UrsulaResetError and drops the outcome, and signal bounds the wait for the outcome but not a body that has started. See Gating outputs. The helper only calls synced(): no server, extension, wire or format change, and it works with any 0.10 extension and server.

Rust library APIs

• Snapshot bytes in usage. StreamStateMachine::snapshot_bytes_by_bucket and InMemoryGroupEngine::snapshot_bytes_by_bucket are new, ursula_runtime::GroupBucketUsage gains snapshot_bytes, and ursula_proto::admin::BucketUsageReport gains snapshot_bytes: Option<u64> (None is unknown, and BucketUsageReport::add keeps it known only when both sides are), so a struct literal of either needs the field. A crate that depends on ursula-proto = "0.10" or ursula-runtime = "0.10" gets this change on cargo update.

Known issues

The known issues of 0.10.0 are unchanged.

0.10.0

0.10.0 is a breaking release. A meta group replaces static membership: it records every node and places every data group, so groups move between nodes and nodes join and leave a running cluster. 0.10 moves the storage and protocol format to format epoch 6. A 0.10 node refuses epoch-5 Raft WAL directories and object-storage prefixes at startup and refuses 0.9 peers, and a 0.9 node refuses 0.10 peers. The backup format stays 5, so ursulactl restore of 0.10 restores a 0.9 backup. There is no in-place upgrade, no rolling upgrade and no rollback.

• If you cannot move your data yet, stay on 0.9.0.
• Deploy 0.10 fresh: empty PVCs and never-used object-storage prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.9, or a failed upgrade attempt, wrote to.
• Move your data with a backup, and keep the 0.9 server.cluster_id. Install 0.10 with the 0.9 cluster's server.cluster_id and raft.group_count. Stop client writes to 0.9, back it up with ursulactl backup-create and backup-verify of 0.9, stop the 0.9 nodes, copy their cold objects into the 0.10 cold root, skipping URSULA_FORMAT_EPOCH and snapshots/, and run ursulactl restore of 0.10, which checks the copy first. Every restored stream answers with a new Stream-Incarnation, as after any restore.
• helm upgrade from 0.9 stops at the first pod. The new pod finds the 0.9 data on its volume and exits at startup, so no second voter is replaced and the refused pod writes nothing. Recover with helm rollback. A values file that still sets a removed value fails validation first.
• Keep the peer lists of 0.9 and 0.10 nodes apart. A node that exchanges a Raft RPC with a node of the other release answers 503 with format_epoch_mismatch on /__ursula/ready until it restarts.
• Recover the meta group first, then the data groups. ursulactl recover-group --target meta, then --target G for each data group, whose plan reads its floor from a confirmed meta view.
• A lost quorum of disks is lost data, for the meta group as for a data group. No recovery flow brings such a group back. Restore a backup into a fresh cluster.

See Upgrading to 0.10.

Formats

• Format epoch 6. The Raft WAL journals and state files, the meta journal, the WAL directory and object-storage markers, group snapshots and the Raft gRPC protocol carry format epoch 6. A node refuses data and peers of another epoch.
• The run state records each journal class. run-state.bin holds one record for the data journals and one for the meta journal, each with its fsync policy and recovery epoch, and the device the WAL root is mounted from. Every journal record, core metadata entry and group snapshot record carries a log generation, so a retired group's log leaves nothing a later life of the group restores.
• A WAL root is bound to its node. topology.bin adds the node ID and a root ID drawn when the root is created. A root opens only under the raft.node_id that created it, and another ID refuses startup (OtherNode). The new identity.bin counts the root's runs and records whether the meta group bound the root. A root that holds prior state without it refuses startup (MissingIdentity). The new lease.probe holds the disk lease's probe.
• The meta group's journal. raft-log/meta/ holds the meta group's journal and its snapshot record group-0.snapshot.json, whose snapshot is always inline. It always runs fsync = always.
• Raft payloads name their plane. A Raft entry's payload carries the meta group's commands beside the data groups', and a data entry keeps its epoch-5 bytes. Every Raft request names its plane and target node, a node refuses a request meant for another node, and barrier answers echo the group they answer for.
• GroupEngineError names the plane. Every GroupInfraError field raft_group_id is now group, a data group's ID or the meta group, at the same position, so a refusal of the meta group no longer reads as one of data group 0. The wire form is frozen within 0.10.
• Backup format 5. Unchanged: a backup holds each group's stream snapshot and no membership or placement.

Removed and replaced

Removed or changed	Kind	Replacement
raft.groups ([[raft.groups]] with raft_group_id and voters)	config, refused as an unknown key	the meta group places every group at its bootstrap: group g on min(raft.placement.replication_factor, N) consecutive raft.peers nodes in ascending ID from index g mod N
raft.init_membership, raft.init_membership_per_group	config, refused as unknown keys	none: the first node of a group's seed voters initializes it once every seed voter reports it empty and no leader
a node with raft.wal.path and an empty raft.peers	config	refused (NodeIdNotInPeers): every node with raft.wal.path, a single node included, lists itself in [[raft.peers]] and sets server.cluster_id
raft.group_count unset on a node whose raft.peers lists another node	config	refused (GroupCountRequired): set it, the same on every node, at most 4,096
raft.wal.fsync defaulting to never	config	unset, always for groups of one or two voters, never otherwise
ursulactl startup-admit	command	none: a node starts unattended
ursulactl pin-incarnations	command	each command's own incarnation check, which stops a command when a node restarts under it, and ursulactl wait between steps
ursulactl activate-maintenance-fence, retire-maintenance-fence	commands	none: placement writes go through the meta group
ursulactl reservation-propose, reservation-cell, reservation-source, reservation-read, reservation-bootstrap, reservation-acknowledge, and reservation-request with reserve-host-recovery, admit-host-termination, record-host-termination, admit-replacement-termination, record-replacement-termination, restage-host-replacement, complete-host-replacement, admit-fenced-pod-retirement, bind-host-replacement, publish-host-inventory, reserve, takeover, admit-pod-deletion, complete-pod-replacement and bind-pod-replacement	commands	a lost disk: start the node under its ID on an empty WAL and, once the old disk is revoked, ursulactl node replace-disk. A node that leaves: ursulactl node decommission. Never start a node on a copy of another node's disk, and never let two hosts mount one volume
ursulactl recover-group --group G	flag, refused as unknown	--target G for a data group, --target meta for the meta group
--expected-groups of wait-ready, verify-quorum and verify-survivors	flags	none: the groups and their target voters come from the meta leader's placement view
verify-survivors output excluded_voter_id, configured_voter_ids, surviving_voter_ids	ursulactl JSON	excluded_node_id, surviving_node_ids. Any manifest node may be excluded
verify-quorum output schema 3	ursulactl JSON	schema 4, without maintenance_executor_certified, maintenance_executor_retired_certified and maintenance_fence
manifest fields expected_process_incarnation, expected_maintenance_fence and the root maintenance_fence	ursulactl manifest	none: ignored, like any unknown field
governance.leadership_balance.peer_timeout	config, refused as an unknown key	none: every question a node asks its peers, for their leadership-shed state, recovery gates, leader counts and WAL-disk and egress states, waits one minimum election timeout (1.5 s), as a Raft append call does
POST /__ursula/raft/{group}/membership?voters=	admin route, answers 404	POST /__ursula/placement/groups/{group}/move, or ursulactl placement move
POST /__ursula/maintenance/fence/activate, /retire	admin routes	none
/__ursula/raft/{group}/...	admin routes	/__ursula/raft/{target}/..., a group ID or meta
x-ursula-maintenance-fence	header	none: an admin mutation that carries it answers 400
maintenance_fence, maintenance_fence_uncertain in /__ursula/metrics	admin JSON	none
raft_maintenance schema version 1	admin JSON	version 2 (Admin API and readiness)
/__ursula/usage version 1, with node-summed buckets	admin JSON	version 2, with one entry per hosted group under groups
a recovery plan without floor	admin JSON, answers 422	floor: "meta", "seeded" or {"certified": ...}
URSULA_STARTUP_RESERVATION, URSULA_STARTUP_PROVIDER, URSULA_STARTUP_STATEFULSET, URSULA_STARTUP_NAMESPACE, URSULA_STARTUP_POD_NAME, URSULA_STARTUP_POD_UID, URSULA_START_MAINTENANCE_DRAINED	environment	none: ignored. To keep a started node drained, run ursulactl drain once it is up
server.updateStrategy, server.podManagementPolicy, server.probes.readiness.enabled, server.onDeleteMigration.*, server.startupOwnership.*, server.gracefulRollout.*, raft.initMembership, raft.initMembershipPerGroup	chart values, refused by the schema	the StatefulSet always updates by RollingUpdate under Parallel, with a readiness probe on /__ursula/ready and the leadership handoff on SIGTERM
server.replicaCount 2, and every pod a voter of every group	chart value	1, or 3 and more. Each group has at most three voters, so 5 replicas keep three copies of a group. Lower the count only after decommissioning every dropped ID, and raise it again once their removals completed
server.podDisruptionBudget.maxUnavailable above 1	chart value	at most 1
raft.walFsync defaulting to never	chart value	"", which renders no raft.wal.fsync
charts/ursula/files/graceful-rollout.sh, maintenance-rollout.sh, graceful-rollout_test.sh, templates/startup-ownership.yaml, graceful-rollout.yaml, ondelete-migration.yaml, scripts/maintenance_rollout_test.py, scripts/helm_ondelete_upgrade_test.sh	scripts and templates	helm upgrade under RollingUpdate. Outside an upgrade, drain, restart, wait and undrain. For a listener move, every server pod deleted at once
the <StatefulSet>-rollout-state and <StatefulSet>-maintenance ConfigMaps	cluster state	none: delete them

Placement and the meta group

• Every node runs the meta group. Every node that runs Raft, a single node included, runs a replica of the meta group, which records the cluster's bootstrap, every node with its lifecycle and WAL root, and each data group's placement. Its voters are raft.meta_voters, by default the smallest min(3, N) node IDs, read at genesis only. Every other node is a meta learner. A fresh cluster forms once every raft.peers node is up and claims the same genesis values. See Placement and the meta group.
• Placement decides what a node hosts. A node opens the groups its journals hold, in parallel, and creates a group only from a placement view a meta read barrier confirmed. A failed open no longer stops the start: the group stays closed under its open back-off and opens again. With more nodes than the replication factor, a node hosts only the groups placed on it, where 0.9 ran every group on every peer. A 0.9 cluster of five nodes therefore runs three voters per group in 0.10, unless it sets raft.placement.replication_factor = 5. A node that raft.peers gains after genesis hosts nothing until groups move onto it.
• Groups move. POST /__ursula/placement/groups/{group}/move and ursulactl placement move move a live group to other voters, as many as it has, under a move limit (raft.placement.max_concurrent_moves, 4 by default, ursulactl placement set-move-limit). The group's leader adds the new voters as learners, promotes each once it caught up and its recovery gate is open, replaces one voter at a time, and reports the new voters to the meta group. A move of a moving group retargets it, and a move back to its source cancels it.
• A learner catches up in rounds. A move's leader counts a learner caught up once it matched, within the group's leader lease (3 s), the commit index as it stood when its catch-up round began, and while its next round has not run longer, the rule of the Raft thesis. A move therefore finishes under steady writes over a long round trip, where the commit index of the moment stays ahead of the learner, and a learner that cannot keep up is still never promoted.
• A dropped replica is torn down. A node tears down a replica that a confirmed view gives no role once the group answered twice that the node is no member. It retires the replica's log and deletes its own snapshot objects of the group. Cold objects stay.
• Nodes are cordoned and decommissioned. ursulactl node cordon, uncordon and decommission change a node's lifecycle. The meta leader moves a decommissioning node's groups to eligible active nodes and removes the node once it has no role. A removal cannot be undone for the node's process and disk, and a meta voter is never decommissioned. With an S3 snapshot store, the meta leader deletes a removed node's snapshot objects of a group once the group's leader confirms that the node is no member. No operator deletes <prefix>/node-<id>/. A removed node restarted with its disk asks each group about every replica it holds, also while an open keeps failing, and keeps closed each one its group no longer counts. No node deletes a snapshot object before the meta group answered its registration in that run, and a removed node deletes none.
• A removed ID is used again. The meta leader completes a removal once every group's leader confirmed the node no member, the meta group lists it in neither membership, its snapshot objects are gone and no process of its root answers, and marks the ID vacant. An empty disk started under a vacant ID registers under a new root and joins as a new node, through the same predecessor wait, vote floor and recovery gate a wiped node rejoins with. The old disk never binds again and is superseded. A disk started under the ID before its removal completed waits with removal_pending and binds with no restart. leader_report.removal_pending in GET /__ursula/placement on the meta leader lists what each removal waits for. Stop the old process and wipe or destroy the old disk before a new disk starts under its ID. A meta plan that cuts the meta log below a removal's completion or a new disk's binding refuses that disk, which must be wiped again.
• NotHosted redirects. A node that hosts no replica of a group, or whose replica knows no leader and does not count as a voter, answers 307 to a node of the group's placement, a different one on successive answers, or to another raft.peers node while it has no bootstrapped view. A replica that counts as a voter and knows no leader answers 503. The gateway takes the named host as a hint and goes on to its next upstream when that host is ejected or does not connect. The backup and cold-flush admin routes on a node that does not host the group answer 503 with NO_LEADER, and ursulactl follows no redirect.
• Local reads need a voter. A local read on a replica that does not count as a voter, an incoming learner or a dropped voter, is answered as a leader-only read.
• One node purges for the cluster. DELETE /__ursula/purge/{bucket} on any node purges every group: the ones the node hosts in place, every other one through its leader. Node-wide passes such as the cold flush and cold GC skip groups the node does not host.
• The leadership balancer settles. The fair share is every data group of the view over the nodes that may campaign, are serving-ready and hold a role, and every handoff (the balancer, the shutdown handoff, the shedding gates, ursulactl drain) goes only to a replica whose role is voter or incoming.
• A restarted node takes a leadership only once it can. A replica takes a leadership from a leader only while its node takes transfers, its recovery gate lets it campaign and it acknowledged a leader of its group since it started. Its node answers that rule for each replica (takes_leadership), the leader reads the answer before it hands a leadership over, and the replica checks the same rule when the transfer arrives. The leadership balancer also waits until the node is serving-ready. In earlier 0.10 builds the balancer handed leaderships to a node seconds after it restarted, the node refused them, and each such group served no write for 3 to 6 seconds.
• A leadership transfer that its target refuses costs one election. A leader whose target refuses a transfer steps down at once and campaigns with a pre-vote round, which it repeats every minimum election timeout (1.5 s) until a leader is known, whether or not the node sheds. A voter that got the transfer grants the round at once, and the target that refused it grants it once its lease of the leader has lapsed, at most 3 s (election_timeout_max) after the leader's last heartbeat. A leader whose target took the transfer but did not win within one minimum election timeout does the same. A leader whose transfer call failed before it left the node, because the target's name did not resolve or its connect failed, steps down and campaigns at once too: the target cannot have received it. Vote rounds and transfers to a peer go over the connection its append stream uses, since a failure streak no longer replaces a channel an append stream lives on, so a leader whose follower answers its appends reaches it with a transfer without dialing again. In earlier 0.10 builds a transfer to a node that had restarted a moment ago could dial a new connection, fail its name lookup, and leave the group with no leader for one lease (3 s). A target that already holds a newer vote, or whose answer does not arrive, leaves a newer election one lease (3 s) from the transfer before the leader campaigns, and a pre-vote never unseats a leader a quorum follows. A refused transfer used to cost two maximum election timeouts (6 s), while the group served no write.
• A meta outage leaves the layout static. While the meta group has no quorum, no group changes a voter, promotes a learner or moves a leadership by choice: a group leader's reconcile shell holds a meta confirmation for 3 seconds, and the leadership balancer confirms the view by a meta read barrier before its handoffs. Raft failover, the disk lease, the egress gate's shed for a node's own fault and a stopping node's handoff go on. In earlier 0.10 builds a confirmation held until the leadership changed, and the balancer moved leaderships from a view no barrier confirmed.
• A damaged meta replica does not stop the node. A node whose meta journal or replica is damaged serves its data groups, reports meta_damaged, and marks the new placement-unavailable leadership shed: it campaigns but takes no leadership transfer.

Disks and processes

• A wiped or replaced disk needs a release. A node on a new disk under a live ID is refused by the meta group (root_refused) and takes part in nothing until ursulactl node replace-disk --node N --expect-root R releases its old root through another node's admin plane. The node then binds with no restart, waits 12 seconds and rejoins. 0.9 let a wiped voter rejoin with no operator step.
• A superseded process fences itself. A process whose root was released, or whose ID the meta group binds to another root, or that a peer saw a newer run of, takes part in no Raft traffic, writes no object, answers clients 503, reports superseded and records no clean end. It does not exit.
• The disk lease. A node syncs a probe of its WAL disk every second. With no synced probe for raft.wal.lease (5s), it stops all Raft traffic, leads no group, writes no object, answers clients 503 and reports disk_lease_lost, until a probe syncs again.
• A possibly live predecessor is waited out. A start after a host crash, a reboot, a remount or a lost run state waits 12 seconds before it votes, campaigns or writes a snapshot object (predecessor_wait). A run that did not end cleanly and is read on the same boot from another device counts as a remount and opens as after a host crash.
• Raft RPCs name the processes. Every Raft request names the sender's process and, when it knows it, the target's (ursula-incarnation, ursula-target-incarnation), and every answer names the responder's (ursula-node-id, ursula-incarnation). A receiver refuses a request for another process of it, from a stale or unbound sender, a duplicate or one that names no sender, and a sender reads every such refusal as a lost RPC.
• x-ursula-process-incarnation keeps its 32 hex digits, now the WAL root ID, the run counter and a nonce. It still changes on every restart.
• A clean shutdown keeps the WAL lock until the process exits, and the node fence closes before clean is recorded. Shutdown stops the data groups, then the meta group, then the WAL.
• A graceful stop answers a request on an idle connection. Once a node stops accepting connections, it marks every HTTP/1 answer it starts from then on Connection: close and keeps a connection that was idle when the stop began open for up to 1 s, so a request a client sends on it within that second is answered. The gateway reuses an idle upstream connection for at most 0.5 s. 0.9 closed idle connections at once, and a write the gateway sent on one at that moment failed with 502. A write can still fail that way on a connection the node had not accepted when it stopped accepting, or on one whose last answer started before the stop. A stop takes up to 1 s longer while a client or a peer keeps a connection open.

Replication

• A slow link slows replication instead of stopping it. Each Append call between nodes waits up to the minimum election timeout (1.5 s) for its answer. 0.9 waited one heartbeat interval (250 ms), so once a round trip between nodes took longer than that, every append failed, the nodes rebuilt their channels to each other over and over, and no write committed. An Append that got no answer in time no longer counts toward rebuilding the channel to its peer. Writes over a slow link now take a few round trips.
• Elections start with a pre-vote. A voter whose election timeout passes first asks its peers whether they would vote for it, and raises its term only once a quorum would. A peer that still hears its leader would not. A restarted voter that campaigns before its leader's first heartbeat reaches it, or a voter cut off from its peers, therefore leaves the group's term alone. In 0.9 each such election raised the term, and the leader stepped down at its next append to that voter, so a clean restart could cost each group the node follows another election and a few seconds of 503 NO_LEADER. A leadership transfer still elects its target at once. A follower whose leader went silent keeps its vote for that leader, so a follower now names a leader only while that leader's appends reach it, within the last 3 s. Otherwise its current_leader in /__ursula/metrics is null, its maintenance report lists leader_unknown for the group, and it answers the group's writes 503 NO_LEADER and forwards no leader-only read, where it would have redirected them to a leader it no longer hears. A restarted follower names its leader once the first heartbeat reaches it. A node counts as knowing a meta leader, for meta_unavailable and for registering again, only while it leads with a quorum that answered within 3 s or hears its leader so. Pre-vote is a new Raft RPC, PreVote, on the cluster listener for every group, data and meta. /__ursula/metrics gains raft_grpc_pre_vote_requests, raft_grpc_pre_vote_request_bytes and raft_grpc_pre_vote_response_bytes.
• Heartbeats go out every 600 ms, up from 250 ms. A leader waits one heartbeat interval at most for each heartbeat's answer, and an idle group's leader takes writes only while those answers renew its lease. Idle groups, ReadIndex reads and the meta leader's view of live nodes now keep working over round trips up to 600 ms. The election timeouts stay 1.5 to 3 s. Raft ticks every 121 ms per group instead of 50 ms.

Leaders and the gateway

• A leader that cannot reach a quorum answers promptly. OpenRaft keeps a leader that no quorum acknowledged within its lease (3 s, its election_timeout_max) in its leader state until a message with a higher term reaches it, which a node cut off from its peers never gets. Such a node now leads nothing: it answers a new write 503 NO_LEADER before the body, and a write it proposed earlier, which may still commit, 503 OUTCOME_UNKNOWN once its lease ends. 0.9 held a write it had proposed, and every later write of the group queued behind it, until the cut healed. Cold flush, compaction and the other leader-gated jobs skip it too.
• The gateway moves on from an upstream that does not answer. --response-header-timeout (gateway.responseHeaderTimeoutSeconds) defaults to 5 s, from 30 s. A read or a producer append (a POST to a stream with a producer key) that gets no response headers within it goes on, once, to an upstream it has not tried, and so does one waiting on an upstream when it is ejected (see the next item), at once, except a producer append on a node that stopped accepting. A producer append is answered from there only with a 2xx receipt, and otherwise, like any other write at its timeout, with 504 UPSTREAM_TIMEOUT: the first node may still apply it. The repeat never goes back to the node that answered nothing, though a redirect names it. A missing answer ejects nothing: a live node can hold one request longer than the timeout, such as a write on a leader just elected over a slow link, while it answers every probe. A read or a producer append sent to a stopped node (SIGSTOP, a frozen host) before its ejection costs about 1.75 s, and any other write 5 s, where 0.9 held both for 30 s.
• Every gateway probes every node. A gateway sends GET /__ursula/live to each upstream every 250 ms, whether or not requests go there. A node answers the new route 204 on its client plane without reading any state. A probe that fails or gets no response headers within 1.5 s ejects the node, and its next answer readmits it, so every gateway leaves a stopped node out of selection within 1.75 s, and a write that a client retries on another gateway no longer waits for it again. 0.9 ejected a node only for a request to it that failed, on that gateway alone, and kept it out for 10 to 60 s, after which a request probed it. Those windows are gone. Besides the probes, only a request that fails at the transport ejects a node. An ejection has a cause. A node that refuses or closes a connection stopped accepting, as a stopping node does once its listener closes, and still answers the requests it accepted: a producer append in flight there waits for its answer, which settles it, and a read goes on elsewhere at once, as a long-poll may outlast the drain. A node that a probe gets no answer from within 1.5 s, or whose connect or send times out, stopped answering, and its reads and producer appends in flight go on elsewhere at once. The ejection is logged at warn with its cause and the readmission at debug. A uniform delay ejects nothing once a connection is pooled: under a 500 ms round trip a probe takes 500 ms. A probe past 1.5 s keeps waiting for its answer, up to the response header timeout, and readmits the node when it comes, so a new connection that takes longer than that ejects the node only until it completes. The gateway no longer follows a leader redirect to an ejected node: the request searches the nodes it has not tried, and with none left answers 503 NO_LEADER with Retry-After. When every upstream is ejected, none is. Upgrading: Gateway::new and Gateway::with_access_control start the probes, so they must be called inside a Tokio runtime, and a gateway probes until its last clone is dropped.
• A stopping voter leads nothing while its listeners drain. Every 50 ms for its five seconds, the SIGTERM handoff asks the peers again which may take a transfer, waiting up to 1.5 s for each answer as every question between nodes now does, and submits each leadership no peer could take yet, including one the node gained during the handoff, only to a peer that answered that its replica of the group may take the leadership. It logs the peers' answers, each group no peer can take with the handoff check's refusals, and what is left when the five seconds end. It then steps down from what is left without campaigning, within two seconds. The other voters elect once their own view of its lease has lapsed, about 3 s (election_timeout_max) after its last heartbeat, since a pre-vote is refused while a voter's last leader may still hold its lease. 0.9 skipped a refused target without a word, and kept the leaderships it could not hand off through the ten-second listener drain, while clients that followed its peers' redirects found its listener closed.
• The cluster-egress gate sheds only for a fault of its own node. A node whose egress probe (governance.cluster_probe) fails to reach a quorum sheds its leaderships, and stops campaigning, only while a voter of a group its probe found short of a quorum answers that its own probe was clean, or too few peers answer it for a quorum of any of its groups, and hands each only to a peer that answered that its replica of the group may take the leadership. A node that too few peers answer sheds on its first such tick, at most one probe interval plus 1.5 s after it is cut off, and its pre-votes keep it from raising a term before that. In 0.9 a delay on every node's path, such as a round trip longer than the probe's 200 ms timeout, made every node shed, and a leadership handed to a peer that did not answer was refused by that peer, itself shedding, so the group had no leader until it re-elected.

Single node

• A single node runs as a cluster node. Its data groups run the gated open path under raft.node_id, with the cluster's election timeouts, every gate and driver of a cluster node, its snapshots on storage.snapshot.backend, and writes refused with RAFT_LOG_PRESSURE past twice raft.snapshot_log_budget. It opens every group at start. Unset raft.wal.fsync is always there, so it restarts unattended after a host crash. Under an explicit never, each group waits behind its recovery gate after a host crash for ursulactl recover-group, meta first.
• A single node grows. With raft.placement.replication_factor = 1 and its ID as raft.meta_voters on every node, new nodes join it and its groups move onto them. A group of one voter moves, since its leader is its own quorum. Each group keeps one voter, and the first node stays the meta group's only voter. The chart refuses a raise from 1. See Adding a node.
• A node without raft.wal.path runs on a temporary WAL as before, and lists itself when raft.peers is empty.

Configuration

• New keys. raft.meta_voters (unset: the smallest min(3, N) IDs), [raft.placement] with replication_factor (3) and max_concurrent_moves (4, recorded at bootstrap), and raft.wal.lease (5s, from 2s to 10s, WalLeaseOutOfRange outside). A meta voter outside raft.peers is refused (MetaVoterOutsidePeers).
• raft.group_count is at most 4,096. A larger value is refused at config load, and the default of the available parallelism times 16 is capped there, since the meta group's snapshot holds one placement per group.
• raft.wal.path is required when raft.peers lists another node. A temporary WAL may list only the node itself.

Admin API and readiness

• New routes. GET /__ursula/placement and GET /__ursula/placement/groups/{group} on both listeners, with ?confirmed=true on the admin listener only. On the admin listener: POST /__ursula/placement/groups/{group}/move, POST /__ursula/placement/move-limit, POST /__ursula/placement/nodes/{node}/cordon, .../uncordon, .../decommission and .../replace-disk. Every refusal is a typed PlacementApiError: 404 for an unknown group or node, 409 for a refusal of the meta group, 503 while it cannot answer and 504 when the write may have applied.
• Readiness reasons. New, in this order before the 0.9 reasons: superseded, disk_lease_lost, root_refused, removal_pending and predecessor_wait.
• Node lifecycles. A node is active, cordoned, decommissioning, removed or vacant, and a registration answers removal_pending for a new disk under a removed ID whose removal has not completed. The meta group refuses a command that names a vacant node with node_vacant.
• Readiness reads only the node's own state. leader_unknown, leader_outside_voters and the new membership_uncommitted block maintenance only, so a meta outage, or a group whose quorum is lost on other nodes, no longer makes every node unready, and a roll no longer replaces healthy pods for it. The new group issue leader_not_acknowledged holds a restarted node unready until each of its replicas accepted an append that took its log to its leader's commit index. It ends after 10 seconds for a replica that no leader contacts. membership_not_applied now means a committed membership entry not yet applied. Each group's metrics carry maintenance.leader_acknowledgement (awaiting, acknowledged or no_leader_contact), and ursulactl wait-ready waits for it through each node's readiness.
• Maintenance report version 2. raft_maintenance names the node's placement, lists expected_groups with their role and target voters and unplaced replicas with the last answer about them and their teardown stage. New node issues: placement_unknown, placement_not_bootstrapped, genesis_mismatch, meta_unavailable, meta_damaged, node_removed, group_count_mismatch and replication_factor_mismatch. New group issues: unplaced_replica, unplaced_member, move_in_progress and move_stalled. unconfigured_member, unexpected_group and empty_expected_inventory are gone, and a node with no group is ready. A replica that does not count as a voter is checked only for being stopped.
• Metrics. /__ursula/metrics gains meta, the meta replica's group metrics, and every group's reconcile status and needs_vote_floor. wal_recovery gains the meta journal's opening and predecessor, and previous_run may read remounted. recovery_gates includes the meta replica's gate under "meta", and stalled lists it as "meta". recovery_stalled_groups in /__ursula/ready stays data groups only. A group's maintenance.accepting_transfers, which ursulactl drain reads, now follows the per-replica rule of takes_leadership: it is false until the replica acknowledged a leader of its group since the node started.
• Leadership shed. GET /__ursula/leadership-shed gains takes_leadership, recovery_gates, leader_count, serving_ready and egress (unprobed, clean or degraded, the node's last egress probe), and the shed reasons placement-unavailable and fence. takes_leadership lists the replicas whose group's leadership the node may take now ("meta" for the meta replica), and every planned handoff (the shedding gates, the balancer, the shutdown handoff) goes only to a replica listed there.
• Recovery plans. floor is required: "meta" for the meta group, "seeded" or {"certified": {"term", "leader", "index"}} for a data group. A plan of the other plane answers 409 floor_other_plane, and a certified plan whose leader does not hold the entry answers 409 floor_not_held. A live group's plan keeps the membership its last convergence certified.
• Usage version 2. /__ursula/usage reports each hosted group with its applied_index. A collector keeps the newest report per group and sums by bucket (UsageReport::bucket_usage).

ursulactl

• New verbs. placement status, placement move, placement cancel, placement set-move-limit, node cordon, node uncordon, node decommission, node verify-removed, node verify-joinable, node verify-vacant and node replace-disk. See ursulactl.
• The view decides. wait-ready waits for every group live with no move, on exactly its target voters with a leader. wait --node N, verify-cluster and drain check the groups placed on the node, and wait-ready, verify-cluster and drain skip removed and vacant nodes. ursulactl of 0.9 cannot read a 0.10 node.
• recover-group plans the meta group with --target meta and a data group with the floor of a confirmed meta view, and leaves out replicas without a vote floor.
• drain waits until the node leads no data group and not the meta group.

Helm chart

• RollingUpdate only. The StatefulSet updates one pod at a time from the highest ordinal, each waiting for readiness. Start an upgrade only when ursulactl verify-cluster passes.
• A recreated pod takes the new revision. The StatefulSet renders rollingUpdate with partition: 0 and maxUnavailable: 1. Without it, the controller recreated a pod that went missing during a roll, such as one deleted while unready, at the old revision when its ordinal was below the count of old-revision pods, and on Kubernetes 1.37 and later deleted it again at once, in a loop that only a rollback ended.
• Gateway pods drain before they stop. A terminating gateway pod keeps serving for gateway.preStopDelaySeconds (10) in a preStop hook, so that Service proxies and load balancers stop sending it new connections before it gets SIGTERM and closes its listener. Its drain, gateway.gracefulShutdownTimeoutSeconds, now defaults to 45 (was 3600), and the new gateway.terminationGracePeriodSeconds (60) must exceed the two together, or the chart refuses to render. A change of server.replicaCount rolls the gateway pods, since their upstream list names every server pod.
• Scaling. Raising server.replicaCount from 3 adds pods that hold no group until groups move onto them, which stays manual. Lowering it is supported after ursulactl node decommission --wait of every dropped ID, and a later raise starts those IDs again as new nodes once their removals completed. The new pre-upgrade,pre-rollback hook (server.scaleCheck.*) refuses a raise from 1 and a lower count whose dropped IDs are not removed. On a raise it waits for each started ID's removal to complete, deletes the left volume of a vacant ID and keeps the volume of an ID in a life, and refuses any other volume. Its Role may get and delete the volumes of the ordinals a raise can start.

Observability

• New OTLP metrics. ursula.meta.leader, ursula.meta.applied_index, ursula.meta.wal.batches, ursula.meta.wal.records, ursula.placement.moving_groups, ursula.placement.retirements_pending, ursula.placement.oldest_move_age_seconds, ursula.placement.reconcile_waiting{reason} and ursula.placement.reconcile_steps_total{step, outcome}. ursula.raft.recovery_gates counts the meta replica's gate too, and ursula.wal.recovering gains the reason remounted.

Rust library APIs

• ursula-control is the placement kernel. Its 0.9 operation model, ClusterNode, DataGroupPlacement, MetaConfig, NodeState, NodeEndpoint, GroupPlacementView and PlacementNode are gone. ControlPlaneState, ControlCommand, ControlResponse, ControlError, Cluster, GroupPlacement, the seed rule, the planners, placement views and the admin types are new, with ursula_proto::placement re-exported. No shim keeps the old names.
• ursula-raft runs the meta group. The single-node meta Raft (MetaRaft, MetaRaftHandle, MetaRaftTypeConfig, MetaSnapshot and the rest) is gone. ursula_raft::meta holds the meta replica, runtime, status reader and placement watch. PlacedRaftGroupEngineFactory, over PlacementInputs, is the one engine factory, in place of StaticGrpcRaftGroupEngineFactory, DurableRaftGroupEngineFactory and the single-node constructors of RaftGroupEngine. RaftTarget (in ursula_shard) names a data group or the meta group on the wire and in errors. OwnerRaftHandle::change_membership_if replaces add_learner and replace_voters.
• The configuration types. RaftGroupConfig and RaftConfig::{groups, init_membership, init_membership_per_group} are gone. RaftConfig.group_count and WalConfig.fsync are Options, read through effective_group_count and wal_fsync. PlacementConfig and RaftConfig::{meta_voters, placement} are new.
• The removed tooling. ursula-ctl's reservation and startup modules, MetricsClient::{pin_nodes, set_maintenance_fence, change_membership} and ursula-proto's maintenance fence and startup admission types are gone. ursula_proto::placement::ProcessIncarnation replaces ursula_proto::admin::ProcessIncarnation. ursula_proto::admin::{MembershipQuery, MembershipResponse} went with the membership route.
• The leader predicate reads the lease. GroupLeadership::of takes the group's leader lease, and LeaderRole::Lapsed names a replica that OpenRaft keeps as leader but that cannot commit: its fence does not admit it, or no quorum acknowledged it within the lease. A fenced leader was LeaderRole::Unknown in 0.9.
• The server. ursula::Topology is Development or Cluster, spawn_runtime takes the config and the topology, and HttpState::with_raft_cluster and router_with_raft_cluster take the node's placement inputs.

Known issues

#461 is narrower: a restarted voter campaigns with a pre-vote, which its peers refuse while they hear their leader, and a group that knows no leader no longer makes a node unready. The other known issues of 0.9.0 remain open: #411, #465, #496, #517, #518, #561, #735, #753 and #754. These are new in 0.10.0:

• #769: a successful local read (consistency=local, and the default catch-up and offset=now reads) on a voter that no longer receives appends, such as a follower cut off from its leader, answers from the prefix the replica holds, with no bound on how stale it is. A learner, and a replica that a confirmed placement no longer counts as a voter, answer such a read as a leader-only read. The issue predates 0.10.
• #862: a SIGTERM handoff that its target refuses is not offered to another peer within the 5 s grace. The stopping node leads the group again at once and steps down at the end of the grace, and the group elects a new leader about 3 s later.

0.9.0

0.9.0 is a breaking release. It changes the storage and protocol format to format epoch 5. A 0.9 node refuses epoch-4 Raft WAL directories and object-storage prefixes at startup and refuses 0.8 peers, and a 0.8 node refuses 0.9 peers. It also moves the backup format to 5. 0.9 does not restore 0.8 backups. There is no in-place upgrade, no rolling upgrade, no backup migration and no rollback.

• If you cannot move your data yet, stay on 0.8.2.
• Deploy 0.9 fresh: empty PVCs and never-used object-storage prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.8, or a failed upgrade attempt, wrote to.
• Move your data through your applications. Read the streams you keep from the 0.8 cluster with your own clients and write them to the 0.9 cluster through the HTTP API. A stream written again is a new stream: a new Stream-Incarnation, offsets from its first written byte, and only the producer states, TTL and published snapshots that the application sets again.
• Give the 0.9 cluster its own server.cluster_id. Its streams are new streams, so a client that compares cluster ids then knows that its 0.8 state does not carry over.
• helm upgrade from 0.8 stops at the first pod. The new pod finds the 0.8 data on its volume and exits at startup, so no second voter is replaced and the refused pod writes nothing. Recover with helm rollback.
• Keep the peer lists of 0.8 and 0.9 nodes apart. A 0.9 node with a configured 0.8 peer refuses to start. A node that exchanges a Raft RPC with a node of the other release answers 503 with format_epoch_mismatch on /__ursula/ready until it restarts.
• Each node owns its snapshot objects. With an S3 snapshot store, a node writes its snapshot objects under <prefix>/node-<id>/group-<gid>/, each under a name no object had before, and only that node deletes them. A follower that installs a snapshot from a peer writes its own copy before it uses it. Snapshot references and pins under <prefix>/group-<gid>/references/ are gone. Snapshot GC reads no modification times, has no grace and no clock assumption, and never waits for a voter that has not stored a reference. A node deletes the object it replaced once no send of it is in flight. snapshot_gc_skipped_unreferenced_voter is removed. raft.install_snapshot_timeout now covers a follower's download and the upload of its copy.
• Delete the 0.8 snapshot prefixes once nothing reads them: <prefix>/group-<gid>/ under the 0.8 cluster's storage.snapshot.s3_prefix (snapshots/ by default). 0.9 never reads or deletes them.
• Optionally scope credentials per node. A node writes and deletes only under <prefix>/node-<id>/ and reads the rest of <prefix>/.
• A node that left the cluster keeps <prefix>/node-<id>/. Delete it once the node is stopped for good. No other node uses it.

See Upgrading to 0.9.

Formats

• Format epoch 5. The Raft WAL journal and state files, the WAL directory and object-storage markers, group snapshots and the Raft gRPC protocol carry format epoch 5. A node refuses data and peers of another epoch.
• A snapshot pointer names one node's object. The S3 location in a snapshot pointer, in group-N.snapshot.json and in the snapshot an install sends, no longer carries shared_object. A FullSnapshot receiver writes its own copy of the object the pointer names before it commits. 0.9 needs no further format epoch for it, since 0.9 already moved to epoch 5.
• Raft membership carries node IDs only. Membership entries, snapshot records and the snapshots an install sends name each member with an empty node instead of its address. Backups hold no Raft membership.
• Compact MessagePack. Raft log entries in the WAL, Raft RPC payloads, forwarded reads and writes, the WAL state files, snapshot records and pointers, and backup group objects are MessagePack that writes a struct as an array of its fields in declaration order instead of a map keyed by field names. Enum variants keep their names. In the WAL a 16-byte append takes 57% fewer bytes, a 100-byte append 48% fewer and a 1 KiB append 18% fewer. Encoding takes about 40% less time and decoding about 43% less. Raft RPC messages of 1 KiB or more were already compressed with zstd and shrink little. Smaller ones, such as a single small append, shrink as the WAL does.
• Backup format 5. 0.9 writes and reads backup format 5: format_version 5 in manifest.json, in GET /__ursula/backup/info and in x-ursula-backup-format, and group objects in compact MessagePack whose first field is the format. ursulactl refuses a backup or a target cluster of another backup format, such as a 0.8 backup (format 4), before the first import, and a server refuses to import a group object of another backup format with 400.

Server

• A stream named . is refused. A request that names a stream . answers 400 with INVALID_STREAM_ID, as a name containing .. or / already does. Its object keys would otherwise carry a /./ segment.
• Snapshot objects are owned per node, and installs copy (#687). A node's snapshot objects live under <prefix>/node-<id>/group-<gid>/, named <index>-<nonce>.snap by the snapshot's last log index and 64 random bits, and are created only if no object has the name. A build holds its object in memory from before its PUT, and a send holds the object it names until the RPC returns. A follower that installs a peer's snapshot reads it, writes its own copy, checks the copy with a HEAD and commits the copy. A source object that is gone answers UNAVAILABLE, and the leader's next attempt carries its current pointer. A GC pass lists the node's own directory for the group and deletes every object the node no longer holds. It runs once the restore at startup ended, again an install timeout later, after every build and on the snapshot driver's ticks. Content-addressed object names, the GC grace, its use of Last-Modified and node clocks, the reference and pin records, and the rule that GC fails closed for a voter without a reference are all gone. So is the race in which one node's GC pass deleted an object that another replica's snapshot still named (#687). A node restores only an object of its own directory. Started with another node's Raft log directory or another storage.snapshot.s3_prefix, it refuses to open the group with ForeignKey, and an operator rejoins it as a wiped node. A snapshot store that does not answer a restore refuses the group with StoreUnavailable, which answers 503 with GROUP_RESTARTING and Retry-After: the group did not open, so nothing of the request was proposed. The node then holds the group closed for a back-off that doubles per failed open, from 1 second to 30 seconds, and opens it again by itself. During the back-off and during that reopen, the node refuses the group's requests at once. open_backoff_groups in /__ursula/metrics counts the groups that wait. A group's engine opens in a task of the core that owns the group, so a store that does not answer holds only that group, and the core's other groups keep serving (#741). A request that reaches a group during its first open waits for the open, up to as many requests as the group's mailbox holds, and a request past them is refused at once. A shutdown of the group or of the node during an open stops the open before the group's Raft starts, so a store that does not answer no longer holds the shutdown. A group that opens while the core journal's previous writer is still stopping, as after another group of the core failed to open, waits for that writer to release the journal's lock instead of failing with Locked. A node whose store does not answer at startup starts and keeps those groups closed. The snapshot store has a request timeout and retries of its own, storage.snapshot.s3_timeout (10s) and storage.snapshot.s3_max_retries (1), apart from the cold store's storage.cold.s3.timeout and max_retries, which stay as they were. The timeout covers one attempt, and a PUT sends the whole object in one attempt, so it must also cover a healthy PUT of the largest snapshot. With the defaults a follower's copy of a snapshot, a GET, up to two PUTs and a HEAD, takes at most 88 seconds, within the default raft.install_snapshot_timeout of 120 seconds. A cluster node with S3 snapshots refuses to start with SnapshotCopyOutlastsInstall when these values let the copy take longer than raft.install_snapshot_timeout (#742). A gateway drops its cached leader route for the group, and the client's retry takes a fresh lookup, which may land on the same node. raft.deleted_data_max_age keeps its floor of eight raft.install_snapshot_timeouts. The replaced object now goes within one install timeout and a drive interval after the snapshot that covers a removal, which a replica takes a quarter of the bound after the removal. That needs a bound of only four thirds of an install timeout and a drive interval, and eight is a margin on top.
• Cold GC deletes only the group's own cold objects. A queued cold GC entry that names objects now deletes a path only if it is an exclusive chunk, a pack or an external payload of the entry's bucket that the group owns. Any other path, such as a snapshot object, a cold-index page, another bucket's object or another group's pack, stays in the store. An entry for a whole stream now reclaims it only if the stream is of the entry's bucket and placed in the group. Before, an entry restored into one group could delete the chunks, pages and external payloads of a live stream in another group. Only a corrupt or hand-edited backup could queue such entries. The entry's other paths are deleted and the entry leaves the queue as before. The check is on the namespace, not on liveness. /__ursula/metrics gains cold_gc_foreign_paths_refused, which counts refusals and counts them again when GC runs an entry again.
• New cold object names end in 64 random bits. Chunks, packs, external payloads and snapshot parts were named by their write time and a counter that restarted with each process. The last 16 hex digits of a new name are now random, so whatever the clocks and restarts of the nodes, two names collide only if they share the nanosecond and draw the same 64 random bits, with probability about 2^-64 per pair. Later cold GC relies on unique names. Names keep their shape and length.
• A leader proposes a cold upload only in the leadership that uploaded it. The Raft entry that names a cold object a leader uploaded, a flushed chunk or pack, a compaction's replacement, an external payload, a snapshot body or a parts commit's copy, is appended only while the group's committed leader, its term and node, is still the one under which the upload began. A write whose leadership ended in between answers 503 with Retry-After, nothing of it applies, and the leader deletes its object at once. Before, a write that waited across a re-election of the same node was appended in the new term. Producer-keyed retries stay exact: the refused write applied nothing, so its retry with the same key applies once. A flush or compaction whose leadership ended deletes its upload, a pack none of whose slices was published included, and a later flush uploads the bytes again.
• 0.9.0 prepares orphan cleanup without the one-day wait. A later 0.9 release may delete orphaned cold objects sooner. That deleter requires every node to fence its uploads to their leadership and to name them with random nonces, as every 0.9.0 node does.
• A cold flush or compaction refused while a group's history is pending cleans up after itself. A leader whose history event has not applied refuses such a publish before proposing it. The flush or compaction now deletes its upload and keeps the old index, instead of leaving them for the orphan sweep, and a compaction pass goes on with the next stream.
• A cold compaction whose input went stale cleans up after itself on a node that runs the in-memory engine. When the chunks a compaction read are no longer in the stream's cold index, the compaction is refused before it is proposed. It now deletes its replacement object, and a compaction pass goes on with the next stream. Before, the pass stopped at that stream.
• JSON writes staged in object storage make small Raft entries. With a cold store, the Raft entry of a JSON create or append whose body was staged in object storage held the end offset of every message. A 32 MiB body of 2-byte messages made an entry of about 80 MiB that weighed 128 MiB against raft.snapshot_log_budget, and a staged body of 1.6 MiB of such messages already made an entry larger than a replication read. The entry now holds the number of messages, which committed_records in /__ursula/usage counts as before, so it stays at about 2 KiB plus its Content-Type whatever the body holds. With a cold store at the default runtime.external_payload_min_size, only a restore now makes an entry larger than a replication read (#518).
• A body sent to a stream of another content type answers 409 whatever its size. A body staged in object storage, a JSON body to a byte stream or a byte body to a JSON stream, answered 400 with INVALID_RECORD_BOUNDARIES. It now answers 409 with CONTENT_TYPE_MISMATCH, as a smaller body does.
• Expired streams leave through one background command. Each group's leader removes expired streams with a dedicated ExpireStreams Raft entry. One entry removes up to 256 streams that expired a second or more before the leader's clock, earliest expiry first. The leader sweeps every minute, or every quarter of raft.deleted_data_max_age if that is shorter. A pass that stopped at its 16 entries or its 250 ms in a group runs again a second later, with 50 ms per group, until the group's backlog is gone. Writes no longer remove expired streams they do not name. In 0.8 a create, an external append, a snapshot publish, a retention advance, a close, a read that renewed a TTL and a read that found its stream expired also removed up to 256 expired streams of its group as it applied, which made that request slower. A request that names an expired stream still removes it, and reads, writes and HEAD of an expired stream answer 404 at once as before. A DELETE of an expired stream now answers 404 at once too, with or without Stream-Incarnation, as for a missing stream. 0.8 answered 204 until a write or the sweep removed the stream. What changes is when an expired stream leaves a group that takes writes: within the sweep's interval and a second after its expiry, as in a quiet group. Until then stream_count and retained_bytes in /__ursula/usage still count it, and a bucket purge counts it among its removed streams. /__ursula/metrics gains ttl_expiry_swept_streams, the streams the node's sweeps removed, and ttl_expiry_sweeps now counts ExpireStreams entries.
• storage.snapshot.s3_prefix is checked when the config is read, whatever the snapshot backend, inline included. Every leading and trailing / is trimmed, and what is left must be one or more components separated by single /s, each of ASCII letters, digits, -, _ and ., and none of them . or ... A node refuses to start with an empty prefix, an empty component such as a//b, or any other character, whitespace included, and the error names the key. Such a prefix used to put snapshot objects where the store rewrote their keys (#724). The Helm chart's schema checks snapshotStore.prefix the same way.
• The snapshot driver always runs. storage.snapshot.drive_interval can no longer be "0s". A config that sets it to "0s" is refused at startup with an error that names the key, and the Helm chart's schema refuses snapshotStore.driveIntervalMs: 0. The driver takes every Raft snapshot on every node that runs Raft, and OpenRaft's own snapshot policy is off on every group. raft.snapshot_logs_since_last, which set that policy's entry count, is removed, and a config that sets it is refused as an unknown key. Remove it, and the chart's raft.snapshotLogsSinceLast, before you deploy. A node without raft.peers now takes raft.snapshot_backstop_logs (100,000 entries) as its far entry backstop, as a cluster node does, instead of 5,000 entries. Set raft.snapshot_backstop_logs = 5000 to keep the old single-node bound. Its groups still snapshot by log bytes under raft.snapshot_log_budget and a quarter of raft.deleted_data_max_age after a removal.
• OpenRaft 0.10.0-alpha.37. Ursula moves from OpenRaft 0.10.0-alpha.28 to 0.10.0-alpha.37, with the changes below.
• A leader without a live quorum refuses writes with 503 NO_LEADER. A leader refuses a proposal when no quorum has acknowledged it within its lease of 3 seconds, as when its followers stop answering. A write that reaches a new leader before its first follower answers waits for that answer, up to the minimum election timeout. OpenRaft appends nothing it refuses, so the answer is 503 with NO_LEADER and Retry-After, or a redirect, and the write did not apply. Only an entry the leader appended and then dropped for a new leader's log still answers 503 with OUTCOME_UNKNOWN.
• A leader no longer grants a vote while a quorum acknowledges it. It refuses a vote request while a quorum has acknowledged it within its lease, unless a leadership transfer sent it. Pre-vote stays off, so a voter whose election timed out still raises its term, and the leader steps down at that voter's next reply.
• A leadership transfer resets the target's backoff, so the leader replicates to the target at once instead of after a backoff from an earlier failure.
• ReadIndex reads share heartbeat rounds and stay bounded. A linearizable read still confirms leadership with a quorum round that begins after it arrives, never with an earlier acknowledgement, and still answers 503 when no quorum confirms it within the minimum election timeout. Reads that arrive together now share one heartbeat round.
• The snapshot record's snapshot_id is empty. OpenRaft no longer keeps a snapshot id in its snapshot metadata. It writes the field empty in group-N.snapshot.json and in the snapshot an install sends, and ignores it on read. Ursula names a snapshot by its group and last log id, and its pointer keeps that name.
• Raft ticks about 7 times as often. OpenRaft now ticks every heartbeat_interval * 13 / 64, 50 ms at the default 250 ms heartbeat instead of 375 ms, at a random phase per group. Heartbeats go out on time, about every 250 ms. A node with many groups spends more CPU on ticks, and a single node, whose heartbeat is 10 ms, ticks every 2 ms per group.
• A follower's install refusals answer with their own gRPC codes. When the snapshot object an install names is missing or does not read, or the install gate is closed, the follower answers UNAVAILABLE, so the leader backs off before it resends its current snapshot. Before, it answered INTERNAL and the leader resent at once. An undecodable pointer answers INVALID_ARGUMENT, a snapshot that does not decode answers DATA_LOSS, and the message carries the store's cause.
• A group restarts only once its previous engine stopped. When a group's engine stops or restarts in a running node, the node now waits for the engine's background Raft work as well: queued applies, snapshot builds, snapshot sends and installs, and replication. The group opens again only once that work ended. A request that reaches the group meanwhile answers 503 with GROUP_RESTARTING and Retry-After. It is a refusal, so nothing of a write applied. A restart whose old engine's work has not ended after 30 seconds fails, and the group stays closed until the work ends. The code can show up briefly during any restart of a group (#709).
• A snapshot record is published only after the WAL is synced. Before a replica persists the record of a snapshot it installed or built, it syncs its core journal. The record was synced on its own, while the WAL truncate of the uncommitted tail that the snapshot replaced waited for the core's next synced batch, under fsync = never and fsync = always alike. A power loss in between kept the record and lost the truncate, and the replica restarted with entries from an older term after its snapshot. Each snapshot publication now costs one journal fsync. At startup a replica now checks its log against its snapshot record and refuses to open a group whose log holds such an entry, with the error LogBelowSnapshot. The group stays closed on that replica until its node rejoins as a wiped node. See A log below its snapshot record.
• A voter that lost entries is caught up in place. A follower that lost Raft log entries it had acknowledged after a host crash under raft.wal.fsync = never stays a voter. Its leader rewinds its replication to it on the first conflict it answers and sends it its log again. A follower on a lost disk also stays a voter and is caught up the same way, once a quorum of the other voters has proved its vote floor, so two lost disks of three still need an operator. In 0.8 the leader removed such a voter, added it back as a learner and promoted it once it caught up, and rewound only a majority. The follower's answer counts as its acknowledgement of the leader, so the leader keeps its quorum and its lease meanwhile, and a group whose surviving leader is idle heals without proposing anything. A bound host replacement that dies after its leaders reached it stays a voter too, so verify-survivors proves its loss and the host is restaged (#698). While followers that lost entries cannot catch up, for example one that needs a snapshot while object storage is unavailable, writes are accepted and can time out with an unknown outcome instead of answering 503 (#728). The new ursula.raft.follower_log_reversions counter counts these rewinds by follower node, and the leader logs each one with the group and the follower.
• A Raft group's cold-index failure answers 502 with COLD_STORE_FAILED. When writing or rolling back a stream's cold-index pages, repairing a group's cold index or indexing an offloaded object fails in object storage, a Raft group answered 500 with INTERNAL. It now answers 502 with COLD_STORE_FAILED, as a single node does, and logs the store's error.
• A forwarded bucket purge keeps the leader's answer. When the leader refused a forwarded purge as a whole, for example while its group's history is pending or after its batch lost its answer, the follower answered 500. It now answers what the leader would: 503 with HISTORY_PENDING and Retry-After, or 503 with OUTCOME_UNKNOWN and no Retry-After, since the purge may have applied.
• A Raft group's stale cold compaction deletes its upload. A compaction whose input no longer matches the stream's cold index proposed nothing, yet left its replacement object for the orphan sweep and ended the compaction pass. It now deletes the object at once, and the pass goes on with the next stream.
• A refused leadership transfer answers a precise gRPC code (#539). A node answered every leadership transfer it refused with FAILED_PRECONDITION. A refusal by the election policy still does. A node whose Raft core stopped now answers UNAVAILABLE, and a group removed from the node while the request was in flight answers NOT_FOUND, as a group the node never ran already did. None of them reads as another format epoch's refusal.
• A recovery barrier covers the leader's whole log. The barrier that opens a gated replica is now the higher of the read index and the leader's last log index, so the gate opens only once the replica holds every entry its previous process could have acknowledged. GET /__ursula/raft/{group}/quorum reports that index as required_applied_index.
• A [[raft.peers]] url that ends in / works for every peer request. The egress probe and the leadership-shed polls used to add their path after the slash and miss their route. Every peer address now comes from one directory built from raft.peers at startup, which drops a trailing / from url and client_url once.
• Peer addresses come from raft.peers only. Replication, forwarded reads and writes, recovery probes, the egress probe, leadership-shed polls and leader redirects all take a peer's address from the node's own [[raft.peers]]. Raft membership names node IDs only. In 0.8 each group recorded every member's url when it was first initialized, and peers dialed the recorded address from then on.
• A peer's address changes with the configuration. Change the node's url and client_url in [[raft.peers]] on every node and restart the nodes. Nothing holds the old address and no admin step migrates it. A change that moves a listener, such as a new cluster port or a new host, needs every voter restarted together. See Changing a peer address. The startup refusal PeerAddressChanged is gone (#633).
• A follower redirects only to a leader that raft.peers lists. A follower whose planes share server.listen used to redirect to the address a group recorded for a leader that its [[raft.peers]] did not list. It now answers 503 with Retry-After, as it already did with server.cluster_listen set, and it no longer forwards reads to such a leader.
• The maintenance report names members that configuration does not list. A group whose membership names a node that is not one of its configured voters reports unconfigured_member in raft_maintenance.group_issues. Serving readiness ignores it and maintenance readiness requires its absence. No node dials a member that [[raft.peers]] does not list, and each node logs that once. The leader never removes such a member. Remove it with the membership endpoint, which now removes learners as well, or list the node again.

Admin API

• recovering_target is gone. A leadership transfer to a follower that lost entries answers 409 with unready_target until the follower caught up, and ursulactl plans again as for any unready target.
• POST /__ursula/raft/{group}/learners/{node} is gone. The group's leader adds a configured voter that is missing from its membership as a learner by itself and promotes it once it caught up. A learner the route added for a node that is not a configured voter of the group was never hosted by that node. The admin listener answers 404.
• POST /__ursula/raft/{group}/membership?voters= also removes learners. It replaces the voter set and removes every learner outside the new voters in the same change. A configured voter it removes comes back, since the group's leader adds it as a learner again.

Rust library APIs

• ursula_stream::msgpack is Ursula's one MessagePack codec. Its encode, encode_into and decode write and read compact MessagePack, with EncodeError and DecodeError. decode_backup_snapshot decodes a backup group object after it checked its backup format, and fails with BackupSnapshotError, Undecodable or Format. BACKUP_FORMAT_VERSION is 5. The source of StateFileError::Undecodable and of ursulactl's BackupError::DecodeSnapshot is ursula_stream::msgpack::DecodeError. ColdGcEntry and StreamVisibleSnapshot encode every field, None and zero included, and StreamSnapshot::format_version has no serde rename.
• Cold uploads carry their leadership. FlushColdRequest, CompactColdRequest, CreateStreamExternalRequest, AppendExternalRequest and ColdSnapshotBody gain upload, an Arc<ProposedUpload> fenced to an UploadFence, and CreateStreamExternalRequest::from_create_request and AppendExternalRequest::from_append_request take it. The requests of one upload share one ProposedUpload through its Arc, and a Raft engine reaches the fence of an upload proposal only through it. ProposedUpload::new and CommittedLeader::new are hidden and not stable API: a later 0.9 release replaces them with a hold an upload takes before its PUT. WriteRoute::Leader names the fence, GroupEngine::upload_fence and ShardRuntime::cold_upload_fence read it, and GroupLeadership::upload_fence takes it from a replica's Raft metrics. uploaded_object names the object a StreamCommand names as new. GroupEngine::upload_fence has no default. ursula::router(runtime) and HttpState::new without a Raft registry serve in-memory engines only: they fence uploads to UploadFence::Unreplicated, which a Raft engine refuses. A Raft runtime needs the registry-backed entry points, such as router_with_raft_registry and HttpState::with_raft_registry. CreateStreamExternalRequest and AppendExternalRequest no longer implement Serialize and Deserialize.
• A write that never reached its Raft core proposed nothing. A Raft engine's write or batch whose submission found the group's Raft core stopped answered OutcomeUnknown. It now answers NotLeader with SteppedDown and NotProposed, since the core handed the write back unsent. A write the core took and whose answer was lost is still OutcomeUnknown.
• The voter rebuild is removed. GroupRejoin::follower_lost_log, reverted_followers and is_reverted_follower, OwnerTrigger::allow_next_revert, GrpcRaftNetworkFactory::with_rejoin, InProcessRaftNetworkFactory::with_rejoin, LeadershipTransferError::RecoveringTarget and TransferRejection::RecoveringTarget are gone. A group whose RaftGroupEngineOptions::recovery is set runs OpenRaft with allow_log_reversion. run_rejoin_heal is no longer public. RaftGroupEngine::publish_recovery binds only the gate the engine started with (RaftGroupEngineOptions::recovery) and refuses any other with RecoveryAlreadyBound. RaftGroupHandleRegistry::follower_log_reversions is new.
• A group's snapshot record comes from its log store. RaftGroupEngineOptions::snapshot_metadata_path is replaced by snapshot_record, a SnapshotRecord that RaftGroupFileLogStore::snapshot_record returns. It names the record's file and syncs the core journal before each record. GroupInfraError::LogBelowSnapshot and LogPosition are new.
• GroupEngineError is typed. GroupEngineError::ForwardToLeader is NotLeader(LeadershipRefusal) with KnownLeader, Proposal and LeadershipReason, and GroupLeaderHint is gone. A refusal names its leader only with the term it leads in. A stream refusal is Stream(StreamRefusal) with code and text, and its text no longer starts with the code. GroupInfraError::Internal, GroupInfraError::internal, GroupEngineError::new and the message methods of GroupEngineError and GroupInfraError are removed, so every engine error is a typed variant and renders its text through Display. GroupInfraError::ProtoDecode is gone: a Raft gRPC request without a required field is refused with INVALID_ARGUMENT.
• One classification. status, write_effect, not_leader and peer_backs_off classify every error. refused_before_proposal holds for a write refused before its command was proposed, and applied_nothing for a write of which nothing applied: such a refusal, or a stream refusal. GroupEngineError::forward_to_leader, forward_to_leader_before_proposal, with_leader_term, leader_hint, is_forward_before_proposal, is_refused_before_proposal, is_backpressure, is_history_pending, stream_parts and stream_from_replicated, and GroupInfraError::is_backpressure, are removed. On RuntimeError, not_leader and applied_nothing take the place of leader_hint and is_forward_before_proposal.
• GroupInfraError gains a typed variant for every failure class. ColdStore names its ColdOperation instead of carrying the store's text, which cold_store_failure logs. The new variants are UnexpectedResponse, MissingStreamId, Unsupported, BootstrapWindowNotHot, ColdCompactionStale, ForwardFailed, ForwardedAnswerInvalid, InvalidRaftConfig, StaticMembership, RecoveryGateOpen, MembershipInit, LeaderWaitTimedOut and Snapshot(GroupSnapshotError). GroupWriteResponse::kind and a TryFrom<GroupWriteResponse> for each response are new. StreamSnapshotError is now Serialize and Deserialize, and its PayloadLengthMismatch::payload_len is a u64.
• The in-memory engine and the engine defaults answer typed errors. An operation a GroupEngine leaves to the trait's default answers Unsupported with its EngineOperation. The in-memory engine answers UnexpectedResponse, MissingStreamId, BootstrapWindowNotHot, ColdCompactionStale, ColdStore and Snapshot(GroupSnapshotError::InvalidContent) where it answered Internal. A second engine for a group a core already runs is refused with the new RuntimeError::GroupAlreadyInstalled.
• The Raft engine's errors are typed. A stopped Raft core is OwnerStopped, an unusable Raft config InvalidRaftConfig, a static membership without a group, voters or peer addresses StaticMembership, a recovery gate that did not open RecoveryGateOpen, and an engine answer of the wrong kind UnexpectedResponse, where each was Internal with text. ResponseKind::answering names the kind of response a command gets. GroupRejoin::durable returns the new GateOpenError, and RecoveryGateError loses StartAsFollower, RaisePlanVote and Cut, which only that function returned.
• GroupSnapshotError names why a snapshot does not restore. It is new, and its variants are InvalidContent, ForeignKey, ObjectMissing, ObjectCorrupt with a SnapshotObjectCheck, PointerUnreadable and StoreUnavailable with a SnapshotStoreOperation. All but StoreUnavailable are persistent, so a group whose snapshot does not restore answers 500 with no Retry-After. StoreUnavailable is Temporary and NotProposed, and answers 503 with GROUP_RESTARTING and Retry-After. They join the frozen wire form. A restored pointer outside the node's own directory is refused with ForeignKey before anything is read.
• SnapshotStore names each node's own objects. The trait requires namespace, stage, write_staged, list_own, delete_own and read_stored, and provides stage_stored, download, verify_uploaded and health_check with defaults. A store writes and deletes only objects under {prefix}/node-{node}/group-{gid}/, named by OwnedKey, and delete_own takes only an OwnedObject the store listed. OwnedPointer is a pointer a node may commit, and the only pointer a restore or an install commits. S3SnapshotStore::new and try_new take a SnapshotPrefix and the node id, and RaftSnapshotConfig::s3_prefix is an Option<SnapshotPrefix>. upload, upload_iter, delete, prune_retired, publish_reference, pin_reference and reconcile_reference_pins are removed, and so are SnapshotKey, SnapshotPrune, SnapshotReferenceConfig, S3SnapshotStore::with_references and with_gc_grace, and the shared_object field of SnapshotLocation::S3. snapshot_store_from_config takes the node id. write_staged returns an OwnedLocation, the location its stage drew, and OwnedPointer::new makes the one pointer a build or a copy commits from it. read_stored reads a body as the store holds it, which stage_stored writes again as it is, and download decompresses it. StoredSnapshot and OwnedLocation are new.
• Snapshot GC has no grace. RaftConfig::snapshot_gc_grace, RaftConfig::MAX_SNAPSHOT_GC_GRACE and DEFAULT_SNAPSHOT_GC_GRACE are removed. SnapshotInstallCoordinator::with_gc_grace is with_late_write and RaftGroupHandleRegistry::with_snapshot_install takes the same wait, raft.install_snapshot_timeout on a server, after which a group runs its second GC pass for a write that failed. SnapshotGcPass::skipped_unreferenced_voter and SnapshotGcSample::skipped_unreferenced_voter are removed. StaticGrpcRaftGroupEngineFactory::with_snapshot_store sets the registry's store, which every engine of the node writes to.
• SnapshotStoreError is typed. The string variants Backend, NotFound and Integrity are gone. NotFound, NameCollision, SizeMismatch and Backend carry the key, and Backend names its SnapshotStoreOperation and keeps the store's error as its source. ErrorChain renders an error with its sources for logs.
• OpenRaft's SnapshotData is Ursula's SnapshotData. It was Cursor<Vec<u8>>. It holds the encoded pointer, read with pointer_bytes() and taken with into_parts(), and a SnapshotLease on the object it names. The snapshot a replica hands OpenRaft to send leases its object, and both networks keep the lease until the RPC returns. KeyHolds keeps in-memory holds on object keys.
• OpenRaft's snapshot types move off the type config. UrsulaRaftTypeConfig and MetaRaftTypeConfig no longer name SnapshotData. The state machines, their snapshot builders and the networks name it instead, as SnapshotData for groups and MetaSnapshotData for the meta group. UrsulaSnapshot and MetaSnapshot are the snapshots with their metadata, and RaftGroupEngine::new_node and MetaRaftHandle::new_node_with_log_store_and_network require a network whose snapshots carry that data. begin_receiving_snapshot is gone.
• LeadershipReason::LeadershipTransfer is new. A leader that is handing its leadership to another node refuses a write with it, and answers 503 with NO_LEADER, since the target does not lead yet. A write OpenRaft refuses before appending it is Proposal::NotProposed, and only an entry it appended and then discarded is Proposal::MayHaveCommitted.
• SnapshotInstallError gains typed refusals for the install gate, the incoming pointer, a missing or unreadable source object, an undecodable snapshot and the node's copy of it (CopyWrite, CopyVerify, and NameCollision after one retry under a new name). Publication is removed, since an install publishes nothing to the store. RaftGroupHandleRegistry::build_snapshot_for_transfer returns SnapshotBuildError.
• The error's wire form is frozen within 0.9. A forwarded GroupRead or GroupWrite answers with the error as compact MessagePack, and no 0.9 patch release changes it. Every field of the error is required. The incarnation of the forwarded responses (CreateStreamResponse, AppendResponse, ReadStreamResponse, PublishSnapshotResponse, AdvanceRetentionResponse and CloseStreamResponse) is required as well.
• Expired-stream removal changes the stream and runtime types. StreamCommand::ExpireStreams and StreamResponse::StreamsExpired are new. StreamCommand::DeleteStream and DeleteStreamRequest carry now_ms, and StreamResponse::Deleted and DeleteStreamResponse say whether the stream had expired. GroupEngine::tidy_streams and ShardRuntime::tidy_streams_all_groups_once return a TidyStreamsPass, the totals with the first error, and TidyStreamsRequest carries expiry_sweep_budget. ursula_stream no longer exports AppendStreamInput and WriteKind.
• Engine shutdown waits for OpenRaft's work. RaftGroupEngine::shutdown returns once the work OpenRaft runs for the engine on tasks of its own ended, or fails with the new GroupInfraError::EngineWorkOutlivedShutdown. GroupInfraError::PreviousEngineWorkRunning refuses a group that does not open while its previous engine's work still runs. Both are Temporary, NotProposed and peer_backs_off, and both are in the wire form. RaftWal::open refuses such a group with the new RaftWalError::PreviousEngineRunning. A RaftGroupEngine dropped before its shutdown stops in the background, and its group opens again once its work ended. GROUP_RESTARTING is a new ErrorCode (#709).
• RaftEngineConfig loses its OpenRaft policy fields. snapshot_drive_interval_ms and snapshot_logs_since_last are removed, and so is RaftConfig::snapshot_logs_since_last. StaticGrpcRaftGroupEngineFactory, and DurableRaftGroupEngineFactory once given a RaftEngineConfig, always run their groups with SnapshotPolicy::Never.
• A group's engine opens off its core's loop (#741). RuntimeError::GroupOpening is new. A core answers it at once for a group whose engine reopens after a back-off, or whose shutdown began during its open, for a request past the group mailbox's capacity during a first open, and to the requests that waited for an open that ended without an answer. It is Temporary and refused before proposal, and answers 503 with GROUP_RESTARTING. RuntimeError::leaves_group_closed tells the open failures that leave a group closed while the node serves its other groups. ShardRuntime::warm_all_groups waits for every group's open and fails only on other errors. A core drops the future GroupEngineFactory::create returns when the group's shutdown begins during the open, so a factory's future must leave nothing running when it is dropped at an await. RaftWal::open_waiting is new: it opens a group's log store as RaftWal::open does, after waiting for the core's previous journal writer to release the journal's lock. RaftWal::open refuses such a group with the new RaftWalError::CoreWriterStopping, which carries the writer's released signal. Both Raft engine factories open through open_waiting.
• The snapshot store has its own request budget (#742). RaftSnapshotConfig gains s3_timeout and s3_max_retries, request_budget, install_copy_budget and INSTALL_COPY_REQUESTS. S3SnapshotStore::try_new takes the RaftSnapshotConfig in place of the prefix. s3_operation_budget moves from ursula_runtime to ursula_config, with S3_RETRY_MIN_DELAY and S3_RETRY_MAX_DELAY. ValidationError::SnapshotCopyOutlastsInstall is new.
• PeerDirectory resolves peers. ursula_raft::PeerDirectory and PeerEndpoints are new, and PeerDirectory::from_config builds the directory from raft.peers. StaticGrpcRaftGroupEngineFactory::new, PeerCluster::bind, Topology::static_cluster, HttpState::with_static_raft_cluster, with_static_raft_cluster_topology, ClientWriteLeaderRouter::new, with_static_topology, router_with_static_raft_cluster and router_with_static_raft_cluster_topology take a PeerDirectory. PeerCluster::directory returns it. A group whose static membership the factory cannot build fails with GroupInfraError::StaticMembership.
• Leader hints carry no address. KnownLeader::address, WriteRoute::Follower::leader_address and LeaderRole::Follows::leader_address are removed, and the wire form of GroupEngineError changes with KnownLeader. RecoveryTransport::probe and barrier, and the probes that run_rejoin_vote_barrier and run_group_bootstrap call, take a node ID only. GrpcRaftNetwork::new takes no address: it dials the cluster URL its transport's PeerCluster lists for the target, and GrpcRaftNetwork::with_threshold is gone. PeerCluster::cluster_url is new. PeerPlanes, HttpState::with_peer_planes, ClientWriteLeaderRouter::with_planes, RaftGroupHandleRegistry::recorded_node_addresses, ursula::PeerAddressChanged, AddLearnerQuery, AddLearnerResponse and MetricsClient::add_learner are removed. RaftMaintenanceIssue::UnconfiguredMember is new. OwnerRaftHandle::change_membership is replaced by replace_voters, which removes every learner outside the new voters in the same change.
• OpenRaft's Node is EmptyNode. UrsulaRaftTypeConfig and MetaRaftTypeConfig name openraft::EmptyNode. RaftGroupEngine::initialize_membership, attach_recovery, RecoveryGate::attach, run_rejoin_vote_barrier, run_group_bootstrap, OwnerRaftHandle::initialize and MetaRaftHandle::initialize_membership take a BTreeSet<u64>. OwnerRaftHandle::add_learner, RaftGroupEngine::new_single_node and its variants, and MetaRaftHandle::new_single_node_with_log_store no longer take a node.

Known issues

Four known issues of 0.8.2 are fixed. A replica's snapshot no longer names an object that another node's GC can delete (#687). A bound replacement candidate that dies mid-rebuild no longer blocks verify-survivors (#698). Engine shutdown waits for OpenRaft's work, so a recovery plan's restart no longer fails with GroupAlreadyOpen (#709). A refused leadership transfer answers a precise gRPC code (#539). #518 is narrower: with a cold store at the default runtime.external_payload_min_size, only a restore makes an entry larger than a replication read. The other known issues of 0.8.2 remain open: #411, #461, #465, #496, #517, #518 and #561. These are new in 0.9.0:

• #735: Raft membership names node IDs only, and every peer address comes from [[raft.peers]]. A DNS name that still resolves can change in an ordinary rolling restart. Moving a node's cluster listener (a new cluster port, server.cluster_listen turned on or off, or a host without a stable name) needs every voter restarted together, since during a roll of one node at a time some pairs of nodes cannot reach each other and a group can lose its quorum.
• #753: a core syncs its journal before every snapshot record, after a local build and after an install. The sync holds the core's journal writer for about 2 ms per build on local ext4, and appends of every group on that core wait behind it. Only a group with a log truncation that is not durable yet needs it, so a leader's builds pay it for nothing. It matters only at high snapshot rates.
• #754: with an S3 snapshot store, every snapshot build ends with a GC pass that lists the node's directory for the group. A build then costs a LIST and a DELETE on top of its PUT and HEAD, about two S3 requests per build per node more than a delete of the replaced object by its key.

0.8.2

0.8.2 removes the experimental event-time indexer and changes nothing else: format epoch 4, the server's storage and protocol formats and the extension's formats (USQ2, USS3, USL1) are unchanged. 0.8.1 and 0.8.2 nodes run in one cluster during a rolling upgrade, and 0.8.1 and 0.8.2 owners and readers of one stream work side by side. Upgrade in place with an ordinary rolling restart (helm upgrade), with no fresh install or data move, once your values no longer set the indexer. Rolling back to 0.8.1 is safe.

Indexer

• The indexer role is gone. The ursula binary no longer has an indexer subcommand, so ursula indexer ... now fails at argument parsing. The ursula-index crate is no longer built or published. Nothing in the server or the gateway depended on it: offsets, Stream-Incarnation and leader reads behave as before.
• The Helm chart drops it. The indexer values block, the deploymentContract.indexerServiceAccountName, indexerRoleArn and indexerS3Prefix keys, and the indexer's Deployment, Service, PodDisruptionBudget, NetworkPolicy and ServiceAccount are removed. The values schema refuses unknown keys, so a values file that still sets any of them, indexer.enabled: false included, fails schema validation in helm install and helm upgrade. Current Helm reports additional properties 'indexer' not allowed, or names the leftover key at /deploymentContract. The ursula-chaos chart drops target.indexerUrl and target.indexSourceBaseUrl the same way, and its agent no longer registers its record streams with an indexer.
• Event-time queries have no replacement. Read a stream by offset, and keep an index in your application if you need lookups by event time, for example a SQLite database replicated through Ursula with @tonbo/sqlite-ursula.
• Upgrading: from 0.8.0, the 0.8.1 upgrade notes apply as well. If you run the indexer, turn it off on 0.8.1 or earlier first. Set indexer.enabled: false and roll that out, then delete the indexer block and the deploymentContract.indexer* keys from your values, and only then upgrade to 0.8.2. Outside Helm, remove every process or manifest that runs ursula indexer before you deploy the 0.8.2 image, which cannot start one. Afterwards the indexer's S3 prefix and its IAM role are unused and nothing in Ursula reads them, so you can delete both. Rolling back to 0.8.1 is safe, and the indexer stays off until you enable it again. If you deleted its prefix, register the streams again, and it rebuilds each index from the stream's retained history.

Known issues

The known issues of 0.8.1 are unchanged.

0.8.1

0.8.1 is a drop-in patch for 0.8.0: format epoch 4, the server's storage and protocol formats and the extension's formats (USQ2, USS3, USL1) are unchanged. 0.8.0 and 0.8.1 nodes run in one cluster during a rolling upgrade, 0.8.0 and 0.8.1 owners and readers of one stream work side by side, and owners upgrade one at a time. Upgrade in place with an ordinary rolling restart (helm upgrade), with no fresh install or data move. Rolling back to 0.8.0 is safe once raft.deleted_data_max_age (Helm raft.deletedDataMaxAgeMs) is unset, since 0.8.0 refuses unknown configuration keys.

Server

• Deleted streams leave snapshots and the Raft log within a bound. After a replica applies a DeleteStream, a PurgeBucket or the removal of an expired stream, its current group snapshot and its retained Raft log drop the stream's records within raft.deleted_data_max_age. Unset, that is the larger of 1 hour and eight times raft.install_snapshot_timeout. Removed bytes count toward a group's next snapshot, so a large removal leaves the snapshot and the log within minutes. See Deleted data.
• Old snapshot objects are deleted within the bound. With an S3 snapshot store, the snapshot object a replica replaced stays pinned for a quarter of raft.deleted_data_max_age (at most an hour, or two raft.install_snapshot_timeouts if that is longer), and then snapshot GC deletes it unless another replica still names or pins it. Any other object of the group that no replica names or pins is deleted once it is that old. 0.8.0 kept the newest unused object and pruned only after a build. With storage.snapshot.drive_interval = "0s" replaced objects are still deleted, after each build. /__ursula/metrics gains snapshot_gc_deleted_objects, snapshot_gc_skipped_unreferenced_voter and snapshot_gc_failed_passes.
• Raft log files follow. A journal segment that holds purged entries is sealed and deleted a quarter of raft.deleted_data_max_age after the purge, once the few live entries other groups keep in it are copied forward. wal_dirty_segments counts such segments. The copies add to wal_rewritten_bytes: quiet groups' live entries move about once per quarter of the bound, at most an eighth of a segment per core per drive interval. The first passes after the upgrade, and after a restart, find every old purged entry at once and copy up to about twice a core's live log, and snapshot the groups that keep much of an old segment.
• Expired streams leave in the background. Each group's leader removes the streams that expired in its group every minute, or every quarter of raft.deleted_data_max_age if that is shorter, about 4,100 a minute per group, so the bound now covers a stream that expires in a quiet group. 0.8.0 removed an expired stream only when a request reached it, or when a create, an external append, a snapshot publish, a retention advance, a close or a read that renews a TTL reached its group. A group that took only plain appends kept its expired streams, because a plain append never sweeps. 0.8.1 removes those too. The leader removes a stream once its own clock is a second past the stream's expiry. It writes the TouchStreamAccess entry that a read of an expired stream already writes, and 0.8.0 replicas apply it the same way. The removal also queues the stream's cold objects for cold GC and takes it out of retained_bytes and stream_count in /__ursula/usage. A DELETE of such a stream then answers 404, as it did once a write had removed it. /__ursula/metrics gains ttl_expiry_sweeps, the sweep entries the node's leaders wrote, and ttl_expiry_sweep_due_groups, the groups the latest pass left with expired streams for the next one.
• A single node runs the snapshot driver. A node without raft.peers now drives its snapshots as a cluster node does, so the bound holds there too. A group still becomes due for a snapshot every raft.snapshot_logs_since_last entries (5,000, the count of OpenRaft's default policy that 0.8.0 ran), and also once its log reaches its byte threshold under raft.snapshot_log_budget or it holds a removal past a quarter of raft.deleted_data_max_age. Its snapshots stay inline whatever storage.snapshot.backend says. /__ursula/metrics now reports the driver's gauges and raft_deleted_data_* there, and the journal's copies add to wal_rewritten_bytes. storage.snapshot.drive_interval = "0s" turns the driver off, and the node then snapshots as 0.8.0 did, with OpenRaft's own policy and 1,000 entries kept below a snapshot. Helm installs run as a cluster, a single replica included, and the default preset runs no Raft, so neither changes.
• What the bound does not cover. Snapshot GC deletes nothing while a configured voter of the group has never stored a snapshot reference. A node that left the cluster keeps its references/node-<id>.json and its pins under references/pins/<id>/ in each group's snapshot prefix, and with them the objects they name, so delete both when you remove a node. On a bucket with object versioning, a deleted snapshot object stays as a noncurrent version until a lifecycle rule expires it. Deleted data lists every case the bound does not cover.
• Metrics. /__ursula/metrics gains raft_deleted_data_oldest_age_ms, raft_deleted_data_overdue_groups and a per-group deleted_data list. Alert when raft_deleted_data_oldest_age_ms nears half of raft.deleted_data_max_age.
• The admin log purge can answer 503. POST /__ursula/raft/{group}/purge answers 503 with Retry-After while a snapshot of the group builds or an earlier purge of it has not landed. Send it again then.
• A restart purges the log through the restored snapshot. A restarted follower may then catch up by a snapshot install rather than from the log.
• A snapshot build that outlives its group engine writes nothing. A build still running when a group's engine stopped, on node shutdown or for a recovery plan's cut, could write group-N.snapshot.json after the engine stopped. After a cut it could replace the restarted replica's newer pointer with an older one whose log the replica had already purged, leaving a gap between the snapshot and the log at the next start. 0.8.0 has the same race. A build that starts after its engine stopped now ends before it uploads anything, and a build already under way is refused before it writes the pointer. Snapshot GC deletes an object such a build uploaded.
• An idle group that pins old log files is purged through its snapshot. When such a group applied nothing since its snapshot, the driver purges its whole log rather than the usual raft.max_in_snapshot_log_to_keep entries below the snapshot, so a follower of that group may catch up by a snapshot install.
• A snapshot range past the end answers 416 under a retired token. A snapshot read whose range starts past the end answers 416 naming the current token under any Stream-Incarnation, on both snapshot resources. 0.8.0 answered 412 under a retired token. A matching If-None-Match still ignores the range and answers 412 or 304.
• Upgrading a single node. These change on a node without raft.peers that runs Raft, the tiny, small, standard and large presets and any config file without peers among them:
  • It keeps raft.max_in_snapshot_log_to_keep entries (64 by default) below a snapshot instead of OpenRaft's 1,000. Set it to 1000 to keep as many as before.
  • It builds one snapshot at a time while its log is below raft.snapshot_log_budget, and up to raft.snapshot_build_max_concurrency past it, where 0.8.0 could build every group at once. A due group waits for the build slot, so under many small appends across many groups the due groups queue and their logs grow past 5,000 entries until their turn comes. Below the budget builds run one at a time whatever raft.snapshot_build_max_concurrency says. The setting adds builds only past the budget. storage.snapshot.drive_interval = "0s" goes back to a build per group, as in 0.8.0.
  • It never refuses writes for its log. Its driver reports the log's pressure in /__ursula/metrics, but a single node does not answer 503 RAFT_LOG_PRESSURE in 0.8.x, as 0.8.0 did not. Only a cluster node does, past twice raft.snapshot_log_budget. The default preset runs no Raft and has no such gate either.
  • It writes no new record, so a 0.8.0 node rolled back onto its data reads its inline snapshot file and its log as before.
• Upgrading: replace the binaries one node at a time. Only a group whose leader runs 0.8.1 sweeps its expired streams. While a group's leader still runs 0.8.0 the group removes them as 0.8.0 did, so in a mixed cluster those groups keep 0.8.0's behavior until their leader is upgraded. Once a group's leader runs 0.8.1, a group that took only plain appends or no writes drains the expired streams it kept, about 4,100 a minute per group: stream_count and retained_bytes in /__ursula/usage drop within minutes, cold GC sees a burst of deletes, and each draining group's actor is held up to 250 ms, plus the commit of its last command, per pass. While 0.8.0 and 0.8.1 nodes run together, a 0.8.0 node does not pin the snapshot object it replaced, so a 0.8.1 node's GC can delete an object a 0.8.0 leader is still sending. That install fails once, with a follower warning that reads prefetch OpenRaft snapshot ... before install: snapshot not found, and is retried with the leader's newer snapshot. Set raft.deleted_data_max_age (Helm raft.deletedDataMaxAgeMs) only once every node runs 0.8.1, since 0.8.0 refuses unknown configuration keys.

SQLite VFS

• A commit above the commit limit fails as too_large. A commit whose pages take more than URSULA_VFS_MAX_COMMIT_BYTES (128 MiB by default, from 4096 bytes to 1 GiB) fails at its COMMIT with SQLITE_IOERR_AUTH (7178), action fix. SQLite rolls the whole transaction back, nothing of it is logged or sent, and the database stays usable. 0.8.0 bounded a commit only by memory. Readers apply every commit a frame holds, whatever their own limit, and a run of parts too long for any commit frame now fails at its first part instead of after it is joined.
• WriteRefusedKind and the error kinds gain too_large. UrsulaWriteRefusedError carries it, and status().refused names it until the next commit is logged. A switch over WriteRefusedKind needs a case for it.
• A read refusal that names the cache's own Stream-Incarnation is decided without a HEAD. Since 0.8.0 the 410 and 416 of a read, and the 404 of a snapshot that is not the latest, name the serving stream's token. When that token is the cache's own, the extension now acts on the refusal directly: a 410 to a synced() probe resets fenced, a 410 to the pre-check read resets stale, and a 416 to a replicator read is resent after backoff. A refusal that names another token or none, and the reads of an open's build and catch-up, still send a HEAD. A malformed or repeated Stream-Incarnation on such a refusal is now a protocol violation and blocks as rejected, like any other malformed typed header. 0.8.0 sent a HEAD whatever the refusal's headers were.
• A provisional reopen of an owner that never wrote goes live. An owner that bound its stream and was closed or crashed before its first commit, then reopened provisionally, reset with stream_state (action fix) once the stream answered, because the stream answered its resent claim as a repeat at the acknowledged offset. It now continues with its next claim, and so does such an owner after an inheriting history event. A repeat whose receipt starts below the acknowledged offset and ends past it still resets stream_state.
• Each open draws its own activation. 0.8.0 drew one activation per process. Each open that starts a binding now draws its own 16 random bytes, as the design states, unless the activation option of ursula_open sets one. A reopen still recognizes its earlier claims by the claim records in its log.
• Writing a commit takes less memory. The commit frame is built in one buffer and the pages are dropped once it is encoded, so the writer holds about the pages plus twice the compressed record. The bytes logged and sent are unchanged.
• Upgrading: check your largest transactions, such as a VACUUM or a rewrite of a large table. Split them, or set URSULA_VFS_MAX_COMMIT_BYTES (at most 1 GiB) on every host that may own the database, with memory for about three times the limit. Commits that 0.8.0 logged above the limit still replicate and are still read.

Known issues

Two known issues of 0.8.0 are fixed. Snapshot GC no longer gives up a pass when a pin it listed is released before it reads it (#490). The snapshot driver now prunes the retired snapshot objects of a group that stopped building (#494), so with storage.snapshot.drive_interval = "0s" a group still prunes only when it builds. The other known issues of 0.8.0 remain open. These were already present in 0.8.0 and are listed for the first time:

• #687: snapshot objects are named by their content, so a build whose bytes match an old object that no replica names reuses that object. If a GC pass deletes the old object between the upload and the build's pin, the new snapshot names an object that is gone, and an install or a restore of it fails with NotFound. The window is short and needs identical bytes, and the shorter grace of 0.8.1 makes the old object's age easier to reach.
• #698: when the bound replacement candidate of a host recovery dies after its group leaders started rebuilding it, they keep it as a learner in those groups, so ursulactl verify-survivors refuses for good and the host restaging cannot proceed without manual work. No acknowledged write is at risk, since the survivors refuse.
• #709: OpenRaft's state-machine worker keeps running for a moment after a group engine stops, and holds the group's log store until it finishes. So the restart that a recovery plan's cut runs can fail at once with GroupAlreadyOpen, or because the core journal's lock is still held, and nothing retries it. The plan then answers 500, and that node's replica of the group stays stopped until the next request to the group opens it again.

0.8.0

0.8.0 is a breaking release. It changes the storage and protocol format to format epoch 4. A 0.8 node refuses epoch-3 Raft WAL directories and object-storage prefixes at startup, refuses 0.7 peers, snapshots and backups, and ursulactl restore refuses 0.7 backups. There is no migration, no in-place upgrade, no rolling upgrade and no rollback.

• If you cannot move your data yet, stay on 0.7.6.
• Deploy 0.8 fresh: empty PVCs and never-used object-storage prefixes, for the cold store, S3 snapshots and the event-time indexer.
• Move data by export and re-import. The applications that own the data read it from 0.7 and write it into 0.8. Ursula provides no converter.
• Set server.cluster_id on every node of a multi-node cluster. A node with raft.peers and no server.cluster_id refuses to start. Use the same value on every node and a different one for every deployment a client can reach. Chart installs must now set server.clusterId, a single replica included: the chart always renders the peer list, and it refuses to render without the id.
• Give nodes with a cold backend at least 64 MiB of server.http_inflight_body_size. A smaller budget refuses to start with InflightBodyBudgetTooSmall. The tiny and small presets set exactly 64 MiB.
• Lower a gateway's --max-request-body-bytes to at most 32 MiB. That is the protocol's request limit, and a gateway configured above it refuses to start. A larger snapshot goes up in parts.
• Clients read write outcomes by one rule. x-ursula-not-applied is gone. A refusal's status and error code say whether its request applied nothing, a leadership loss after a proposal answers 503 with OUTCOME_UNKNOWN, and ursula_headers::classify implements the rule. A proxy in front of Ursula must not answer a write it forwarded with its own 4xx other than 408, or with its own redirect, must not resend a write it forwarded unless the answer proves nothing applied (a server's redirect does), and must not accept a body above the server's request limit (write outcomes).
• Update recovery runbooks. accept-unsynced-loss takes a recovery plan that counts every voter set the replicas report, so it recovers across a rolled-back membership change. A replica whose committed entries a plan dropped waits for an operator to rejoin its node as a wiped node, after checking the other groups on that node (Recovering after a host crash).
• The SQLite VFS has new formats and a new API. The 0.8 extension refuses 0.7 streams and 0.7 servers. Export each database with 0.7 and import it with 0.8, as described under SQLite VFS.

See Upgrading to 0.8.

Write keys and typed answers

Producer writes now get a typed answer to every repeat, so a client resolves an unknown outcome by sending the same write again. See write keys and typed answers.

• x-ursula-not-applied and x-ursula-not-applied-checked are removed. Neither the server nor the gateway sends them. A refusal's status and error code say whether its request applied nothing, by one rule for every client, and ursula_headers::classify implements it. Repeat a write whose outcome is unknown with its producer key. See write outcomes.
• A leadership loss after a proposal answers OUTCOME_UNKNOWN. A node that loses the leadership after it proposes a write now answers 503 with OUTCOME_UNKNOWN, where it used to answer NO_LEADER (or redirect). A node answers NO_LEADER, or redirects, only for a write it refused before proposing it, so a client reads NO_LEADER as a refusal and repeats an OUTCOME_UNKNOWN write with its key. A read never gets OUTCOME_UNKNOWN: it still redirects or answers NO_LEADER. A follower now checks its leadership right before it proposes any write, a delete, a plain create and a batch of appends included, so those still redirect or answer NO_LEADER. A parts commit whose key check after its body meets a leadership refusal answers 503, never a redirect.
• A success without its typed answer is unknown. classify reads a 2xx as applied only with the typed answer Ursula always sends for the write: a producer write's receipt, Stream-Next-Offset of another append or close, Stream-Incarnation of a create or a discard of staged parts, the published snapshot of a publish or a commit, Stream-Snapshot-Part of a part, and Stream-Retained-Offset with Stream-Incarnation of a retention advance. A 2xx without it comes from a layer outside Ursula and says nothing of the write. A delete of a stream and a create of a bucket have no typed answer, so their 2xx holds by its status alone.
• Rust library APIs. ursula_headers gains classify, WriteOutcome and WriteKind. WriteKind is #[non_exhaustive], since a later 0.8 release may add a kind of write, so a match on it needs a wildcard arm.
• Gateway embedders can refuse before forwarding. Gateway::refuse answers a request with PAYMENT_REQUIRED (402), ACCOUNT_REVOKED (403), FORBIDDEN (403) or RATE_LIMITED (429), after reading its body as the gateway's own refusals do, and answers a CORS preflight as Gateway::handle does. Clients read it as a refusal, never as a producer answer. /__ursula/gateway/metrics counts them in embedder_refusals. The gateway no longer uses a system proxy from HTTP_PROXY for its upstreams.
• One receipt per producer. The server keeps the receipt of each producer's newest write only. A repeat of that write answers 204 with its receipt. A repeat of an older write of the epoch answers 204 with Producer-Outcome: receipt-evicted and no range. 0.7 kept the ranges of up to 1,024 earlier writes per stream.
• Producer-Seq on 200 and 204 is the epoch's highest applied seq, the base protocol's meaning. 0.7 echoed the request's seq on a duplicate. Receipts carry Stream-Offset, where the write's bytes start, next to Stream-Next-Offset, where they end, Stream-Closed: true when the write closed the stream, and Stream-Incarnation.
• A claim reports the epoch it closed. The answer to seq 0 of a later epoch, and to every repeat of it, carries Producer-Closed-Epoch and Producer-Closed-Seq: the epoch it closed and that epoch's last applied seq.
• A write of an older epoch always answers 403, even when it applied, with Producer-Epoch and one Producer-Outcome. A write of the epoch that the current claim closed gets epoch-closed;by=claim with Producer-Closed-Epoch and Producer-Closed-Seq, which say whether it applied. Any older epoch gets stale-epoch. A seq above 0 of a later epoch that is not claimed yet answers 400 with Producer-Expected-Seq: 0.
• Producer-Digest makes a write's body part of its identity. It is optional: the BLAKE3-256 hash of the body as 64 lowercase hex digits. The server checks it against the body (400 on a mismatch, nothing applied). Another identity under the key of a producer's newest write, or of its current epoch's claim, answers 403 with Producer-Outcome: key-conflict.
• A forgotten epoch never opens again to a write with Stream-Incarnation. When a stream forgets a producer, after 7 idle days, past the 4,096-producer cap or when it tidies an idle stream, it raises its epoch floor to that producer's epoch. A write with Stream-Incarnation from a producer the stream holds no record for, at or below the floor, answers 403 with Producer-Epoch set to the floor and Producer-Outcome: producer-expired. Claim an epoch above it. A write without Stream-Incarnation keeps the base protocol's rule: a forgotten producer is new at any epoch. A record that such a write opens at or below the floor answers producer-expired to writes with the token until one of them claims above the floor. That claim reports no closed epoch, since the epoch it replaces may be the forgotten one. A live producer's claims are never refused for the floor. Upgrading: a client that sends Stream-Incarnation and starts new producer ids on long-lived streams should be ready to claim above Producer-Epoch on producer-expired, as a base client's auto-claim does.
• Repeats are answered before the body. The leader answers a repeat of an applied write, or a key conflict, from its applied state before it reads the body, when the write carries producer headers and the stream's current Stream-Incarnation, after Expect: 100-continue or for a declared body of at least 64 KiB. A client that sent Expect: 100-continue sends no body. Over HTTP/2 the rest of the body is not read. Over HTTP/1.1 without Expect: 100-continue, the server reads and drops the body, up to 32 MiB, 5 s without data and 30 s in all, and closes the connection past those bounds: the early answer saves the staging and the proposal, not the upload. Smaller writes are read and answered as before. Write admission's pressure checks now run after the early answer, so a repeat gets its answer under load.
• A producer close of a closed stream answers 409. Only a close without producer headers is idempotent on a closed stream. A repeat of the producer's own close still gets its receipt. A close-only producer write that applies answers 204, as the base protocol requires.
• A create with producer headers answers 400. Create the stream, then claim an epoch.
• Producer metrics changed. Each record keeps one receipt, so the per-group state gauges in /__ursula/metrics no longer report receipts, receipt_items or max_receipt_items_per_stream. New: the floored_streams gauge counts streams with an epoch floor, and the producer_expired_answers counter counts writes answered producer-expired.

History identity

Stream-Incarnation now changes whenever bytes the server served at an offset may differ, not only on a delete and recreate. See history rewrites.

• Tokens carry the group's history. Every token is derived from the stream's creation time and a random history id of its Raft group, drawn when the group is created and again at every history rewrite: a restore and an accepted loss. Every token lies in [2^52, 2^53), so no 0.7 token is valid against 0.8: a request carrying one answers 412 with the 0.8 token, or 404. Keep treating the token as opaque.
• A restore and an accepted loss change every token of the group. A write or read that carries the old token answers 412 with the new one. Long-polls and SSE sessions pinned to the old token end: a long-poll answers 412, an SSE stream ends with an error event. An SSE session without a precondition is now pinned to the token its first response announced, so it also ends on a delete and recreate. Read ETags include the token.
• HEAD reports what survived. After a rewrite, HEAD carries Stream-Previous-Incarnation, the stream's token before it, and Stream-Inherited-Offset: the stream's bytes below it are the same under both tokens. A restored stream names its token on the source cluster with the restored tail. A stream the source had not written since a rewrite of its own, such as a restore of a cluster that was itself restored, names the token from before that rewrite, as after any two rewrites with no write between them. A client that finds its own token there keeps what it derived below the offset.
• A rewrite closes every producer epoch claimed before it. A producer's next new write of such an epoch answers 403 with Producer-Outcome: epoch-closed;by=history, Producer-Closed-Epoch and Producer-Closed-Seq, the epoch's last seq that survived. A repeat of a write that survived still gets its receipt. Claim a new epoch and resend what follows Producer-Closed-Seq, as after epoch-closed;by=claim. A producer whose record did not survive, and that claims without Stream-Incarnation, is new on the stream and its claim applies, as the base protocol requires. A producer that sends the token gets 412 instead, so the SQLite VFS never applies a write from before a rewrite.
• A group serves nothing before its history starts. A new group's first leader commits the group's history event before any other write, and a leader that holds a recovery plan does the same. Until then the leader answers writes and reads with 503 and Retry-After, after waiting up to 2 seconds for the event. This takes one round trip after the first election. Every group's log now starts with this entry, so commit indexes in group metrics are one higher than in 0.7 for the same writes.
• Restore is idempotent and keyed by the backup. POST /__ursula/backup/group/{group}/import requires x-ursula-backup-blake3, the BLAKE3 digest of the group's backup object as 64 hex digits (400 when it is missing or does not match the body), and answers a typed body: buckets, streams, group_commit_index, history and already. A repeat of an import that applied answers 200 with "already": true and starts no second history. ursulactl restore sends the digest.
• accept-unsynced-loss takes a recovery plan. The operator reads every reachable replica of a stalled group and computes one plan: the replica to lead, the log index the group continues from, and whether every replica cuts its log back to what it knew was committed. The plan goes to its leader first with the highest vote term the operator read, and the leader answers its plan term: that term or its own vote term, whichever is higher. Then the plan goes to every other replica with the plan term. A replica with a plan votes only for the plan's leader, only in an election above the plan term, and only while the leader's log holds no entry above the plan's point at or below the plan term, until the group's new history event applies. A member that holds a plan takes replication only from a leader above the plan term. The plan's leader campaigns but votes for no other candidate, and while it leads it serves nothing until the event applies. A follow whose plan term is below the vote term its expected names is refused with plan_term_below_vote, and a plan that cuts a group with one voter with single_voter_cut. A replica whose plan cuts its log, or that leads or campaigns when it takes the plan, restarts its group in the process before it answers, and reports cutting meanwhile. A cut that cannot run drops the plan and keeps the gate closed, across restarts too, until a new plan. A replica that leads or follows a live leader, a plan's leader that is no voter of the membership it would campaign under, or a request whose step is not the replica's is refused with 409 and a typed body naming the refusal. A 0.7 request body answers 422. A replica takes one plan at a time: a second call while one runs is refused with plan_in_flight. Group metrics gain voter_configs and committed_voter_configs, the voter sets of the replica's membership and of the one it keeps when a plan cuts its log, history (event_index, imported), recovery (leader, point, leader_term, cut_pending), recovery_gate and leads_live, which says whether a leader's quorum acknowledged it within the last 3 seconds. A gate held by a plan reports planned, or leads_plan on the plan's leader. ursulactl recover-group explains the plan for a group and sends nothing unless --accept-loss is given. With it, it sends the plan in this order and waits until it is satisfied. See Recovering after a host crash.
• A plan whose leader lost entries it had replicated wedges until a second plan. This happens, for example, when the plan's leader crashes right after its election and loses its first entries, which its members already hold. The members then elect no one. After 30s without applying anything, every replica that holds the plan reports its group stalled (recovery_stalled, recovery_gate: stalled). Plan again. ursulactl recover-group --accept-loss plans again by itself when it sees, in two observations in a row, a member's log ahead of the plan's leader's, or a stalled replica that holds the plan.
• A plan counts every voter set the replicas report. A membership change that only a minority appended before the loss, such as a promotion of a new voter, leaves the replicas reporting different voter sets. So does a committed change above a replica's commit marker, which is not synced on its own. A plan needs a majority of each set, and never counts a replica whose group stopped. Its leader must be a voter of the membership it campaigns under: its own, or with a cut the one it keeps after the cut, which the server reads from its log and committed_voter_configs reports. Only the plan's leader checks that, and refuses with leader_not_voter. A plan's leader whose cut leaves it no vote, or whose restart never shows its membership, drops the plan and answers cut_left_no_vote or cut_membership_unknown. Other replicas no longer refuse a leader outside their own membership, because the leader counts their votes under its membership. Before this, the replicas without a rolled-back promotion refused the promoted voter as the plan's leader, and ursulactl recover-group stopped with no majority. leader_not_voter and a member's recovery_point_behind now make it read the replicas again and plan again, and once the plan is satisfied it names the replicas outside the leader's membership and those whose group stopped.
• A plan never continues below a reached replica's commit. The tool refuses a plan whose point is below a committed or applied index that any replica it reached reports, learners, stopped and excluded replicas included. When a learner knows a later committed entry than every voter's marker, and a voter's log still holds that entry, the plan cuts every log back to that entry instead of to the voters' markers, and its leader checks that it holds exactly that entry. A replica refuses an append or a snapshot install that would replace entries at or below its commit marker. Each history event now records the history it replaced and the last index it kept from it, in the log and in snapshots. A snapshot of another history than the one a replica's commit marker lies in installs only if it directly replaced that history and kept every entry up to the marker. A replica that no plan reached while two or more plans ran in a row refuses the newer history and waits. Its committed entries above the plan's point are gone, and an operator rejoins it as a wiped node, once no other group on the node depends on it (see the operations guide). The history event also records the term and node of the leader that proposed it, so a replica that saw that leader's first entries committed, but not the event, still takes its snapshot when the event came in that leader's term. When the event was proposed again at a later term, the replica waits for an operator.
• A replica holding a plan serves no client read. Until it applied the plan's history event, a replica that does not lead answers catch-up and live reads, HEAD, snapshots and forwarded reads with 503 and Retry-After at once. Usage and metrics keep answering. The plan's leader whose vote term is below the plan term restarts its group and starts with the plan term as its vote, so it no longer climbs to it one election at a time.
• Upgrading: clients that keep offsets, producer state or caches keyed by Stream-Incarnation must expect the token to change without a delete, and should read Stream-Previous-Incarnation and Stream-Inherited-Offset on HEAD after a 412 to keep what survived. A single-voter group under fsync = "never" takes a new history after every unclean restart that needs a plan, so use fsync = "always" for single-voter groups whose clients keep state. Recovery runbooks and scripts that sent expected_last_log_index and expected_current_term must send a plan instead. Tools that call the import route directly must send x-ursula-backup-blake3.

Server identity, routing and request limits

• Every HEAD answer names the protocol and the deployment. It carries Ursula-Protocol: 2 and Ursula-Cluster-Id, 404, 307 and 503 included, so a client tells a host alias routed to another deployment from a deleted stream. The id is the new server.cluster_id setting (Helm server.clusterId), 1 to 64 characters from A-Z, a-z, 0-9, ., _ and -. A single node without it generates u followed by 32 hex digits at its first start. The WAL records the id at its first start, and a node whose configured id differs refuses to start, so a disk cannot move to another cluster unnoticed. The chart requires server.clusterId. See server metadata.
• Peers check the cluster id. Every Raft gRPC request carries the sender's cluster id and node id as request metadata, and a node refuses a peer of another cluster with FAILED_PRECONDITION before any Raft RPC runs, so a node of another cluster can no longer move this cluster's Raft terms. Both sides count the refusals in raft_grpc_peer_cluster_refusals (OTLP ursula.raft.peer_cluster_refusals), log each peer once with its address, and list refused peers of raft.peers under peer_cluster_mismatches in /__ursula/ready and /__ursula/metrics. A refusal never changes readiness or its reason, and a GET /__ursula/leadership-shed answer, which now carries cluster_id, counts only when the id matches the poller's. The cluster id is public (every HEAD carries it), so the check guards against misconfiguration, not a hostile sender. A forwarded read, HEAD or purge refused by a leader of another cluster answers 503 with Retry-After. A request without a cluster id, from a 0.7 node or a stray client, gets a format-epoch refusal and is counted in raft_grpc_unidentified_peer_refusals, but no longer makes the receiving node unready: only a node's own contact with a configured peer of another format epoch does. A node whose WAL holds another cluster id now refuses to start before it contacts any peer. /__ursula/ready and /__ursula/metrics carry the node's cluster_id, and OTLP exports it as the cluster_id label of ursula.cluster.info. See Peers of another cluster.
• HEAD advertises the request and snapshot limits. Every successful HEAD of a stream carries Ursula-Max-Request-Bytes: 33554432, the largest request body any request to the stream may carry, and, on a node with a cold backend, Ursula-Max-Snapshot-Bytes: 1073741824, the largest snapshot it accepts in parts. The gateway lowers Ursula-Max-Request-Bytes to its own limit for the request, the smaller of --max-request-body-bytes and the tenant's max_request_body_bytes, on every response that carries it. See server metadata.
• Writes are routed before their body. A node that does not lead the stream's group answers a create, an append or close, or a snapshot publish with 307 to the leader before it reads any of the body, so a follower no longer reads or stages a misdirected upload. The redirect carries x-ursula-raft-leader-id as before, and now x-ursula-raft-group and x-ursula-raft-term. With no leader known the node answers 503 with Retry-After: 1 and x-ursula-raft-group. A node that hosts no replica of the group redirects to one that does with x-ursula-raft-group alone, where 0.7 sent x-ursula-raft-leader-id naming that replica. A node never redirects a write after it read the body: one that lost the leadership meanwhile answers 503 with Retry-After, where 0.7 could redirect a write it had proposed. The 503 is NO_LEADER when the node refused the write before proposing it and OUTCOME_UNKNOWN when it may have proposed it. Write admission's pressure checks run after routing, so a follower under disk or Raft-log pressure redirects instead of refusing. A node routes a write from its own view of the group's leadership, which can lag an election by a moment: a write it routes to itself while it no longer leads answers 503 after the body. See routing answers.
• Every 413 names the limit it applied. A node's and a gateway's 413 carry Ursula-Max-Request-Bytes, or Ursula-Max-Snapshot-Bytes for a snapshot assembled from parts above 1 GiB, set to the limit. 0.7 sent a bare 413 with a text body. A node refuses a declared Content-Length above the limit before it reads the body, and a snapshot publish above 32 MiB on a node without a cold backend is now refused from its Content-Length too, where 0.7 read 32 MiB of it first. A tenant's max_request_body_bytes now stops the gateway's reading at that limit instead of at --max-request-body-bytes.

Network planes

• The chart serves Raft on its own port. server.clusterListen defaults to true: every voter renders server.cluster_listen = "0.0.0.0:4439" (server.ports.cluster), each generated [[raft.peers]] entry names that port in url, and the headless peer Service gains a cluster port. The client Service still exposes only the client port, so a caller that reaches it no longer reaches Raft. Without the NetworkPolicy below, pods can still dial a voter's pod IP on the cluster port. server.clusterListen=false serves both planes on the client port, as 0.7 charts did. See Network planes.
• Peers have a client_url. Leader redirects name a peer's [[raft.peers]].client_url, or its url when the entry has none. With server.cluster_listen set, every peer needs a client_url that differs from its url, or the node refuses to start, and a leader that [[raft.peers]] does not list gets no redirect: the follower answers 503 with Retry-After. 0.7 redirected clients to the peer url, which named the cluster listener when server.cluster_listen was set, and that listener serves no client route.
• The egress probe moved to the cluster plane. POST /__ursula/cluster-probe is served on the cluster plane only. 0.7 served it on the client plane while peers sent it to their peer url, so a 0.7 node with server.cluster_listen set saw every probe fail and shed all its leaderships.
• The chart can render a NetworkPolicy. server.networkPolicy.enabled (off by default) admits only the release's voters to the cluster port, and server.networkPolicy.clientIngressFrom (every source when empty) plus the release's gateway, indexer and Helm test pods to the client port. It refuses to render with server.adminListen off loopback. The Helm test pod's name label is now <chart>-test, with the release label.
• ursulactl reads metrics at client_url, never at http_url. A manifest node gains client_url, the node's client-plane URL. Metrics and readiness use metrics_url, then client_url, then admin_url. 0.7 fell back to http_url, the Raft address, which serves no metrics on a node with server.cluster_listen. A manifest that relied on http_url for metrics must add client_url. In the chart's cluster-manifest.json and the graceful-rollout manifest, http_url names the cluster port, and the chart's manifest sets client_url. The graceful-rollout job takes CLUSTER_PORT in place of CLIENT_PORT. It also takes CLUSTER_DOMAIN and HEADLESS_SERVICE from the chart, so the learners it attaches carry the same peer address as [[raft.peers]]. It used to hardcode cluster.local.
• Upgrading: 0.8 installs fresh (see above), so no running cluster moves its peers to the new port. Each voter's Raft membership records its peers' url when a group is first initialized, so choose server.clusterListen and server.ports.cluster before the first install and do not change them on an initialized cluster. A node whose url differs from the address its groups recorded for it refuses to start and names both, which also catches a pre-release 0.8 install that the new chart default would move to the cluster port. Keep the gateway's --upstream URLs on the client port. Outside the chart, set server.cluster_listen and client_url together on every node of a fresh cluster.

Error bodies

• Every error answer has one JSON body. Every 4xx and 5xx from a node or from the gateway carries Content-Type: application/json and {"error":{"code":"<CODE>","message":"<detail>"}}. It is the shape the Durable Streams protocol uses for its subscription API errors. The protocol's stream operations define no error body. error is now an object, so a 0.7 client that reads error as a string breaks. The code is stable and names the refusal. A 4xx message is the detail 0.7 sent as plain text, such as which header was malformed and why, for the refusals that name only what the request named. Every other answer carries a fixed message per code: 0.7's text named node and peer addresses and the cold store's keys and endpoints, and the gateway passed it to tenants. The message is informative only and may change in any release, so match the code. Clients ignore members and treat codes they do not know by the status, so later 0.8 releases can add both. A live SSE read's event: error carries the same body. Error answers are never compressed. The gateway's own answers now carry Cache-Control: no-store, as a node's do, so a cache in front of the gateway keeps no refusal, such as a concealing 404 (#640). The typed headers are unchanged. The operator routes under /__ursula/ keep their own answers. ursula_headers::ErrorBody parses the body and ErrorCode lists the codes. See error bodies. What a 0.7 client saw and what 0.8 sends:

  0.7	0.8
  503 {"error":"WalDiskPressure"}	503, code WAL_DISK_PRESSURE
  503 {"error":"RaftLogPressure"}	503, code RAFT_LOG_PRESSURE
  503 {"error":"IngressBodyBytesLimitReached"}	503, code INGRESS_BODY_BYTES_LIMIT_REACHED
  429 with a text body starting producer_limit	429, code PRODUCER_LIMIT
  413 with a text body	413, code REQUEST_TOO_LARGE or SNAPSHOT_TOO_LARGE
  503 with text when no leader is known	503, code NO_LEADER, only for a write refused before its proposal
  503 with text, or a redirect, when a node lost the leadership after it proposed a write	503, code OUTCOME_UNKNOWN: the write may have applied
  500 with text for a failed cold-tier read on a read or a bootstrap	502, code COLD_STORE_FAILED
  503 with text when a write needs a cold backend the node lacks	501, code NO_COLD_STORE
  any other plain-text 4xx or 5xx of a node	its code in error bodies, such as STREAM_NOT_FOUND or STREAM_SEQ_CONFLICT
  gateway 401 with no body	401, code INVALID_CREDENTIAL or CREDENTIAL_EXPIRED
  gateway 403 with no body	403, code FORBIDDEN
  gateway 404 with no body for a route it does not serve	404, code NOT_FOUND
  gateway 404 with no body for a resource its authorizer conceals	404, code BUCKET_NOT_FOUND, the node's answer for a bucket that does not exist
  gateway 429 with text	429, code RATE_LIMITED or LIVE_READS_EXHAUSTED
  0.7.6 gateway 429, 503 and 408 of its body budget, with the token in x-ursula-not-applied	429, code UPLOADS_EXHAUSTED or UPLOAD_BYTES_EXHAUSTED, 503, code UPLOAD_BUDGET_EXHAUSTED, and 408, code BODY_TOO_SLOW
  gateway 503 with no body or text	503, code AUTHENTICATION_UNAVAILABLE, AUTHORIZATION_UNAVAILABLE, NO_UPSTREAM, UPSTREAM_UNREACHABLE or NO_LEADER
  gateway 502 and 504 with text	502, code UPSTREAM_FAILED, and 504, code UPSTREAM_TIMEOUT
  SSE event: error with text	the same event with the JSON body

  The snapshot commit 503s that 0.8 adds are SNAPSHOT_COMMITS_BUSY and SNAPSHOT_COMMIT_IN_PROGRESS.

• A Stream-Seq conflict names the stream's last seq. Its 409 carries the code STREAM_SEQ_CONFLICT and the new header Stream-Last-Seq, the stream's last accepted Stream-Seq, so a writer using it as a compare-and-set retries from that value. A content-type mismatch 409 carries CONTENT_TYPE_MISMATCH and no Stream-Last-Seq, so the two no longer look alike. ursula_headers::StreamSeqConflict parses it. See conditional writes.

Gateway request body budget (carried from 0.7.6)

0.7.6 bounded the request bodies the gateway buffers. 0.8 keeps that behaviour, its flags, its chart value and its defaults. The refusals' wire form follows 0.8's error bodies, and --max-request-body-bytes is now at most the protocol's 32 MiB (see below).

• The gateway bounds the request bodies it buffers across requests. Before it reads a body, a request reserves the body's Content-Length from a node-wide budget, --body-budget-bytes (256 MiB by default), and holds it until the gateway has forwarded the body. The gateway copies each body into one buffer of that size as it arrives, so the memory a body holds stays within its share however small the pieces it arrives in. A body without a declared length reserves --small-body-max-bytes (1 MiB by default) at first, and more as it arrives. --small-body-headroom-bytes (32 MiB by default) of the budget is kept for bodies of at most --small-body-max-bytes, so SQLite commits do not wait behind snapshot uploads. A request that finds the budget spent waits --body-budget-wait-ms (1000 by default) and then gets 503 with Retry-After: 1 and the code UPLOAD_BUDGET_EXHAUSTED.
• --max-request-body-bytes is at most 32 MiB. That is the protocol's request limit (ursula_headers::MAX_REQUEST_BYTES), which every node applies. A node refuses a larger body from its Content-Length before it reads it, so a gateway that forwarded one saw a reset and answered 502 UPSTREAM_FAILED, which a client reads as an unknown outcome and resends forever. A gateway configured above 32 MiB now refuses to start with LimitAboveProtocol, and the chart refuses such a gateway.maxRequestBodyBytes. Upgrading: a 0.7 gateway configured above 32 MiB must lower --max-request-body-bytes (gateway.maxRequestBodyBytes) to at most 33554432 before it runs 0.8. A larger snapshot goes up in parts.
• The body budget follows --max-request-body-bytes and the headroom. With --body-budget-bytes unset, the budget is 256 MiB or the limit plus the small-body headroom, whichever is larger, so a gateway whose headroom was raised keeps starting, with a warning. Give the gateway about twice its budget in memory. The budget cannot be set below the limit plus the headroom, so if that is more than the pod's memory can hold, lower --max-request-body-bytes (the chart's gateway.maxRequestBodyBytes) and set --body-budget-bytes (gateway.bodyBudgetBytes) with it, or give the pod more memory. At startup the gateway also warns when its budget, set or grown, exceeds half of its cgroup memory limit, which it looks up along its own cgroup path. A budget set explicitly below the limit plus the headroom makes the gateway exit at startup with LimitAboveSharedBudget, and the chart refuses such a gateway.bodyBudgetBytes. The chart renders --body-budget-bytes only when the value is set. The other flags (--small-body-headroom-bytes, --small-body-max-bytes, --body-budget-wait-ms, --body-stall-timeout, --body-min-bytes-per-sec) have no chart value and go in gateway.extraArgs.
• The gateway cuts bodies whose client abandoned them. It cuts a body once no byte of it arrived for --body-stall-timeout (30 s by default), or once less than --body-min-bytes-per-sec (1 KiB/s by default) arrived over one such window, and answers 408 with the code BODY_TOO_SLOW, with nothing forwarded. Nothing bounds a reading in all, so a slow upload that keeps arriving is never cut, and with the defaults the gateway never cuts a commit or a snapshot the SQLite extension still waits for. Over HTTP/1.1 the connection closes after the 408. Keep --body-stall-timeout above the extension's 10 s commit timeout. The gateway warns at startup otherwise.
• A quota policy can bound each bucket's uploads. max_concurrent_uploads and max_upload_bytes, under [default] or a [[bucket]] of --quota-policy, bound the bucket's bodies over --small-body-max-bytes while the gateway reads them. A request past either gets 429 with Retry-After: 1 and the code UPLOADS_EXHAUSTED or UPLOAD_BYTES_EXHAUSTED. A body stops counting once the gateway has buffered it, and smaller bodies never count. A policy that sets neither limit, and a gateway without access control or a quota policy, answer no such 429.
• The refusals above reach a client that sends its body first. Like the refusals the gateway decides from the head, each is sent once the gateway has read the body to its end and dropped it, here at the pace of a forwarded body. A body declared over its bucket's own max_request_body_bytes gets its 413 before it reserves any of the budget. Only a body past --max-request-body-bytes still gets its 413 with the rest unread. Every 413 names the request's limit in Ursula-Max-Request-Bytes.
• Gateway metrics. /__ursula/gateway/metrics reports upload_refusals_budget, upload_refusals_tenant_uploads, upload_refusals_tenant_bytes and upload_refusals_too_slow, the refusals above by reason, upload_budget_bytes, the budget in effect, upload_reserved_bytes, the budget reserved now (each body's share in whole KiB), upload_tenants, the buckets with a large body under way under an upload limit, and body_bytes_buffered, the bytes of bodies it read to forward them. Its refusal_drains and refusal_drains_cut count the refusals above too, and refusal_drain_bytes counts the bytes of refused bodies it read and dropped.
• Rust library APIs, as in 0.7.6. In ursula-gateway, GatewayConfig::max_request_body_bytes moved into the required field uploads, an uploads::UploadLimits that only UploadLimits::try_from(UploadLimitsSpec) builds, failing with UploadLimitsError. admission::TenantLimits has max_concurrent_uploads and max_upload_bytes, AdmissionRejection has UploadsExhausted and UploadBytesExhausted, and Admission::admit_upload and UploadGuard exist. ursula_headers::ErrorCode gains UploadsExhausted, UploadBytesExhausted, UploadBudgetExhausted and BodyTooSlow.

Two known issues of the body budget remain in 0.8:

• A few slow clients can hold the gateway's body budget. A request reserves its declared length before it sends the body, and the gateway gives the reservation back only when the body stalls or arrives below --body-min-bytes-per-sec. With the defaults, about seven connections that each send 1 KiB/s can hold a 256 MiB budget, and other large bodies then get 503 with UPLOAD_BUDGET_EXHAUSTED. That is the cost of never cutting a slow upload that keeps arriving. The per-bucket upload limits are keyed by bucket only, with no limit per principal, and without access control there is no limit per client address.
• One bucket's small bodies can take the whole small-body headroom. Every bucket shares the headroom, so it keeps small writes from waiting behind large uploads, but not one bucket's small writes from another's.

Snapshots

• One request limit for every request. Every request body is at most 32 MiB (Ursula-Max-Request-Bytes), snapshot publishes included. 0.7 accepted a snapshot of up to 1 GiB in one request on a node with a cold backend. Such a publish now answers 413 with Ursula-Max-Request-Bytes, from its Content-Length before the body is read.
• Snapshots above the request limit go up in parts. On a node with a cold backend, POST {stream}/snapshot/{offset}/parts stages one part and answers 201 with its id in Stream-Snapshot-Part. A PUT {stream}/snapshot/{offset} with Stream-Snapshot-Parts and Stream-Snapshot-Digest, whose body lists the part ids one per line, commits them: Ursula copies the parts into one snapshot of up to 1 GiB, checks the digest and publishes it. A commit names at most 4,096 parts. It answers 400 for an unknown part, another stream's part, a part listed twice or a digest that differs, and 413 with Ursula-Max-Snapshot-Bytes above 1 GiB. A commit is answered from its offset and digest before its body is read when the latest snapshot decides it: 204 for the same snapshot, 409 naming the latest for another one, even after its parts are gone. Ursula deletes the parts once the commit's answer is final or its copy found a digest mismatch, keeps them after a 503 or a 400 that copied nothing, and its orphan sweep reclaims unused parts after a day, the parts of deleted streams included. The node owns a commit once it copies, so a client or gateway that gives up cancels nothing: a repeat answers 503 SNAPSHOT_COMMIT_IN_PROGRESS while it runs and 204 after. A node copies at most two commits at once (503 SNAPSHOT_COMMITS_BUSY past them, counted by snapshot_commits_busy in /__ursula/metrics). The orphan sweep now lists each bucket's part directory at the start of every cycle, one more LIST request per bucket and group, at most 4,096 parts per directory per cycle. Without a cold backend, parts and commits answer 501. The gateway authorizes a part as a snapshot publish. See upload in parts.
• A stream's unused parts are bounded. A stream holds at most 8,192 staged parts that no commit used, and at most 2 GiB of them, the parts of its earlier incarnations included until they are deleted. A part past either bound answers 429 Too Many Requests before its body is read, with the error code STAGED_PARTS_LIMIT, Stream-Incarnation and Ursula-Max-Staged-Parts or Ursula-Max-Staged-Bytes naming the bound, and no Retry-After. A commit frees its parts once it has deleted them, the new DELETE {stream}/snapshot/parts discards them at once, a delete of the stream deletes them after it answers, and the orphan sweep reclaims the rest. The gateway authorizes a discard as a snapshot publish. A node now runs one commit per stream incarnation at a time: another commit of the stream, not only a repeat, answers 503 SNAPSHOT_COMMIT_IN_PROGRESS. The new snapshot_parts_staged_limit counter in /__ursula/metrics counts the 429 answers. See staged parts.
• Usage reports staged snapshot parts. /__ursula/usage gains node_id, raft_group_count and staged_snapshot_parts: for each group the node leads, its term, counting_finished, and the bytes of staged, unused parts per bucket. A new leader seeds the totals from one listing of the group's part directories in the background, then keeps them up to date. A collector keeps the highest term per group and treats the total as partial while a group has not finished counting or no report names it. /__ursula/metrics and the OTLP gauge ursula.snapshot_parts.staged_bytes report the node's sum. Staged parts do not count toward the other usage fields. Bounding or billing them is the operator's policy.
• A refused publish names the latest snapshot. A 409 to a snapshot publish carries the latest snapshot's Stream-Snapshot-Offset and Stream-Snapshot-Digest: a later offset means the publish was superseded, the same offset means other content is published there. A 409 without them means the offset is beyond the tail.
• A node with a cold backend needs server.http_inflight_body_size of at least 64 MiB. A snapshot commit's copy is charged against it, 32 MiB at the least, and starts only while one request of 32 MiB still fits beside it, so a 64 MiB node copies one commit at a time. A smaller budget is refused at startup. The tiny and small presets set exactly 64 MiB.
• Retention only moves forward. A retention request at or below the current retained offset answers 204 with the current Stream-Retained-Offset and changes nothing. 0.7 answered 409.
• Snapshot reads honor one byte range. A GET {stream}/snapshot/{offset} with Range: bytes=a-b, bytes=a- or bytes=-n answers 206 with Content-Range and only those bytes, and a range past the end answers 416 with Content-Range: bytes */total. Several ranges, a malformed range or If-Range get the whole snapshot. 404, 410 and 412 come before the range. The snapshot headers and the digest describe the whole snapshot on a 206 too. Snapshot answers carry Accept-Ranges: bytes and are no longer compressed. Stream reads ignore Range.

Immutable reads

• Two new read resources that a cache can store. GET {stream}/incarnation/{token}/chunk/{offset} returns the stream's bytes from offset to the next multiple of 4 MiB, or to the tail, and GET {stream}/incarnation/{token}/snapshot/{offset} returns the latest snapshot, with byte ranges. The token in the path is the read's Stream-Incarnation precondition (412 when it is not the current one, after 404, 410 and 416), and the offset is the 20-digit form the server sends. A query, a malformed token or a malformed offset answers 400. See immutable reads.
• Stream-Immutable: true marks what a cache may keep. It is on a chunk that ends at its 4 MiB boundary below the tail, on the chunk at the tail of a closed stream (with Stream-Closed: true), and on every snapshot answer of the new resource, 206 and 304 included. A chunk that holds the tail of an open stream is unmarked: continue there with an ordinary read. Every chunk answer carries Stream-Offset, the offset it starts at, and none carries Stream-Up-To-Date or Stream-Cursor. A follower answers a chunk only when it can mark it, and leaves the others to the leader. The leader answers a chunk it cannot mark, and any read that would answer 404, 410 or 416, only after a ReadIndex round, so a just-elected or deposed leader never answers short of a chunk another node marked, nor for a stream a recreate replaced.
• Every refusal of the immutable routes is a typed error answer. A query answers 400 QUERY_NOT_ACCEPTED, a malformed token or offset 400 INVALID_RESOURCE, the earlier query form 400 UNSUPPORTED, another method 405 METHOD_NOT_ALLOWED and an unknown kind 404 NOT_FOUND, each with Cache-Control: no-store, so no cache in front stores it.
• Nodes still send Cache-Control: no-store and never Stream-Read-Origin. A layer in front decides what a cache stores and where the read origin is. These answers are never compressed. A restore or an accepted loss changes the token, so a cache that kept a marked answer never serves it for the new history, and nothing needs purging.
• The query form of the unreleased design answers 400. incarnation and aligned on a stream read (catch-up, long-poll and SSE) and incarnation on GET {stream}/snapshot/{offset} answer 400, naming the path form, instead of being ignored.
• ETags. Snapshot answers now carry an ETag, the snapshot digest in quotes, and answer If-None-Match with 304 before any range. Stream reads and chunks carry an ETag in a new form, a BLAKE3 hash of the token, the range, where the read ended, the content type and the body, which every node and release computes alike for one answer. Closing a stream no longer changes it below the tail. An ordinary read at the open tail is tagged differently by a follower and by the leader, since only the leader can say the read is at the tail. An ordinary read the server may gzip (JSON or NDJSON, at least 256 bytes) gets a weak tag (W/"…"), because one strong tag must not cover a compressed and an uncompressed body. If-None-Match is compared weakly. Chunks are never compressed, so their tags stay strong.
• Volume-level restores are unsupported. Restore with ursulactl restore only. A cluster brought back from disk or EBS snapshots of its WAL or data keeps its tokens while its tails go back, so clients and caches take later bytes for those they hold. See backup and disaster recovery.
• 410 and 416 name the token. Every 410 and 416 of a read (catch-up, long-poll, the immutable resources and an unsatisfiable snapshot range) and the 404 of a snapshot that is not the latest carry Stream-Incarnation with the token of the stream that answered, which a follower keeps when it forwards the leader's answer. These come before the precondition, so a client tells a replaced stream from a real overshoot or a trimmed offset without a HEAD. A missing stream carries none.
• No 404, 410 or 416 from a state that lags. A follower forwards a read that would answer 404, 410 or 416 to the leader, and the leader answers one only after a ReadIndex round, as it does for a chunk it cannot mark. A just-elected leader that has not applied a recreate therefore never answers for the stream before it.
• Ordinary reads change on a leader that cannot reach a quorum. A catch-up read or an offset=now read that would answer 404, 410 or 416 now answers 307 to the leader or 503 with Retry-After while the leader cannot confirm its leadership, where 0.7 answered from its own state. An offset=now read with consistency=local confirms a missing stream with a linearized HEAD before it answers 404. Reads that succeed are unchanged and take no extra round.
• SSE sends Cache-Control: no-cache, no-store. The base protocol asks an SSE answer for no-cache, and no node answer may be stored, so it carries both, where 0.7 sent no-cache alone.

SQLite VFS

The SQLite VFS is redesigned for 0.8. A commit is local: the extension appends it to a log on the same host and replicates the log to the stream in the background, one append at a time, resent unchanged until the stream answers it. await db.synced() is the output gate. See SQLite on Ursula and docs/architecture/sqlite-vfs.md.

• New formats. Stream frames are USQ2 under the content type application/vnd.ursula.sqlite-vfs.v2, snapshot bodies are USS3, and the local cache is a generation directory with a USL1 commit log. Every append, a claim included, carries Producer-Id: sqlite-ursula-vfs/2, Producer-Digest and Stream-Incarnation. The 0.8 extension refuses 0.7 streams (stream_format) and 0.7 servers (server_too_old), and never reads 0.7 caches.
• New API. open() is asynchronous and the extension owns its files. synced(), status() and the reset listeners replace attach, setToken, status(file), drainStats and instrument. Four error classes replace six, and network failures never throw. importSqlite and ursula_import import a plain SQLite file as the first snapshot of a new stream.
• Outages are lag. Commits continue while the stream is unreachable, and writes fail with backlog_full only past maxPendingBytes. There is no poisoned state, no outcome_unknown and no retry budget. A trusted cache opens provisionally on an unreachable stream.
• History changes keep or salvage. A cache whose history the stream kept is kept under the new token, with no reset. Otherwise it is moved aside into salvage/ as a plain SQLite database. Another cluster id is refused (wrong_cluster).
• Large commits and snapshots. A commit above the request limit goes out in parts, and a snapshot above it goes up in parts with a cold backend. URSULA_VFS_MAX_FRAME_BYTES becomes URSULA_VFS_MAX_REQUEST_BYTES, and URSULA_VFS_RETRY_MS and URSULA_VFS_SNAPSHOT_MIN_BYTES are gone.
• Pi Durable >=1.0.0 <1.4.0. From Pi 1.2 a commit refused with backlog_full waits for synced() and runs again.
• Upgrading: export each database with 0.7 using VACUUM INTO, then import the file into a new stream with 0.8 (importSqlite or ursula_import). Keep the 0.7 cache until the import is checked. See Moving from 0.7.

Known limits of 0.8.0:

• Commit size is bounded by memory. Capture holds the whole transaction, and a build holds a joined commit of parts twice. Bounded by the commit limit in 0.8.1.
• Two history events with no write between them, plus a crash, a clean close or a reset between a re-key's log roll and its claim, move aside a cache that lost nothing.
• Offline commits after a history event. Commits logged before an open and after its latest acknowledgement are moved aside after any history event, since they may have been sent. A clean close records an exact acknowledgement, and commits made while provisional still replicate.
• Idle expiry. An owner idle past producer expiry whose own claim the extension cannot find on the stream resets stale. Reopen it.
• No resume after a memory restore. A process restored from a memory snapshot is safe (a copy that diverges resets fenced before anything it wrote is acknowledged), but nothing checks its memory against its disk, and each restored copy needs its own disk.
• Transport. HTTP/1.1 only, so a retry resends the whole body. An IPv6 path that connects and then stalls fails every attempt: use an IPv4-only host name. Each database uses two HTTP threads and a snapshot thread.
• One token per process, and no separate read origin.
• Snapshots are not published above 1 GiB compressed, or above 32 MiB without a cold backend.
• Local durability. A power loss drops unacknowledged commits since the last log fsync. Network filesystems and a fork after open are unsupported.
• A misrouted host name on a fresh host opens the other deployment's stream, since there is no cache yet to compare its cluster id with.
• Import. Opens of a stream fail with stream_state while an import into it runs, and a failed import leaves a stream that refuses opens until it is deleted or recreated.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests show that a failed write of the poisoned status fails closed.
• #461: a voter restarted without a graceful drain wins elections for dozens of groups within seconds of starting. For about 5 s every node then answers /__ursula/ready with 503 raft_maintenance_unready, and clients get 503s. Graceful rollouts avoid it, because the new pod starts drained, except a reserved rollout (server.gracefulRollout.maintenanceReservation) without server.startupOwnership, whose new pod starts undrained.
• #465: replaying the WAL at startup takes longer as the WAL grows, 17 s for 1.2 GB in one test, so in a test of repeated pod deletions over three hours, the recovery after a deletion grew from about 25 s to between 40 and 74 s.
• #490: a pass that prunes retired S3 snapshots prunes nothing when a pin it listed is released before it reads it, and logs failed to prune retired OpenRaft snapshots with NotFound, so retired snapshot objects can stay in S3 past their hour of grace. No data is at risk.
• #494: a group prunes its retired S3 snapshots only when it builds a snapshot, so a group that stops taking writes keeps the snapshots it retired in its last busy hour until it builds again.
• #496: under moderate load, leaders log raft append stream exceeded the OpenRaft hard TTL for replication and heartbeat calls to followers. No commit failed where this was seen, but under a heavier load a busy group could elect a new leader.
• #517: a group sends each follower one Append call at a time, and since 0.7.2 a call carries at most 4 MiB of entries, so one group replicates to a follower at most about 4 MiB per round trip. That can slow the catch-up of a very busy group over a link with a long round trip.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described under 0.7.2.
• #539: a node answers every leadership transfer it refuses with the gRPC status FAILED_PRECONDITION, also when its Raft core has stopped, so the sending leader cannot tell a stopped core from a refusal by policy.
• #561: the snapshot driver's gauges are reported only at /__ursula/metrics, not over OTLP. In the validation runs of 0.7.4 with two builds, /__ursula/ready answered 503 for apply_lag in one or two 5 s samples per run, too briefly to fail a probe that needs three failures in a row.

0.7.6

0.7.6 is a drop-in patch for 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4 and 0.7.5: format epoch 3 is unchanged, with no storage, snapshot, WAL or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5 and 0.7.6 voters run side by side during the upgrade. 0.7.6 changes only the gateway. With unchanged values, the 0.7.6 chart renders the same server and gateway configuration as the 0.7.5 chart. From 0.7.0, a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.6 together with the chart, as for 0.7.1. From 0.7.4 or earlier, the upgrade notes of 0.7.5 apply too, from 0.7.3 or earlier those of 0.7.4, and from 0.7.2 or earlier those of 0.7.3.

Before the upgrade, check the gateway's memory. The gateway now bounds the request bodies it buffers at once with a body budget, 256 MiB by default, and may buffer that much. If you raised --max-request-body-bytes (the chart's gateway.maxRequestBodyBytes), an unset budget grows to that limit plus the 32 MiB small-body headroom, and the gateway warns at startup. Set --body-budget-bytes (gateway.bodyBudgetBytes) to at least that grown value, and give the gateway about twice its budget in memory. A budget set below the limit plus 32 MiB makes the gateway exit at startup, and the chart refuses it. If the pod cannot hold twice that, lower --max-request-body-bytes or give the pod more memory. A gateway before 0.7.6 exits on --body-budget-bytes and on the other new flags, so unset gateway.bodyBudgetBytes and remove those flags from gateway.extraArgs before you roll a gateway back.

Clients can now get three new refusals from the gateway, each marked as one that applied nothing: 503 UploadBudgetExhausted with Retry-After: 1 when the budget stays spent, 408 RequestBodyTooSlow for a body its client abandoned, and 429 TenantUploads or TenantUploadBytes with Retry-After: 1 where a quota policy sets the new per-bucket keys max_concurrent_uploads or max_upload_bytes. The SQLite extension takes each as a refusal that applied nothing, so none of them poisons a file, and SQLite VFS owners need no upgrade: 0.7.6 changes neither the extension nor @tonbo/sqlite-ursula. These changes are described below.

• Rust library APIs changed. In ursula-gateway, GatewayConfig::max_request_body_bytes moved into the new required field uploads, an uploads::UploadLimits that only UploadLimits::try_from(UploadLimitsSpec) builds, failing with UploadLimitsError. admission::TenantLimits gains max_concurrent_uploads and max_upload_bytes, so code that builds it with a struct literal must set them, AdmissionRejection gains UploadsExhausted and UploadBytesExhausted, and Admission::admit_upload and UploadGuard are new. None of these changes touches the wire or the disk.

Gateway

• The gateway bounds the request bodies it buffers across requests. It buffered each body up to --max-request-body-bytes, with no bound across requests, so concurrent or slow uploads could exhaust its memory. Before it reads a body, a request now reserves the body's Content-Length from a node-wide budget, --body-budget-bytes (256 MiB by default), and holds it until the gateway has forwarded the body. The gateway copies each body into one buffer of that size as it arrives, so the memory a body holds stays within its share however small the pieces it arrives in. A body without a declared length reserves --small-body-max-bytes (1 MiB by default) at first, and more as it arrives. --small-body-headroom-bytes (32 MiB by default) of the budget is kept for bodies of at most --small-body-max-bytes, so SQLite commits do not wait behind snapshot uploads. A request that finds the budget spent waits --body-budget-wait-ms (1000 by default) and then gets 503 with Retry-After: 1, marked UploadBudgetExhausted. The SQLite extension retries it as a refusal that applied nothing.
• The body budget follows a raised --max-request-body-bytes. With --body-budget-bytes unset, the budget is 256 MiB or the limit plus the 32 MiB small-body headroom, whichever is larger, so a gateway whose limit was raised (for example to 1 GiB for large snapshots) keeps starting, with a warning, and may buffer that much at once. Give the gateway about twice its budget in memory. The budget cannot be set below the limit plus the headroom, so if that is more than the pod's memory can hold, lower --max-request-body-bytes (the chart's gateway.maxRequestBodyBytes) and set --body-budget-bytes (gateway.bodyBudgetBytes) with it, or give the pod more memory. At startup the gateway also warns when its budget, set or grown, exceeds half of its cgroup memory limit, which it looks up along its own cgroup path. Once grown, the budget holds one body at the limit beside the headroom, so a request near the limit that waits for the whole budget takes every share released meanwhile, and large bodies behind it can get 503 too. A budget set explicitly below the limit plus the headroom makes the gateway exit at startup with LimitAboveSharedBudget, and the chart refuses such a gateway.bodyBudgetBytes. The chart renders --body-budget-bytes only when the value is set, and a gateway before 0.7.6 exits on that flag, so unset the value before you roll a gateway back. The other new flags (--small-body-headroom-bytes, --small-body-max-bytes, --body-budget-wait-ms, --body-stall-timeout, --body-min-bytes-per-sec) have no chart value. Add them to gateway.extraArgs only once every gateway runs 0.7.6, and remove them before a rollback, for the same reason.
• The gateway cuts bodies whose client abandoned them. It put no time limit on the body of a request it forwards. It now cuts a body once no byte of it arrived for --body-stall-timeout (30 s by default), or once less than --body-min-bytes-per-sec (1 KiB/s by default) arrived over one such window, and answers 408, marked RequestBodyTooSlow, with nothing forwarded. Nothing bounds a reading in all, so a slow upload that keeps arriving is never cut, and with the defaults the gateway never cuts a commit or a snapshot the SQLite extension still waits for. Over HTTP/1.1 the connection closes after the 408. Keep --body-stall-timeout above the extension's 10 s commit timeout. The gateway warns at startup otherwise.
• A quota policy can bound each bucket's uploads. max_concurrent_uploads and max_upload_bytes, under [default] or a [[bucket]] of --quota-policy, bound the bucket's bodies over --small-body-max-bytes while the gateway reads them. A request past either gets 429 with Retry-After: 1, marked TenantUploads or TenantUploadBytes. A body stops counting once the gateway has buffered it, and smaller bodies never count. A policy that sets neither limit, and a gateway without access control or a quota policy, answer no new 429.
• The refusals above reach a client that sends its body first. Like the refusals the gateway decides from the head, each is sent once the gateway has read the body to its end and dropped it, here at the pace of a forwarded body. A body declared over its bucket's own max_request_body_bytes gets its 413 before it reserves any of the budget. Only a body past --max-request-body-bytes still gets its 413 with the rest unread.
• New gateway metrics. /__ursula/gateway/metrics reports upload_refusals_budget, upload_refusals_tenant_uploads, upload_refusals_tenant_bytes and upload_refusals_too_slow, the refusals above by reason, upload_budget_bytes, the budget in effect, upload_reserved_bytes, the budget reserved now (each body's share in whole KiB), upload_tenants, the buckets with a large body under way under an upload limit, and body_bytes_buffered, the bytes of bodies it read to forward them. Its refusal_drains and refusal_drains_cut count the refusals above too, and the new refusal_drain_bytes counts the bytes of refused bodies it read and dropped.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests show that a failed write of the poisoned status fails closed.
• #452: the HTTP clients of the experimental event-time indexer have no connect or request timeouts, so an indexer call to a node that loses power can hang for minutes, until the kernel gives up on the connection.
• #461: a voter restarted without a graceful drain wins elections for dozens of groups within seconds of starting. For about 5 s every node then answers /__ursula/ready with 503 raft_maintenance_unready, and clients get 503s. Graceful rollouts avoid it, because the new pod starts drained, except a reserved rollout (server.gracefulRollout.maintenanceReservation) without server.startupOwnership, whose new pod starts undrained.
• #465: replaying the WAL at startup takes longer as the WAL grows, 17 s for 1.2 GB in one test, so in a test of repeated pod deletions over three hours, the recovery after a deletion grew from about 25 s to between 40 and 74 s.
• #484: when the replacement voter of a host recovery dies after its group leaders started rebuilding it, they keep it as a learner in those groups, so ursulactl verify-survivors refuses for good and the host restaging cannot proceed. No acknowledged write is at risk, since the survivors refuse. There is no automatic host-recovery consumer yet, and the chart keeps server.startupOwnership off by default.
• #490: a pass that prunes retired S3 snapshots prunes nothing when a pin it listed is released before it reads it, and logs failed to prune retired OpenRaft snapshots with NotFound, so retired snapshot objects can stay in S3 past their hour of grace. No data is at risk.
• #492: after a peer loses power, the surviving nodes keep its dead inbound connections until they restart, because the server sets no TCP or HTTP/2 keepalive on the connections it accepts.
• #494: a group prunes its retired S3 snapshots only when it builds a snapshot, so a group that stops taking writes keeps the snapshots it retired in its last busy hour until it builds again.
• #496: under moderate load, leaders log raft append stream exceeded the OpenRaft hard TTL for replication and heartbeat calls to followers. No commit failed where this was seen, but under a heavier load a busy group could elect a new leader.
• #497: a SQLite VFS attach that installs a snapshot is slower than one that replays as many bytes of log, about twice as long for a 1 GB database in one test.
• #517: a group sends each follower one Append call at a time, and since 0.7.2 a call carries at most 4 MiB of entries, so one group replicates to a follower at most about 4 MiB per round trip. That can slow the catch-up of a very busy group over a link with a long round trip.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described under 0.7.2.
• #539: a node answers every leadership transfer it refuses with the gRPC status FAILED_PRECONDITION, also when its Raft core has stopped, so the sending leader cannot tell a stopped core from a refusal by policy.
• #550: a SQLite VFS snapshot whose body is over a request size limit in front of the stream, such as the gateway's --max-request-body-bytes, is still sent. It usually meets a closed connection rather than the 413, is reported as a failed write and backs off. Compare snapshot_bytes with the limit. Commits stay below such a limit with URSULA_VFS_MAX_FRAME_BYTES.
• #561: the snapshot driver's gauges are reported only at /__ursula/metrics, not over OTLP. In the validation runs of 0.7.4 with two builds, /__ursula/ready answered 503 for apply_lag in one or two 5 s samples per run, too briefly to fail a probe that needs three failures in a row.
• #566: a SQLite VFS commit's append must be sent and answered within 10 s, its upload included, so a commit frame near the 32 MiB default of URSULA_VFS_MAX_FRAME_BYTES needs about 27 Mbit/s of upload. On a slower uplink a large commit times out with part of its body sent, which leaves its outcome unknown, so it poisons the database once the retry budget runs out, whether or not the stream would have taken it. At 10 Mbit/s up that happens from about 12 MiB. On a slow uplink, set URSULA_VFS_MAX_FRAME_BYTES well below what the uplink sends in 10 s, so that a larger transaction fails alone, with refused_kind too_large, instead. Snapshots have 120 s and are not affected.
• A node shed for failed writes can shed again every few minutes. A node shed because only its own cold-flush writes fail, while its probes answer, hands off its leaderships and then flushes nothing, so it heals about a minute after its last failed write even if its writes still fail. It sheds again once the leadership balancer hands leaderships back and a flush fails, for as long as such an outage lasts.
• A few slow clients can hold the gateway's body budget. A request reserves its declared length before it sends the body, and the gateway gives the reservation back only when the body stalls or arrives below --body-min-bytes-per-sec. With the defaults, about seven connections that each send 1 KiB/s can hold a 256 MiB budget, and other large bodies then get 503. That is the cost of never cutting a slow upload that keeps arriving. The per-bucket upload limits are keyed by bucket only, with no limit per principal, and without access control no limit per client address.
• One bucket's small bodies can take the whole small-body headroom. The headroom is shared by every bucket, so it keeps small writes from waiting behind large uploads, not one bucket's small writes from another's.

0.7.5

0.7.5 is a drop-in patch for 0.7.0, 0.7.1, 0.7.2, 0.7.3 and 0.7.4: format epoch 3 is unchanged, with no storage, snapshot, WAL or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4 and 0.7.5 voters run side by side during the upgrade. With unchanged values, the 0.7.5 chart renders the same server and gateway configuration as the 0.7.4 chart. From 0.7.0, a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.5 together with the chart, as for 0.7.1. From 0.7.3 or earlier, the upgrade notes of 0.7.4 apply too, and from 0.7.2 or earlier those of 0.7.3.

Before the upgrade, check raft.snapshot_build_max_concurrency (#561). A configuration file that sets it to 0 or to more than 16 no longer loads, so set it to a value between 1 and 16 or remove it. The 0.7.5 chart's schema refuses such a raft.snapshotBuildMaxConcurrency, so helm upgrade fails before any pod restarts.

SQLite VFS owners upgrade one at a time, as within any minor: the 0.7.x extensions take over each other's streams. Upgrade @tonbo/sqlite-ursula together with the extension. 0.7.5 changes no format. Pi users install @earendil-works/pi-durable themselves, in the range >=1.0.0 <1.2.0 (npm refuses a version outside it with ERESOLVE), and import openUrsulaPiStorage from @tonbo/sqlite-ursula/pi. The extension now sends User-Agent: sqlite-ursula-vfs/0.7.5, so update any proxy, WAF or log rule keyed on the User-Agent before upgrading owners. Through openUrsulaPiStorage, a fenced commit or read now rejects with an UrsulaFencedError and an unknown outcome with an UrsulaReattachError, both subclasses of UrsulaReplicationError whose name is their own, so code that matches the name or the class a logger prints must match with instanceof UrsulaReplicationError instead. On Linux and macOS, attach now refuses (in_use) while the process has the file, its -wal, -shm or -journal open outside the extension, so close every such connection, such as one opened before the extension was loaded, before the attach. An attach that cannot reach the endpoint before its claim throws UrsulaWriteRefusedError where it threw SQLite's error, and attach refuses a stream URL with a query or a fragment. These changes are described below.

• Rust library APIs changed. In ursula-raft, SnapshotBuildCoordinator::start_snapshot takes the log the snapshot is asked for, a check of the snapshot OpenRaft published where it took covered, and a future that ends once a published snapshot covers the request. SnapshotBuildCoordinator::acquire and acquire_below fail with the new SnapshotPermitError where they failed with GroupEngineError, and SnapshotStartError::Permit carries it too. SnapshotBuildCoordinator gains busy, publishing and add_publishing, build_ended is now public, and hand_off_and_trigger also answers Handoff::Busy while a build of the group holds a permit. ursula_config::validate::ValidationError gains SnapshotBuildConcurrency, which is not #[non_exhaustive], and ursula_config::RaftConfig::MAX_SNAPSHOT_BUILD_CONCURRENCY is new. ursula_observability::serve::serve_until_shutdown takes any axum::serve::Listener of TcpStreams, where it took a tokio::net::TcpListener. None of these changes touches the wire or the disk.

Snapshot builds

• raft.snapshot_build_max_concurrency must be between 1 and 16 (#561). 0.7.4 took 0 as 1 and set no maximum. From 0.7.5 a configuration file that sets 0 or more than 16 no longer loads, and the node exits at startup with an error such as raft.snapshot_build_max_concurrency (17) must be between 1 and 16. Before the upgrade, set it to a value between 1 and 16 or remove it. The 0.7.5 chart's schema refuses such a raft.snapshotBuildMaxConcurrency, so helm upgrade fails before any pod restarts. See Cold storage.
• The snapshot driver and an operator's snapshot see a build that has just ended (#561). A group whose trigger went unclaimed is now held out of the driver's passes for 0.5 s, doubling for each such trigger in a row up to 16 s, where it sat out 2 to 32 passes. A group whose build has just ended, with a snapshot OpenRaft has not published yet, is left out of the passes for up to 5 s and counts in the pass log's building. raft_snapshot_pressure_passes no longer counts a pass that only carries the same held groups as the pass before it, so it counts the pressure passes that planned a build, ended because the log outgrew them, or changed the groups held out. The pass log raft snapshot pressure pass completed gains busy, the groups skipped because a build of them runs or a permit was just handed to them. Such a group is skipped at once and not held. 0.7.4 counted it in not_started when a permit was handed to it. When a build of it ran, 0.7.4 triggered it all the same and OpenRaft dropped the trigger: the pass waited 5 s, logged did not claim its build permit within 5s, then counted the group in unclaimed and held it. POST /__ursula/raft/{group}/snapshot no longer starts a second build when a build of the group that has just ended covers the request, and in that case no longer waits 5 s on a trigger OpenRaft dropped or answers 503 although the covering snapshot exists. 0.7.4 lists both as a known issue. Unless the snapshot OpenRaft has published already covers the request, it now waits up to 10 s for the group's running build to end before its 10 s wait for the snapshot, so a request can take up to about 30 s where it took up to about 20 s. It still answers 500 when the snapshot does not show in time. See Manually trigger a snapshot.

SQLite VFS

• openUrsulaPiStorage moved to @tonbo/sqlite-ursula/pi, and the package has no runtime dependencies. @tonbo/sqlite-ursula used to depend on @earendil-works/pi-durable, so an application that only attaches files for node:sqlite installed Pi Durable and the model SDKs it pulls in, over 100 MB. The root entry now imports nothing outside Node.js, and Pi Durable is an optional peer dependency in the range >=1.0.0 <1.2.0, which npm installs only when the application depends on it. npm refuses the install with ERESOLVE when the application's version is outside that range, where 0.7.4 installed a second copy for the package without a warning, so move it into the range before or with the upgrade. This is a breaking change for Pi users: install @earendil-works/pi-durable next to the package, and import openUrsulaPiStorage and UrsulaPiStorageOptions from @tonbo/sqlite-ursula/pi. Every other export, the error classes included, stays in @tonbo/sqlite-ursula. See Pi Durable.
• The extension identifies itself. Every request it sends, including the CONNECT to a proxy, carries User-Agent: sqlite-ursula-vfs/0.7.5, the extension's name and version, so a gateway or proxy can attribute traffic and require a minimum version. Extensions up to 0.7.4 send ureq's default, ureq/3.4.2. Update any proxy, WAF or log rule keyed on the User-Agent before upgrading owners.
• Attach refuses a file open outside the extension, and a failed attach locks the file out of the process (#570). Attach refuses (in_use), changing nothing, while the process has the file, its -wal, -shm or -journal open outside the extension, read-only or not, such as a connection opened before the extension was loaded. It checks where it can read the process's open files: on Linux with /proc mounted, and on macOS. After a failed attach the file opens in the process only once an attach succeeds, with or without a sidecar, and a failed attach that wrote nothing into a new file removes the sidecar it created, which never touches the db file, -wal or -shm. As before, a failed attach may already have discarded local files it did not trust or folded their WAL into the db file, and the next attach resumes the file when its sidecar names it and rebuilds it otherwise. An attach refused because of the file itself before it changed anything (in_use, unreplicated_file, other_stream) is no failed attach and changes nothing, so a database that holds data but was never replicated still opens to copy its rows out. Writes to an attached file outside the extension are still not detected, as in 0.7.4.
• Fenced owners stop serving reads. Once an owner knows it is fenced, its new read transactions fail too (errcode 266), and through openUrsulaPiStorage they reject with UrsulaFencedError, an UrsulaReplicationError (fenced: true, a message that starts with read refused). A read transaction already open keeps its snapshot until it ends.
• Errors name a stable kind. Every error of the extension's SQL functions starts with <function>: <kind>: , and status(file).error reports kind, action (retry, reattach, stop or fix) and message while writes fail. An attach refused with a 4xx other than 404 carries the server's reason, and a token refused when the retries run out keeps the layer's explanation. A database without a sidecar is refused as unreplicated_file with a message that says how to copy its rows out. An attach that fails after its claim reports outcome_unknown, since the claim may have fenced the earlier owner, and a local file the setup refuses (no permission, a read-only file system, no space left) reports local_setup, whose action is fix. See Health and alerts and the design doc for the kinds.
• open(streamUrl) opens a replicated database in one call, and its errors say what to do. open loads the extension, attaches a local file named after the stream (or the path it is given, made absolute), opens it, and returns a node:sqlite DatabaseSync. A failure of the extension in it throws an UrsulaError, whose action is retry, reattach, stop or fix and whose cause is the error it surfaced as, and every other error, such as a constraint violation, is SQLite's own, unchanged. The ROLLBACK that a transaction helper (drizzle's, say) runs after such a failure, which SQLite has rolled back already, does not replace it. A read-only open keeps a file of its own, so it opens next to the owner on its host. The new classes are UrsulaError, UrsulaFencedError (stop), UrsulaReattachError (reattach) and UrsulaSetupError (fix). UrsulaReplicationError, UrsulaWriteRefusedError and UrsulaPaymentRequiredError are now subclasses of UrsulaError. attach, status, setToken and drainStats throw UrsulaErrors too, with the extension's message, and with SQLite's code, errcode and errstr copied, so code that reads them keeps working. openUrsulaPiStorage reports through the same mapping: a fenced commit or read is an UrsulaFencedError and an unknown outcome an UrsulaReattachError, both UrsulaReplicationErrors as before. The docs no longer ask for PRAGMA journal_mode=WAL: attach leaves the file in WAL mode. What a caller can see change, see SQLite on Ursula:
  • An attach whose claim is refused other than with a 402 has kind refused where it had null.
  • An attach that fails before it sends its claim because the endpoint cannot be reached, puts it off or refuses the credentials throws UrsulaWriteRefusedError (unreachable, unavailable or unauthorized) where it threw SQLite's error.
  • attach and openUrsulaPiStorage refuse a stream URL with a query or a fragment (invalid_url) before sending anything, where the attach created the stream, appended a claim and failed. openUrsulaPiStorage refuses :memory:, "" and file: URIs as its file (invalid_argument), and makes a relative one absolute.
  • The errors openUrsulaPiStorage builds for a failed commit or read have messages that end with one sentence that says what to do, after commit not replicated: <reason>, read refused, the file is fenced: <reason> and commit refused ....
  • Those errors for a fenced commit or read, and for an unknown outcome, have the name UrsulaFencedError or UrsulaReattachError, where it was UrsulaReplicationError, which shows in String(error), the stack and loggers that print the class. instanceof UrsulaReplicationError holds as before: match with it.
  • A statement of Pi's storage on a poisoned file that fails with an error that is not an I/O error, such as a constraint violation, rejects with that error where it rejected with UrsulaReplicationError. A failure of the extension that Pi wraps in its StorageRejected still rejects with the Ursula error.

Event-time indexer

• The indexer sets TCP_NODELAY on the connections it accepts, as the server and the gateway already did. Without it, a reply of ursula indexer (the experimental event-time indexer) that followed another on the same connection could wait for the client's delayed acknowledgement of the first, 40 ms or more on Linux.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests show that a failed write of the poisoned status fails closed.
• #452: the HTTP clients of the experimental event-time indexer have no connect or request timeouts, so an indexer call to a node that loses power can hang for minutes, until the kernel gives up on the connection.
• #461: a voter restarted without a graceful drain wins elections for dozens of groups within seconds of starting. For about 5 s every node then answers /__ursula/ready with 503 raft_maintenance_unready, and clients get 503s. Graceful rollouts avoid it, because the new pod starts drained, except a reserved rollout (server.gracefulRollout.maintenanceReservation) without server.startupOwnership, whose new pod starts undrained.
• #465: replaying the WAL at startup takes longer as the WAL grows, 17 s for 1.2 GB in one test, so in a test of repeated pod deletions over three hours, the recovery after a deletion grew from about 25 s to between 40 and 74 s.
• #484: when the replacement voter of a host recovery dies after its group leaders started rebuilding it, they keep it as a learner in those groups, so ursulactl verify-survivors refuses for good and the host restaging cannot proceed. No acknowledged write is at risk, since the survivors refuse. There is no automatic host-recovery consumer yet, and the chart keeps server.startupOwnership off by default.
• #490: a pass that prunes retired S3 snapshots prunes nothing when a pin it listed is released before it reads it, and logs failed to prune retired OpenRaft snapshots with NotFound, so retired snapshot objects can stay in S3 past their hour of grace. No data is at risk.
• #492: after a peer loses power, the surviving nodes keep its dead inbound connections until they restart, because the server sets no TCP or HTTP/2 keepalive on the connections it accepts.
• #494: a group prunes its retired S3 snapshots only when it builds a snapshot, so a group that stops taking writes keeps the snapshots it retired in its last busy hour until it builds again.
• #496: under moderate load, leaders log raft append stream exceeded the OpenRaft hard TTL for replication and heartbeat calls to followers. No commit failed where this was seen, but under a heavier load a busy group could elect a new leader.
• #497: a SQLite VFS attach that installs a snapshot is slower than one that replays as many bytes of log, about twice as long for a 1 GB database in one test.
• #517: a group sends each follower one Append call at a time, and since 0.7.2 a call carries at most 4 MiB of entries, so one group replicates to a follower at most about 4 MiB per round trip. That can slow the catch-up of a very busy group over a link with a long round trip.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described under 0.7.2.
• #539: a node answers every leadership transfer it refuses with the gRPC status FAILED_PRECONDITION, also when its Raft core has stopped, so the sending leader cannot tell a stopped core from a refusal by policy.
• #550: a SQLite VFS snapshot whose body is over a request size limit in front of the stream, such as the gateway's --max-request-body-bytes, is still sent. It usually meets a closed connection rather than the 413, is reported as a failed write and backs off. Compare snapshot_bytes with the limit. Commits stay below such a limit with URSULA_VFS_MAX_FRAME_BYTES.
• #561: the snapshot driver's gauges are reported only at /__ursula/metrics, not over OTLP. In the validation runs of 0.7.4 with two builds, /__ursula/ready answered 503 for apply_lag in one or two 5 s samples per run, too briefly to fail a probe that needs three failures in a row.
• #566: a SQLite VFS commit's append must be sent and answered within 10 s, its upload included, so a commit frame near the 32 MiB default of URSULA_VFS_MAX_FRAME_BYTES needs about 27 Mbit/s of upload. On a slower uplink a large commit times out with part of its body sent, which leaves its outcome unknown, so it poisons the database once the retry budget runs out, whether or not the stream would have taken it. At 10 Mbit/s up that happens from about 12 MiB. On a slow uplink, set URSULA_VFS_MAX_FRAME_BYTES well below what the uplink sends in 10 s, so that a larger transaction fails alone, with refused_kind too_large, instead. Snapshots have 120 s and are not affected.
• A node shed for failed writes can shed again every few minutes. A node shed because only its own cold-flush writes fail, while its probes answer, hands off its leaderships and then flushes nothing, so it heals about a minute after its last failed write even if its writes still fail. It sheds again once the leadership balancer hands leaderships back and a flush fails, for as long as such an outage lasts.

0.7.4

0.7.4 is a drop-in patch for 0.7.0, 0.7.1, 0.7.2 and 0.7.3: format epoch 3 is unchanged, with no storage, snapshot, WAL or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1, 0.7.2, 0.7.3 and 0.7.4 voters run side by side during the upgrade. With unchanged values, the 0.7.4 chart renders the same server and gateway configuration as the 0.7.3 chart. From 0.7.0, a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.4 together with the chart, as for 0.7.1. From 0.7.2 or earlier, the upgrade notes of 0.7.3 apply too.

A gateway with access control (--auth-issuer) now answers 404 on its metrics path, /__ursula/gateway/metrics (#548). If something scrapes that path, add --auth-public-metrics to gateway.extraArgs in the same helm upgrade that moves the gateways to 0.7.4. A gateway before 0.7.4 exits at startup on the unknown flag, so do not add it before that upgrade, and remove it before rolling a gateway back to an earlier release. The flag makes the counters public: the gateway then serves them to every caller, with or without a token.

The SQLite VFS is no longer experimental (#468). Within a minor release (0.7.x), the formats the extension writes to the stream and to the local files do not change, and the 0.7.x extensions take over each other's streams, so owners upgrade one at a time. The API of the extension and of @tonbo/sqlite-ursula can still change in a patch release, as it did in 0.7.3, so upgrade the package together with the extension. A new minor release may also change the formats. Either way, the release notes say how to upgrade. 0.7.4 changes no format. Its API changes are the refusal kind unreachable and an attach that fails as refused when nothing of it went out, both described below. The @tonbo/sqlite-ursula README on npm drops the experimental label with the 0.7.4 publish. See SQLite on Ursula.

• Rust library APIs changed. In ursula-raft, SnapshotBuildCoordinator no longer exposes reclaimable_log_progress, try_acquire, hand_off, handoff_pending or reclaim_handoff, and gains max_concurrency, node_log_bytes, builds_in_flight, running_builds, acquire_below, hand_off_and_trigger and start_snapshot, with the new Handoff, SnapshotStartError and SNAPSHOT_HANDOFF_WAIT. hand_off_and_trigger takes one hand-off per group at a time (Handoff::Busy otherwise) and gives its permit back when its future is dropped. In ursula_raft::snapshot_cadence, SnapshotPlan::pressure is a LogPressure instead of a bool, and SnapshotCadence::plan_with_lagging gives way to plan_pass, which takes GroupDriveStates within DriveLimits, next to the new pressure, pass_step (with PassStep), hard_limit_bytes and resume_bytes. In ursula-runtime, RuntimeMetrics::record_raft_snapshot_pressure takes the pass's time as a second argument, RuntimeMetrics::record_raft_log_pressure and RaftLogPressureSample are new, and RuntimeMetricsSnapshot has seven new public fields, so code that builds it with a struct literal must set them. ursula_gateway::auth::AccessControl::with_public_metrics is new, and a gateway built with Gateway::with_access_control answers 404 on its metrics path unless it is set. ursula_config::RaftConfig now defaults snapshot_build_max_concurrency to 2. None of these changes touches the wire or the disk.

Snapshot builds

• Snapshot builds escalate before the Raft log reaches its hard limit (#519). Under a sustained write load, the unsnapshotted Raft log of every node could reach its hard limit, twice raft.snapshot_log_budget, at about the same time, and the nodes then refused writes with 503 RaftLogPressure for about 20 seconds until snapshots caught up. By default the snapshot driver ran one build at a time, and its pressure passes took the groups that free the most log per snapshot byte first, which put near-empty groups first. They now snapshot the groups that hold the most log first, and at or past the budget they run more builds at once the further the log is past it, up to raft.snapshot_build_max_concurrency, whose default rises from 1 to 2. With 4, a node runs one build at the budget, two just past it, three past 1.33 budgets and four past 1.67 budgets. As the builds bring the log back down the driver runs fewer of them, and below the budget it starts a build only when no other runs. At or past the budget a pass starts up to raft.snapshot_pressure_max_groups_per_tick groups for each build it keeps in flight. The hard limit and the point where writes resume, 1.5 budgets, are unchanged. On a 3-voter test cluster with real S3 and 128 SQLite VFS owners writing flat out, 0.7.3, and 0.7.4 set to one build, crossed the hard limit on every node and refused writes for 11 to 63 seconds at a time, while 0.7.4 with the default of 2 stayed well below it.
• What the new default costs. A build holds its group's captured state and, while it uploads, its compressed snapshot, so two builds hold twice what one did. With snapshots of 2.8 MB on average and 67 MB at most, as under the load of #519, two builds hold about 11 MB in the common case, about 270 MB when both build the largest groups, and about 400 MB if both uploads fail and fall back to an inline snapshot. At or past the budget a node also sends up to two builds' requests to the snapshot store at once. To keep one build at a time, set raft.snapshot_build_max_concurrency = 1 (the chart's raft.snapshotBuildMaxConcurrency: 1), which is safe at any time. A node with memory to spare can set 4. Set a value above 1 only after every node runs 0.7.4: servers before 0.7.4 read the setting too, and a helm upgrade changes the chart's ConfigMap in place, so a value set in the same upgrade reaches an older pod that restarts before the rollout replaces it. That pod's earlier driver then runs up to that many builds at once at any log level, which was never validated. The chart renders the setting only when raft.snapshotBuildMaxConcurrency is set, so with it unset each server keeps its own default, which is 1 before 0.7.4. With the snapshot driver off (storage.snapshot.drive_interval = 0), OpenRaft's own policy may now run two builds at once too. See Cold storage.
• Metrics for the snapshot driver. /__ursula/metrics reports the node's unsnapshotted Raft log and the most it held (raft_log_bytes, raft_log_bytes_high_watermark), the driver's pressure level (raft_snapshot_pressure_level: 0 below three quarters of the budget, 1 below the budget, 2 at or past it), the builds a pass keeps in flight at that level (raft_snapshot_pressure_builds), the builds that run now and the most that ran at once (raft_snapshot_builds_in_flight, raft_snapshot_builds_in_flight_max), and the time of its pressure passes (raft_snapshot_pressure_pass_ns). Only raft_snapshot_pressure_passes changed meaning. It counts the pressure passes that planned a build or held out groups whose last trigger went unclaimed, where it counted every pressure pass, so the mean pass time, raft_snapshot_pressure_pass_ns over raft_snapshot_pressure_passes, covers those passes only. raft_snapshot_pressure_groups did not change. The driver logs each pass it counts as raft snapshot pressure pass completed. See Metrics.
• An operator's snapshot waits for a build permit (#529). POST /__ursula/raft/{group}/snapshot now waits while a build of the group runs and while other builds hold every snapshot build permit. It answers 200 when a snapshot covers what the group had applied when the request arrived, and 503 with Retry-After: 1 when no build of the group can start within 10 s, where it answered 500 after about 10 s. A request can now take up to about 20 s, where it took up to about 10 s: up to 10 s for a build of the group to start, then up to 10 s for the snapshot, after which it answers 500 as before. A build of the group that has just ended can go unseen for a moment, so the request may then start another build (#561). See Manually trigger a snapshot.

Gateway

• The gateway's metrics follow its access control (#548). A gateway with access control (--auth-issuer) no longer serves /__ursula/gateway/metrics. It answers 404 to every caller, with a token or without, like every other /__ursula/* path, unless --auth-public-metrics is set (with the Helm chart, in gateway.extraArgs). The flag requires --auth-issuer. Public metrics are served without a token check, so an invalid or expired token gets them too. The counters hold no tenant data but show traffic volume, so set the flag only where untrusted clients cannot reach the gateway, or where a proxy in front of it denies /__ursula/*. A gateway without access control serves its metrics as before. While the gateways roll, a scraper gets both answers. See Gateway access control.
• The gateway reads the body before a refusal it decided from the request head (#540). ursula gateway decides some refusals from a request's head alone: the access-control refusals (401, 403, 404, and the 503s while the token issuer or the authorizer is unavailable) and a tenant's 429. It used to send them with the body unread and close the connection, so a client that sends its whole body before it reads the answer, as the SQLite VFS does, often met a broken pipe or a reset instead of the refusal. The gateway now reads the body and discards it before it sends such a refusal, with the same status and headers, and stops after 5 s without new data or 30 s in all. A SQLite VFS commit larger than one read that the gateway refuses this way now fails alone with the refusal's kind, as a small one already did, instead of poisoning the database once the retry budget runs out. Only the gateway needs 0.7.4 for this: a 0.7.3 extension behind it gets the refusal too.
  • The gateway reads at most --max-request-body-bytes of such a body. A body that goes past the limit, stops arriving for 5 s or is still arriving after 30 s gets the refusal at once, and the connection closes. A refused request whose Content-Length is over the limit gets its refusal at once, with its body unread, and its connection closes. A request the gateway would forward still gets its 413 for a body over the limit, as before: the gateway stops reading at the limit and closes the connection, so a client still sending the body may meet the closed connection instead. Keep commits below the limit with URSULA_VFS_MAX_FRAME_BYTES.
  • A refused upload costs the gateway the body's ingress, but no memory, and once the body was read to its end the connection stays open for the client's next request. Over HTTP/2 a refused request also uploads its body before it gets the refusal, and a reading cut short resets only that request's stream. A client that sends Expect: 100-continue (curl does for uploads over 1 MiB) now gets 100 Continue and uploads its body before the refusal, unless its Content-Length is over the limit. Programs that embed the gateway get the same from Gateway::handle.
  • /__ursula/gateway/metrics reports the refused requests with a body as refusal_drains, and those whose body the gateway did not read to its end as refusal_drains_cut. Only a gateway with access control refuses requests from their head, and it serves these counters only with --auth-public-metrics, which makes them public.
  • A SQLite VFS snapshot over a limit in front of the stream is still sent, and still usually meets a closed connection (#550).

SQLite VFS

• A commit whose attempts never went out fails alone. An attempt that failed before any byte of it went out, because the stream's host did not resolve, no connection was made or the TLS handshake failed, now proves that nothing of it was applied, like the refusals of 0.7.3. A commit whose every attempt failed this way or was refused fails alone with SQLITE_IOERR_AUTH (errcode 7178) once the retry budget runs out, instead of poisoning the database. refused_kind is unreachable when the last attempt never went out, and refused_reason gives its failure, such as timeout: resolve. The commit's producer sequence stays free, and the next commit succeeds once the endpoint is reachable, with no new attach. A failure after a connection was made, on a pooled connection too, still leaves the commit's outcome unknown, and poisons once the retry budget runs out. An attach, and an owner's later claim of its producer, whose every attempt failed before it went out now fail as refused, with nothing claimed: ursula_attach fails with SQLITE_IOERR_AUTH and attach throws UrsulaWriteRefusedError, where both failed with an unknown outcome. An attach usually meets an unreachable endpoint at its first HEAD, which fails as before. TypeScript's WriteRefusedKind gains unreachable, so a switch that handles every kind needs a case for it. Troubleshooting says what to check, with the TLS failures as rustls reports them.
• Host addresses are cached. Every request used to look its host up, also on a pooled connection. In Kubernetes that could exhaust the connection tracking of the node that runs CoreDNS, and a DNS outage of 30 s poisoned every owner whose commits met it. The extension now looks a host up at most once every 30 s, one request at a time, while the host's other requests go on with its last addresses. The request that looks the host up waits for the lookup, up to the 5 s resolve timeout, even with a pooled connection ready. While lookups fail, the last addresses serve requests for up to 10 minutes after the last successful lookup, the host is looked up again 5 s after each failed lookup, and the extension logs dns_stale. A host with no addresses to use fails its requests as never sent until a lookup succeeds, with one lookup every 5 s. A failed connection makes the host's next lookup due, no sooner than 5 s after the last one, so an endpoint that moved is found at the next attempt.
• A reset that macOS reports as Invalid argument is retried (#542). The extension ships prebuilt for macOS on Apple silicon. macOS reports a connection reset as Invalid argument (os error 22) when it lands between two of the extension's calls on the socket, and as a reset otherwise. The extension took that error as final, so a read, HEAD, create or snapshot GET whose pooled connection the server had just closed failed the attach or the snapshot pass. Such a request is now sent again on a new connection within URSULA_VFS_RETRY_MS, as on Linux, where nothing changes. This fixes the known issue listed under 0.7.3.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests show that a failed write of the poisoned status fails closed.
• #452: the HTTP clients of the experimental event-time indexer have no connect or request timeouts, so an indexer call to a node that loses power can hang for minutes, until the kernel gives up on the connection.
• #461: a voter restarted without a graceful drain wins elections for dozens of groups within seconds of starting. For about 5 s every node then answers /__ursula/ready with 503 raft_maintenance_unready, and clients get 503s. Graceful rollouts avoid it, because the new pod starts drained.
• #465: replaying the WAL at startup takes longer as the WAL grows, 17 s for 1.2 GB in one test, so in a test of repeated pod deletions over three hours, the recovery after a deletion grew from about 25 s to between 40 and 74 s.
• #484: when the replacement voter of a host recovery dies after its group leaders started rebuilding it, they keep it as a learner in those groups, so ursulactl verify-survivors refuses for good and the host restaging cannot proceed. No acknowledged write is at risk, since the survivors refuse. There is no automatic host-recovery consumer yet, and the chart keeps server.startupOwnership off by default.
• #490: a pass that prunes retired S3 snapshots prunes nothing when a pin it listed is released before it reads it, and logs failed to prune retired OpenRaft snapshots with NotFound, so retired snapshot objects can stay in S3 past their hour of grace. No data is at risk.
• #492: after a peer loses power, the surviving nodes keep its dead inbound connections until they restart, because the server sets no TCP or HTTP/2 keepalive on the connections it accepts.
• #494: a group prunes its retired S3 snapshots only when it builds a snapshot, so a group that stops taking writes keeps the snapshots it retired in its last busy hour until it builds again.
• #496: under moderate load, leaders log raft append stream exceeded the OpenRaft hard TTL for replication and heartbeat calls to followers. No commit failed where this was seen, but under a heavier load a busy group could elect a new leader.
• #497: a SQLite VFS attach that installs a snapshot is slower than one that replays as many bytes of log, about twice as long for a 1 GB database in one test.
• #517: a group sends each follower one Append call at a time, and since 0.7.2 a call carries at most 4 MiB of entries, so one group replicates to a follower at most about 4 MiB per round trip. That can slow the catch-up of a very busy group over a link with a long round trip.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described under 0.7.2.
• #539: a node answers every leadership transfer it refuses with the gRPC status FAILED_PRECONDITION, also when its Raft core has stopped, so the sending leader cannot tell a stopped core from a refusal by policy.
• #550: a SQLite VFS snapshot whose body is over a request size limit in front of the stream, such as the gateway's --max-request-body-bytes, is still sent. It usually meets a closed connection rather than the 413, is reported as a failed write and backs off. Compare snapshot_bytes with the limit. Commits stay below such a limit with URSULA_VFS_MAX_FRAME_BYTES.
• #561: the snapshot driver's new gauges are reported only at /__ursula/metrics, not over OTLP, and raft.snapshot_build_max_concurrency = 0 is taken as 1 rather than refused. In the validation runs with two builds, /__ursula/ready answered 503 for apply_lag in one or two 5 s samples per run, too briefly to fail a probe that needs three failures in a row. An operator's snapshot can miss a build of its group that has just ended and start another one. When that build covered the request, the request can wait 5 s and answer 503 although a covering snapshot exists, and a retry then answers 200.
• A node shed for failed writes can shed again every few minutes. A node shed because only its own cold-flush writes fail, while its probes answer, hands off its leaderships and then flushes nothing, so it heals about a minute after its last failed write even if its writes still fail. It sheds again once the leadership balancer hands leaderships back and a flush fails, for as long as such an outage lasts.

0.7.3

0.7.3 is a drop-in patch for 0.7.0, 0.7.1 and 0.7.2: format epoch 3 is unchanged, with no storage, snapshot or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1, 0.7.2 and 0.7.3 voters run side by side during the upgrade, and the 0.7.3 chart passes the same flags as the 0.7.2 chart. From 0.7.0, a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.3 together with the chart, as for 0.7.1. A configuration file that sets storage.cold.s3.probe_timeout, storage.cold.s3.unhealthy_ticks, storage.cold.s3.heal_ticks or governance.leadership_balance.peer_timeout to 0 no longer loads, so remove such a setting before the upgrade.

Upgrading gateways before voters, and rolling back voters before gateways, is recommended, not required. Clients trust the new x-ursula-not-applied marker only together with x-ursula-not-applied-checked, which only gateways from 0.7.3 on send, so no order in which a helm upgrade or helm rollback replaces gateways and voters can make a client count a committed write as never applied. Upgrading gateways first keeps the marker useful during the rollout. Before relying on the marker, check the layers around the gateway. A proxy or service mesh in front of it must not retry writes or follow redirects for them, and may answer 401, 402 or 429 only on a request it did not pass on. Nothing between the gateway and the nodes may resend requests or answer 401, 402 or 429, and a 0.7.3 gateway must not sit in front of an older gateway. See redirects and retries.

SQLite VFS owners upgrade one at a time, as within any minor: the 0.7.x extensions take over each other's streams. Upgrade @tonbo/sqlite-ursula together with the extension, because a 0.7.2 package over a 0.7.3 extension reports every refused commit as UrsulaPaymentRequiredError. The SQLite VFS stays experimental (#468).

• Rust library APIs changed. ursula_gateway::GatewayError has two new variants, LeaderUnreachable { upstream, source, applied_nothing } and LeaderNotFound { followed, applied_nothing }, and ursula-gateway now depends on ursula-proto. In ursula-raft, RaftGroupHandleRegistry::handle_transfer_leader returns the new IncomingTransferError instead of GroupEngineError, OwnerRaftHandle::handle_transfer_leader is no longer public, and RaftGroupHandleRegistry::request_self_election, SelfElectionError and OwnerRaftHandle::handoff_settle_time are new. ursula::SpawnRuntimeError gains PeerStatusClient, and ursula_config::validate::ValidationError gains ZeroS3HealthSetting (with the new S3HealthSetting) and ZeroPeerTimeout. None of these enums is #[non_exhaustive], so an exhaustive match on one outside its crate needs the new variants. The rest are additions: ursula_proto::headers with NOT_APPLIED_HEADER and NOT_APPLIED_CHECKED_HEADER, ursula_runtime::GroupEngineError::refused_before_proposal with ursula_runtime::RefusedBeforeProposal, and ursula_runtime::s3_operation_budget. None of these changes touches the wire or the disk.

Refusals that applied nothing

• Responses can prove that a write attempt applied nothing. A response that carries both x-ursula-not-applied and x-ursula-not-applied-checked answers an attempt that was refused before anything of it was proposed, so nothing of it can apply later, whatever the status. The first header, the marker, names the refusal with a token such as RaftLogPressure. The server marks the refusals of its write admission: 503 with WalDiskPressure, RaftLogPressure or IngressBodyBytesLimitReached, and 413 with RequestBodyTooLarge for a Content-Length over the route's limit. The gateway marks the requests no node received: its own refusals before forwarding, such as a 503 while its token issuer is down or a 403 from its authorizer, and the 503 it answers when it cannot connect to the first upstream it picked (UpstreamUnreachable). The second header is the gateway's vouch. A gateway from 0.7.3 on sets it on every response it returns, after it dropped any copy from an upstream and any node marker that a leader redirect may have made stale. Nodes never set it. So a client can rely on the marker only behind a 0.7.3 or newer gateway. It keeps counting 401, 402 and 429 as never applied by their status, and any other 5xx as an unknown outcome. The headers speak for one attempt only. The chart's TLS ingress example now sets nginx.ingress.kubernetes.io/proxy-next-upstream: "off", so ingress-nginx never resends a request to another gateway pod. In a service mesh, turn off retries for the stream routes on both sides of the gateway. Istio before 1.24, for example, resends a request up to twice on a 503 by default. Istio 1.24 to 1.26 does too when istiod runs with compatibilityVersion=1.23.
• Refusals raised before a write was proposed are marked, redirects included. A node now marks a create, append, close or delete that it refuses before it proposes the write, when it can tell that it did: a 307 to the leader, or a 503 while no leader is known, from its own leadership check (NotLeader), a node that does not host the stream's Raft group (GroupNotHosted, a 307 to a node that does or a 503), and the 503 of uncommitted-bytes or cold-write backpressure (RaftUncommittedBackpressure, ColdBackpressure). NotLeader covers appends and closes, and creates unless the body is staged in object storage or storage.cold.max_hot_size_per_group is 0. A redirect or 503 that can follow a proposal stays unmarked, and so does a follower's redirect for a delete or for one of those creates. Snapshot and retention PUTs stay unmarked, because repeating them is harmless. Through a 0.7.3 gateway with several upstreams, a write that reaches a leader refusing at admission, such as 503 RaftLogPressure under sustained write load, through a follower's redirect now keeps the leader's marker, with the vouch.

Gateway redirects

• The gateway follows leader redirects until a node answers (#460). During leader changes, clients behind ursula gateway no longer get 307 redirects. The gateway follows Raft leader redirects itself, up to three hops after the first, none started later than 5 s after the first redirect. When a redirect names an address that is none of its upstreams, as happens when --upstream names the nodes otherwise than their peer URLs, or one the request already went to, it tries the upstreams it has not tried yet, in the configured order from the one after the upstream that redirected, and skips ejected ones. When it finds no leader within those limits, it answers 503 with Retry-After: 1, which Durable Streams clients retry. Before, it followed one redirect to a configured upstream and passed any other on with a path-only Location, which redirect-following clients looped on. Other 307s, which name no Raft leader, still reach the client with a path-only Location.
  • The final answer keeps its x-ursula-not-applied only when every redirect on the way carried one. Only then are the gateway's own 503s after redirects marked: LeaderNotFound when it stops following, and UpstreamUnreachable for a node it then could not connect to. A hop that was sent and then failed ends the request with an unmarked 502 or 504.
  • Each hop sends the request again. A plain append, without a producer or Stream-Seq, can apply once for each node that proposed it and then redirected, up to four times, where a gateway before 0.7.3 applied it at most twice. Use a producer for exactly-once writes.
  • The gateway learns a leader route only from a 2xx after redirects, not when it follows a redirect. /__ursula/gateway/metrics gains leader_searches and leader_not_found, and leader_redirects now counts each hop that follows a redirect.

Leadership shedding

• One failed S3 health probe no longer moves every leadership (#508). The snapshot-store health gate used to shed after one bad tick: a probe that missed its 2s deadline while the S3 client was still retrying it moved every leadership on the node to one peer within a second, and commits on several groups stalled. The gate now sheds only after three bad ticks with no good tick between them. A tick is bad when the probe fails or misses its deadline, or when a cold-flush write failed and no write has succeeded since, for as long as the S3 client can take to give up on a write (55 seconds with the defaults). So a store that refuses writes sheds the node even while the probe answers. With the defaults a node sheds about 10 to 20 seconds after its probes start failing, about 10 to 30 seconds after its writes start failing (sooner when the store refuses them outright), and about a minute after its write attempts start hanging until their timeout. While the gate judges the store unhealthy, routine snapshots are held back. A node shed for failed writes may lead again about two ticks after a write succeeds, and a node that has handed off every leadership waits until its last failed write is about a minute old.
• A shedding node moves its leaderships a few at a time, and only to peers that may campaign. The store gate and the cold-health gate each hand off at most four leaderships per tick, the groups with the most unflushed data first, and keep a handoff pending until another node leads the group. A node with N leaderships evacuates in about ceil(N / 4) ticks. Both gates pass over a peer that is draining, behind its recovery gate or shedding for its own store, cluster egress or WAL disk, and a peer that does not report its leadership-shed state within governance.leadership_balance.peer_timeout (500ms by default). The cluster-egress gate still moves every leadership at once, and now also reads its peers' leadership-shed state. It passes over a peer that answered that it may not campaign, tries a peer that did not answer at most once per leadership in each shed, and skips the peers that did not answer on a tick when another peer answered that it may not campaign.
• S3 health settings. storage.cold.s3.probe_timeout (default now 5s, was 2s), storage.cold.s3.unhealthy_ticks (now 3, was 1) and storage.cold.s3.heal_ticks (2) are now documented under Cold storage. A configuration that sets them keeps its values, and the flush rule and the pace of four handoffs per tick apply either way. 0 is rejected for each of them and for governance.leadership_balance.peer_timeout.

Raft

• Self-election on the leader is refused (#516). POST /__ursula/raft/{group}/self-election on a node that already leads the group now answers 409 Conflict and leaves the term unchanged. Before, it started a new election that could trip an OpenRaft invariant, the leader's vote falling behind the node's vote. In debug builds that stopped the core thread, and in release builds the invariant was silently violated. The endpoint also answers 503 Service Unavailable instead of 409 when the replica's Raft core or apply loop has stopped. A follower's self-election is unchanged, and no ursulactl command calls the endpoint.

SQLite VFS (experimental)

• A commit refused before it reaches the stream no longer poisons the database. When every attempt of a commit is refused in a way that proves nothing was applied, the commit fails alone with SQLITE_IOERR_AUTH (errcode 7178 in node:sqlite), as a 402 already did. That covers a token refused until URSULA_VFS_RETRY_MS runs out, a 429, and a 4xx or 5xx with both x-ursula-not-applied and the vouch of a 0.7.3 or newer gateway, such as the 503 RaftLogPressure of #519. Without a gateway, or behind an older one, the marker counts for nothing and only a 401, 402 or 429 fails a commit alone. A 403 without both headers still poisons, because a filtering proxy can send one after the stream applied the write. Reads keep working, and the next commit succeeds once the refusals stop, with no new attach. A refused commit with any attempt whose outcome is unknown, such as no answer, a 307 or a 5xx without both headers, still poisons, as before. status(file) reports writes_refused, refused_kind and refused_reason, the extension logs writes_refused, and instrument() adds the gauge sqlite_ursula.writes_refused. openUrsulaPiStorage rejects such a commit with the new UrsulaWriteRefusedError, and UrsulaPaymentRequiredError is its subclass for a 402. An attach whose claim is refused this way throws UrsulaWriteRefusedError, and ursula_attach now fails with SQLITE_IOERR_AUTH instead of SQLITE_ERROR for such a claim, while a 402 keeps SQLITE_AUTH. See Writes refused before they reach the stream.
• The commit after a refused one checks the refusal. If a layer in front of Ursula refuses a commit that the stream applied, the next commit now usually poisons the database, with a reason that says a commit refused as never applied may have been applied after all, instead of being acknowledged without its pages. Attaching again replays the commit the stream holds. The same poison can follow a commit that a gateway sent again to a new leader right after a refused commit, which costs an attach and loses nothing. When an attempt of that next commit got no answer first, the commit is still acknowledged, and the extension logs refused_commit_unchecked. Turn off retries and redirect following for writes in any proxy or service mesh in front of Ursula, because such a proxy can relay a refusal for a write that an earlier exchange applied.
• A transaction too large for one request fails alone. The extension never sends a commit frame over URSULA_VFS_MAX_FRAME_BYTES, 32 MiB after compression by default, the server's request body limit. That is about 8,000 changed 4 KiB pages of incompressible data, and about three times as many of typical text. A larger transaction fails alone with SQLITE_IOERR_AUTH, status() reports refused_kind: "too_large" with the frame's size and the limit, and later, smaller commits succeed without a new attach. A 413 with both headers counts the same. Before, the server's 413, or a connection it closed, poisoned the database.
  • Split bulk imports into smaller transactions. CREATE INDEX, REINDEX, a rebuilding ALTER TABLE and VACUUM are one transaction too. To VACUUM a database above the limit, first delete rows so that what it writes compresses below the limit, or copy the rows into a new database in transactions below the limit.
  • Behind a proxy or gateway with a lower body limit, set URSULA_VFS_MAX_FRAME_BYTES at or below it. It takes 64 KiB to 32 MiB. Otherwise a large commit usually meets a closed connection rather than a 413 (#540), and still poisons once the retry budget runs out.
• The extension never follows redirects. During a leadership change behind a gateway with access control, a redirected request used to go out without its bearer token, so an attach could fail as if the stream were missing, a snapshot read-back could fail, and a stream create could fail with "redirect failed". A 307 is now sent again to the same URL with the token, like a 503, within URSULA_VFS_RETRY_MS. Any other redirect, a 308 from a proxy included, fails the request at once with an error that names where it pointed. Point the stream URL at a gateway or at a single-node server, at the address a proxy would redirect to. A URL that names one node of a multi-node cluster works only while that node leads the stream, and a URL that a proxy redirects (from http:// to https://, say) fails. Snapshot publishes and retention moves now wait for the Retry-After of an answer they retry.
• Connections are kept for 90 s between requests, up from 15 s (URSULA_VFS_IDLE_MS, in milliseconds, 0 turns reuse off). A database that commits every 15 to 90 s reuses its connection instead of opening a new TCP and TLS connection for almost every commit. A request that gets no answer, such as one sent on a connection a proxy closed at that moment, is now sent again on a new connection: reads and HEADs within URSULA_VFS_RETRY_MS, where they used to fail the attach at once. Behind a NAT, a load balancer or a firewall that drops idle connections sooner than 90 s without telling either end, set URSULA_VFS_IDLE_MS below its limit. Resolving the stream's host and connecting to it, the TLS handshake included, now time out after 5 s each, so a host that does not answer no longer holds a request for its whole timeout.
• A snapshot that keeps failing backs off. It used to be read, compressed and sent again every 30 seconds, for example while it was larger than the gateway's --max-request-body-bytes. The next attempt now waits 100 ms after a failure and twice as long after each further one, up to an hour while the body is too large and up to five minutes otherwise, give or take a fifth. It starts at once after the database shrinks by more than a quarter, as after a VACUUM.
  • A snapshot over 1 MiB, and larger than any this owner published or installed, whose publish fails while the stream still answers counts as too large, and one that fails while the stream does not answer as an outage. A publish the server puts off, with a 503 that carries Retry-After or the marker, is an outage too, and the next attempt waits at least as long as it asked. After an outage, the first commit that succeeds brings the next attempt forward once, to within 30 seconds.
  • A snapshot publish over 1 MiB is sent twice at most per attempt, so a short outage during one now counts in snapshot_failures. A failed retention move is logged as retention_failed and tried again after the next snapshot, without counting as a failed snapshot. snapshot_failed gains retry_ms and kind (size, outage or other).
  • status(file) reports the latest snapshot body's compressed size as snapshot_bytes, and instrument() as the gauge sqlite_ursula.snapshot.bytes. Alert when it nears the gateway's limit. Behind Ursula's gateway, a snapshot a little over the limit is reported with a 413 and its size, and one far over it as a failed write (#540).
• Measurements are bounded, and instrument() drains them every second. An attached database no longer grows by up to 64 MB when nothing drains its measurements. Between drains it keeps the newest 8,192 commit, checkpoint and snapshot measurements, 512 KiB of commits. drainStats reports what it dropped in dropped. instrument() still counts every commit's bytes and retries, and counts the commits whose durations were dropped in the new counter sqlite_ursula.commit.unmeasured. It now drains every second by default instead of every 10 seconds, so its duration histograms see every commit up to about 8,000 commits a second. A file that commits faster needs a shorter intervalMs, and intervalMs: 10000 keeps the old cadence.
• A panic inside the extension no longer aborts the application that loaded it. The call fails instead: a database operation with an I/O error, after which the database must be attached again, and a SQL function with an error. The extension logs event=panic at level=error, with the entry point and the panic's message.
• @tonbo/sqlite-ursula declares the Pi Durable versions it supports: @earendil-works/pi-durable >=1.0.0 <1.2.0, where it was ^1.0.0. openUrsulaPiStorage passes Pi's storage conformance suite through the extension on 1.0.0 and on 1.1.0, including the scan order that 1.1.0 added to the Storage contract, so either version works without a code change. Keep your application's @earendil-works/pi-durable in the same range, so that npm installs one copy for both. With a version outside it, npm installs a second copy for @tonbo/sqlite-ursula without a warning, and openUrsulaPiStorage returns that copy's SqliteStorage. It implements the Storage contract of its own version, so a harness from another version can get wrong results from it. npm ls @earendil-works/pi-durable lists the copies.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests show that a failed write of the poisoned status fails closed.
• #517: a group sends each follower one Append call at a time, and since 0.7.2 a call carries at most 4 MiB of entries, so one group replicates to a follower at most about 4 MiB per round trip. That can slow the catch-up of a very busy group over a link with a long round trip.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described under 0.7.2.
• #519: under a sustained write load, the unsnapshotted Raft log of every node can reach its hard limit, twice raft.snapshot_log_budget, at about the same time, and the nodes then refuse client writes for about 20 seconds until snapshots catch up. These refusals now carry the marker RaftLogPressure, so behind a 0.7.3 gateway a SQLite VFS commit refused this way fails alone instead of poisoning the database.
• #529: POST /__ursula/raft/{group}/snapshot answers 500 after about 10 seconds when another group's snapshot build holds the node's build permit, because the deferred build is not retried for the request. Send the request again.
• #539: a node answers every leadership transfer it refuses with the gRPC status FAILED_PRECONDITION, also when its Raft core has stopped, so the sending leader cannot tell a stopped core from a refusal by policy.
• #540: the gateway answers some refusals before it reads the request body, such as an access-control refusal or a 413 for a body over --max-request-body-bytes, and then closes the connection. A client that sends the whole body before it reads the answer often sees a reset instead of the refusal. A SQLite VFS commit larger than one read that is refused this way is retried as an unknown outcome and poisons the database once the retry budget runs out, as in 0.7.2, and such a snapshot is reported as a failed write rather than a 413.
• #542: on macOS, a write to a socket that the peer already reset can fail with EINVAL, which the SQLite VFS does not retry as a request that got no answer. A read, HEAD, create or snapshot GET sent on a pooled connection that the server had just closed then fails the attach or the snapshot pass, where Linux retries it. Appends are not affected.
• A node shed for failed writes can shed again every few minutes. A node shed because only its own cold-flush writes fail, while its probes answer, hands off its leaderships and then flushes nothing, so it heals about a minute after its last failed write even if its writes still fail. It sheds again once the leadership balancer hands leaderships back and a flush fails, for as long as such an outage lasts.

0.7.2

0.7.2 is a drop-in patch for 0.7.0 and 0.7.1: format epoch 3 is unchanged, with no storage, snapshot or Raft protocol change. Upgrade in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0, 0.7.1 and 0.7.2 voters run side by side during the upgrade.

• Rust library API changed. ursula_raft::RaftGrpcMetricsSnapshot has a new field, raft_grpc_append_stream_frame_bytes_max.

Fixed

• Server memory no longer grows with uptime (#507). Each voter keeps one long-lived gRPC stream per peer and core for Raft replication, and the gRPC library keeps a stream's encode and decode buffers at the largest message the stream ever carried. A replication message could batch 32 calls of up to 300 entries each, so one burst of large writes, or one follower catching up, left every stream holding buffers of up to hundreds of MiB until a peer restarted. In the 0.7.0 soak, memory grew by about 0.2 GB per hour on nodes whose peers stayed up, and it stayed up after the load stopped. A replication read now returns at most 4 MiB of entries, and a message stops taking calls once it holds 4 MiB, so it stays under about 8 MiB. After the same burst of large writes, the replication buffers a node kept fell from between 1.2 and 1.7 GB to about 0.4 GB, and its memory from about 2.7 GB to between 1.2 and 1.8 GB. /__ursula/metrics reports the largest message sent as raft_grpc_append_stream_frame_bytes_max. A single entry larger than a read still travels alone and leaves its stream's buffers at its size (#518). With a cold store, writes of at least runtime.external_payload_min_size (1 MiB by default) are staged in object storage, so such an entry comes only from a restore or from a JSON append of several hundred thousand messages, whose message offsets stay in the entry. Without a cold store, request bodies of up to 32 MiB go through Raft, so leave memory headroom on such nodes.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests now show that a failed write of the poisoned status fails closed.
• #518: a single Raft entry larger than a replication read leaves its stream's buffers at its size until a peer restarts, as described above.
• #519: under a sustained write load, the unsnapshotted Raft log of every node can reach its 2 GiB hard limit at about the same time, and the nodes then refuse client writes for about 20 seconds until snapshots catch up. It happened once in two runs of 128 SQLite owners writing flat out on three voters.

0.7.1

0.7.1 is a drop-in patch for 0.7.0: format epoch 3 is unchanged, with no storage, snapshot or Raft protocol change. Upgrade from 0.7.0 in place with an ordinary rolling restart (helm upgrade). No fresh install or data move is needed.

0.7.0 and 0.7.1 voters run side by side during the upgrade. The 0.7.1 chart passes the gateway the new --upstream-tcp-user-timeout flag, which a 0.7.0 gateway refuses at startup, so a values file that pins global.image.tag or gateway.image.tag must move it to 0.7.1 together with the chart. SQLite VFS owners upgrade one at a time, as within any minor: the 0.7.0 and 0.7.1 extensions take over each other's streams.

• Rust library APIs changed. The ursula-control, ursula-raft (MetaRaftHandle migration methods, and RaftWal::open, which now returns RaftWalError with new variants), ursula-runtime (GroupInfraError::WalOpen), ursula-proto (ProcessIncarnation::try_from error type) and ursula-gateway Rust APIs changed, with no wire or on-disk change. GatewayConfig gains the public field upstream_tcp_user_timeout_secs (NonZeroU64), so code that builds it with a struct literal must set it.

Changed

• SQLite VFS: a 402 no longer poisons the file, and files can attach read-only. A layer in front of the stream, such as an authorizer or a quota or billing service, may refuse writes with 402 Payment Required. A commit it refuses now fails alone with SQLITE_IOERR_AUTH (UrsulaPaymentRequiredError through openUrsulaPiStorage). The file is not poisoned, reads keep working, and the next commit succeeds once the layer accepts writes again, without a new attach. status(file) reports payment_required and payment_reason. An attach it refuses fails with SQLITE_AUTH (UrsulaPaymentRequiredError from attach). attach(file, url, { readOnly: true }) (ursula_attach(path, url, 'read_only')) catches a file up without claiming the stream. It never writes the stream and fences no owner, and every write to the file fails with SQLITE_READONLY. See SQLite VFS.
• Release images carry their provenance and SBOM as GitHub attestations. The ursula and ursula-chaos-agent image indexes on GHCR hold only their linux/amd64 and linux/arm64 images, with no attestation manifests. Build provenance and an SPDX 2.3 SBOM are published as GitHub artifact attestations instead. Verify them with gh attestation verify oci://ghcr.io/tonbo-io/ursula:<version> --repo tonbo-io/ursula, and add --predicate-type https://spdx.dev/Document/v2.3 for the SBOM. GHCR receives stable releases only, with no 0.0.0-* candidate images or charts, and latest moves only to the highest stable release.

Fixed

• The gateway gives up on a connection to a vanished node within seconds (#495). A voter pod deleted with --force, or a host that loses power, leaves the gateway's pooled connections to it open without an answer. A request sent on one waited up to 30 s, reqwest's default TCP_USER_TIMEOUT, which is longer than the SQLite VFS's 10 s request timeout, so owners stalled for 10 to 22 s in failover tests. On Linux, upstream connections now give up after 5 s of unacknowledged data, set by the new --upstream-tcp-user-timeout flag (Helm gateway.upstreamTcpUserTimeoutSeconds). The request fails with 502, and the client decides whether to retry it. TCP keepalive follows the flag: an idle connection is probed after 2/5 of it and then every 1/5, each at least 1 s. With the default, three probes go out at 2, 3 and 4 s and a dead connection closes at about 5 s, so one lost probe does not close a healthy long-poll or SSE read. The flag takes at least 1 s.
• The gateway stops picking a node that does not answer (#454). After a transport failure it dropped the cached leader route but picked the next upstream at random, often the dead one again, until that node came back. After a host power-off, one stream failed through the gateway for about 60 s. An upstream that refuses or times out a connection, or breaks an exchange, is now skipped for 10 s, and each failure after a window doubles the next one, up to 60 s. A missing response within --response-header-timeout does not count, because one slow Raft group on a live node causes it too. When a window expires, one request probes the upstream while the others keep skipping it, for at most --connect-timeout plus --upstream-tcp-user-timeout. Any response puts it back. If every upstream is skipped, requests still go out, along their cached leader route first. /__ursula/gateway/metrics reports upstream_ejections and upstream_ejected.
• A node that lost its WAL run state recovers above the epochs its cores verified (#411). Without run-state.bin, 0.7.0 restarted the recovery epoch at 1, so a core that had already verified a later epoch took its journal as verified and skipped the recovery rewrite of its durable prefix. The next epoch now starts above the highest one any core verified. Empty journals whose core metadata records a verified epoch count as earlier history, so the node recovers behind the recovery gate instead of starting as a new disk.
• Snapshot pointers survive power loss and failed writes (#411). When a group's snapshot pointer write failed after its rename, for example in the directory fsync, the node could delete the snapshot object that the new pointer names, and a restart that found that pointer could not restore the snapshot. That object now stays in object storage. Later pointer writes of that group fail with snapshot metadata publication stopped after an earlier failure until the node restarts, and a restart fsyncs the pointer's directory before it releases older snapshots. A directory created for a pointer is now fsynced before the pointer counts as published.

Known issues

• #411: residual risks from the adversarial review of recovery stay open. An acknowledgement that a replica sent before a crash still counts toward commit if it reaches the leader after the replica's post-restart barrier, which needs a network delay longer than a reboot. The ordering of directory operations on real filesystems has not been reviewed. Tests now show that a failed write of the poisoned status fails closed.
• #507: in a soak with a steady write load, server memory grew by about 0.2 GB per hour of uptime and stayed up after the load stopped, while a restarted node used less. Leave headroom in the servers' memory limit and watch their memory. Fixed in 0.7.2.

0.7.0

0.7.0 is a breaking release. It changes the Raft WAL format (format epoch 3) and removes the memory WAL, so the per-core disk journal is the only Raft WAL. 0.7 cannot read 0.6 data: a node refuses 0.6 Raft WAL directories and object-storage prefixes at startup, ursulactl restore refuses 0.6 backups, and a node will not join a cluster with 0.6 nodes. There is no in-place upgrade, no automatic migration and no rollback.

• If you run 0.6 with data you need to keep, stay on 0.6.2. 0.7 provides no migration path from 0.6.
• Deploy 0.7 fresh: empty PVCs and a never-used object-storage prefix.
• raft.wal.fsync now defaults to never. The 0.6 disk WAL fsynced every batch. Set always to keep that behavior.
• helm upgrade from 0.6 stops before a second voter is replaced. Recover with helm rollback.

This release adds an fsync policy and a recovery gate to the Raft WAL, splits each core's journal into segments with bounded memory, binds the WAL to the routing configuration, and removes the memory-WAL restart tooling.

Installing 0.7.0

See Upgrading to 0.7.

1. Install fresh. Install with a new s3.prefix and empty PVCs (with cold storage off and S3 snapshots, also a new snapshotStore.prefix). With the event-time indexer, use a new or emptied indexer.s3.prefix and register the indexes again.
2. Never reuse a prefix that an earlier release, or a failed upgrade attempt, wrote to.
3. Update the configuration. Remove the keys and chart values listed under Removed in 0.7.0. Set raft.wal.path on every node of a multi-node cluster: 0.6 defaulted to the memory WAL, and a 0.7 node with raft.peers refuses to start without a WAL path. Choose raft.wal.fsync (Helm raft.walFsync) as described under WAL fsync policy.
4. Update runbooks and tooling. Take the removed ursulactl verbs out of scripts. Custom tooling that calls admin mutations must send x-ursula-process-incarnation (Changed).

A values file that still sets a removed chart value fails validation, and nothing changes. Otherwise helm upgrade from 0.6 replaces the first pod, which finds the 0.6 data on its volume and exits at startup, so no second voter is replaced. Recover with helm rollback. With OnDelete, also delete the refused pod after the rollback so that it starts from the 0.6 template again. The refused pod wrote nothing. A 0.7 node that contacts 0.6 peers makes each of them answer 503 with format_epoch_mismatch on /__ursula/ready until it restarts, so keep the peer lists of 0.6 and 0.7 nodes apart.

WAL fsync policy

raft.wal.fsync (Helm raft.walFsync) chooses when journal appends reach stable storage. Votes, each group's log state and the node's run state are always written with fsync.

• never, the new default. A replica acknowledges an ordinary data append after writing it to the page cache, without fsync. A data write is acknowledged once a quorum of voters holds it in the page cache. Membership batches are always fsynced before acknowledgment, including the bootstrap membership. A process crash loses nothing, and a graceful shutdown fsyncs every journal. A host crash or power loss can cost that replica the unsynced tail of its journals. While at most a minority of a group's voters lose their tail, the other voters hold every acknowledged write. If a majority of a group's voters crash at once, writes acknowledged after their last fsync can be lost, and the group stops until an operator accepts that loss.
• always. A replica acknowledges a batch only after fsync, as the 0.6 disk WAL did. A host crash loses no acknowledged write. Each batch waits for an fsync, which costs write latency.

See Configuration and Recovering after a host crash.

Recovery gate

A replica that may have lost acknowledged writes recovers behind a gate for each of its groups. That covers a host crash under never, a run stopped by a journal I/O error, a wiped or replaced WAL directory, a missing run state next to a non-empty journal, and a damaged journal tail.

• A gated replica neither votes nor campaigns until it has applied a fresh ReadIndex barrier from the group's leader. A replica that led the group restarts as a follower. Meanwhile /__ursula/ready answers 503 with reason recovery_gate_closed, and recovery_gates in /__ursula/metrics lists the gated groups.
• The leader rebuilds a replica that lost entries it had acknowledged. It removes the voter, adds it back as a learner and promotes it once it has caught up. While a majority of the group's voters are intact, no operator step is needed.
• A replica whose WAL directory was wiped also lost its vote. It refuses replication until the group's leader proves a fresh quorum without it. In a 3-voter group that needs both other voters up.
• If a majority of a group's voters are gated, the group has no leader and refuses writes. After 30 seconds the gated replicas report the group stalled (readiness reason recovery_stalled). An operator then runs POST /__ursula/raft/{group}/recovery/accept-unsynced-loss on the gated replicas with the most up-to-date last log id (highest last_log_term, then last_log_index) until the open replicas form a majority. The request names the replica's last_log_index and current_term. It answers 409 and changes nothing when the gate is not stalled, when the replica no longer holds that log, or when the replica lost its vote and has no durable vote floor.
• Under always, a host crash can still leave a complete invalid frame after the last fsync in a journal's newest segment, for example a torn sector inside an unacknowledged batch. Recovery cannot tell it from media corruption, so it gates every group on that core. A single-voter group, or a group gated on a majority of its voters, then needs accept-unsynced-loss although no acknowledged write was lost.
• A single-voter group has no other replica to recover from. Under never it needs accept-unsynced-loss after every host crash, once its gate reports the group stalled.

See Recovering after a host crash and Rejoining a wiped node.

New

• Segmented journal with bounded memory. Each core's journal is a run of segments (raft.wal.segment_size, default 64 MiB, Helm raft.walSegmentBytes). Purge deletes whole segments, and the few entries quiet groups leave in old segments are copied forward. Each group keeps an index of its entries and a cache of recent ones (raft.wal.cache_size, default 256 MiB per node, Helm raft.walCacheBytes). Older entries, such as a lagging follower's catch-up, are read from disk. /__ursula/metrics reports the new work under wal_segments, wal_rotations, wal_rewritten_bytes, wal_pinned_segments, wal_lagging_groups and the wal_cache_* and wal_disk_read* counters.
• WAL layout guard. {raft.wal.path}/raft-log/topology.bin records runtime.core_count and raft.group_count. A node refuses to start when either differs from its configuration, and when an existing WAL has no record or a corrupt one. Before, a changed core count moved groups to other journals and a changed group count remapped streams. Keep both values explicit. See WAL routing configuration.
• Graceful voter shutdown. On the first SIGTERM or Ctrl-C a voter marks itself drained and hands each leadership to a caught-up voter for up to 5 seconds. It then drains its listeners for up to 10 seconds, stops its groups, fsyncs every journal and records a clean shutdown. The whole shutdown has a deadline of 20 seconds from the signal. A shutdown that has not finished by then exits without recording a clean shutdown, and the next start treats the run as a crash. The chart's new server.terminationGracePeriodSeconds defaults to 30 to fit. See Graceful voter shutdown.
• Raft work stays on the owning core. Raft RPCs from peers run on the core that owns the group, and each core keeps its own peer connections. A group's journal append returns once the batch is queued, and Raft counts the entries once the fsync policy is met, so a group no longer waits for one journal write before it queues the next. raft.append_transport_budget_bytes (default 128 MiB) caps the append bytes queued for Raft transport in each direction across the node.
• CPU pinning. On Linux, each core's worker thread is pinned to a CPU when the process may use exactly runtime.core_count CPUs. runtime.cpu_affinity = true forces pinning and false turns it off.
• Fresh-quorum checks. ursulactl verify-quorum confirms each group's leader with a fresh ReadIndex round and waits until every replica has applied that prefix (admin route GET /__ursula/raft/{group}/quorum). ursulactl verify-survivors runs the same check on the two remaining voters when one is down. See ursulactl.
• Maintenance fencing for rollout tooling. ursulactl pin-incarnations, activate-maintenance-fence, retire-maintenance-fence, startup-admit and the reservation-* verbs, with the chart values server.gracefulRollout.maintenanceReservation and server.startupOwnership, serialize planned voter replacements through a shared reservation and bind them to process and host identity. Both chart values default to false. Reservation stores and verify-quorum proofs each have one schema version, and ursulactl refuses any other.
• Restore checks cold objects first. ursulactl restore asks the target cluster to check every group's cold references against its cold store (POST /__ursula/backup/group/{group}/cold-check). If any object is missing, it imports nothing and names the first missing keys. See Backup and disaster recovery.
• SQLite VFS on npm (experimental). The TypeScript helpers are published as @tonbo/sqlite-ursula, with the extension prebuilt for linux-x64-gnu, linux-arm64-gnu, linux-x64-musl and darwin-arm64. The VFS is experimental: its formats and API may change in a minor release (#468). It speaks https:// and sends a bearer token to a gateway with access control (URSULA_VFS_TOKEN_FILE or setToken). Attach refuses a file whose stream is missing, which may hold the only copy of its data, instead of starting it over. A WAL write-back that the local file lost poisons the database before a commit builds on it, and on Linux before a published snapshot does. status(file), instrument(file, meter) and logfmt lines on stderr report the owner's health. See SQLite VFS.

Changed

• Admin mutations are bound to one server process. Every mutating admin request needs x-ursula-process-incarnation, the process_incarnation that GET /__ursula/metrics reported before the operation. Without it a node answers 428. A node that restarted since then answers 412 and changes nothing. ursulactl sends the header. Custom tooling must add it, bucket purge included. See Admin endpoints.
• Format epoch 3. The Raft WAL journal and state files, the WAL directory and object-storage markers, group and stream snapshots, snapshot references, backups (format_version 3) and the Raft gRPC protocol carry format epoch 3. A node refuses data, backups and peers of another epoch.
• A journal write or fsync error stops the process. The journal writer retries nothing and aborts the process. The next start keeps the verified prefix of the journals and puts the node's groups behind the recovery gate. Journal frames now carry checksums over their header, payload and journal sequence, so a damaged length is reported as corruption. Recovery refuses damage in sealed segments and repairs only the tail of the newest one.
• Bootstrap. A replica that ever held a group never initializes it again. A replica that holds nothing initializes the group only when every configured voter reports an empty group and no leader. This replaces the 0.6.2 restart guard, and 0.7 neither writes nor reads its URSULA_GROUP_INITIALIZED/ markers in object storage.
• Raft without a WAL path. 0.6 defaulted to the memory WAL. Every preset other than default, and a config file loaded without --preset default, now runs Raft on the disk WAL. A single node without raft.wal.path keeps its WAL in a temporary directory that it logs at startup and removes on a clean shutdown. The Docker image's default configuration runs this way.
• Chart rollouts. The graceful-rollout hook runs drain, delete, wait and undrain for one voter at a time. An external OnDelete controller runs the same sequence.
• Leadership transfer errors. POST /__ursula/raft/{group}/leader/transfer/{node} answers 409 for a changed leader or a recovering target, 400 for an invalid target, 404 for an unregistered group and 500 for a stopped Raft. ursulactl drain observes again and plans again after a 409.
• Leader forwarding is bounded. A follower that forwards a read, HEAD or bucket purge to a leader it cannot reach answers 503 with Retry-After within about 3 seconds, or 10 seconds when the leader accepts the call and never answers. Before, it answered 500 once the kernel gave up on the connection, about 16 minutes after the leader's host lost power, and the group's later requests waited behind it. Connections between nodes now send HTTP/2 PINGs to detect a silent peer.
• Gateway upstream failures. ursulagw forgets a cached leader when its upstream fails at the transport. An upstream it cannot connect to answers 503 with Retry-After, and one that sends no response headers in time answers 504. Both used to answer an empty 502.
• Readiness checks the node's own Raft replicas. /__ursula/ready answers 503 with the reason raft_replica_unready while one of the node's replicas is not a healthy voter, for example a wiped replica that its leaders rebuild as a learner. It also adds the reasons recovery_gate_closed and recovery_stalled and the list recovery_stalled_groups, and the admin plane now serves it next to /__ursula/metrics. Readiness does not report lost redundancy: healthy surviving voters stay ready while a replica rebuilds. Alert on incomplete_voter_set in raft_maintenance.group_issues instead (see Observability).
• Metrics. /__ursula/metrics adds process_incarnation, raft_maintenance, wal_recovery, recovery_gates and each group's last_log_term, and OpenTelemetry adds the gauges ursula.wal.recovering and ursula.raft.recovery_gates. memory_wal_full_restart_groups and the wal_backend metric are gone.
• ursulactl wait-ready waits for full redundancy. Besides a leader for every group, it requires each node's raft_maintenance report to show the complete voter set, no uncertain maintenance fence, and 200 from /__ursula/ready. It retries connection failures until its timeout.
• Retention reclaims external payloads it drops from state. An append of at least runtime.external_payload_min_size keeps its object's locator in replicated state until the leader moves it into a cold-index page. Retention that passed such an object before that dropped the locator without queueing the object, which then stayed in object storage until the cold orphan sweep deleted it, a day later. Retention now queues it for the cold GC with the same 5-minute grace as dropped pack slices (#475).

Removed in 0.7.0

• The memory WAL and its configuration keys: raft.wal.backend, raft.wal.allow_volatile_multi_peer and raft.memory_bootstrap_marker_dir, and also raft.rejoin_probe. A config file that still sets one fails to load.
• Chart values raft.storageMode, raft.allowVolatileMultiPeer and persistence.enabled. Every server pod gets a raft-data PVC, and raft.logDir must be set. The values schema refuses unknown keys under raft and persistence.
• ursulactl verbs prepare-restart, restart-quiesce-capability, finish-prepared-restart, abort-prepared-restart, prepare-amnesiac-restart, repair-restarted-voter, prepare-recovery-handoff and classify-amnesiac. A planned restart is drain, restart, wait and undrain (see Restarting a voter). A voter that lost entries needs no verb: its group leaders rebuild it.
• Admin endpoints POST /__ursula/raft/quiesce-for-restart, POST /__ursula/raft/{group}/rejoin/adopt-survivor/{node} and POST /__ursula/raft/{group}/rejoin/reinitialize. accept-unsynced-loss replaces the last two.

Known issues

• #411: an adversarial review of recovery left residual risks open, mostly liveness. For example, a node that loses its run state file can restart a recovery epoch that its cores already verified, and a failure to record the poisoned status is not yet covered by tests. 0.7.1 fixes the run state case.
• #454: after a voter host loses power, ursulagw keeps choosing the dead upstream at random until its pod returns. Each such attempt waits for the 5 second connect timeout, so a stream can fail through the gateway for about a minute although the cluster elected new leaders within seconds. Clients that connect to the servers directly are not affected. Fixed in 0.7.1.

0.6.2

0.6.2 is a drop-in patch for 0.6.x: format epoch 2 is unchanged, with no change to existing storage records, snapshots or the Raft protocol. It adds one small marker object per memory-WAL group in object storage, which 0.6.0 and 0.6.1 ignore. Upgrade from 0.6.0 or 0.6.1 in place with an ordinary rolling restart (helm upgrade); no fresh install or data move is needed.

0.6.0 or 0.6.1 voters and 0.6.2 voters run side by side safely during the upgrade; a group whose leader still runs 0.6.0 or 0.6.1 is rebuilt by the graceful-rollout hook as before, which runs the ursulactl of the chart's image.

Fixed

• A memory-WAL node restarted alone heals itself. It no longer initializes the groups it bootstraps again (it does so only when every configured voter reports an empty group and no leader), and the leaders rebuild it with no operator: a leader that sees a follower lose entries it had acknowledged commits a voter set without it, adds it back as a learner, and promotes it once it has caught up, the same sequence as ursulactl repair-restarted-voter. The steps are read off the current membership, so a leader change, a second restart or repair-restarted-voter running at the same time converge. Leadership handoffs skip such a follower until it is rebuilt.
• An empty memory-WAL replica never votes before it has caught up. Until it holds the log its leader had committed when it first heard from it, it refuses every candidate with a non-empty log, so it can no longer help elect a leader that lacks entries committed with its lost copy. A fresh group's first election is unaffected.
• A majority of a memory-WAL group restarted at once stops the group instead of losing writes. No leader is elected from the incomplete logs and writes are refused (503 while no leader exists) until an operator adopts the survivor's log with POST /__ursula/raft/{group}/rejoin/adopt-survivor/{node} on every node of the group. See Restarting a memory-WAL node.
• A restart of every memory-WAL node stops the groups that held writes instead of re-initializing them empty. Each memory-WAL group keeps an "initialized" marker in object storage, URSULA_GROUP_INITIALIZED/group-{id} beside the format-epoch marker; a leader writes it before the group's first write and acknowledges no write without it, and a replica of a group that already holds writes (created by 0.6.1) writes it as well. When every voter reports an empty log and the marker exists, the group stays without a leader, refuses writes (503), and is listed under memory_wal_full_restart_groups in GET /__ursula/ready on its initializer, until an operator accepts the loss with POST /__ursula/raft/{group}/rejoin/reinitialize?accept_data_loss=true on that node. Without a marker (a fresh install, or every node crashed before the first write) the group bootstraps as before; if the marker cannot be read, it retries and never initializes blind. Without object storage a full restart cannot be detected and the groups still initialize again, empty. 0.6.1 ignores the marker, so rolling upgrade and downgrade stay safe; an upgraded cluster is protected once every pod runs 0.6.2.
• ursulactl repair-restarted-voter accepts a group the leader already rebuilt.

0.6.1

0.6.1 is a drop-in patch for 0.6.0: format epoch 2 is unchanged, with no storage, snapshot or Raft protocol change. Upgrade from 0.6.0 in place with an ordinary rolling restart (helm upgrade); no fresh install or data move is needed.

The chart's graceful-rollout hook runs the ursulactl of the chart's image, so the upgrade to 0.6.1 already uses the fixed readiness check below.

Fixed

• A memory-WAL voter restarted empty no longer passes readiness. On restart it rejoins as a voter with an empty log for the groups it bootstraps, and an idle leader never backfills it. ursulactl counted such a voter ready whenever the peers' commit index was within --lag-tolerance, so repair-restarted-voter skipped those groups and a rollout continued with two real copies. A voter that has applied nothing while its peers have committed entries is now never ready, and repair-restarted-voter rebuilds it through the learner catch-up path; classify-amnesiac counts such groups as missing for amnesiac recovery, so an empty voter outside --lag-tolerance is now recoverable instead of refused.
• The leadership balancer hands a group only to a caught-up peer. It used to hand leadership to a voter that had not replicated the leader's log; the transfer then left the leader muted while the target could not win, and the next restart fence timed out with "leader … is stale". The balancer now waits until the leader's replication metrics show the target matched its last log entry.
• The restart fence waits out an in-flight leadership transfer. The restart fence (prepare-restart, prepare-amnesiac-restart, repair-restarted-voter) now waits 1.5 s (the Raft minimum election timeout) after fencing the surviving voters before it checks leadership, so a transfer accepted just before the fence settles first.

Known issue (fixed in 0.6.2): a restarted memory-WAL node rejoins as an empty voter

A node on raft.wal.backend = "memory" restarts with an empty Raft log, but the group leaders still count it as a voter, do not resend what it lost, and it grants its vote to any candidate. Until it is rebuilt, each of its groups holds two real copies while reporting three, and one more failure can lose acknowledged writes. A prepared rollout rebuilds it (repair-restarted-voter, fixed above); a crash, OOM kill, eviction or node loss does not. Production clusters must use the disk WAL. On a memory-WAL cluster, after any unplanned restart run ursulactl verify-cluster and, if it reports the node unready, ursulactl repair-restarted-voter --node N. See Restarting a memory-WAL node.

0.6.0

0.6.0 is a breaking release. It changes the storage format (format epoch 2) and removes several protocol features (listed in the release notes). It refuses to start on data written by 0.5.x (Raft WAL directories, object-storage prefixes, snapshots and backups) and will not join a cluster with 0.5.x nodes. There is no in-place upgrade, no automatic migration and no rollback.

• If you run 0.5.x with data you need to keep, do not upgrade yet. Stay on 0.5.1 and open an issue so we can work out a migration path with you.
• New deployments install on fresh storage: empty PVCs and a never-used object-storage prefix.
• helm upgrade from 0.5.x stops at the first pod. Recover with helm rollback.

This release changes the storage format to epoch 2, removes the protocol features listed under Removed, stores JSON messages exactly as written, and stops per-stream replicated memory from growing with message count or cold history, without retention (see Memory without retention).

Installing 0.6.0

See Upgrading to 0.6.

1. Install fresh. Install with a new s3.prefix and empty PVCs (with cold storage off and S3 snapshots, also a new snapshotStore.prefix). With the event-time indexer, use a new or emptied indexer.s3.prefix and register the indexes again (Event-time indexer).
2. Never reuse a prefix that an earlier release, or a failed upgrade attempt, wrote to.
3. helm upgrade from 0.5.x stops at the first pod. The new pod exits at startup or reports not ready (/__ursula/ready answers 503 with format_epoch_mismatch), so no second voter is replaced and the refused pod writes nothing. Recover with helm rollback.
4. SQLite VFS. Stop every owner of a stream before upgrading the extension, then delete <db>, <db>-wal and <db>-ursula or attach under a new stream URL (SQLite VFS).

Always on

Every group runs one behaviour from its first command; there are no feature levels to raise. Always on: bounded producer receipts (1,024 receipt items per stream plus each producer's newest; older duplicates are answered 204 without ranges), 7-day idle-producer expiry, the 4,096-producer cap, unique incarnations, external locators committed before indexing, the compact hot window (no per-message bookkeeping), and cold-tier snapshot bodies up to 1 GiB.

Observable changes for JSON streams

Every application/json stream now follows JSON message text: Ursula validates the body and stores each message as the writer's text with insignificant whitespace removed, followed by one LF. Compared with earlier releases, which re-serialized every message:

Input	Before	Now
{"b":1,"a":2}	{"a":2,"b":1} (members sorted)	{"b":1,"a":2} (writer's order)
{"a":1,"a":2}	one a kept	both members kept
1.50e3	1500.0	1.50e3
1e400	400 Bad Request	stored as written
"\ud800" (unpaired surrogate escape)	400 Bad Request	stored as written
[{"a":[…127 levels…]}]	depth counted including the outer array	depth counted per message: a message inside an array body may be as deep as a bare one (127)

• Every read surface returns the stored text: NDJSON reads, SSE and bootstrap parts.
• Offsets, ETags and byte counts refer to the stored text, so a message's size can differ from what the previous release would have stored for the same input.
• Clients that parse records must accept unpaired surrogate escapes and numbers outside the double range. Ursula's event-time indexer does.

New

• Stream-Incarnation as a request precondition. Every request that addresses a stream may carry Stream-Incarnation; the server compares it with the stream's incarnation atomically with the request (a write when it is applied) and answers 412 Precondition Failed (a refused write applies nothing) when the stream was deleted and recreated, including by TTL expiry and a later create. A PUT with it for a missing stream answers 412 and creates nothing; a 404, 410 or 416 stays as it is; a long-poll or SSE read with it ends when its stream is recreated. Without the header nothing changes. A read refused by the precondition returns no data but, like any request that reaches a stream, may still renew the sliding TTL of the stream now at that path. Every successful stream response except a DELETE now carries Stream-Incarnation, not only HEAD. The SQLite VFS sends it on every request after attach, so an owner of a deleted stream can no longer reach the stream recreated at its path, and the event-time indexer sends it on reads. See Stream incarnation.
• Metrics. /__ursula/metrics adds group_state_gauges (per-group state sizes).
• External payload locators. External appends commit their locator in replicated state first and the leader indexes it afterwards, so runtime.external_payload_min_size may be lowered.
• Compact hot window. Replicated state keeps nothing per message, and cold admission and flush thresholds charge hot payload only.
• Snapshot bodies in the cold tier. With a cold backend, snapshot bodies of at least runtime.external_payload_min_size are streamed into the cold tier instead of replicated state, and bodies of up to 1 GiB are accepted (32 MiB inline). Snapshot reads and /bootstrap stream them back and send Content-Length. Superseded bodies are deleted five minutes after the publish that replaced them.

Changed

• Bootstrap is an honest partial. /bootstrap never reads cold storage and never skips a byte. When updates after the snapshot are already cold, it returns the snapshot part alone with Stream-Next-Offset at the snapshot point and no Stream-Up-To-Date; when updates exceed 8 MiB, a JSON stream stops at the last message boundary within the cap and any other stream answers the snapshot part alone. Clients continue from Stream-Next-Offset with ordinary reads, as the specification already required. On a JSON stream each update part holds exactly one message, split on LF, so a JSON array append now appears as one part per element.
• Producer receipts are bounded. Each stream keeps exact receipts for 1,024 items plus each producer's newest one. An older duplicate is still deduplicated, but answered 204 without range headers. A producer idle on a stream for 7 days is forgotten and must restart at sequence 0 or bump its epoch. See Exactly-once writes.
• Producer-Id and Stream-Seq are capped at 256 bytes on every write path, with 400 above it.
• Unique incarnations. A stream's created_at_ms strictly increases within a Raft group, so a stream deleted and recreated at the same path is always a distinguishable incarnation, and stream GC only removes the deleted incarnation's objects.
• Snapshots are written in a compact MessagePack envelope that earlier releases cannot read.
• Producer cap. A stream holds at most 4,096 producers; a new producer evicts producers idle for at least an hour, and when none is, the write answers 429 with a body starting producer_limit.
• Reads are capped at 8 MiB. A larger max_bytes is clamped. A capped offset read may end inside a message, on a JSON stream too: an offset read without max_bytes, or with max_bytes above 8 MiB, no longer ends at a message boundary. Continue from Stream-Next-Offset.
• Binary bootstrap returns one part. On a stream that is not application/json, /bootstrap returns the hot updates after the snapshot as one part, which may contain several messages; a part boundary is not a message boundary. When those updates exceed 8 MiB, or an external append among them is not hot, the response is the snapshot part alone.
• Non-JSON snapshot and retention offsets need no message alignment. Any offset between the retained offset and the tail is accepted; an offset inside a message used to answer 400 or 204 depending on whether it had been flushed.
• JSON snapshot and retention offsets must follow an LF, and the check fails closed. On an application/json stream, PUT …/snapshot/{offset} and PUT …/retention/{offset} answer 400 for an offset inside a message, whether its bytes are hot or already flushed (an offset below the flushed point used to be accepted), and 503 when the server cannot read the byte before the offset to check it (it used to skip the check). Retention is exact: it no longer lands below its target. Snapshot publishes and retention advances no longer cost a linearizable HEAD on every stream; a flushed JSON boundary costs one leader read.
• Hot tails are flushed after storage.cold.flush_max_hot_age (default 5 minutes, 0 disables), even when the group is below its flush threshold.
• Defaults. storage.cold.compaction_enabled now defaults to true, and storage.snapshot.backend defaults to auto (S3 when the cold backend is S3, inline otherwise). Raft snapshots are taken by log bytes against a node-wide raft.snapshot_log_budget (default 1 GiB) with a raft.snapshot_backstop_logs entry backstop.
• Format epoch 2. The Raft WAL directory and the object-storage prefix carry a format-epoch marker, every snapshot starts with a format-epoch frame, snapshot pointers are MessagePack only, the WAL journal header, the Raft gRPC protocol (now also on forwarded reads and writes), S3 snapshot references and backups are version 2, and ursulactl checks the target cluster's backup format before the first import. A node checks its WAL directory, its object-storage prefix and its peers before it writes anything, and /__ursula/ready answers 503 with format_epoch_mismatch after any later Raft protocol mismatch.
• Reserved bucket IDs. snapshots (the default raft-snapshot namespace in the object store) and IDs starting with __ answer 400.
• PUT /{bucket} validates the bucket ID. It answers 400 for an ID that does not match [a-z0-9_-]{4,64} and 201 otherwise, and still changes no state: buckets are implicit namespaces. See Buckets.
• Helm chart defaults. The chart now renders coldStorage.compaction.enabled: true and snapshotStore.backend: auto (previously false and inline), so a release with cold storage gets compaction and S3 snapshots unless its values set the old ones.
• TTL renewal is monotonic. A read or write never moves a stream's expiry earlier. Before, a request stamped by a node whose clock lagged the node that last renewed the stream could shorten the stream's life by that lag.
• consistency=leader reads are linearizable. Before it answers a consistency=leader read, HEAD, /bootstrap, a snapshot read or a long-poll/SSE registration, the leader confirms with a majority of voters that it still leads (Raft ReadIndex) and waits until it has applied everything committed up to that point. Each such request costs one round trip to the voters, shared by concurrent requests to the same Raft group. On the leader, the confirmation runs before the request reaches the group, so appends and other reads to that group do not queue behind it. offset=now without consistency=leader takes no confirmation and may resolve to a lagging tail right after a leader change. A long-poll or SSE read takes its offset=now position, existence, content type and incarnation from its registration, and its first read is served only by the leader that registered it. A long-poll also answers 416 at registration for an offset beyond the tail (an SSE session reports it as an error event from its first read), ends when the stream it registered against is recreated even without Stream-Incarnation, and on timeout answers 204 at the requested offset. A leader cut off from the majority, or one that cannot confirm in time, answers 503 with Retry-After instead of serving state that may miss appends a newer leader acknowledged. Before, a deposed leader could serve these reads stale, and a follower dropped consistency=leader when it forwarded the read.
• SQLite VFS: opaque offsets. ursula_attach, ursula_status, ursula_stats and @tonbo-io/sqlite-ursula (attach, status, drainStats) return stream offsets as the server's strings ("-1" for none) instead of numbers; compare them as strings. The extension never computes an offset: it verifies its claim by reading from the tail it caught up to, and every commit carries a Stream-Seq, so a foreign append with a higher one fences the owner. The sidecar (<db>-ursula, format 2) and the snapshot body (USS2) changed: the first attach after the upgrade rebuilds the local file, and a stream whose latest snapshot is an older body (USS1) cannot be attached: attach under a new stream URL (deleting the local files is not enough).
• Bucket purge is admin-plane only. DELETE /__ursula/purge/{bucket} is served only on the admin listener (server.admin_listen, loopback by default); the client listener answers 404. A Raft group whose leader is unknown or changing answers 503 with Retry-After instead of redirecting (307) to a peer's client URL; purge is idempotent, so retry it. /__ursula/usage stays on both listeners. See bucket purge.

SQLite VFS

• Trusted network only. The extension speaks plain HTTP to Ursula, with no TLS and no credentials. Run it, and the Ursula or ursulagw endpoint it talks to, on a network you trust.
• The local file is a cache. A commit is durable once Ursula acknowledges it; the extension no longer fsyncs the WAL or the sidecar on every commit. After every reboot (a new kernel boot id), and whenever the local files cannot be verified, attach rebuilds the database from its latest snapshot plus the tail since it. In the 0.6 benchmarks a fresh host took 16 to 27 s to attach a 1 GB database, so a fleet-wide rolling reboot is a burst of snapshot reads, and a stream whose snapshots cannot be published replays from the last one it has. See the SQLite VFS design.
• Owners are bound to a stream incarnation. The extension's Producer-Id is sqlite-ursula-vfs/<Stream-Incarnation> and every request after attach carries Stream-Incarnation, so an owner of a deleted stream is fenced instead of writing into the stream recreated at its path. Versions with the old Producer-Id and this one do not fence each other: stop every owner of a stream before upgrading (or downgrading) the extension.

Event-time indexer (experimental)

• API break. Query and status return (offset, len) locators with opaque next and through tokens instead of record ordinals. The 0.5.1 query parameters after_captured_at_ms, after_record and through_record answer 400, and unknown registration fields answer 422. Registration takes extract (JSON pointers, with fallbacks and units) and start; the legacy timestamp_field is still accepted. Re-register every index under a new or emptied indexer.s3.prefix.
• NDJSON sources (application/x-ndjson) are indexed, and nested or unit-ful timestamps (OTLP JSON) are supported. Messages without a usable event time are skipped and counted instead of blocking the index.
• Chart values. indexer.workers.segmentBytes replaces readBatchRecords and segmentRecords.
• Polling cost. Every indexer pod sends a HEAD (always confirmed by a ReadIndex) for every registration on every poll (indexer.ingest.pollIntervalMs, default 250 ms), and each one costs the stream's leader a ReadIndex round trip to its voters, even when the stream is idle. Keep registrations few, or raise the poll interval.

Removed

• Feature levels. /__ursula/feature-level has no route any more (the admin listener answers 404), the 409 with feature_not_enabled is gone, the per-group feature_level gauge is gone from /__ursula/metrics, and ursulactl cluster feature-level and cluster enable-feature are gone.
• Keyed streams and clients/pi-durable-ursula. profile=keyed-batch-v1 is an inert media-type parameter, {stream}/keyed-state answers 404, and [keyed_state], server.keyed_state_upstream and the indexer's keyed mode are gone. Pi Durable runs on clients/sqlite-vfs and clients/sqlite-ursula.
• Data-plane bucket quotas. PUT /__ursula/quota/{bucket} is gone from both listeners, and creates and appends no longer answer 429 for a bucket's stream count or retained bytes. The per-group limits were a backstop worth limit x group_count cluster-wide. Nothing in Ursula replaces them: the gateway's --quota-policy limits request rate, live reads and body size only (Security). Cap a tenant's streams or bytes in your own control plane, using /__ursula/usage. The 429 with a producer_limit body is unchanged.
• Bucket lifecycle rules in the extensions specification. Explicit bucket creation, bucket metadata, stream listing and bucket delete (Sections 1.2 to 1.5) were never implemented and are withdrawn; see Buckets. Removing a bucket stays an operator action (bucket purge), which permanently fences the bucket ID: later stream creates in it answer 404.
• Stream attributes. GET and PUT /{bucket}/{stream}/attrs are gone, a Stream-Attrs header on create is ignored, and create idempotency no longer compares attributes. Keep application metadata about a stream in your own store.
• Snapshot delete, the latest-snapshot redirect and Stream-Snapshot-Match. DELETE /{bucket}/{stream}/snapshot/{offset}, which never succeeded, and GET /{bucket}/{stream}/snapshot, which redirected (307) to the latest snapshot, now answer 405 from a node (an access-controlled ursulagw answers 404 for the DELETE, as for any unrecognized route). To read the latest snapshot, take Stream-Snapshot-Offset from HEAD and GET /{bucket}/{stream}/snapshot/{offset}. A snapshot publish carrying Stream-Snapshot-Match answers 400 rather than publishing unconditionally.
• Append Batch and Prefer: return=minimal. POST /{bucket}/{stream}/append-batch and its affinity form are gone, and Prefer: return=minimal is ignored. To send several messages in one request to a JSON stream, POST a JSON array: it is committed atomically under one producer sequence. /__ursula/metrics no longer reports the 18 raft_write_many_* series.
• Path affinity and $transaction. Three-segment stream paths (/{bucket}/{affinity}/{stream} and their subresources), POST /{bucket}/{affinity}/$transaction, the admin route /__ursula/flush-cold/{bucket}/{affinity}/{stream} and the path-affinity-v1 and group-append-transaction-v1 extension tokens are gone; such paths answer 404. Streams created under the three-segment form are not carried over. Address every stream as /{bucket}/{stream}; there is no atomic append across streams, but a JSON array POST is atomic within one stream. A stream ID may no longer start with $ (400); that prefix is reserved for future bucket-level subresources. $transaction, append-batch, attrs, bootstrap, retention and snapshot stay reserved as stream subresource names.
• JSON record coordinates. Record ordinals and everything built on them are gone: the read parameters record= (including record=now), tail_records, max_records and record_view (with the application/vnd.durable-stream-records+ndjson envelope and its SSE form) now answer 400 on catch-up, long-poll and SSE reads; Stream-Record-Match on an append answers 400; the Stream-Record-First, Stream-Record-Start and Stream-Record-Next headers, the record fields of SSE control events and the Stream-Extensions header are no longer sent; PUT …/snapshot?record= and PUT …/retention?record= are gone (405 and 404 from a node; an access-controlled ursulagw answers 404 for both); and record_coordinate_corruptions and the record gauges (record_marks, dense_record_entries, hot_records, hot_real_bytes) left /__ursula/metrics. Resume from the opaque offsets in Stream-Next-Offset and SSE streamNextOffset; append conditionally with Stream-Seq (conditional writes). JSON messages are still validated, stored one per line, and counted in usage committed_records.
• Compatibility-only configuration keys. raft.snapshot_pressure_unpurged_logs (chart value raft.snapshotPressureUnpurgedLogs) and storage.snapshot.drive_flush_concurrency had no effect and are gone. A config file that still sets either now fails at startup; the Helm chart ignores raft.snapshotPressureUnpurgedLogs if a values file still sets it.
• Bucket purge's legacy-pack pass. Purge no longer rewrites slices of packs written before buckets were erasure domains, and its response no longer carries legacy_shared_chunks_pending: every pack of this release lives under its bucket's prefix.
• Stream-Integrity-* headers. HEAD no longer returns the seven undocumented Stream-Integrity-* headers. They exposed a per-stream setsum tally that snapshots carried forward without recomputing it from stored bytes, so it could not detect cold-tier or read-path corruption, and it cost a SHA3-256 pass over every inline-appended byte on every replica. HEAD still returns Stream-Cold-Hot-Start-Offset.

Fixed

• /bootstrap silently dropped bytes after a checkpoint that retention had not followed.
• The /bootstrap multipart boundary was a hash of the response's shape, so a writer could predict it and forge part breaks inside an update. Each response now uses a fresh 128-bit random boundary.
• After a stream with small hot bytes received an append of 1 MiB or more and then flushed, the large append could become unreadable and snapshots holding the stream could not be restored.
• Cold-index pages could keep entries of rejected large appends and later serve wrong bytes; existing pages are clipped and repaired.
• Stream GC after a delete and recreate could delete objects of the new incarnation.
• Staged external payloads were deleted after ambiguous failures, even when the append might have committed.
• Several node-local structures grew without bound: the cold-index page cache's LRU, cold-read readahead readers, per-stream append counts and the TTL index.

Documentation

• Read path and HEAD are specified. The Read path section of the extensions specification now defines consistency=leader, the 8 MiB read cap with max_bytes, JSON stream reads and binary SSE, next to timeout_ms. Snapshot metadata on HEAD adds Stream-Retained-Offset and requires Stream-Snapshot-Offset and Stream-Snapshot-Digest whenever a snapshot exists. HEAD also lists Stream-Cold-Hot-Start-Offset, as a diagnostic header that may change in any release. The HEAD, read, bootstrap and snapshot-read pages list the 307 and 503 answers of leader-confirmed requests.
• Stream-Seq is documented as the compare-and-set it is. Conditional writes describes the per-stream, bytewise check, the prefix-count pattern for several writers, and publishing committed facts with producer headers. Ursula adds no If-Match tail precondition.
• Corrected statements. A creating PUT stores Stream-Seq without checking it. POST never creates a stream. ?cursor= is an echo of Stream-Cursor, not an alternative to offset, and ?stream-cursor= is not an alias for it. A catch-up read at the tail answers 200 with an empty body, not 204. SSE sends no heartbeat comments. A stale Producer-Epoch answers 403, not 409. offset=-1 answers 410 once retention has trimmed the start of the stream. The 122-byte stream-ID limit is enforced. Stream-Up-To-Date is only ever sent as true; a read response without it is partial.