Operations

The first tool to reach for is ursulactl. It covers the raft-aware verbs operators run most often: drain, status, readiness gates. This page covers everything around it: the metrics shape ursulactl reads, the admin endpoints it (and your custom tooling) can call, and the operational policies ursulactl does not encode (backups, log levels). Deployment and process lifecycle belong to the platform, such as Helm.

Tooling map

Surface	When to reach for it
ursulactl	Day-2 cluster ops over HTTP: drain, catch-up waits, status, wait-ready, group moves, node cordon and decommission, disk replacement and recovery plans.
/__ursula/metrics and the /__ursula/raft/... admin endpoints	Custom tooling. ursulactl uses these underneath. The surface is small and stable enough to script directly.

There is no Prometheus scrape and no general-purpose orchestrator yet.

Metrics

curl http://127.0.0.1:4437/__ursula/metrics | jq .

The JSON snapshot covers per-core mailbox depth, append and apply counters, cumulative timings (*_ns), each group's leader, term and applied index (raft_groups), hot bytes waiting for cold flush, cold-flush uploads and errors, WAL physical/reclaim/recovery work, WAL disk-pressure admission, SSE counters, live-read waiters, and cold-write backpressure. Start here when triaging slowness, lag, or 503s. Troubleshooting lists jq selectors for the common fields. GET /__ursula/ready provides the local serving signal with a typed reason on refusal, on the client and admin planes alike. Healthy surviving voters can serve while a voter that lost entries catches up and remains unready. Maintenance also requires a ready raft_maintenance report in the metrics. ursulactl wait-ready checks the metrics, this readiness endpoint and the meta leader's placement view. ursulactl status presents leadership fields across the cluster.

Peers of another cluster

Every Raft gRPC request a node sends carries the id its WAL holds (server.cluster_id) and its raft.node_id as the ursula-cluster-id and ursula-node-id request metadata. That covers replication, votes, snapshots, leadership transfers, recovery probes, forwarded reads and HEADs, and forwarded bucket purges. A node checks the metadata before it decodes the request and refuses a peer with another id with FAILED_PRECONDITION, before any Raft RPC runs. The refusal names the refusing node's id, so the sender learns it as well. A mistyped server.cluster_id or a peer URL that points at another deployment therefore fails loudly instead of exchanging Raft traffic between two clusters, and a misconfigured node of another cluster cannot move this cluster's Raft terms. The cluster id is public, since every HEAD answer carries it as Ursula-Cluster-Id, so the check guards against misconfiguration, not against a hostile sender: keep the cluster plane on a private network.

What each side does with a refusal:

• Counts it. /__ursula/metrics counts every refusal, both ways, in raft_grpc_peer_cluster_refusals.
• Logs it once. The node logs each refused peer once at error level, with its node id, cluster id and address, while it keeps being refused, not once per RPC.
• Reports configured peers. /__ursula/ready and /__ursula/metrics list, under peer_cluster_mismatches, the peers of raft.peers refused within the last minute, with their node_id and the cluster_id they sent or answered with. A peer this node dialed always counts. A request this node refused counts when its sender claims a configured node id and, where that peer's URL names an IP address, comes from that address. Senders outside raft.peers are counted and logged, never listed.
• Keeps its readiness. A refusal never changes readiness or its reason. One misconfigured node does not take its healthy peers out of service, and an unready node still names its own cause, such as recovery_stalled.
• Backs off. OpenRaft treats a refusing peer as unreachable. The node keeps its connection to it, since the answer is final rather than a broken channel.
• Leaves it out of handoffs. GET /__ursula/leadership-shed answers carry the node's cluster_id, and a node counts a peer's answer only when the id is its own. A node of another cluster at a peer's address therefore never becomes a handoff target, even before any Raft traffic reached it.
• Forwards answer 503. A read, HEAD or bucket purge a follower forwards to a leader of another cluster answers the client 503 with Retry-After, as for a leader it cannot reach. Nothing ran on the refusing node.

A request that names no cluster id comes from a node before format epoch 4 or a stray client. It is refused with the Raft protocol-mismatch text, which such a node reads as a format-epoch mismatch. The node counts it in raft_grpc_unidentified_peer_refusals and logs each sender address once. It does not make the node unready, so a misrouted older node cannot take a healthy one out of service. A node that itself contacts a configured peer of another format epoch still becomes unready with format_epoch_mismatch.

At startup, a node checks that its WAL holds its configured server.cluster_id before it contacts any peer, and its format-epoch probe reports a configured peer of another cluster at error level and starts anyway. A refusal lasts until an operator fixes the configuration: correct server.cluster_id or the peer URL on the wrong node. A node whose WAL already holds another id refuses to start, so give a replaced node an empty WAL (see Rejoining a wiped node).

cluster_id is in both /__ursula/ready and /__ursula/metrics, so tooling can check that every node of a deployment reports the same id:

for node in NODE1 NODE2 NODE3; do
  curl -s http://$node:4437/__ursula/ready | jq -c '{cluster_id, peer_cluster_mismatches}'
done

Placement and the meta group

Every node that runs Raft, a single node included, runs a replica of the meta group. The meta group records the cluster's bootstrap, every node by its ID with its lifecycle and the WAL root it is bound to, and each data group's placement: the voters the group runs on and whether it moves. Its journal lives in raft-log/meta/ under raft.wal.path and always runs fsync = always, whatever raft.wal.fsync says. It takes a few MiB. Its voters are raft.meta_voters, by default the smallest three node IDs of raft.peers (every node of a smaller cluster), read only when the meta group forms. Every other node is a meta learner. No route changes the meta voters: a meta voter is never decommissioned, and its host is replaced under its node ID (Rejoining a wiped node).

Forming a cluster. A fresh cluster forms its meta group once every raft.peers node answered that it holds nothing and claims the same genesis values: the meta voters, the raft.peers IDs, raft.group_count and the replication factor, capped at the node count. A node that claims other values keeps the meta group from forming, and the maintenance report names genesis_mismatch (Troubleshooting). Once every raft.peers node registered, the meta leader bootstraps the cluster. Group g is seeded on min(raft.placement.replication_factor, N) consecutive nodes in ascending node ID from index g mod N, and the first of them initializes it. The group's leader then reports the voters it formed on, and the group turns live. raft.meta_voters and the replication factor count only at genesis, and the meta group's records rule after it. A node whose raft.group_count or capped replication factor differs from the cluster's reports group_count_mismatch or replication_factor_mismatch.

Reading the placement. GET /__ursula/placement, on the client and the admin listener, answers the node's view: the stage of the cluster, the last meta entry the view includes, the node's meta replica with its role and the meta leader it knows, and its registration. On the meta leader leader_report adds each live node's eligibility, the moves the decommission driver could not make, what each removed node's removal still waits for (removal_pending) and the age of the oldest move. Every other node answers null there. ?confirmed=true, on the admin listener only, confirms the view with a meta read barrier first. GET /__ursula/placement/groups/{group} answers one group's placement with the node's replica of it. ursulactl placement status prints both (ursulactl).

What a node hosts. Placement decides it. A node opens the groups its journals hold at start, and creates a group only once a view a meta read barrier confirmed places it there. A node that a confirmed view gives no role in a group tears its replica down once the group's leader answered twice, before and after the replica closed, that the node is no member. It retires the replica's log and deletes its own snapshot objects of the group under node-<id>/group-<gid>/. Cold objects belong to the group and stay. A node runs at most 8 teardowns at once. A node that hosts no replica of a group, or whose replica knows no leader and does not count as a voter, answers a client with 307 to a node of the group's placement, a different one on successive answers, or to another raft.peers node while the node has no bootstrapped view, as before the cluster bootstrapped or while its meta replica is damaged. With more nodes than the replication factor this is routine. Clients that reach the nodes without the gateway follow it, and try another node when the one named is down. The gateway does the same.

While the meta group is down. Data groups keep serving, electing leaders and restarting. No placement write commits, so no move starts or ends, no node is removed, no teardown starts and no group is created. The layout stays as meta left it: within 3 seconds of the meta group losing its quorum no group changes a voter or promotes a learner, and no leadership moves by choice, the leadership balancer's included. Raft still elects a leader when one is gone, the disk lease and the egress gate still fence a node for its own fault, and a stopping node still hands its leaderships off, so restart one pod at a time. A teardown planned from a confirmed view before the outage can still retire its replica's log, since that needs only the data group's answers, and its report waits for the meta group. Placement writes answer 503, or 504 when they may have applied, and the maintenance report names meta_unavailable after 30 seconds without a meta leader, which blocks nothing. Readiness never reflects the outage, nor a group whose quorum is lost on other nodes, so Kubernetes replaces no healthy pod for it. A meta group whose voters stall behind their recovery gates, as after a majority of them lost their run state, is recovered with a plan before any data group (Recovering after a host crash).

A damaged meta replica. A node whose meta journal does not open, whose meta snapshot does not restore, whose meta replica does not start, or whose meta replica holds committed entries the meta group no longer has, runs no meta replica for the rest of the run. It keeps raft-log/meta/ as it is and serves its data groups. Its placement status shows meta as damaged with a kind and the error, the maintenance report names meta_damaged, and the node marks the placement-unavailable leadership shed: it still campaigns, takes no leadership transfer and hands its leaderships to its peers. To bring the replica back while the other meta voters keep a quorum, stop the node, remove the whole raft-log/meta/ directory, group-0.snapshot.json included, and start the node. A meta journal that is gone while the data journals hold data opens behind its recovery gate, so the replica rejoins through the meta leader. On a node that is the meta group's only voter, a single node or a cluster grown from one, do not remove the directory: its damaged meta journal is lost, and the node serves its data groups with no placement change. A new disk whose meta replica does not open takes part in no data group in that run, since only the meta group binds its root.

A lost quorum of disks. A group whose majority of voters lost their disks at once has lost its data, and no recovery flow brings it back. This holds for the meta group as for a data group: a meta group that lost a quorum of its voters' disks has lost the cluster's placement and node records. Restore the latest verified backup into a fresh cluster (Backup and disaster recovery).

Adding a node

Raft membership names node IDs only, and every address comes from [[raft.peers]]. To add a node, add its entry to [[raft.peers]] on every node, restart the running nodes one at a time (Restarting a voter), then start the new node with the same list. It registers with the meta group, joins it as a learner and holds no group. Then move groups onto it, for example ursulactl placement move --config cluster.toml --group 4 --group 9 --replace 3=4 --wait. A move keeps each group's voter count. Each node claims the smaller of raft.placement.replication_factor and its own [[raft.peers]] count, and its maintenance report compares that claim with the factor the cluster recorded when it formed. A cluster that formed with fewer nodes than raft.placement.replication_factor recorded its node count. Once its list grows, every node started or restarted with the longer list claims more and reports replication_factor_mismatch, which blocks maintenance (wait-ready, drain) on it. Before you grow such a cluster, set raft.placement.replication_factor to the recorded factor on every node, the new ones included. With the chart, raise server.replicaCount (Deploy a Cluster). A raise ends with ready pods that hold no group: moving groups onto them stays manual, as for any added node.

Using a removed ID again. The ID of a decommissioned node is used again the same way, on an empty disk, once its removal completed and the meta group shows it vacant (Cordon and decommission). ursulactl node verify-joinable --node N waits for that. The new disk registers under a new root and joins as a new node: it waits out a predecessor, joins the meta group as a learner and holds no group until a move places one on it. Stop the old process and wipe or destroy the old disk first, and never start that disk again. A disk started under the ID before its removal completed waits: its registration and its readiness report removal_pending (503), its data replicas take part in nothing, and it binds with no restart once the ID is vacant.

Growing a single node. A single node with raft.wal.path grows the same way. It formed with a replication factor of 1 and itself as the meta group's only voter, and growing changes neither. Give every node, the new ones included, the single node's server.cluster_id, the longer [[raft.peers]], raft.placement.replication_factor = 1, raft.meta_voters set to the single node's ID, and raft.group_count set to the count it formed with, which the params of GET /__ursula/placement show. Start the new nodes, restart the single node with the longer list, and move groups onto the new nodes. A group of one voter moves like any other, since its leader is its own quorum. With the factor at 1 an unset raft.wal.fsync stays always. Every group keeps one voter, so growing spreads the groups and their load over the nodes but gives no group a second copy. The single node stays the meta group's only voter: while it is down no placement changes, and it is never decommissioned. A node on a temporary WAL does not grow, since it forms a new cluster at every start. The chart does not grow a single pod.

Cordon and decommission

ursulactl node cordon --node N keeps node N's replicas and places no new one on it, and ursulactl node uncordon --node N makes it active again. A move that adds a cordoned node to a group is refused with ineligible_target.

ursulactl node decommission --node N --wait removes a node for good. It cordons the node, then the meta leader moves the node's groups away, every second and within the move limit, keeping one move for operators, to active nodes that acknowledged it within the last 10 seconds and do not shed for their WAL disk. Once the node has no role in any group the meta group marks it removed. uncordon aborts a decommission while the node still has a role. A removal cannot be undone for the node's process and disk: they never take part again, and a move that names the node answers 409 with node_removed. A meta voter's decommission is refused with meta_voter. The node may be down: its groups move away without it while each keeps a quorum of its other voters. GET /__ursula/placement on the meta leader lists the moves the driver cannot make under decommission_blocked.

A removed node marks the placement-unavailable leadership shed, closes each replica once it stops counting as a voter, and reports node_removed, which blocks no readiness. Restarted with its disk, it keeps the view it had before its removal, but no group places it: it asks each group about every replica it holds, also while the replica's open keeps failing, as when the meta leader took its snapshot objects, and keeps closed each one its group no longer counts. A removed node deletes no snapshot object, and no node deletes one before the meta group answered its registration in that run.

Completing a removal. The meta leader completes a removal and marks the ID vacant once nothing of the node's life is left: every data group's leader confirmed with a fresh quorum that the node is no member, the meta group lists it as neither voter nor learner, its snapshot directory is empty, and no process of its root answers at its raft.peers address. The leader runs a pass at the start of its leadership, at once when a node is newly removed, and every 30 seconds while a node is removed. With an S3 snapshot store, a pass deletes the removed node's snapshot objects of each group that confirmed it no member, after it confirmed the view again. leader_report.removal_pending in GET /__ursula/placement on the meta leader lists, by removed node, what its removal still waits for: the groups no live leader confirmed (a group without one is down or lost its quorum), the groups whose objects remain, whether the meta group still lists the node, and whether its old process still answers. Stop the node's process for its removal to complete. ursulactl node verify-removed --node N passes for a removed or vacant node, so run it before you delete its volume. Do not delete <prefix>/node-<id>/ yourself. With the chart, decommission every ID above the new server.replicaCount before you lower it (Deploy a Cluster).

A vacant ID. A vacant ID refuses every command but the registration of an empty root other than its released one, which begins a new life (Using a removed ID again). A move that names it answers 409 with node_vacant. The old disk never binds again: started under the ID, it is refused for the released root and superseded, and readiness reports superseded. Before a new disk starts under a decommissioned ID, stop the old process and wipe or destroy the old disk. A process of the old root that still runs, cut off from the meta group, can still answer a peer whose meta replica lags, the same exposure a released disk has (Rejoining a wiped node). On Kubernetes a StatefulSet runs one pod per ordinal, which keeps this rule, and a force-deleted pod voids it. A late snapshot upload of the old process that lands after the completion stays under <prefix>/node-<id>/ until a new life of the ID collects it at its start.

Admin endpoints

These are the primitives ursulactl and any custom operator tooling builds on. They live on the admin plane (server.admin_listen, loopback 127.0.0.1:4438 by default), not the public client plane. Nodes expose no cluster-mutation surface on the network. Reach them over a tunnel to the admin port (see ursulactl's operation providers). The examples below assume a forward from local 4438 to the node's admin plane. Each call is local to one node. To act on every group, loop over the IDs in metrics.

Every admin mutation requires x-ursula-process-incarnation, pinned from GET /__ursula/metrics before the operation. Missing identity returns 428; a restarted target returns 412 before the handler changes state. Keep the same identity for the entire plan and stop on either response. This precondition is public instance identity, not authentication. Updated ursulactl attaches it automatically and refuses to refresh it during an operation.

# Observe once for this operation; do not refresh after a 412.
process_incarnation=$(curl --fail http://127.0.0.1:4438/__ursula/metrics | jq -er .process_incarnation)

# Force a cold flush for one stream (skip the timer)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/flush-cold/demo/hello

# Trigger a Raft snapshot for one group
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/42/snapshot

# Purge log entries through index 1000 for one group, at most through its
# last snapshot. Answers 503 with Retry-After while a snapshot of the group
# builds or a purge of it has not landed. Send it again then.
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/42/purge?upto=1000"

# Hand leadership of one group to another voter (used by `ursulactl drain`)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/42/leader/transfer/2

# Read group 42's placement and this node's replica of it, with what its reconcile shell does
curl --fail http://127.0.0.1:4438/__ursula/placement/groups/42

# Move group 42 to nodes 2, 3 and 4, while its placement is still at the key the status shows
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d '{"expected": {"term": 3, "leader": 1, "index": 57}, "to": [2, 3, 4]}' \
  -X POST http://127.0.0.1:4438/__ursula/placement/groups/42/move

# Let at most two groups move at once
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d '{"limit": 2}' -X POST http://127.0.0.1:4438/__ursula/placement/move-limit

# Cordon node 4, make it active again, or decommission it. No body, and the answer is its lifecycle
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/placement/nodes/4/cordon
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/placement/nodes/4/uncordon
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/placement/nodes/4/decommission

# Release root 9e3779b97f4a7c15 of node 3 once its old disk is revoked (see Rejoining a wiped node)
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d '{"expect_root": "9e3779b97f4a7c15"}' -X POST http://127.0.0.1:4438/__ursula/placement/nodes/3/replace-disk

# Put this node into maintenance drain / lift it again
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/leadership-shed/maintenance
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X DELETE http://127.0.0.1:4438/__ursula/leadership-shed/maintenance

# Send one group's recovery plan to this node, a member of a plan whose leader answered plan term 9,
# naming the log its metrics show (see Recovering after a host crash)
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d '{"leader": 2, "point": 1234, "cut": false, "step": {"follow": {"leader_term": 9}}, "floor": "seeded", "expected": {"last_log_index": 1230, "last_log_term": 7, "current_term": 8}}' \
  -X POST http://127.0.0.1:4438/__ursula/raft/42/recovery/accept-unsynced-loss

# The same for a plan that cuts to an entry another replica reported committed, at index 1240
# of term 7 led by node 3: every request of the plan names that entry
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d '{"leader": 2, "point": 1240, "cut": true, "point_entry": {"term": 7, "node_id": 3}, "step": {"follow": {"leader_term": 9}}, "floor": "seeded", "expected": {"last_log_index": 1241, "last_log_term": 7, "current_term": 8}}' \
  -X POST http://127.0.0.1:4438/__ursula/raft/42/recovery/accept-unsynced-loss

# The same for the meta group: `meta` in place of the group id, and the floor `meta`
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d '{"leader": 2, "point": 310, "cut": false, "step": {"follow": {"leader_term": 4}}, "floor": "meta", "expected": {"last_log_index": 305, "last_log_term": 3, "current_term": 3}}' \
  -X POST http://127.0.0.1:4438/__ursula/raft/meta/recovery/accept-unsynced-loss

Each /__ursula/raft/{target}/... route takes a data group's id or meta, the meta group. The snapshot, leader-transfer, quorum, self-election and recovery routes act on this node's meta replica for meta, and answer 404 when the node runs none. The purge route serves data groups only and answers 400 for meta. The meta group answers as raft_group_id 0 in their bodies.

The snapshot endpoint answers once the group has a snapshot of everything it had applied when the request arrived. It waits while a build of the group runs or other snapshot builds hold all of the node's build permits, and answers 503 with Retry-After: 1 when no build of the group can start within 10 seconds (see Manually trigger a snapshot).

The leader-transfer endpoint refuses with 409 Conflict if the receiving node isn't the current leader and 400 Bad Request if the target isn't a voter. This is why ursulactl is the safer way to chain these calls: it consults metrics first.

Placement decides which nodes vote in each group, and only a move changes it. GET /__ursula/placement/groups/{group} answers the group's placement in the node's view, with ?confirmed=true on the admin plane after a meta read barrier, and the node's replica of the group as its metrics show it, null when the node hosts none. On the group's leader, the replica's reconcile says which membership step the leader takes toward the placement or why it waits. The client plane serves the same read without confirmed.

POST /__ursula/placement/groups/{group}/move starts a move of a live group to the voters to, as many as it has now, when its placement is still at expected, the since of the placement the status shows. A move of a moving group retargets it, and a move back to its source cancels it. The meta leader takes the request from any node, and the answer is the group's new since. POST /__ursula/placement/move-limit sets how many groups may move at once. Every refusal names its reason and whether the write may have applied: 404 for an unknown group or node, 409 for a refusal of the meta group, such as stale_placement with the placement as current, move_limit or no_change, 503 while the meta group cannot answer, and 504 when the write may have applied, after which a confirmed status shows whether it did. The group's leader adds the new voters as learners. It changes the voters only once each learner caught up and its recovery gate is open, which it asks of the learner's node (recovery_gates in GET /__ursula/leadership-shed). A learner is caught up once it matched, within the group's leader lease (3 s), the leader's commit index as it stood when its catch-up round began, so it catches up under steady writes too, and one that falls behind by more than that is not promoted. A promotion of a group of one or two voters needs the incoming voter's acknowledgement for the joint configuration, so the group's writes can pause for the learner's remaining lag, which the lease bounds to about two leases of commits. A learner whose gate is closed waits as gate_closed, and one whose node does not answer waits as gate_unknown. The leader of a move that makes no progress for five minutes reports move_stalled in readiness.

The node routes answer {"lifecycle": L}, the node's lifecycle after the change: active, cordoned, decommissioning, or removed for a decommission of a node with no role left. They answer 404 for an unknown node and 409 for a meta voter's decommission (meta_voter), a removed node (node_removed), a vacant node (node_vacant) or a cordon of a decommissioning node (decommissioning). The replace-disk route answers the node's record, and 409 with stale_root when the node is bound to another root than expect_root. A release cannot be undone (Rejoining a wiped node).

ursulactl placement move --config cluster.toml --group 42 --to 2,3,4 --wait sends the move from a confirmed view and waits until the group is stable on its new voters, printing what its leader does as it changes. --replace 3=4 over several --group values moves every one of them from node 3 to node 4, queued under the move limit, and a rerun finishes a batch it interrupted. ursulactl placement status shows each moving group's leader and why it waits, ursulactl placement cancel --group 42 moves a group back to its source, and ursulactl placement set-move-limit --limit 2 sets the limit (ursulactl).

Maintenance drain

POST /__ursula/leadership-shed/maintenance marks the node as draining: it stops campaigning and accepting transfers until the mark is lifted with DELETE, so no handoff and no balancer moves a leadership to it. Both verbs return the node's leadership-shed state so tooling can confirm the transition took effect. The state also lists the replicas whose group's leadership the node may take now (takes_leadership, the meta replica as "meta"), the recovery gate of each replica the node runs (recovery_gates), how many data groups it leads (leader_count) and whether it is serving-ready (serving_ready, as /__ursula/ready answers). A replica takes a leadership only while the node takes transfers, its gate lets it campaign and it acknowledged a leader of its group since the node started, and every planned handoff (the health gates' sheds, the leadership balancer and the shutdown handoff) goes only to a replica listed in takes_leadership. The commit-stall watchdog and the WAL-disk shed do not ask the peers: they hand a leadership to a voter the leader's own handoff check accepts, and a target that refuses costs the group one election. A group's leader promotes a learner on the node only once its gate there is open, and the peers' leadership balancers read the count and move leaderships only to a serving-ready node:

{
  "bits": 8,
  "state": "maintenance-drain",
  "should_accept_transfer": false,
  "should_campaign": false,
  "should_shed_current_leaders": true,
  "takes_leadership": [],
  "recovery_gates": { "7": "open", "meta": "open" },
  "leader_count": 3,
  "serving_ready": true
}

ursulactl drain wraps exactly this pair: mark first so the node stops re-acquiring groups, then transfer leaderships away explicitly. It hands each leadership to a caught-up voter whose own report gives it the role voter or incoming in the group, never to a replica a move takes out. The mark stays set until ursulactl undrain clears it or the process restarts, since it lives in the process. If a rollout ever dies mid-way, a node can be left refusing leadership. Run ursulactl undrain (or DELETE the mark directly) to recover. Peers read each other's shed state (GET /__ursula/leadership-shed on the cluster-plane listener) when deciding which nodes are eligible to pick up leaderships, so a drained node stops attracting groups cluster-wide, not just locally.

Graceful voter shutdown

On the first SIGTERM or Ctrl-C, the voter marks itself maintenance-drained to prevent re-acquiring leadership. It keeps Raft transport available while it hands off each leadership to an eligible voter (see the handoff rule under Recovering after a host crash). Every 50 ms it asks its peers again which of them may take a transfer, waiting up to one minimum election timeout (1.5 seconds) for a peer's answer, logs their answers when they change, and submits each leadership that no peer could take yet, only to a peer that answered that its replica of the group may take the leadership. A submitted transfer is not repeated within the handoff: it waits longer than five seconds to settle. It logs the groups no peer can take, with the reason the handoff check gave for each peer. It waits for observed leadership changes for up to five seconds and then logs each leadership it still holds. It steps down from those, without campaigning, for up to two seconds, so the other voters of those groups elect a leader within one election timeout (1.5 to 3 seconds) instead of following a node whose listeners are closing. Then it closes Raft transport and drains HTTP listeners for up to ten seconds. Each listener stops accepting connections and marks every HTTP/1 answer it starts from then on Connection: close. A connection that was idle when the drain began stays open for up to one second, so a request a client sends on it within that second is answered. The gateway reuses an idle connection for at most half a second. A gateway request can still fail with 502 in two cases. One is a connection the node's kernel completed but the node had not accepted when its listener closed, which is reset. The other is a connection whose last answer started before the drain, which closes after that answer without Connection: close. A connection still open after the ten seconds, such as one from a peer that lost power, is abandoned. It then stops its Raft groups, fsyncs every journal and records a clean shutdown, so the next start reads the journals strictly and the node holds every entry it acknowledged. The whole shutdown has a deadline of twenty seconds from the first signal. If it has not finished by then, the process exits without recording a clean shutdown, and the next start treats the run as a crash. A second signal also exits at once without recording a clean shutdown. The chart gives voter pods thirty seconds by default. This reduces planned-stop election pauses. It cannot hand off leadership on SIGKILL or OOM.

runtime.node_memory_abort_cap_size is an emergency process-abort threshold, not HTTP admission control. Exceeding it calls abort() without graceful leadership handoff, like an OOM kill. It does not return 503. Leave room below the container memory limit for normal operation and recovery, and use write admission and backpressure to manage load.

Restarting a voter

Restart one voter at a time:

1. ursulactl drain --node N moves every leadership away and keeps the node from taking one back.
2. Restart the process through the platform (Helm, systemd). The graceful shutdown above keeps every journal intact.
3. ursulactl wait --node N returns once the node is a voter again of every group the meta leader's placement view places it in, and has caught up.
4. ursulactl undrain --node N lets it hold leaderships again.

Do not stop another voter before wait returns. Two overlapping outages of a three-voter group leave it without a quorum even if the restarts were issued separately.

With the chart, helm upgrade rolls the voters by RollingUpdate. The StatefulSet controller replaces one pod at a time, from the highest ordinal down, and moves on only once the replaced pod passes its readiness probe. A pod the controller recreates during the roll, after a crash or an eviction, takes the new revision. On Kubernetes 1.37 and later, the controller first replaces every pod that is already unready on the old revision, all at once, and replaces a ready pod only while every other pod is ready. Each voter hands off its leaderships on SIGTERM (Graceful voter shutdown). The controller does not look at the other pods, so start an upgrade only when ursulactl verify-cluster passes.

Changing a peer address

Raft membership names node IDs only. Every node dials its peers at the url its own [[raft.peers]] lists, and a follower redirects clients to the leader's client_url there. A peer's address therefore changes with the configuration, and no admin step migrates it. Keep [[raft.peers]] the same on every node.

• A new address behind the same name needs no change. Name peers by DNS names that follow each node, as the chart's headless Service does.
• A new name that the old one still reaches rolls like any restart. Change [[raft.peers]] on every node and restart the nodes one at a time (Restarting a voter).
• A listener that moves, such as a new cluster port, server.cluster_listen turned on or off, or a host without a stable name, needs every voter restarted together. Stop every voter, change [[raft.peers]] on every node, and start them all. The cluster serves nothing until a quorum is up, which takes about as long as one restart. Restarting one node at a time does not work here. Until the last node restarted, some nodes cannot reach each other, and a group can lose its quorum. Rolling listener moves are tracked in #735.

With the chart, server.ports.cluster and server.clusterListen move every listener. Run that helm upgrade without --wait or --atomic, then delete every server pod at once with the release's server selector labels:

kubectl delete pod -l app.kubernetes.io/instance=<release>,app.kubernetes.io/name=<chart name or nameOverride>

The gateway's -gateway name does not match that selector. The upgrade starts the rollout itself. The controller replaces the highest ordinal, which comes up alone on the new port and stays unready, so the rollout cannot finish on its own. --wait would hang until its timeout, --atomic would roll the change back, and a GitOps sync shows Progressing until the delete. Expect one voter isolated in between. The recreated pods take the new revision, all together under the chart's Parallel pod management policy, and each records a clean shutdown on SIGTERM. global.clusterDomain rolls like any restart while the old domain still resolves.

A node that a group's membership names but [[raft.peers]] does not list is never dialed. Each node logs it once, and the group reports unplaced_member in the maintenance report when the node has no role in the group's placement (unplaced_member).

WAL routing configuration

A WAL root is bound to runtime.core_count and raft.group_count in <raft.wal.path>/raft-log/topology.bin. Startup holds the WAL lock and checks both values before recovering any group or writing the run state. A mismatch, missing record on an existing WAL, or corrupt record refuses startup.

The same record binds the root to the raft.node_id that created it, and a root opens only under that id. A volume moved to a node with another id refuses startup. Each start also counts itself in identity.bin beside it. A root that holds prior state without that file refuses startup too, so start the node on an empty WAL instead.

Keep both values explicit and unchanged when moving a persistent volume or resizing its host. Without explicit values, core_count follows the process's available parallelism and group_count defaults to that parallelism times 16 (resource presets set a fixed group count). Helm defaults to 4 and 64. Changing cores changes the journal directory of some groups. Changing group count changes stream routing. Neither is an online resharding operation.

For a mismatch, configure the stored counts again. The error reports them. Do not manufacture a record from the new configuration or remove the check's files. A replica whose record is missing or corrupt can be rebuilt on an empty WAL directory from a healthy quorum, with the cluster's routing counts. Follow the rejoin procedure below and wait for full catch-up before replacing another replica. A single node has no quorum to rebuild from, so a fresh WAL does not preserve its data.

Rejoining a wiped node

A voter whose WAL directory was wiped or replaced, for example after a lost disk, starts with an empty log for every group under a new root. The meta group still binds the node id to the old root, so the node's data replicas stay out of every group: they grant no vote and their recovery gates stay closed. Its placement status shows the registration root_refused with the old root's slot, and readiness reports root_refused. Once the old disk is revoked (the old process ended, its volume was detached or destroyed, or its host is shut down, on Kubernetes a pod deleted normally, a force-deleted pod whose VolumeAttachment is gone, or a node tainted out of service), release it from any machine that reaches another node's admin plane:

# R is the old root, as the refused node's placement status names it
ursulactl node replace-disk --config cluster.toml --node 3 --expect-root 9e3779b97f4a7c15

The release and the new disk may come in either order. The node binds its new root within a few seconds of the release with no restart, waits 12 seconds for any process of the old disk to fall silent (readiness predecessor_wait), and then rejoins. A release cannot be undone: the released root never binds the node id again. A process that keeps its disk past a release, or that holds state and finds its id bound to another root, fences itself for good and reports superseded. Release a disk, never a process. Then, for each group:

• It does not initialize the group again. A group's initializer, the first node of its seed voters, runs Initialize only while a confirmed view shows the group seeded, and only when every other voter of the group answers that it holds nothing for the group either and no leader is seen. A fresh install still bootstraps this way. One voter holding entries or following a leader means the group exists, and the node waits to be replicated to. It waits for every voter, not a quorum, so a fresh install with a voter that never comes up stays uninitialized (it logs "waits for every voter before initializing").
• It refuses replication until the leader proves a quorum without it. The node lost its vote with its disk, so it cannot tell the current leader from a stale one. It refuses appends, heartbeats, snapshots and votes until the group's leader confirms a fresh ReadIndex barrier with a quorum of the other voters. The node then records that leader's vote as its vote floor and accepts replication. In a 3-voter group that quorum needs both other voters up. While the wiped voter waits and another voter is down, the group has no quorum and cannot commit writes.
• It stays behind the recovery gate. It does not campaign and refuses every vote once it knows the group holds entries, until it has applied a fresh barrier from the group's leader (see Recovering after a host crash).
• The leader catches it up in place. The leader sees the node answer that it lacks entries it had acknowledged. It rewinds its replication to the node and sends it its log again, and the node stays a voter throughout. Leadership handoffs (the balancer, the commit-stall watchdog) skip the node until it holds every committed entry.

While the other voters are up, the release is the only operator step. ursulactl wait --node N returns once the node caught up in every group placed on it and opened its gates, and ursulactl verify-cluster shows it unready until then. Do not stop another voter before that.

If a majority of a group's voters lost their disks at once, the group's data is lost. The voters that kept their disks are not a quorum, no leader can prove a quorum to the wiped replicas, so they never obtain a vote floor, and accept-unsynced-loss cannot replace lost voting history. No recovery flow brings the group back, and a live group is never initialized again, even when every voter lost its disk. The same holds for the meta group (A lost quorum of disks). Restore the latest verified backup into a fresh cluster on a new object-storage prefix (see Backup and disaster recovery).

The disk lease

Every node that runs Raft rewrites and syncs lease.probe at its WAL root every second. With no synced probe for raft.wal.lease (5s by default, from 2s to 10s), or after a probe error, the node stops every Raft request and answer of every group and of the meta group, leads no group, sends no cold or snapshot object write, hands off its leaderships (shed reason fence) and answers clients and mutating admin routes 503 with Retry-After, local reads included. Readiness reports disk_lease_lost. The process does not exit, and the next probe that syncs lifts the fence. Raise the lease on slow disks.

A start whose previous run may still be alive waits 12 seconds from its open before it grants a vote, campaigns, takes a transfer, opens its recovery gates or writes a snapshot object, and readiness reports predecessor_wait meanwhile. That covers a start after a host crash or reboot, a remount, a lost run state, and a new disk that joins a bootstrapped cluster, once the meta group answers its registration. A clean restart and a process crash on the same boot wait for nothing. A fresh cluster forms without the wait. Keep raft.wal.path on a volume: a run that did not end cleanly and is read on the same boot from another device counts as a remount, and a container's writable layer has a new device at every container start, so every crash there would count as a host crash.

A log below its snapshot record

A replica refuses to open a group whose log holds an entry, or whose commit marker names one, at or after the index of its restored snapshot record but below that snapshot in log-id order. It logs LogBelowSnapshot with the snapshot and the entry, and the group stays closed on that replica while its other replicas keep serving it. No such entry can be committed. In 0.9 a log reaches this state only through a bug or disk corruption, because the WAL is synced before every snapshot record. To recover, rejoin the node as in Rejoining a wiped node, after the checks of its other groups in Recovering after a host crash.

Recovering after a host crash

With raft.wal.fsync = "never", the default for groups of three or more voters, a node acknowledges ordinary data appends from the page cache. Membership batches are always fsynced before acknowledgment, so a full-cluster power loss cannot erase the voter configuration. A process crash loses nothing. A host crash or power loss, or a journal I/O failure, can cost the node the unsynced tail of its journals. The node detects it on the next start from its run state (wal_recovery in GET /__ursula/metrics, the ursula.wal.recovering gauge) and keeps the frames that still verify. A node whose run state file is missing while its journals hold records is treated the same way. Every group the node held enters the recovery gate, which is recorded in the node's metadata and survives further restarts:

• The node does not campaign, takes no leadership transfer and grants no vote. A node that led the group starts as a follower. It still accepts appends from a leader of its current term or a later one.
• It asks the group's leader for a fresh ReadIndex barrier, applies the log up to the barrier's index and then opens the gate. Until then readiness answers 503 with reason recovery_gate_closed, and recovery_gates in GET /__ursula/metrics lists the gated groups.
• The leader sees that the node lost entries it had acknowledged, rewinds its replication to the node and sends it its log again. The node stays a voter, and nothing is proposed to repair it. The node's answer still counts as its acknowledgement of the leader, so the leader keeps its quorum while the node catches up. The leader logs a warning naming the group and the node, and ursula.raft.follower_log_reversions counts it per node. A node that shows up there without a host crash or a lost disk may have a disk that drops synced writes.

Leadership handoffs (draining, balancing, a node that sheds its leaderships, the commit-stall watchdog, the admin transfer endpoint) go only to a voter that acknowledged the leader within the last 1.5 seconds (the minimum election timeout) and holds every committed entry. A handoff whose target refuses it is abandoned at once, and one whose target took it but has not won within 1.5 seconds right after: the leader steps down and campaigns again with a pre-vote round. One whose target holds a newer vote or does not answer is abandoned the same way after 6 seconds (two maximum election timeouts).

While at most a minority of a group's voters lost their tail, the other voters hold every acknowledged write and nothing is lost. Wait for every group to open the gate on a node before you stop another voter.

A majority of a group's voters lost their tail at once. If the group's leader kept its log, it stays leader, catches the gated replicas up and its barrier opens them as above. The gated replicas renew its lease before they caught up. If they cannot catch up, for example when one needs a snapshot while object storage is unavailable, the leader stays live: writes are accepted and can time out with an unknown outcome instead of answering 503, and recover-group refuses to fence the group while it leads with a live quorum. Otherwise the gated voters refuse every vote, so the group elects no leader and writes answer 503. After 30s without a leader barrier and without applying anything, each gated replica reports its group stalled: readiness reason recovery_stalled, the group under recovery_stalled_groups in GET /__ursula/ready and under recovery_gates.stalled in the metrics, the ursula.raft.recovery_gates{status="stalled"} gauge, and an error log naming the group. Writes acknowledged after the last fsync of the surviving replicas may be gone. Only an operator can accept that, with a recovery plan. The plan elects one replica as the group's leader, and that leader starts a new history of the group: every stream of the group gets a new Stream-Incarnation, and every producer epoch claimed before is closed (see history rewrites). For each stalled group:

Recover the meta group first, then the data groups. A plan names the plane it is for in its floor: meta for the meta group, seeded or certified for a data group. A replica refuses a floor of the other plane with floor_other_plane. A data group's floor comes from a confirmed meta view, GET /__ursula/placement?confirmed=true on a node's admin plane: seeded while the view shows the group seeded, and {"certified": converged_at} once it shows the group live, where converged_at ({"term", "leader", "index"}) is the membership entry the group's last convergence certified committed. Without a confirmed view, for example while the meta group itself is stalled, a data group cannot be planned. When the meta group stalls, its replicas report under meta in GET /__ursula/metrics, with the same fields as a group under raft_groups. Plan it with ursulactl recover-group --target meta, or by hand with meta in place of the group id and the floor meta, and wait until it is satisfied before you plan any data group. A meta plan that cuts the meta log below a removal's completion or below a new disk's binding refuses that disk afterwards, since its root is no longer empty: wipe it and start it again, and every group moved onto it loses that replica until then. The meta replica's gate shows in recovery_gates in the metrics under "meta", and a stalled one is listed as "meta" under recovery_gates.stalled and counted by the ursula.raft.recovery_gates gauge. It never makes the node unready: recovery_stalled_groups in GET /__ursula/ready lists data groups only.

A live group's plan keeps its certified membership. Commit markers are not synced on their own, so after a power loss of every voter under never each marker can lie below a membership change the group committed, while the replicas the change removed are already torn down. A plan that cut every log back to its marker would bring back the old membership. Under a certified floor every replica that holds the certified entry keeps it through a cut as if its marker reached it, and the plan's leader refuses the plan with floor_not_held unless it holds the entry at or below the plan's point.

ursulactl recover-group --config cluster.toml --target 42 explains the plan it would send for one group and sends nothing: the floor, why it cuts or not, each replica's log and voter sets the plan expects, the replicas without a vote floor, and the voters that did not answer. A replica whose metrics show needs_vote_floor lost its vote and refuses every plan until a live leader's barrier proves one: the tool neither counts it toward a majority nor sends it the plan. With --accept-loss it runs the steps below: it reads every voter, computes the plan, sends it to the plan's leader first and then to every other replica that answered, and waits until the plan is satisfied. It stops with an error when a replica leads the group with a live quorum and no plan was sent. While a replica's gate may still open without a plan, it waits, and once it sent a plan, it waits while a replica leads with a live quorum or follows a live leader it did not reach. It plans again when a plan wedges in two observations in a row (see below), when the plan's leader refuses it, or when replicas refuse it so that the plan's leader and the replicas that took it are no majority, up to --max-plans plans. It never plans again while a replica leads at a term above the last plan's term: that leader was elected under the plan. When the plan's leader refuses for a reason no plan fixes, missing_vote_floor, step_mismatch or floor_other_plane, it stops with an error. A member that refuses so is left out of the next plan, but the voter sets it reports still count, and a plan that a majority took stands. A leader_not_voter refusal changed nothing: the membership moved since the tool read it, so it reads the replicas again and plans again. Nor did the leader's floor_not_held: the tool leaves that replica out and plans again. Once the plan is satisfied, it names the replicas it reached outside the leader's membership, which keep the replaced history. When the plan's leader does not answer in time, it reads the plan the leader recorded and sends the other replicas that plan term. --accept-timeout-secs bounds one call, which a cut or a demotion restart makes slow (ursulactl). By hand:

1. On every node that holds a replica of the group, learners included, read GET /__ursula/metrics, and under raft_groups the group's recovery_gate, last_log_term, last_log_index, current_term, committed_index, recovery, leads_live, voter_configs, committed_voter_configs and maintenance.running. voter_configs are the voter sets of the replica's membership, two while a change is joint. committed_voter_configs are the voter sets of the membership it keeps when a plan cuts its log back to its committed entry. Plan without a replica whose maintenance.running is false: its group stopped, and it reports no membership. Plan without a replica whose needs_vote_floor is true too, but count the voter sets it reports. Keep its last_applied_index for step 2. Read also committed_term and committed_node, which with committed_index name a replica's committed entry, and last_applied_index. Go on only when, in every voter set any replica reports in either field, a majority of its voters answer, only while no replica's gate is awaiting_barrier, catching_up or cutting, and only while no replica reports leads_live: such a replica may still open through a leader's barrier without losing anything, is running a plan's cut, or leads the group. A replica whose own metrics name it leader without leads_live lost its quorum: the plan demotes it.
2. Compute the plan from the replicas that answered:
   • The term floor: the highest current_term among the replicas whose gate is not open or whose recovery is not null. It is not the request's floor field, which names the plane.
   • The leader: among the replicas some voter set names, the one with the greatest last log id, highest last_log_term first, then highest last_log_index. An empty log (null) ranks last. Its last_log_index is the plan's point, and cut is false.
   • When that replica's voter_configs and committed_voter_configs name more than one voter and its last_log_term is below the term floor, its log may hold an entry that a lost leader overwrote. When its own voter_configs do not name it, it cannot campaign. In both cases cut is true, the leader is the replica with the greatest committed_index among those whose own committed_voter_configs name them, and its committed_index is the point. Every replica then keeps only the entries it knew were committed.
   • When cut is true and a replica, a learner say, reports a committed_index above the point, the voters may still hold that entry in their logs although their markers lie below it. Then the leader is instead the replica with the greatest committed_index among those whose own committed_voter_configs and voter_configs both name them and whose last_log_index reaches that index. The point is that index, and point_entry names the entry: {"term": committed_term, "node_id": committed_node} of the replica that reported it. The leader refuses the plan unless its log holds exactly that entry at the point.
   • Under a certified floor, with the certified entry at index C and term T (converged_at.index and converged_at.term): a replica holds the entry when its last_log_index reaches C and its last_log_term reaches T. A log that reaches C at a lower last term holds an older leader's entry there instead. Count the certified voters as one more voter set, and no voter set of a replica that does not hold the entry. A replica whose committed_index is below C but that holds the entry keeps the certified membership through a cut, in place of its committed_voter_configs. When the plan cuts below C, the leader is instead the certified voter with the greatest committed_index among those that hold the entry, the point is C, and point_entry is {"term": converged_at.term, "node_id": converged_at.leader}. Stop when no certified voter that answered holds the entry.
   • Stop when any replica that answered, learners and replicas whose group stopped included, reports a committed_index or a last_applied_index above the point: the plan's leader would overwrite that replica's committed entries.
   • highest_term: the highest current_term among all the replicas that answered.
3. Send the plan to the leader first: {"leader", "point", "cut", "step": {"lead": {"highest_term": H}}, "floor", "expected": {"last_log_index", "last_log_term", "current_term"}}, where H is highest_term, floor is the data group's floor from the confirmed view (seeded, or {"certified": converged_at}) and meta for the meta group, and expected is what you read on the leader. A cut to a reported entry adds "point_entry": {"term", "node_id"}, the same in every request of the plan. Leave it out otherwise. The leader answers outcome leads and leader_term, the plan term.
4. Then send it to every other replica that answered, with "step": {"follow": {"leader_term": T}} where T is the leader_term the leader answered, and expected as you read it on that replica. Each answers follows. Take T from the leader's answer, never from the metrics. A replica refuses a T below the current_term in its expected with plan_term_below_vote, but it cannot tell a wrong T above it. Send no follow before the leader answered. A replica that takes the plan before its leader can elect the plan's leader at a term below the plan term, before that leader took the plan. The leader then keeps leading below the plan term, and the plan never completes. No replica elects a leader that lacks the plan even then (#643). ursulactl and the script below always send to the leader first.
5. When the plan cuts, each replica first stops voting and taking replication, restarts its group in the process with the log cut back to its committed entry, and answers once the restarted group holds the plan. Two replicas keep more. The plan's leader keeps its log up to a point_entry. Under a certified floor, a replica whose committed entry lies below the certified entry, and that holds that entry at or below the point, keeps its log up to the certified entry. A replica that leads or campaigns when it takes a plan restarts its group the same way, so it drops that leadership or campaign. The plan's leader also restarts when its vote term is below the plan term, and starts with the plan term as its vote, so its first election is above the plan term.

The plan term is the higher of highest_term and the leader's own vote term, so it is at least the vote term of every replica that holds the plan. From the moment it takes the plan, the plan's leader campaigns but votes for no other candidate, and while it leads it serves nothing until the plan's history event applies. A replica that holds the plan votes only for the plan's leader, and only in an election at a term above the plan term: an election the leader started after it took the plan. So no replica elects a leader that has not taken the plan, even one that campaigns while the plan reaches the other replicas. It also refuses the plan's leader while the leader's log holds an entry above the point at or below the plan term, and a replica that holds the plan takes replication only from a leader above the plan term. A plan's leader that restarts after it lost entries comes back with a closed gate instead: it votes and campaigns for no one, and the group waits for another plan. A leader elected before the plan at or below the plan term, which may still believe it leads, thus never extends the plan's leader, and none of its entries is applied. A leader at a higher term could only have been elected by replicas that hold no plan: an open replica refuses a leader whose log is behind its own, and a closed one votes for no one, so it cannot get the plan's leader elected over its entries either. Once elected, the leader proposes the group's history event before anything else and answers 503 with Retry-After to writes and reads until the event applies. On each replica the plan is satisfied once it applied the event: history.event_index above the plan's point and recovery back to null in its metrics. The plan is recorded in the node's metadata, so a restart keeps it until then, and the leader serves nothing until its record is gone. A replica that lost power again after taking the plan keeps its vote restricted to the plan's leader across restarts, and once the event applies its gate stays closed until a leader's barrier opens it. Leaders after that hold the event, so a later leader change starts no new history.

If the plan's leader loses entries it had replicated, for example in a crash right after its election, the other replicas' logs are longer than its own and they elect no one. The plan wedges. After 30s in which a replica holding the plan applied nothing, it reports its group stalled like a closed gate does: readiness reason recovery_stalled, and stalled as its recovery_gate. Plan again. The new plan replaces the first, and since the wedged leader's failed elections raised its vote term, it usually cuts every log back to its committed entry. ursulactl recover-group --accept-loss plans again by itself when it sees, in two observations in a row, a replica holding the plan with a longer log than the plan's leader, or reporting the group stalled.

# group 42: one admin URL per node with a replica of the group, learners included, through your forwards
admins=(http://127.0.0.1:5441 http://127.0.0.1:5442 http://127.0.0.1:5443)
for admin in "${admins[@]}"; do
  curl -sS --fail "${admin}/__ursula/metrics" | jq -c --arg admin "${admin}" '
    .process_incarnation as $incarnation
    | .raft_groups[] | select(.raft_group_id == 42)
    | {admin: $admin, process_incarnation: $incarnation, node_id,
       running: .maintenance.running,
       voters: .voter_configs, kept: .committed_voter_configs,
       gate: .recovery_gate, recovery, committed_index, committed_term, committed_node,
       last_applied_index,
       expected: {last_log_index, last_log_term, current_term}}'
done > replicas.jsonl

jq -sc 'def holds($n): any(.[]; . == $n);
  def votes($cut): .node_id as $n | any((if $cut then .kept else .voters end)[]; holds($n));
  . as $all | map(select(.running))
  | (map(.node_id)) as $answered
  | ([.[] | (.voters + .kept)[] | select(length > 0)] | unique) as $sets
  | if all($sets[]; (map(select(. as $n | $answered | holds($n))) | length) * 2 > length) then .
    else error("a reported voter set has no majority of its voters answering") end
  | ($sets | add | unique) as $voters
  | (map(select(.gate != "open" or .recovery != null) | .expected.current_term) | max // 0) as $term_floor
  | map(select(.node_id as $n | $voters | holds($n))) as $candidates
  | ($candidates | max_by([.expected.last_log_term // -1, .expected.last_log_index // -1, votes(false), .node_id])) as $longest
  | ((($longest.voters + $longest.kept) | add // [] | unique | length) == 1) as $single
  | ((($single | not)) and ((($longest.expected.last_log_term // 0) < $term_floor) or ($longest | votes(false) | not))) as $cut
  | (if $cut then ($candidates | map(select(votes(true))) | max_by([.committed_index // -1, .node_id])) else $longest end) as $leader
  | if $leader == null then error("no replica the plan may elect is a voter of the membership it would campaign under") else . end
  | {leader: $leader.node_id, cut: $cut,
     point: (if $cut then $leader.committed_index else $leader.expected.last_log_index end // 0)} as $plain
  | ($all | map(select(.committed_index != null and .committed_term != null))
     | max_by([.committed_index, .node_id])) as $known
  | (if $cut and $known != null and $known.committed_index > $plain.point then
       ($candidates | map(select(votes(true) and votes(false)
          and (.expected.last_log_index // -1) >= $known.committed_index))
        | max_by([.committed_index // -1, .node_id])) as $reaches
       | if $reaches == null then $plain else
           {leader: $reaches.node_id, cut: true, point: $known.committed_index,
            point_entry: {term: $known.committed_term, node_id: $known.committed_node}} end
     else $plain end) as $plan
  | if any($all[]; ([.committed_index // 0, .last_applied_index // 0] | max) > $plan.point) then
      error("a replica knows a later committed entry than the plan would keep") else . end
  | sort_by(.node_id != $plan.leader)[] | . + {plan: ($plan + {expected})}' replicas.jsonl > plan.jsonl

# Review plan.jsonl, then send it: the leader first, then every other replica with the
# leader's plan term. Stop at the first refusal.
leader_term=""
while read -r replica; do
  if [ -z "${leader_term}" ]; then
    body=$(jq -c --argjson highest "$(jq -s 'map(.expected.current_term) | max' replicas.jsonl)" \
      '.plan + {step: {lead: {highest_term: $highest}}, floor: "seeded"}' <<<"${replica}")
  else
    body=$(jq -c --argjson term "${leader_term}" '.plan + {step: {follow: {leader_term: $term}}, floor: "seeded"}' <<<"${replica}")
  fi
  answer=$(curl -sS --fail-with-body -H "x-ursula-process-incarnation: $(jq -r .process_incarnation <<<"${replica}")" \
    -H "content-type: application/json" -d "${body}" \
    -X POST "$(jq -r .admin <<<"${replica}")/__ursula/raft/42/recovery/accept-unsynced-loss")
  status=$?
  echo "${answer}"
  [ "${status}" -eq 0 ] || break
  leader_term=${leader_term:-$(jq -r .leader_term <<<"${answer}")}
done < plan.jsonl

Do not plan again while a replica reports itself leader at a term above the last plan's term: it was elected under that plan, and a member that stalled under it is cut off, not wedged. If a live leader elected before a plan keeps a minority of replicas that took the plan, they take no replication from it, since its term is at or below the plan term. Transfer the leadership away from that leader (POST /__ursula/raft/{target}/leader/transfer/{node}). Every transfer raises the term by one: once a leader's term is above the leader_term those replicas show under recovery, they follow it.

A refusal answers 409 with a typed body: {"raft_group_id", "node_id", "refusal": {"reason", ...}, "message"}. If the leader refuses the plan, nothing changed: read the metrics again and plan again. A replica that refuses after the leader took the plan does not undo it. A replica whose reason is follows_leader with the plan's leader as leader refuses because the replicas that took the plan elected it, and it rejoins through it. For any other refusal, read the metrics again and plan again once the group has no leader. A refusal with reason not_stalled means the replica may still open through a leader's barrier: wait until its gate reports stalled before you plan again.

The script plans one seeded data group. Plan a live group with ursulactl recover-group, or by hand with the certified rules of step 2. For the meta group, read .meta instead of the group under .raft_groups, send floor meta, and post to /__ursula/raft/meta/recovery/accept-unsynced-loss. For many stalled groups, as after a rack power loss, plan the meta group first and then each data group, one group at a time, with the script or ursulactl recover-group --accept-loss.

The call is refused with 409 Conflict, and changes nothing, when:

• the replica lost its vote and has not established a durable vote floor. Restore connectivity to an intact quorum so it can obtain a fresh barrier. Accepting unsynced data loss cannot authorize a replica to acknowledge an unknown leader.
• the replica leads the group with a quorum that acknowledged it within the last 3 seconds or as its only voter, which is its own quorum, or follows a leader whose replication reached it within the last 3 seconds, or its gate awaits or applies a leader's barrier or runs a plan's cut. The group has a live leader or may still recover without loss.
• the request's step is not the replica's: lead goes to the plan's leader only, and follow to every other replica.
• the plan's floor is of the other plane (floor_other_plane): meta sent to a data group, or seeded or certified sent to the meta group.
• the replica is the plan's leader, the floor is certified, and it does not hold the certified entry at or below the plan's point (floor_not_held). Plan again without it: another certified voter that holds the entry leads.
• a follow names a plan term below the current_term its expected names (plan_term_below_vote): the term did not come from the plan's leader.
• the plan cuts a replica whose voter_configs and committed_voter_configs name only one voter (single_voter_cut): no other voter could have overwritten an entry it holds, so its log is kept whole. The tool decides the cut the same way, from the voter sets of the replica with the longest log.
• the replica is taking another plan (plan_in_flight). Wait for its answer and observe again.
• the replica is the plan's leader and is no voter of the membership it would campaign under (leader_not_voter): its voter_configs, or its committed_voter_configs when the plan cuts. The membership moved since you read the metrics: read them again and plan again. Only the plan's leader checks membership. Another replica takes the plan whatever its own membership says, since the leader counts its vote under the leader's membership.
• the replica no longer holds the log expected names, because it moved since you read the metrics. A replica that campaigns, one with an open gate or the plan's leader, raises its current_term with every election it starts, so only its last log id is compared. Read the metrics again and plan again.
• the replica committed or applied an entry above the plan's point. The plan picked the wrong leader: read the metrics again and plan again.
• the replica is the plan's leader and its log does not continue from the point: its last log index, or its committed index when the plan cuts. When the plan cuts to a later entry another replica reported committed, the leader must hold exactly that entry at the point.
• the plan cuts and the replica knows no committed entry, or no longer holds the committed entry it recorded.
• the plan cuts and its cut cannot run when the group restarts, because the log no longer holds the entry it keeps. The replica drops the plan, records its log as recovering so that its gate stays closed across restarts, and answers 409. Wait until the gate reports stalled, then read the metrics again and plan again.

Across a membership change. A membership entry is synced before it is acknowledged, also under never, so a power loss cannot drop it from a replica that holds it. The replicas may still report different voter sets when you plan:

• A change that only a minority appended, such as a promotion the leader sent only to the new voter before it was cut off, has three outcomes. A cut removes it. Without a cut, the plan's leader either holds it and commits it, or lacks it and overwrites it.
• A cut can also undo a committed change. A replica's commit marker is not synced on its own, so after a crash it can sit below a change the group committed. The replica reports the change in voter_configs but keeps the membership before it, which committed_voter_configs shows. A removed voter can come back, and a promoted voter can go back to being a learner. Under a certified floor the membership of the group's last convergence survives the cut, and a change a move made after it may be undone. The group's leader then takes the move's steps again, since its reconcile shell pursues the placement from the membership it finds. Check the group with ursulactl placement status --group G after a cutting plan.

Count every voter set, as step 1 says, and pick the leader by the membership it campaigns under, as step 2 says. A plan's leader whose cut leaves it no voter of the membership it keeps drops the plan and answers cut_left_no_vote: plan again. Every replica that applies the plan's history event holds the leader's membership. Every replica that holds a plan and does not lead answers client reads of the group (catch-up, live reads, HEAD, snapshots and forwarded reads) with 503 and Retry-After at once until it applied the event, as the plan's leader does after a short wait, so a replica outside the leader's membership, which never receives the event, never serves the history the plan replaced. Its /__ursula/usage and /__ursula/metrics keep answering. No leader adds it back as a learner: a live group's leader changes no voter outside a move, so the group runs on the voters the plan's leader kept, and the maintenance report names incomplete_voter_set. To give the group its replication factor back, move it onto voters that replace the left-out node with another one, ursulactl placement move --group G --replace X=Y: the moving group's leader adds the new voter, and the left-out replica is torn down. A cluster with no node outside the group needs one added first (Adding a node). If its log committed entries above the plan's point, it cannot follow: rejoin its node as in Rejoining a wiped node, which affects every group on that node. The same holds for a replica that no plan reached while two or more plans ran in a row, or whose committed history a later plan dropped: it refuses the leader's appends and snapshots, since nothing shows that the new history kept its committed entries. It logs the refused snapshot with UnrelatedHistory when two or more plans ran, and with CommittedOverwrite when one plan's history replaced its own below its commit marker. Its own committed entries above the first such plan's point are gone for good: that plan replaced them, and wiping the node only stops it from waiting for them.

Rejoining as a wiped node is an operator action, and it wipes every group on the node. Before you wipe, check every other group the node holds a replica of in /__ursula/metrics. Do not wipe while the node may hold the only copy of another group's acknowledged writes: a group whose other voters are gated, stalled or down, or whose leader is this node. Do not wipe while another group on the node would lose its quorum without this replica: while it is rebuilt, each of its groups needs a quorum of the other voters, and a group whose other voters are gated stalls. Recover or wait for those groups first.

A voter set that names nodes that never come back can block every plan. The tool needs a majority of each set's voters, because it cannot rule out that the set committed, and the plan's leader must win one. When those nodes are gone for good, no plan can prove what the group committed: restore the group from a backup.

A replica that cannot lead knows more is committed. A learner, or a promoted voter whose cut would make it a learner again, can know a later committed entry than every replica that could lead. Commit markers are not synced on their own, so after a power loss the voters can still hold that entry in their logs while their markers lie below it. The tool then plans a cut that keeps every log up to that entry, led by a voter whose log reaches it, and the leader refuses the plan unless it holds exactly that entry. When no voter it reached holds the entry, or a replica reports a committed or applied index above every log that could lead, the tool stops with an error naming that replica: a plan from any voter would overwrite those committed entries. Each replica also refuses an append, or a snapshot install, that would replace entries at or below its commit marker. Waiting does not help, since the voters' logs no longer hold the entry. Bring back a voter that was not reached, if one may hold it, and plan again. Otherwise restore the group from a backup.

A body of the 0.7 form, or one without a floor as 0.9 sent, answers 422. A second plan replaces a first one that was never satisfied, as long as the term floor counts the replicas the first one holds.

A voter that was never gated does not report recovery_stalled. It stays ready, and raft_maintenance.group_issues shows leader_unknown for the group, which blocks maintenance.

With fsync = "always", acknowledged appends survive a host crash, but the newest journal segment can still hold bytes written after the last fsync: an unacknowledged batch, or a commit or truncate marker. Recovery refuses damage in sealed segments. In the newest segment it removes an incomplete final frame without gating anything. A complete frame that fails verification, such as a hole, a zero-filled tail or a torn sector inside an unacknowledged batch, looks the same as media corruption of an acknowledged write. Recovery therefore records the recovery gate of every group on that core before it cuts the frame, and those groups rejoin as described above.

A single-voter group has no other replica to recover from, so it needs a plan after every host crash under never, once its gate reports the group stalled, and every such plan gives every stream of the group a new Stream-Incarnation. Stream-Previous-Incarnation and Stream-Inherited-Offset on HEAD then tell clients how much of each stream survived. Use fsync = "always" for single-voter groups whose clients keep state. Under always it needs the call only after a host crash that leaves a complete invalid frame in the newest segment. So does a group whose voters lose power together, for example in a rack power loss, when that gates a majority of them. No acknowledged write is lost in these cases, but only an operator can open the gates.

POST /__ursula/cluster-probe is served on the cluster plane, but it is machinery rather than an operator surface. Nodes send each other probe bodies through it at their peer url to verify cluster-plane egress. The cluster plane runs no ingress admission, so a node under write pressure still answers it.

Memory without retention

Per-stream replicated state, on every replica, is at most about 32 KiB plus bounded producer state, beyond the unflushed payload itself, without retention. It grows neither with the number of messages nor with the length of the cold history. The bounds that make this hold are always on: one receipt per producer with 7-day idle-producer expiry and a 4,096-producer cap, unique stream incarnations with incarnation-scoped cold objects, external payload locators committed first and indexed after, and a compact hot window with no per-message bookkeeping, so cold admission and the flush thresholds charge hot payload only. Snapshot bodies of at least runtime.external_payload_min_size go to the cold tier when a cold backend is configured, up to 1 GiB uploaded in parts. Retention stays available as a storage-cost tool, but it is not the way to bound memory.

The per-group gauges under group_state_gauges in /__ursula/metrics show what each group holds.

Usage export

GET /__ursula/usage returns committed counters and current gauges from local replicated state, by bucket, for each Raft group the node hosts a replica of. The usage of each stream version is in Per-stream usage. A report:

{
  "version": 2,
  "node_id": 1,
  "write_unit_bytes": 10240,
  "groups": [
    {
      "raft_group_id": 0,
      "applied_index": 42,
      "buckets": {
        "tenant-a": {
          "committed_append_bytes": 24576,
          "committed_records": 3,
          "committed_write_units": 4,
          "retained_bytes": 24576,
          "stream_count": 1,
          "snapshot_bytes": 8192
        }
      }
    }
  ],
  "raft_group_count": 1,
  "staged_snapshot_parts": [
    {
      "raft_group_id": 0,
      "term": 7,
      "counting_finished": true,
      "bytes": { "tenant-a": 33554432 }
    }
  ]
}

The raw byte and record counters are pricing-neutral facts. committed_write_units is a convenience derived meter: every committed, non-deduplicated create or append contributes at least one unit and rounds its canonical payload up independently by write_unit_bytes. Consumers must validate both version and write_unit_bytes before interpreting it; the unit is data in the contract, not encoded into the counter name. The same unit is persisted in Raft snapshots, and a build refuses to restore a snapshot carrying a different unit. Changing it therefore requires a new contract version and an explicit state migration, never a configuration edit. Ursula does not attach currency, a price, or an account owner.

Counters are monotonic and survive snapshots, stream deletion, and bucket purge. retained_bytes, stream_count and snapshot_bytes are current-state gauges and may decrease. A node serves local applied state, and only for the groups it hosts, so a collector polls every node. For each group it keeps the report with the highest applied_index, the newest of the group's replicas, and sums the kept groups by bucket. Of two reports at the same applied_index, it keeps the one that carries snapshot_bytes for every bucket. It treats the total as partial when no report names one of the raft_group_count groups. ursula_proto::admin::UsageReport::bucket_usage applies these rules. One response is never a cluster-wide linearizable read.

snapshot_bytes is the summed body length of the latest published snapshot of each of the bucket's live streams, as the client uploaded it: an inline body's length, or the body length recorded for the cold object (its payload_len, not the object's size), up to 1 GiB per stream. It counts the image each stream keeps, not the bytes published over time. A replaced body during its cold-GC grace, staged parts (below), Raft group snapshots and backups do not count, and the value drops when a stream is deleted, swept after expiry, or purged. Each replica derives it from its applied state whenever usage is read, so nothing persists it, replicas at the same applied_index report the same value, and a restore reports it again from the restored streams. A node of 0.10.1 or later reports the field for every bucket, zero included. A report without it, such as one from a 0.10.0 node, means the value is unknown, not zero: ursula_proto::admin::BucketUsageReport decodes it as None, a sum with an unknown part is unknown, and UsageReport::bucket_usage reports a bucket's total as None when any kept group's report lacks it. Ursula never writes the field as null: an unknown value is left out, so a summed report keeps the shape of a 0.10.0 report. A collector must not bill an unknown value as 0. During a rolling upgrade from 0.10.0 it waits until every kept report carries the field.

staged_snapshot_parts reports the staged snapshot parts that no commit has used yet, separately from the replicated fields: for each Raft group the answering node leads, the group, the node's term and the bytes of staged parts per bucket. A new term's totals start at the node's first part, commit or discard in the group, or when its Raft metrics show it leads, whichever comes first. The node then lists the group's part directories in the background and reports counting_finished: false until that listing ends. From then on it follows every part it stages, every commit, discard and delete, and every part the orphan sweep reclaims. The bytes are exact except for parts that were still uploading to the previous leader when the term began, parts a failed delete may have removed, and a part the sweep deletes while the seed listing runs, each until its stream is listed again. A collector polls every voter. For each group it keeps the report with the highest term, so a deposed leader that has not yet seen the new term is not counted twice. It treats the total as partial when a kept group has not finished counting, or when no report names one of the raft_group_count groups, for example a group without a leader. ursula_proto::admin::UsageReport::staged_part_bytes applies these rules. node_id is the answering node's id. Ursula bounds staged parts per stream only. A bound per bucket, or a charge for staged parts, is the operator's policy.

Per-stream usage

POST /__ursula/usage/streams lists, per Raft group, the usage of each stream version that changed after a cursor: its committed counters and appends, its last write, its counters when the group's latest history began, its retained and snapshot bytes while it lives, and how and when it ended. It is served on the admin listener only, because it names every tenant's streams. The client listener answers 404. It is a read, so it takes no x-ursula-process-incarnation. Each node answers from its own applied state, leader or follower, for the groups it hosts a replica of, so a metering consumer polls every node. ursula_proto::admin holds the request and answer types.

A stream version is one life of a stream: a create, its writes, and its end by a delete, an expiry or a bucket purge. A removal leaves an end record with the version's final usage, which stays listed for end_record_retention_ms, half of raft.deleted_data_max_age, and then folds into its bucket's retired usage (Deleted data). Two cases fold it sooner. A group that holds 65,536 end records folds the oldest one at once, which usage_end_records_folded_early counts. A node that restarts or installs a snapshot dates the end records it holds then by their removal times on its own wall clock, so a clock ahead of the removing node's folds them early. A bucket's totals in /__ursula/usage are its retired usage plus its live versions plus its end records.

curl --fail -H 'content-type: application/json' http://127.0.0.1:4438/__ursula/usage/streams \
  -d '{"version": 1, "limit": 10000, "groups": [{"raft_group_id": 0, "cursor": {"kind": "start"}}]}'

The request names each group once with its cursor, and limit, 1 to 10,000, the most items, entries and bucket rows together, of each group's page. A cursor is {"kind": "start"}, or {"kind": "after", "genesis", "history", "applied", "retired_through", "position"} taken from the last page kept, with position {"kind": "entries", "after_seq": n} or {"kind": "rows", "after_bucket": "b"}. {"kind": "rows"} without a bucket asks for a relisting. An answer holds at most 20,000 items over all its groups.

{
  "version": 1,
  "node_id": 1,
  "write_unit_bytes": 10240,
  "end_record_retention_ms": 1800000,
  "raft_group_count": 4,
  "groups": [
    {
      "state": "listed",
      "raft_group_id": 1,
      "genesis": "5d0e9f3a71c2b864",
      "history": { "id": "5d0e9f3a71c2b864", "event_index": 1 },
      "applied": { "term": 3, "index": 57 },
      "usage_seq": 43,
      "retired_through": 37,
      "live_versions": 1,
      "end_records": 1,
      "rows": [],
      "entries": [
        {
          "seq": 43,
          "bucket_id": "tenant-a",
          "stream_id": "notes",
          "created_at_ms": 1700000000100,
          "counted_since": "5d0e9f3a71c2b864",
          "incarnation": 5124095576030430,
          "counters": { "committed_append_bytes": 10250, "committed_records": 2, "committed_write_units": 3, "appends": 1 },
          "history_baseline": { "committed_append_bytes": 0, "committed_records": 0, "committed_write_units": 0, "appends": 0 },
          "last_write_at_ms": 1700000000200,
          "lifecycle": { "state": "ended", "end": "deleted", "ended_at_ms": 1700000000300 }
        }
      ],
      "page": { "status": "complete" }
    }
  ],
  "omitted": []
}

A group answers listed with one page read from one applied state, behind when the replica has not applied the entry the cursor's page was read at, or unavailable with a reason: apply_stopped, recovery_pending, raft_stopped, no_history (the group's first history has not applied on the replica) or other. A group the node hosts no replica of is absent, and a group the answer's item budget left unread is in omitted. History ids are 16 lowercase hex digits. Fields without a value are left out, never written as null. A page that starts a relisting carries relist: start for a start cursor, requested, retired when an end record the cursor had not listed was folded, or history when the cursor names another history or another cluster's group. A relisting lists every bucket the group holds retired usage for first, by bucket id, with its retired usage and its /__ursula/usage totals at the same read, and then every live version and unretired end record from seq 0. A later page of the release line can carry a value this build does not know, in a group state, a page status, a next position, a relist reason, a lifecycle or an end. It decodes as unknown (other for a reason).

Answer	Meaning	Action
400 or 415	The body is not JSON, or it comes without Content-Type: application/json.	Fix the request.
422	The body does not decode: version not 1, limit 0 or above 10,000, an unknown cursor or position kind, a history id that is not 16 hex digits or is 0, an empty bucket in a position, no group or a group named twice, or a group at or above raft_group_count.	Fix the request.
503 with Retry-After	The node's runtime does not serve, for example while it shuts down.	Ask the same node later, or another node.
404	The client listener, or a node of an earlier release.	Poll a 0.11 node's admin listener.
group behind	This replica lags the cursor.	Use another node's answer, or ask later.
group unavailable	This replica could not be read.	Use another node's answer.
group absent, or in omitted	The node hosts no replica of it, or the answer budget left it unread.	Another node, or ask again for those groups.

A metering consumer follows these rules:

1. Poll every node's admin listener. Name every group, with {"kind": "start"} the first time.

2. Per group, keep the listed answer with the highest applied, by term and then index. behind, unavailable, unknown, an absent group and an omitted group are no answer from that node. An answer that holds unknown anywhere, in a page status, a next position, a relist reason, a lifecycle or an end, is no answer: read it with a reader of the server's release line. Answers of one group with different genesis come from different clusters: poll one cluster.

3. The next cursor takes genesis, history.id, applied and retired_through from the kept page, and its position from page.next when the page is more, or {"kind": "entries", "after_seq": usage_seq} when it is complete. GroupUsagePage::next_cursor builds it. Ask for a more page's next position at once, without waiting for your next poll. A fold of an end record that the listing has not reached yet restarts it as retired, and during a relisting's rows every fold does. A group's leader folds at each sweep, every minute or every sixteenth of raft.deleted_data_max_age if that is shorter, so a relisting longer than one page has to finish within one sweep interval.

4. A version's key is (bucket_id, stream_id, created_at_ms, counted_since). Never key by incarnation, which changes at every history event. incarnation is the join between an entry and a token that PUT or HEAD answered: a live entry's is the token the server answers now, an ended entry's the one it answered when the version ended.

5. Hold, per version, its billed level: the counters you have accounted for. Hold, per bucket, a retired view: the bucket's retired usage you have accounted for.

6. A page without relist continues your listing. For each entry, account counters - billed level, which is never negative within a history, and set the level to counters. An ended entry closes its version: write its zero gauge sample at ended_at_ms, and hold it until retired_through reaches its seq. Then add its final counters to its bucket's retired view, keep its key and final counters as folded for one more end_record_retention_ms, and then forget it. Within one history no version reappears after its end record.

7. A page with relist starts a relisting: rows first, then every live version and unretired end record from seq 0. Collect its pages until one is complete, asking for each next page at once (rule 3), and drop a relisting that a later relist page interrupts. Then apply it once, by its history compared with that of the last listing you applied:

   • Nothing applied yet, your first start: take it as your baseline. Every listed version's billed level is its counters, and every bucket's retired view is its row's retired. Account nothing.
   • The same history (requested or retired): for each listed version, account counters - billed level, with a level of 0 for a version you did not hold. Close each version you held that it does not list: its end record was folded before you read it. Write its zero sample at your previous complete poll. Per bucket, account retired - retired view - Σ billed levels of the versions it closed to the bucket, not to a version. It is never negative: alarm if it is. Set the retired view to retired.
   • Another history (history: an accepted loss, a restore, or another cluster's group): a folded version that it lists again, whose fold the loss replaced, is held again. Take its final counters out of its bucket's retired view, and its billed level is its final counters. Then, for each listed version, lower its billed level to its history_baseline when that is lower, with a level of 0 for a version you did not hold, and account counters - billed level. Close each version you held that it does not list. Per bucket it lists a row for, account max(0, retired - retired view - Σ billed levels of the closed versions) to the bucket, and set the retired view to retired. A bucket it lists no row for keeps its retired view, and the billed levels of the versions it closed are added to that view, so a later relisting that lists the bucket's row accounts against both. Rows are never removed within one history, so only another history lists a bucket without its row, for example a new cluster's group read before ursulactl restore imported it.

   Then treat the relisting's end records as in rule 6, including those it lists at or below its retired_through.

8. Check on every complete page that live_versions and end_records equal the live and ended versions you hold of the group. On a relisting that completed in its first page, also check each row: its totals counters equal its retired view plus the counters of every version you hold of the bucket, and its totals gauges equal the sums of the live versions' gauges. Alarm on a mismatch. To check a group at any time, send its cursor with {"kind": "rows"} as the only group of a request with limit 10,000: it relists (requested) and you apply it as in rule 7. A group whose bucket rows, live versions and end records pass 10,000 items cannot complete a relisting in one page, so it checks its counts only.

9. A poll is complete when every group from 0 to raft_group_count - 1 has a kept complete page and no relisting is in progress.

10. Read every group completely at least once per third of the smallest end_record_retention_ms among the nodes' answers, 10 minutes at the defaults, and alarm at half. A retired relisting means that an end record folded before you listed it: you did not read in time, a relisting did not finish within one sweep interval (rule 3), the group held 65,536 end records (usage_end_records_folded_early), or a node dated its end records by its wall clock after a restart or an install. What ended unread is accounted to its bucket.

11. On each complete poll, sample retained_bytes and snapshot_bytes of each live version.

12. A purge is complete for metering when, for each group of the purge report's groups, your kept complete page has history.id equal to the group's history and applied.index at or above its commit_index. Otherwise a history event may have replaced the purge: purge again, which is idempotent, and use the new report.

Why these rules account every committed unit exactly once:

• Within one history, every committed unit is accounted once: to its version by the level rule, or to its bucket when its end record was folded before you read it. The bucket's share is retired - retired view less what you had already accounted to the closed versions, which is the formula of rule 7.
• The bucket share and this exactness cover the three committed counters. appends is exact only for the versions you read before their end record folds: a row's retired holds no appends, so the appends of a version whose end record folded unread, or that a restore closed, are accounted nowhere.
• A history baseline is recorded at the version's first write after the history event, together with the bytes it inherited. So a version's usage in the new history is counters - history_baseline whenever you read it, and lowering the level to the baseline keeps every unit you accounted in a replaced suffix accounted, since it was observed, without counting the kept prefix twice. Without it, the units written between the event and your first read would depend on when you read.
• A restore gives every restored version a new key (counted_since), because a backup holds no per-version usage. So every version you held closes, and the restored buckets' retired, which carries the backup's totals, gives the bucket whatever you had not accounted. The new versions count from zero. Point the consumer at the new cluster once ursulactl restore has finished: a start baseline taken before a group's import holds none of the backup's totals, so the import's relisting would account them all again.
• A lagging replica's answers are behind, and you keep the highest log id, so you never apply a listing from an older state than one you applied. Every entry of a lineage that replaced an accepted loss carries a higher term than every entry it replaced, so (term, index) orders replica states across a loss too.
• One case stays inexact, and in the tenant's favor. A version a loss removed and one whose end record was folded before you read it both vanish from a history relisting, and the second's usage is in retired while the first's is gone. The bucket share subtracts both levels, so when a loss coincides with a read missed past the retention, the bucket can be accounted less than it holds, never more. Reading within the retention (rule 10) leaves only the first kind.

Tenant offboarding (bucket purge)

DELETE /__ursula/purge/{bucket} erases one tenant's complete bucket domain: every stream in every Raft group, the bucket, cold chunks and indexes, external payloads, and any uncommitted object-store staging orphan below {bucket}/. The replicated purge also installs a permanent bucket-erasure fence in every group. It retains only the bucket's aggregate usage counters, because deleting a counter before an asynchronous meter observes it would erase committed billable work. Purge is served only on the admin listener (server.admin_listen, loopback 127.0.0.1:4438 by default); the client listener answers 404. On Kubernetes the chart keeps the admin listener on pod loopback, so reach it through kubectl port-forward pod/<voter-pod> 4438:4438 or kubectl exec; any one node accepts the request for the whole cluster. It purges the groups it hosts itself and sends each other group's purge to that group's leader.

The response is a completion report an operator can attest to:

curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X DELETE http://127.0.0.1:4438/__ursula/purge/tenant-a
# {"bucket":"tenant-a","removed_streams":12,"groups_with_streams":[0,3],"cold_gc_entries_reclaimed":12,"cold_gc_pending_entries":0,"cold_gc_complete":true,"cold_gc_error":null,"bucket_prefix_absent":true,"groups":[{"raft_group_id":0,"commit_index":812,"history":"5d0e9f3a71c2b864"},{"raft_group_id":1,"commit_index":640,"history":"0c41a7e2d95b3f18"},{"raft_group_id":2,"commit_index":577,"history":"7b2f60c8e1a4d935"},{"raft_group_id":3,"commit_index":903,"history":"2e9d4b17a6c0f583"}]}

Semantics:

• Retry on 503. A Raft group whose leader is unknown or changing answers 503 with Retry-After, never a redirect to a peer. Because purge is idempotent, retry the same request.
• Idempotent and resumable. Re-purging an erased bucket returns the same report shape with zero removals. removed_streams counts what the answering request removed, so it may undercount a purge that a lost answer or a 503 interrupted. A purge interrupted mid-way converges on re-run: the permanent fence and stream removal are replicated commands per group, while cold reclamation and the complete bucket-prefix erasure are repeatable object-store operations.
• Permanently fenced. Commands linearized after purge cannot create the bucket or append to its streams. The fence survives Raft snapshots and restore; data-plane access continues to conceal the erased namespace as 404 Not Found instead of recreating it.
• Isolated. Other buckets' streams, offsets, checksums, snapshots, and object-store prefixes are untouched. Every cold object, shared packs included, lives under its own bucket's {bucket}/ prefix.
• Accounting residue. The retained entry contains only aggregate counters and zero-valued content gauges, not stream data. Ursula does not currently provide an acknowledgement protocol for erasing it after an external meter persists the final delta. Each purged stream also leaves an end record with its id and final usage, which stays in its group's state for half of raft.deleted_data_max_age and then folds into the bucket's counters, so the purged stream ids leave every replica's snapshot and log within raft.deleted_data_max_age (Deleted data). Use opaque bucket IDs if retaining the identifier itself would violate an erasure requirement.
• Physical absence proof. cold_gc_entries_reclaimed is diagnostic only. After a second replicated purge proves every group's bucket-specific GC queue is empty, Ursula recursively deletes the complete {bucket}/ erasure domain and lists the authoritative object store to prove it contains no files. Only cold_gc_complete=true, cold_gc_pending_entries=0, bucket_prefix_absent=true, and no error constitute the proof. If object storage is unavailable or still lists a file, the response remains incomplete and a later idempotent request retries the proof.
• Purge index. groups names every group the purge reached, ascending, with the index of the entry that purged the bucket there. A metering consumer holds the purge complete once it read each group at or past that index (Per-stream usage, rule 12).
• Backups. A purged tenant may persist inside backups taken before the purge until those backups age out; restore replays whatever the manifest contains. Deleting old backup sets is part of completing an offboarding with erasure obligations.
• Raft snapshots and logs. Each replica drops the purged streams from its current Raft snapshot and its Raft log within raft.deleted_data_max_age. See Deleted data for what that bound covers.

Deleted data

Deleting a stream and purging a bucket remove the stream from its group's state at once. An expired stream reads as gone at once and leaves its group's state within about a minute, as described below. Copies of its records can stay for a while in each replica's Raft snapshot, its Raft log and the log's files, and in older snapshot objects of an S3 snapshot store. They are cleaned up in the background. raft.deleted_data_max_age (1 hour by default) bounds how long they stay.

A removal also leaves an end record in its group's state: the stream's id, how and when it was removed, and its final usage, which the per-stream usage export lists (Per-stream usage). Each group's leader folds the end record into its bucket's usage counters half of raft.deleted_data_max_age after the removal, by the leader's own monotonic clock, and the fold counts as a removal. The stream's id then leaves the replica's snapshot and log as described below, so it too is gone within raft.deleted_data_max_age: at the defaults about 49 minutes after the removal. A node that restarts, or installs a snapshot, dates the end records it holds then by the removal time each carries, and folds each one half of raft.deleted_data_max_age after that time by its own wall clock, so keep the nodes' clocks in sync. A group holds at most 65,536 end records: a removal past that folds the oldest one at once. The server refuses to start with a raft.deleted_data_max_age too short for an end record to leave within it (Raft configuration). usage_end_records_retired and usage_end_records_folded_early in /__ursula/metrics count the folds the node's replicas applied, and group_state_gauges reports each group's end_records (Observability).

Within that time after a replica applies the removal:

• The replica's current snapshot no longer holds the stream. That covers the snapshot in memory, the snapshot file in the Raft log directory and, with an S3 snapshot store, the snapshot object that file names.
• The replica's Raft log holds no entry with the stream's records.
• No file in the replica's Raft log directory holds the stream's records, as long as the journal can copy the entries it keeps fast enough. A purged entry's bytes stay in its journal segment until the segment is deleted. A segment that has held such bytes for a quarter of raft.deleted_data_max_age since the purge is sealed, the few entries other groups still keep in it and in the segments before it are copied into the newest segment at the rate below, and the segments are deleted. At the defaults that takes far less than the rest of the bound. With a short raft.deleted_data_max_age, a slow drive interval and large live logs to copy, it can take longer.
• The snapshot store no longer holds the snapshot objects the replica used before.

Removed bytes count toward a group's next snapshot, so a large removal leaves the snapshot and the log within minutes. A small removal makes its group take a snapshot a quarter of raft.deleted_data_max_age after the replica applied it, and the replica then purges its Raft log through that snapshot. For a large and a small removal alike, the segments that held the purged entries go a quarter of raft.deleted_data_max_age after that purge, plus the time the copies take. Cold chunks, cold indexes and external payloads are deleted by cold GC, usually within seconds, and a bucket purge reports when they are gone.

Each group's leader removes the streams that expired in its group in the background, every minute, or every sixteenth of raft.deleted_data_max_age if that is shorter. Each time it writes up to 16 small Sweep Raft entries per group, and starts no more after 250 ms. Each entry removes up to 256 expired streams on every replica, earliest expiry first, and then folds up to 256 end records that are due into their buckets' usage. An end record is due half of raft.deleted_data_max_age after its removal, by the monotonic clock of the node that leads the group. A pass that stopped at its 16 entries or its 250 ms in a group runs again a second later, with 50 ms per group so that client writes keep their latency, until the group's backlog is gone. A pass that stopped on an error waits for the usual interval. No other write removes an expired stream it does not name, while a request that names one removes it at once. The leader removes a stream once its own clock is a second past the stream's expiry, so keep the nodes' clocks in sync. As for a deletion, the time counts from when a replica applies the entry that removed the stream. The removal also queues the stream's cold objects for cold GC and takes the stream out of retained_bytes and stream_count in /__ursula/usage. ttl_expiry_sweeps in /__ursula/metrics counts the Sweep entries the node's leaders wrote, those that only fold end records included, ttl_expiry_swept_streams the streams they removed, and ttl_expiry_sweep_due_groups the groups the latest pass left with expired streams or due end records.

Freeing segments this way copies the live entries of quiet groups forward about once per quarter of raft.deleted_data_max_age, which shows in wal_rewritten_bytes. The snapshot driver drives the copies at most once per storage.snapshot.drive_interval, an eighth of raft.wal.segment_size per core each time, so a mass removal does not saturate the disk. At the defaults that is 8 MiB per second per core with inline snapshots and 1.6 MiB per second with an S3 snapshot store. A group that keeps more than a quarter of a segment there is reported in wal_lagging_groups, and the driver snapshots it first, or purges its log through its snapshot when it applied nothing since. wal_dirty_segments counts the segments that hold purged entries. The first passes after an upgrade or a restart find all the old purged entries at once and copy the live entries of every kept segment forward, up to about twice a core's live log, and snapshot the groups reported lagging, before the journal settles.

The time counts per replica, from when that replica first applies the removal. A replica that is down when the removal commits keeps its copies until it is back. It then applies the removal from the leader's log, or installs a newer snapshot, and the time counts from there. For the snapshot and the log, a restart after the replica applied the removal does not extend the bound: a removal it applies again while it replays its log at startup is due at once, and it purges its log through the snapshot it restored. The log's files follow the purge. Purged entries in a journal segment keep their age across a plain restart, since it is read from the segment's last change, but two cases count from the restart. Entries the replica purges only after it restarts leave the files a quarter of raft.deleted_data_max_age after that purge. A restart that rewrites the journal's segments, after a host crash under raft.wal.fsync = "never" or when the journal ended in a damaged frame, starts their age again. One exception: a replica that loses the removal from its log in a host crash under raft.wal.fsync = "never" learns it again from the leader, and its time counts from then.

The snapshot driver enforces the bound and reports its gauges. It runs on every node that runs Raft, a single node included, and cannot be turned off. A single node follows storage.snapshot.backend and refuses writes for its log's pressure as a cluster node does.

While the driver purges a group's log through a snapshot, the group takes no new snapshot until that purge lands, which is usually at once. An operator's POST /__ursula/raft/{target}/snapshot waits for it, and answers 503 with Retry-After when the wait passes 10 seconds. An operator's POST /__ursula/raft/{target}/purge?upto=<index> follows the same rule: while a snapshot of the group builds, OpenRaft has not published the one it built, or an earlier purge of the group has not landed, it answers 503 with Retry-After. Send it again then.

Under write pressure the driver first snapshots the groups that free the most log, and groups that owe a removal wait their turn among the due groups. They lead the driver's passes again once the node's log is below three quarters of its budget.

With an S3 snapshot store, each node writes its snapshot objects under <prefix>/node-<id>/group-<gid>/, each under a name no object had before, and only that node deletes them. The one exception is a node that was decommissioned and removed: the meta leader deletes its objects of a group once the group's leader has confirmed with its quorum that the node is no member, and no node deletes an object before the meta group answered its registration in that run. A node that installs a snapshot from a peer writes its own copy of the object before it uses it, so no node ever uses another node's object for long: a peer reads the object only while the leader's send of it is in flight. What a node may still use of its own objects lives in its memory: the object its current snapshot names, the object a build or an install copy is writing, and the object a send of its snapshot names until that send returns. A snapshot GC pass lists the node's directory for the group and deletes every object in it that the node no longer uses. So a node deletes the object it replaced at its next pass once no send of it is in flight, which a send's raft.install_snapshot_timeout bounds, and with it whatever an earlier build or install left behind. A write the store answered with an error can still land later, so a node runs another pass one raft.install_snapshot_timeout after such a write and after a restart. GC reads no modification times and assumes nothing about clocks. Each build runs a GC pass as it ends. The snapshot driver runs the passes a group still owes after its snapshot builds, and holds them back while the node's log is under pressure or the store health gate reports trouble. snapshot_gc_deleted_objects in /__ursula/metrics counts the objects GC deleted, and snapshot_gc_failed_passes the passes that failed, which run again later.

A node restores only a snapshot object of its own directory. Started with another node's Raft log directory or with another storage.snapshot.s3_prefix, it refuses to open each group whose snapshot names another object, and logs ForeignKey with the object and its own directory. Start it with its raft.wal.path emptied to rejoin as a wiped node (Rejoining a wiped node).

The bound does not cover:

• Retention advances. Records a stream's retention trims are not a removal and leave the snapshots and the log on the group's usual cadence.
• A leader whose follower lags. A leader purges no log entry a follower still needs, so a slow or unreachable follower holds the leader's log through the removal until it catches up or the leader sends it a snapshot.
• A group whose Raft is not running on the replica, for example while the replica restarts or after it stopped on a storage error. The time keeps counting, and the driver cleans the group up once its Raft runs again. Until then its live log entries also keep the journal segments they share with other groups.
• Journal segments while the snapshot store is in trouble. While the store health gate holds routine snapshots back and the node's log is not under pressure, a group that keeps more than a quarter of such a segment and applied entries since its snapshot is not snapshotted, so the segment stays until the store recovers or the log comes under pressure.
• A replica that is down for longer than raft.deleted_data_max_age. It catches up when it is back, and its time counts from when it applies the removal.
• Sustained write pressure. While the node's log stays above three quarters of its budget, the driver snapshots the groups that free the most log first and holds snapshot GC back, and the time can run past the bound.
• A node that is down. Its snapshot objects stay until it is back and runs its GC passes.
• A removed node's snapshot objects. The meta leader's removal pass runs every 30 seconds while a node is removed and deletes a group's objects of the node only once the group's leader answers that the node is no member. leader_report.removal_pending lists what each removal waits for (Cordon and decommission).
• Object versioning. On a bucket with versioning, a deleted snapshot object stays as a noncurrent version until a lifecycle rule expires it.
• Backups taken before the removal. Delete them to complete an erasure.
• Freed disk blocks and memory. Ursula deletes and replaces files but does not overwrite the blocks they used, and read caches keep blocks of a deleted stream until they are evicted. Use encrypted volumes and buckets where that matters.
• A group whose committed application failed. It keeps its whole log for replay until it is repaired.

raft_deleted_data_oldest_age_ms in /__ursula/metrics reports how long ago the oldest removal on the node applied whose stream its current snapshot or its Raft log still holds, and raft_deleted_data_overdue_groups counts the groups whose snapshot still holds one past a quarter of raft.deleted_data_max_age. deleted_data lists each such group with what it still holds: owed or replayed while its snapshot holds the stream, with the bytes and age, and purging while only its log does, with the age and the index the log is purged through. Alert when raft_deleted_data_oldest_age_ms nears half of raft.deleted_data_max_age.

Cleaning S3

Cold data lives under the storage.cold.root prefix. Benchmark and test runs should use a date-stamped root so cleanup is one prefix delete with standard S3 tooling:

aws s3 rm "s3://my-ursula-bucket/ursula-test-20260518T000000Z" --recursive

There is no automatic retention policy. Application checkpoints and explicit stream-retention advances control which cold objects become eligible for GC. After a retention advance, each group's leader deletes the stream's cold objects that lie wholly below the retained offset (chunks and offloaded external payloads, in whole 64 MiB cold-index pages, so up to one page just below the retained offset can remain until retention moves past it) once a 5-minute grace has passed, so reads planned before the advance still complete; its cold-index cursor visits 16 streams per group per minute, so on groups with many streams collection can lag further. The grace restarts after a leader change. Retention is a storage-cost tool only: replica memory does not depend on it (see Memory without retention).

Cold GC deletes a path a queued entry names only if it lies in the entry's own namespace: an exclusive chunk or an external payload of a stream in the group, or a pack of the group, each in the entry's own bucket. It refuses any other path, such as a snapshot object, a cold-index page or another bucket's object, and it refuses an entry for a whole stream that is not of the entry's bucket or is placed in another group. Only a corrupt or hand-edited backup can queue these. A refused path or stream stays in the store, the entry's other paths are deleted as usual, and the entry leaves the queue, so GC does not stall. Each refusal is logged as an error and counted in cold_gc_foreign_paths_refused in /__ursula/metrics. Alert on any increase, and check the backup the group was restored from.

The check is on the namespace, not on liveness. It does not prove that nothing references an object of the right shape in the group's own namespace. Ursula's own entries are safe because each one is queued only after the state drops its references, and every object name is unique.

A leader proposes an object it uploaded only in the leadership it uploaded it in. When that leadership ends between the upload and the proposal, even by a re-election of the same node, the proposal is refused before anything of it is written and the leader deletes the object at once. Objects of writes whose outcome is unknown stay until the orphan sweep finds them unreferenced and more than a day old.

Incomplete multipart uploads

Ursula writes large cold objects with S3 multipart uploads. A crash or shutdown in the middle of a cold write leaves an incomplete upload behind. Ursula never aborts these uploads, and S3 bills for their uploaded parts until they are aborted. Snapshot commits from parts make long cold writes more common, so these leftovers show up more often.

Add an AbortIncompleteMultipartUpload lifecycle rule to every cold bucket, for example after 1 day:

aws s3api put-bucket-lifecycle-configuration --bucket my-ursula-bucket \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "abort-incomplete-multipart-uploads",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 1}
    }]
  }'

This call replaces the bucket's whole lifecycle configuration. If the bucket has other rules, add this rule to them. MinIO and other S3-compatible stores need their own equivalent rule. Check the store's lifecycle documentation for how to expire incomplete multipart uploads.

Backup and disaster recovery

ursulactl provides a verifiable whole-cluster backup and restore workflow:

# Create: one MessagePack snapshot object per raft group + manifest.json,
# into a local directory or s3://bucket/prefix.
ursulactl backup-create --config cluster.yaml --location s3://backups/ursula/2026-07-25

# Verify: manifest completeness, byte sizes, BLAKE3 checksums, and deep
# state-machine validation of every snapshot. Touches no cluster.
ursulactl backup-verify --location s3://backups/ursula/2026-07-25

# Restore: into a FRESH, EMPTY cluster with the same raft group count and the
# source cluster's server.cluster_id.
ursulactl restore --config new-cluster.yaml --location s3://backups/ursula/2026-07-25

An s3:// location uses the AWS SDK credential chain and AWS_REGION. For MinIO or another S3-compatible store, also set AWS_ENDPOINT_URL, for example AWS_ENDPOINT_URL=http://minio.internal:9000.

The recovery contract:

• Consistency boundary. Each group export is the deterministic state-machine snapshot the raft snapshot path persists: internally consistent per group while writes continue. Cross-group consistency is not promised. The boundary is per stream. Acknowledged writes in the exporting replica's applied state are included whether or not they were cold-flushed, so RPO is bounded by export time, not by the cold-flush interval. backup-create asks every node and keeps the reply with the highest group commit index.
• ursulactl restore is the only supported restore. Do not restore a node or a cluster from volume-level copies, such as disk or EBS snapshots of the Raft WAL or the data directories. A cluster brought back from such copies keeps its history ids, so its streams keep their Stream-Incarnation tokens while their tails go back, and new writes then put other bytes at offsets the old cluster already served under the same tokens. Clients that kept state, and caches that stored immutable reads, take those bytes for the ones they hold. A single rolled-back volume also breaks Raft itself. A restore through ursulactl restore starts a new history, so every token changes.
• Keep the cluster id. Give the target cluster the source cluster's server.cluster_id (Helm server.clusterId) before its first start. Clients keep the id with their state and treat a different id as another deployment, so a restore under a new id looks to them like a move to another cluster, not a restore. A target node whose WAL was created with another id refuses to start, so set it before the nodes first run.
• Fresh identity on restore. Restore replays each snapshot as one replicated write on the target cluster, which keeps its own raft membership and log identity. Nothing from the source cluster's raft metadata is reused. Groups that already hold buckets refuse the import with 409, and group-count mismatches fail closed before the first import. Group commit indexes and per-stream usage restart under the new identity: every restored stream version starts at zero counters with no last write, and each bucket's committed counters in /__ursula/usage continue from the backup. Point a metering consumer at the restored cluster only after restore has finished (Per-stream usage). Stream bytes, offsets, close/TTL state, published snapshots, retention floors, and producer dedup states are preserved exactly.
• A restore is a new history. Each group's import starts a new history of the group, so every restored stream answers with a new Stream-Incarnation. A client that holds a stream's token from the source cluster gets 412 with the new token, and HEAD names the source token as Stream-Previous-Incarnation with the restored tail as Stream-Inherited-Offset (or, for a stream the source had not written since a rewrite of its own, the token from before that rewrite): what the client derived from bytes below it still holds. Every producer epoch claimed on the source is closed: a producer's next new write of it answers 403 with Producer-Outcome: epoch-closed;by=history and the last seq the backup kept, and the producer claims a new epoch. The import requires x-ursula-backup-blake3, the BLAKE3 digest of the group's backup object, which ursulactl restore sends. A repeat of an import that applied answers 200 with "already": true and starts no second history, so a restore interrupted by a timeout or a leader change can run again.
• Cold objects are part of the backup set. Snapshots reference cold-tier chunks, cold-index pages and external payloads by object key. The backup holds the references, not the objects. The restored cluster's cold store must hold the same objects under the same keys: point it at a copy (see Copying cold objects). backup-verify validates snapshots but does not dereference cold objects. restore asks the target cluster, before the first import, to check every group's references against its own cold store (POST /__ursula/backup/group/{group}/cold-check). If any object is missing, it stops with nothing imported and prints how many are missing and the first keys.
• Format compatibility. Backups have a format version of their own: format_version 5 in the manifest and in /__ursula/backup/info, and the first field of every group object, which is compact MessagePack. It changes only when what a backup holds changes (the manifest, the group snapshots or the cold objects they reference), not with every format epoch. Ursula 0.9, 0.10 and 0.11 write and read backup format 5, so 0.11 restores a 0.10 or a 0.9 backup. None of them restores backup format 4, which Ursula 0.8 wrote. A backup holds no membership and no placement. ursulactl refuses a manifest or a target cluster of another backup format before the first import, and a server refuses to import a group of another backup format (400).
• RTO is dominated by transferring and importing the group snapshots. Each import is a single replicated write per group.

The recovery drill runs in CI (backup_restore_drill_preserves_streams_and_allows_continued_appends): write, snapshot, retain and close, back up, destroy the cluster, restore, verify checksums, offsets and boundaries, and continue appending.

What you can additionally rely on:

• Quorum durability. Acknowledged writes survive as long as a majority of voters survives. Three voters across AZs tolerate any single-AZ outage.
• Cold-tier durability. Once flushed to S3, a chunk inherits S3-grade durability. The unflushed window is bounded by the flush interval (seconds by default).
• Format epoch 7, no migration. The Raft WAL, the meta journal, snapshots, the object-storage prefix and the Raft gRPC protocol carry format epoch 7. A node refuses data and peers of another format epoch and does not migrate them. Backups have a format of their own (see Format compatibility above).

Node-level loss: replace the host with the same node_id and cluster config, and once the old disk is revoked release it with ursulactl node replace-disk (see Rejoining a wiped node). Each group's leader catches the replacement up from the other voters, and it stays a voter (see Rejoining a wiped node). ursulactl wait --node N returns once the replacement has caught up in every group placed on it, and ursulactl verify-cluster then requires every group to run on exactly its target voters, each caught up. ursulactl wait-ready checks the whole cluster: every group live on its target voters with a leader, a ready maintenance report and 200 from every node's /__ursula/ready. Run it before declaring the recovery done. Total-cluster loss: restore the latest verified backup into a fresh cluster as above.

Copying cold objects

A restored cluster reads cold data under the keys the source cluster wrote, relative to storage.cold.root. Copy the source's objects into the target's root after the target has started and written its URSULA_FORMAT_EPOCH marker, and before restore. Skip two things:

• URSULA_FORMAT_EPOCH, the marker. The target's own marker stays.
• snapshots/, the source's Raft snapshot objects. The target writes its own there. With a custom storage.snapshot.s3_prefix inside the cold root, skip that prefix instead.

With the AWS CLI, from the source root to the target root:

aws s3 sync s3://my-bucket/source/cold/ s3://my-bucket/target/cold/ \
  --exclude 'URSULA_FORMAT_EPOCH' --exclude 'snapshots/*'

For MinIO or another S3-compatible store, add --endpoint-url. The copy must match the backup, so the source must not change its cold objects in between. Stopping writes is not enough: an idle cluster still compacts small cold chunks and deletes the ones it replaced a few minutes later. If the source cluster still runs, stop its nodes once the backup verifies, then copy. restore checks that the copy is complete.

Never point two clusters at one prefix, and never point a new cluster at the source's prefix in place. After a restore both clusters hold the same streams under the same keys, but each one deletes cold objects by its own state alone: the cold GC after a delete, a retention advance or a compaction, the orphan sweep for unreferenced objects older than a day, and each node's snapshot GC of its own directory under snapshots/. Two clusters on one prefix delete objects the other still reads. A copy also leaves the source cluster and its prefix as they were, so the source stays a way back until clients switch.

Upgrading to 0.11

0.11.0 is a breaking release. It moves the storage and protocol format to format epoch 7. A 0.11 node refuses epoch-6 Raft WAL directories and object-storage prefixes at startup and refuses 0.10 peers, and a 0.10 node refuses 0.11 peers. The backup format stays 5, so the data moves by backup and restore: ursulactl restore of 0.11 restores a 0.10 backup into a fresh 0.11 cluster. There is no in-place upgrade, no rolling upgrade and no rollback. If you cannot move your data yet, stay on 0.10.1.

1. Install 0.11 fresh with the 0.10 cluster's server.cluster_id and raft.group_count (Helm server.clusterId and raft.groupCount). Use empty PVCs and never-used prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.10, or a failed upgrade attempt, wrote to. Wait until ursulactl wait-ready passes.
2. Stop client writes and every SQLite session, then back up the 0.10 cluster with ursulactl backup-create and ursulactl backup-verify of 0.10 (Backup and disaster recovery). A running 0.10 session does not notice the move by itself, and a write the 0.10 cluster acknowledges after a group's export is missing from 0.11.
3. Stop the 0.10 nodes and copy their cold objects into the 0.11 cluster's cold root once the 0.11 cluster wrote its URSULA_FORMAT_EPOCH marker, skipping URSULA_FORMAT_EPOCH and snapshots/ (Copying cold objects).
4. Restore with ursulactl restore of 0.11. It checks every group's cold references against the 0.11 cold store before the first import. Every restored stream answers with a new Stream-Incarnation, as after any restore: a client's old token answers 412 with the new one, an SSE session ends, and every producer epoch claimed on 0.10 is closed.
5. Check usage continuity. With writes stopped, each bucket's committed counters in the last 0.10 /__ursula/usage report equal those of the first 0.11 report. Once ursulactl restore has finished, a metering consumer starts from cursor start in every group of /__ursula/usage/streams: every version lists with zero counters, and each bucket's row carries the backup's totals as retired (Per-stream usage). Do not start it earlier: a group read before its import gives a baseline without the backup's totals, and the import's relisting then accounts them again.
6. Upgrade the SQLite extension to 0.11 together with the switch.
7. Switch the clients to the 0.11 cluster.
8. Retire the 0.10 cluster once no client reads it. Delete its volumes and its object-storage prefixes.

helm upgrade from 0.10 stops at the first pod. The new pod finds the 0.10 data on its volume and exits at startup, so no second voter is replaced and the refused pod writes nothing. Recover with helm rollback. The 0.10 cluster and its prefix stay as they were, so they remain a way back until clients switch. Keep the peer lists of 0.10 and 0.11 nodes apart: a node that exchanges a Raft RPC with a node of the other release answers 503 with format_epoch_mismatch on /__ursula/ready until it restarts.

Upgrading to 0.10

0.10.0 is a breaking release. It moves the storage and protocol format to format epoch 6, for the meta group and placement. A 0.10 node refuses epoch-5 Raft WAL directories and object-storage prefixes at startup and refuses 0.9 peers, and a 0.9 node refuses 0.10 peers. The backup format stays 5, so the data moves by backup and restore: ursulactl restore of 0.10 restores a 0.9 backup into a fresh 0.10 cluster. There is no in-place upgrade, no rolling upgrade and no rollback. If you cannot move your data yet, stay on 0.9.

1. Update the configuration. Remove raft.groups, raft.init_membership and raft.init_membership_per_group, which 0.10 refuses as unknown keys: each group's voters come from the meta group's placement. Remove governance.leadership_balance.peer_timeout too, which 0.10 also refuses: every question a node asks its peers waits one minimum election timeout (1.5 s). Every node with raft.wal.path, a single node included, lists itself in [[raft.peers]] and sets server.cluster_id. A node whose raft.peers lists another node sets raft.group_count, at most 4,096. Every node of the cluster sets the same raft.peers, raft.group_count, raft.meta_voters and raft.placement.replication_factor. Decide raft.wal.fsync: unset, it is always for groups of one or two voters and never otherwise. With Helm, remove the chart values the 0.10.0 release notes list as removed, which the values schema refuses.
2. Install 0.10 fresh with the 0.9 cluster's server.cluster_id and raft.group_count (Helm server.clusterId and raft.groupCount). Use empty PVCs and never-used prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.9, or a failed upgrade attempt, wrote to. Keep the cluster id: a restore keeps every stream, and clients that compare cluster ids then see the same deployment. Wait until ursulactl wait-ready passes.
3. Back up the 0.9 cluster. Stop client writes to it, or accept that writes it acknowledges after a group's export are missing from 0.10. Run ursulactl backup-create and ursulactl backup-verify of 0.9 (Backup and disaster recovery).
4. Stop the 0.9 nodes and copy their cold objects into the 0.10 cluster's cold root once the 0.10 cluster wrote its URSULA_FORMAT_EPOCH marker, skipping URSULA_FORMAT_EPOCH and snapshots/ (Copying cold objects).
5. Restore with ursulactl restore of 0.10. It checks every group's cold references against the 0.10 cold store before the first import. Every restored stream answers with a new Stream-Incarnation, as after any restore: a client's old token answers 412 with the new one, an SSE session ends, and every producer epoch claimed on 0.9 is closed.
6. Switch the clients to the 0.10 cluster. With more nodes than raft.placement.replication_factor, a node that hosts no replica of a stream's group answers 307 to one that does. Clients that reach the nodes without the gateway follow it.
7. Update alerts, runbooks and tooling. Every removed command, route, header, config key, chart value and script is listed with its replacement in the 0.10.0 release notes. The changes operators meet first:
   • The membership route is gone. A group's voters change only by a move: ursulactl placement move, cancel and set-move-limit. A node leaves by ursulactl node decommission, which cannot be undone for its disk, and its ID is used again by an empty disk once its removal completed. A meta voter is never decommissioned, only replaced under its ID (Placement and the meta group).
   • A node on a wiped or replaced disk takes part in nothing until ursulactl node replace-disk releases its old root (Rejoining a wiped node). 0.9 let it rejoin with no operator step.
   • ursulactl recover-group takes --target G or --target meta. Recover the meta group first, then each data group (Recovering after a host crash). A data group's plan carries the floor a confirmed meta view gives it.
   • --expected-groups is gone from wait-ready, verify-quorum and verify-survivors: every verb reads the groups and their target voters from the meta leader's placement view. ursulactl of 0.9 cannot read a 0.10 node.
   • raft_maintenance is schema version 2, with new and renamed issues, readiness has the new reasons superseded, disk_lease_lost, root_refused and predecessor_wait, and /__ursula/usage is version 2 with one entry per group. Alerts keyed on issues or reasons change (Troubleshooting).
   • Readiness reads only the node's own state. A node whose group knows no leader, as during a meta outage or after a group lost its quorum on other nodes, stays ready and reports leader_unknown, which blocks maintenance only. A restarted node stays unready with leader_not_acknowledged until each of its replicas caught up with its group's leader. Alert on ursulactl wait-ready, not on pod readiness, for lost redundancy.
   • A single node runs every gate of a cluster node, follows storage.snapshot.backend and refuses writes past twice raft.snapshot_log_budget.
   • A lost quorum of disks is lost data, for the meta group as for a data group. Restore a backup into a fresh cluster then.
8. Retire the 0.9 cluster once no client reads it. Delete its volumes and its object-storage prefixes, and with the chart the <StatefulSet>-rollout-state and <StatefulSet>-maintenance ConfigMaps it left.

helm upgrade from 0.9 stops at the first pod. A values file that still sets a removed chart value fails validation, and nothing changes. Otherwise the new pod finds the 0.9 data on its volume and exits at startup, so no second voter is replaced and the refused pod writes nothing. Recover with helm rollback. If the 0.9 release ran OnDelete, also delete the refused pod after the rollback so that it starts from the 0.9 template again. The 0.9 cluster and its prefix stay as they were, so they remain a way back until clients switch. Keep the peer lists of 0.9 and 0.10 nodes apart.

0.10.1 upgrades 0.10.0 in place with an ordinary rolling restart (helm upgrade), with no fresh install. See the 0.10.1 release notes.

Upgrading to 0.9

0.9.0 is a breaking release. It changes the storage and protocol format to format epoch 5 and the backup format to 5. A 0.9 node refuses epoch-4 Raft WAL directories and object-storage prefixes at startup and refuses 0.8 peers, and a 0.8 node refuses 0.9 peers. 0.9 does not restore 0.8 backups. There is no in-place upgrade, no rolling upgrade, no backup migration and no rollback. If you cannot move your data yet, stay on 0.8.2.

1. Install 0.9 fresh. Use empty PVCs and never-used prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.8, or a failed upgrade attempt, wrote to. Give the 0.9 cluster its own server.cluster_id (Helm server.clusterId). Its streams are new streams, so a client that compares cluster ids then knows that its 0.8 state does not carry over.
2. Move your data through your applications. Read the streams you keep from the 0.8 cluster with your own clients and write them to the 0.9 cluster through the HTTP API. A stream written again is a new stream: a new Stream-Incarnation, offsets from its first written byte, and only the producer states, TTL and published snapshots that the application sets again.
3. Switch the clients to the 0.9 cluster. A client that keeps state, such as offsets or producer sequences, starts it again from the streams of the 0.9 cluster.
4. Drop the removed snapshot settings from any config and Helm values you carry over from 0.8. Remove raft.snapshot_logs_since_last (Helm raft.snapshotLogsSinceLast) and any storage.snapshot.drive_interval = "0s" (Helm snapshotStore.driveIntervalMs: 0). 0.9 refuses both at startup, and the chart's schema refuses them at render. The snapshot driver always runs. To keep the 5,000-entry bound a 0.8 single node had, set raft.snapshot_backstop_logs = 5000 (Helm raft.snapshotBackstopLogs).
5. Check the snapshot store's request budget. S3 snapshot requests no longer follow storage.cold.s3.timeout and max_retries. They use storage.snapshot.s3_timeout (10s) and storage.snapshot.s3_max_retries (1), Helm snapshotStore.s3TimeoutMs and snapshotStore.s3MaxRetries. A follower's install copy, up to four of these requests, must fit inside raft.install_snapshot_timeout. A carried-over cluster config with S3 snapshots and an install timeout below that bound, 88 seconds with the defaults, is refused at startup with SnapshotCopyOutlastsInstall. Raise raft.install_snapshot_timeout or lower the snapshot store's timeout or retries. The timeout covers one attempt, and a PUT sends the whole object in one attempt, so keep it above a healthy PUT of your largest snapshot.
6. Optionally scope credentials per node. With an S3 snapshot store, a node writes and deletes only under <prefix>/node-<id>/ and reads the rest of <prefix>/, so an IAM policy per node can enforce that.
7. Retire the 0.8 cluster once no client reads it. Then delete its volumes and its object-storage prefixes, among them its snapshot objects under <prefix>/group-<gid>/, where <prefix> is the 0.8 cluster's storage.snapshot.s3_prefix. 0.9 never reads or deletes them.

helm upgrade from 0.8 stops at the first pod. The new pod finds the 0.8 data on its volume and exits at startup, so no second voter is replaced and the refused pod writes nothing. Recover with helm rollback. With OnDelete, also delete the refused pod after the rollback so that it starts from the 0.8 template again. The 0.8 cluster and its prefix stay as they were, so they remain a way back until clients switch.

Upgrading to 0.8

0.8.0 is a breaking release. It changes the storage and protocol format to format epoch 4. A 0.8 node refuses epoch-3 Raft WAL directories and object-storage prefixes at startup, refuses 0.7 peers, snapshots and backups, and ursulactl restore refuses 0.7 backups. There is no migration, no in-place upgrade, no rolling upgrade and no rollback. If you cannot move your data yet, stay on 0.7.6.

1. Install fresh. Use empty PVCs and never-used prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.7, or a failed upgrade attempt, wrote to.
2. Set the cluster id. Set server.cluster_id on every node of a multi-node cluster, the same on every node and different for every deployment a client can reach. The chart requires server.clusterId for every install, a single replica included.
3. Choose the network planes before the first install. The chart serves Raft on server.ports.cluster (4439) by default. Do not change server.clusterListen or server.ports.cluster on an initialized cluster. Outside the chart, set server.cluster_listen and client_url together on every node.
4. Check the inflight body budget. A node with a cold backend needs server.http_inflight_body_size of at least 64 MiB, or it refuses to start with InflightBodyBudgetTooSmall.
5. Lower the gateway's request limit. A gateway whose --max-request-body-bytes (gateway.maxRequestBodyBytes) is above 33554432 refuses to start. Larger snapshots go up in parts.
6. Move data by export and re-import. The applications that own the data read it from 0.7 and write it into 0.8. For the SQLite VFS, export each database with 0.7 using VACUUM INTO and import the file with 0.8 (Moving from 0.7).
7. Update clients and proxies. Error answers have a JSON body, x-ursula-not-applied is gone, a leadership loss after a proposal answers 503 with OUTCOME_UNKNOWN, and Stream-Incarnation changes on a restore or an accepted loss. A proxy in front of Ursula follows the intermediary rules of write outcomes. Also check clients and tooling for these changes:
   • A snapshot publish above 32 MiB sent straight to a node answers 413. Send it in parts.
   • A failed cold-tier read answers 502 COLD_STORE_FAILED, where 0.7 answered 500, and a write that needs a cold backend the node lacks answers 501 NO_COLD_STORE, where 0.7 answered 503.
   • Producer-Seq on 200 and 204 is the epoch's highest applied seq, not the request's.
   • A create with producer headers answers 400. Create the stream, then claim an epoch.
   • A producer close of a closed stream answers 409. Only a close without producer headers is idempotent there.
   • A retention request at or below the retained offset answers 204 and changes nothing, where 0.7 answered 409.
   • An SSE session ends on a delete and recreate, since it is pinned to the token its first response announced.
   • Read and snapshot ETags have a new form.
   • /__ursula/metrics no longer reports receipts, receipt_items or max_receipt_items_per_stream.
   • The gateway no longer uses a system proxy from HTTP_PROXY for its upstreams.
   • The chart's Helm test pod's name label is now <chart>-test.
   • ursulactl reads metrics at a manifest node's client_url, never at http_url. Add client_url to custom manifests.
8. Update recovery runbooks. accept-unsynced-loss takes a recovery plan, and ursulactl recover-group --accept-loss computes it. A replica whose committed entries a plan dropped waits for its node to be rejoined as a wiped node. See Recovering after a host crash for the checks before a wipe.

Every behavior change, with what to change for each area, is listed in the 0.8.0 release notes.

0.8.1 upgrades 0.8.0 in place with an ordinary rolling restart (helm upgrade), with no fresh install. See the 0.8.1 release notes.

0.8.2 upgrades 0.8.0 and 0.8.1 in place the same way, once your Helm values no longer set the indexer it removes. See the 0.8.2 release notes.

Upgrading to 0.7

0.7.0 is a breaking release. It changes the Raft WAL format (format epoch 3) and removes the memory WAL, so the per-core disk journal is the only Raft WAL. 0.7 cannot read 0.6 data: a node refuses 0.6 Raft WAL directories and object-storage prefixes at startup, ursulactl restore refuses 0.6 backups, and a node will not join a cluster with 0.6 nodes. There is no in-place upgrade, no automatic migration and no rollback.

• If you run 0.6 with data you need to keep, stay on 0.6.2. 0.7 provides no migration path from 0.6.
• Deploy 0.7 fresh: empty PVCs and a never-used object-storage prefix.
• helm upgrade from 0.6 stops before a second voter is replaced. Recover with helm rollback.

Every behavior change is listed in the 0.7.0 release notes.

0.7.6 upgrades 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4 or 0.7.5 in place with an ordinary rolling restart (helm upgrade), with no fresh install. See the 0.7.6 release notes.

To install 0.7:

1. Install fresh. Install with a new s3.prefix and empty PVCs. With cold storage off and S3 snapshots, also use a new snapshotStore.prefix.
2. Never reuse a prefix that an earlier release, or a failed upgrade attempt, wrote to.
3. Update the configuration.
   • Remove raft.wal.backend, raft.wal.allow_volatile_multi_peer, raft.memory_bootstrap_marker_dir and raft.rejoin_probe. A config file that still sets one fails to load.
   • Set raft.wal.path on every node of a multi-node cluster. 0.6 defaulted to the memory WAL, and a 0.7 node with raft.peers refuses to start without a WAL path.
   • Choose raft.wal.fsync (Helm raft.walFsync). The default never acknowledges a write once a quorum of voters holds it in the page cache, and a host crash can cost a replica its unsynced tail (see Recovering after a host crash). Set always to keep the 0.6 disk WAL behavior of one fsync per batch.
   • With Helm, remove raft.storageMode, raft.allowVolatileMultiPeer and persistence.enabled. Every pod gets a PVC for its WAL. The values schema refuses unknown keys under raft and persistence.
   • Keep runtime.core_count and raft.group_count explicit and unchanged (see WAL routing configuration).
4. Update runbooks and tooling. ursulactl 0.7 removes the memory-WAL restart verbs prepare-restart, restart-quiesce-capability, finish-prepared-restart, abort-prepared-restart, prepare-amnesiac-restart, repair-restarted-voter, prepare-recovery-handoff and classify-amnesiac. A planned restart is drain, restart, wait and undrain (Restarting a voter), and a voter that lost entries is rebuilt by its group leaders. Custom tooling that calls admin mutations must send x-ursula-process-incarnation (Admin endpoints).

A values file that still sets a removed chart value fails validation, and nothing changes. Otherwise helm upgrade from 0.6 replaces the first pod, with the default RollingUpdate strategy and also with OnDelete plus the graceful rollout. That pod finds the 0.6 data on its volume and exits at startup, so no second voter is replaced. Recover with helm rollback. With OnDelete, also delete the refused pod after the rollback so that it starts from the 0.6 template again. The refused pod wrote nothing, so the old pods are unaffected.

How a 0.7 node refuses data of another release, and what it prints, is listed under Startup failures. Every startup refusal happens before the node writes anything:

• Raft WAL directory. {raft.wal.path}/raft-log/FORMAT_EPOCH records the epoch. A directory that holds data without the marker, or a marker of another epoch, is refused. The marker is written after every check passes.
• Object storage. URSULA_FORMAT_EPOCH sits at storage.cold.root when the cold backend is S3, otherwise at the S3 snapshot namespace (storage.snapshot.s3_prefix) when snapshots live on S3. A namespace that holds objects without the marker is refused, so it must be used only by this cluster. With a memory cold store and inline snapshots there is no object-storage check.
• Peers. Before it writes the markers, a node with configured peers asks each one, in parallel (2 seconds to connect plus 2 seconds for the answer, about 4 seconds at most), whether it speaks Raft protocol 3. A peer that refuses ends startup. An unreachable peer is skipped, because a fresh cluster starts its pods in parallel.
• Readiness. Any later Raft protocol mismatch, inbound or outbound, is logged at error level and makes /__ursula/ready answer 503 with "reason": "format_epoch_mismatch" until the node restarts. This also holds for a 0.6 node that a 0.7 node contacts, including through the startup probe. Keep the peer lists of 0.6 and 0.7 nodes apart.
• Backups. Backups use format version 3. ursulactl restore and backup-create refuse a cluster that reports another version, and a server refuses to import a group of another format epoch (400).

Bucket IDs snapshots and any ID starting with __ are reserved (400): snapshots is the default raft-snapshot namespace in the object store.

Logs

RUST_LOG=ursula=info,ursula_runtime=info,ursula_raft=info is the baseline. Bump to debug for one crate when chasing a subsystem:

RUST_LOG=ursula_raft=debug ./target/release/ursula server ...

debug is verbose under sustained load, so redirect to a file.

Committed application failures

A stopped group requires corrected-code replay. Retain the original WAL and referenced snapshot objects. Never skip a committed record to recover readiness.