Operations

The first tool to reach for is ursulactl. It covers the raft-aware verbs operators run most often: drain, status, readiness gates. This page covers everything around it: the metrics shape ursulactl reads, the admin endpoints it (and your custom tooling) can call, and the operational policies ursulactl does not encode (backups, log levels). Deployment and process lifecycle belong to the platform, such as Helm.

Tooling map

Surface	When to reach for it
`ursulactl`	Day-2 cluster ops over HTTP: drain, catch-up waits, status, wait-ready.
`/__ursula/metrics` and the `/__ursula/raft/...` admin endpoints	Custom tooling. ursulactl uses these underneath. The surface is small and stable enough to script directly.

There is no Prometheus scrape and no general-purpose orchestrator yet.

Metrics

curl http://127.0.0.1:4437/__ursula/metrics | jq .

The JSON snapshot covers per-core mailbox depth, append and apply counters, cumulative timings (*_ns), each group's leader, term and applied index (raft_groups), hot bytes waiting for cold flush, cold-flush uploads and errors, WAL physical/reclaim/recovery work, WAL disk-pressure admission, SSE counters, live-read waiters, and cold-write backpressure. Start here when triaging slowness, lag, or 503s. Troubleshooting lists jq selectors for the common fields. GET /__ursula/ready provides the local serving signal with a typed reason on refusal, on the client and admin planes alike. Healthy surviving voters can serve during a rebuild while the learner remains unready. Maintenance also requires a complete raft_maintenance report and certain maintenance authority from metrics. ursulactl wait-ready checks both the metrics and this existing readiness endpoint. ursulactl status presents leadership fields across the cluster.

Peers of another cluster

Every Raft gRPC request a node sends carries the id its WAL holds (server.cluster_id) and its raft.node_id as the ursula-cluster-id and ursula-node-id request metadata. That covers replication, votes, snapshots, leadership transfers, recovery probes, forwarded reads and HEADs, and forwarded bucket purges. A node checks the metadata before it decodes the request and refuses a peer with another id with FAILED_PRECONDITION, before any Raft RPC runs. The refusal names the refusing node's id, so the sender learns it as well. A mistyped server.cluster_id or a peer URL that points at another deployment therefore fails loudly instead of exchanging Raft traffic between two clusters, and a misconfigured node of another cluster cannot move this cluster's Raft terms. The cluster id is public, since every HEAD answer carries it as Ursula-Cluster-Id, so the check guards against misconfiguration, not against a hostile sender: keep the cluster plane on a private network.

What each side does with a refusal:

Counts it. /__ursula/metrics counts every refusal, both ways, in raft_grpc_peer_cluster_refusals.
Logs it once. The node logs each refused peer once at error level, with its node id, cluster id and address, while it keeps being refused, not once per RPC.
Reports configured peers. /__ursula/ready and /__ursula/metrics list, under peer_cluster_mismatches, the peers of raft.peers refused within the last minute, with their node_id and the cluster_id they sent or answered with. A peer this node dialed always counts. A request this node refused counts when its sender claims a configured node id and, where that peer's URL names an IP address, comes from that address. Senders outside raft.peers are counted and logged, never listed.
Keeps its readiness. A refusal never changes readiness or its reason. One misconfigured node does not take its healthy peers out of service, and an unready node still names its own cause, such as recovery_stalled.
Backs off. OpenRaft treats a refusing peer as unreachable. The node keeps its connection to it, since the answer is final rather than a broken channel.
Leaves it out of handoffs. GET /__ursula/leadership-shed answers carry the node's cluster_id, and a node counts a peer's answer only when the id is its own. A node of another cluster at a peer's address therefore never becomes a handoff target, even before any Raft traffic reached it.
Forwards answer 503. A read, HEAD or bucket purge a follower forwards to a leader of another cluster answers the client 503 with Retry-After, as for a leader it cannot reach. Nothing ran on the refusing node.

A request that names no cluster id comes from a node before format epoch 4 or a stray client. It is refused with the Raft protocol-mismatch text, which such a node reads as a format-epoch mismatch. The node counts it in raft_grpc_unidentified_peer_refusals and logs each sender address once. It does not make the node unready, so a misrouted older node cannot take a healthy one out of service. A node that itself contacts a configured peer of another format epoch still becomes unready with format_epoch_mismatch.

At startup, a node checks that its WAL holds its configured server.cluster_id before it contacts any peer, and its format-epoch probe reports a configured peer of another cluster at error level and starts anyway. A refusal lasts until an operator fixes the configuration: correct server.cluster_id or the peer URL on the wrong node. A node whose WAL already holds another id refuses to start, so give a replaced node an empty WAL (see Rejoining a wiped node).

cluster_id is in both /__ursula/ready and /__ursula/metrics, so tooling can check that every node of a deployment reports the same id:

for node in NODE1 NODE2 NODE3; do
  curl -s http://$node:4437/__ursula/ready | jq -c '{cluster_id, peer_cluster_mismatches}'
done

Admin endpoints

These are the primitives ursulactl and any custom operator tooling builds on. They live on the admin plane (server.admin_listen, loopback 127.0.0.1:4438 by default), not the public client plane. Nodes expose no cluster-mutation surface on the network. Reach them over a tunnel to the admin port (see ursulactl's operation providers). The examples below assume a forward from local 4438 to the node's admin plane. Each call is local to one node. To act on every group, loop over the IDs in metrics.

Every admin mutation requires x-ursula-process-incarnation, pinned from GET /__ursula/metrics before the operation. Missing identity returns 428; a restarted target returns 412 before the handler changes state. Keep the same identity for the entire plan and stop on either response. This precondition is public instance identity, not authentication or an exclusive maintenance reservation. Updated ursulactl attaches it automatically and refuses to refresh it during an operation.

# Observe once for this operation; do not refresh after a 412.
process_incarnation=$(curl --fail http://127.0.0.1:4438/__ursula/metrics | jq -er .process_incarnation)

# Force a cold flush for one stream (skip the timer)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/flush-cold/demo/hello

# Trigger a Raft snapshot for one group
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/42/snapshot

# Purge log entries through index 1000 for one group, at most through its
# last snapshot. Answers 503 with Retry-After while a snapshot of the group
# builds or a purge of it has not landed. Send it again then.
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/42/purge?upto=1000"

# Add a learner (non-voting replica) to one group without blocking on catch-up.
# addr is the address peers dial: the learner's cluster listener (its
# [[raft.peers]] url), here on the chart's cluster port 4439.
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/42/learners/4?addr=http://node-4:4439&blocking=false"

# Hand leadership of one group to another voter (used by `ursulactl drain`)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/42/leader/transfer/2

# Replace the voter set of one group (call on the group's current leader)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/42/membership?voters=1,2,3"

# Put this node into maintenance drain / lift it again
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/leadership-shed/maintenance
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X DELETE http://127.0.0.1:4438/__ursula/leadership-shed/maintenance

# Send one group's recovery plan to this node, a member of a plan whose leader answered plan term 9,
# naming the log its metrics show (see Recovering after a host crash)
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d '{"leader": 2, "point": 1234, "cut": false, "step": {"follow": {"leader_term": 9}}, "expected": {"last_log_index": 1230, "last_log_term": 7, "current_term": 8}}' \
  -X POST http://127.0.0.1:4438/__ursula/raft/42/recovery/accept-unsynced-loss

# The same for a plan that cuts to an entry another replica reported committed, at index 1240
# of term 7 led by node 3: every request of the plan names that entry
curl --fail-with-body -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d '{"leader": 2, "point": 1240, "cut": true, "point_entry": {"term": 7, "node_id": 3}, "step": {"follow": {"leader_term": 9}}, "expected": {"last_log_index": 1241, "last_log_term": 7, "current_term": 8}}' \
  -X POST http://127.0.0.1:4438/__ursula/raft/42/recovery/accept-unsynced-loss

The snapshot endpoint answers once the group has a snapshot of everything it had applied when the request arrived. It waits while a build of the group runs or other snapshot builds hold all of the node's build permits, and answers 503 with Retry-After: 1 when no build of the group can start within 10 seconds (see Manually trigger a snapshot).

The leader-transfer endpoint refuses with 409 Conflict if the receiving node isn't the current leader and 400 Bad Request if the target isn't a voter. This is why ursulactl is the safer way to chain these calls: it consults metrics first.

The group leader keeps every group at its configured voters: if the membership endpoint drops a configured voter, the leader adds it back as a learner and promotes it again once it has caught up (see Recovering after a host crash). Membership changes are for repair, not for shrinking a group.

The membership endpoint has the same leader-only contract: it answers 409 Conflict (with the current leader's id in the body) when the receiving node does not lead the group, and 400 Bad Request without a non-empty voters query parameter. On success it returns the log index at which the new voter set was committed. It replaces the whole voter set in one call, so include every node that should remain a voter.

Maintenance drain

POST /__ursula/leadership-shed/maintenance marks the node as draining: it stops campaigning and accepting transfers, and the background leadership balancer migrates its current leaderships to eligible peers one transfer per tick until the mark is lifted with DELETE. Both verbs return the node's leadership-shed state so tooling can confirm the transition took effect:

{
  "bits": 1,
  "state": "maintenance-drain",
  "should_accept_transfer": false,
  "should_campaign": false,
  "should_shed_current_leaders": true
}

ursulactl drain wraps exactly this pair: mark first so the node stops re-acquiring groups, then transfer leaderships away explicitly instead of waiting for the balancer. The mark stays set until ursulactl undrain clears it. If a rollout ever dies mid-way, a node can be left refusing leadership. Run ursulactl undrain (or DELETE the mark directly) to recover. Peers read each other's shed state (GET /__ursula/leadership-shed on the cluster-plane listener) when deciding which nodes are eligible to pick up leaderships, so a drained node stops attracting groups cluster-wide, not just locally.

Graceful voter shutdown

On the first SIGTERM or Ctrl-C, the voter marks itself maintenance-drained to prevent re-acquiring leadership. It keeps Raft transport available while attempting to hand off each leadership to an eligible voter (see the handoff rule under Recovering after a host crash). It waits for observed leadership changes for up to five seconds, then closes Raft transport and drains HTTP listeners for up to ten seconds. A connection still open after that, such as one from a peer that lost power, is abandoned. It then stops its Raft groups, fsyncs every journal and records a clean shutdown, so the next start reads the journals strictly and the node holds every entry it acknowledged. The whole shutdown has a deadline of twenty seconds from the first signal. If it has not finished by then, the process exits without recording a clean shutdown, and the next start treats the run as a crash. A second signal also exits at once without recording a clean shutdown. The chart gives voter pods thirty seconds by default. This reduces planned-stop election pauses. It cannot hand off leadership on SIGKILL or OOM.

runtime.node_memory_abort_cap_size is an emergency process-abort threshold, not HTTP admission control. Exceeding it calls abort() without graceful leadership handoff, like an OOM kill. It does not return 503. Leave room below the container memory limit for normal operation and recovery, and use write admission and backpressure to manage load.

Restarting a voter

Restart one voter at a time:

ursulactl drain --node N moves every leadership away and keeps the node from taking one back.
Restart the process through the platform (Helm, systemd). The graceful shutdown above keeps every journal intact.
ursulactl wait --node N returns once the node is a voter of every group again and has caught up.
ursulactl undrain --node N lets it hold leaderships again.

Do not stop another voter before wait returns. Two overlapping outages of a three-voter group leave it without a quorum even if the restarts were issued separately. The chart's graceful-rollout hook runs the same sequence for every pod.

WAL routing configuration

A WAL root is bound to runtime.core_count and raft.group_count in <raft.wal.path>/raft-log/topology.bin. Startup holds the WAL lock and checks both values before recovering any group or writing the run state. A mismatch, missing record on an existing WAL, or corrupt record refuses startup.

Keep both values explicit and unchanged when moving a persistent volume or resizing its host. Without explicit values, core_count follows the process's available parallelism and group_count defaults to that parallelism times 16 (resource presets set a fixed group count). Helm defaults to 4 and 64. Changing cores changes the journal directory of some groups. Changing group count changes stream routing. Neither is an online resharding operation.

For a mismatch, configure the stored counts again. The error reports them. Do not manufacture a record from the new configuration or remove the check's files. A replica whose record is missing or corrupt can be rebuilt on an empty WAL directory from a healthy quorum, with the cluster's routing counts. Follow the rejoin procedure below and wait for full catch-up before replacing another replica. A single node has no quorum to rebuild from, so a fresh WAL does not preserve its data.

Rejoining a wiped node

A voter whose WAL directory was wiped or replaced, for example after a lost disk, starts with an empty log for every group. For each group:

It does not initialize the group again. A per-group membership initializer (raft.init_membership_per_group = true, the chart default) runs Initialize only when every other configured voter answers that it holds nothing for the group either and no leader is seen. A fresh install still bootstraps this way. One voter holding entries or following a leader means the group exists, and the node waits to be replicated to. It waits for every voter, not a quorum, so a fresh install with a voter that never comes up stays uninitialized (it logs "waits for every voter before initializing").
It refuses replication until the leader proves a quorum without it. The node lost its vote with its disk, so it cannot tell the current leader from a stale one. It refuses appends, heartbeats, snapshots and votes until the group's leader confirms a fresh ReadIndex barrier with a quorum of the other voters. The node then records that leader's vote as its vote floor and accepts replication. In a 3-voter group that quorum needs both other voters up. While the wiped voter waits and another voter is down, the group has no quorum and cannot commit writes.
It stays behind the recovery gate. It does not campaign and refuses every vote once it knows the group holds entries, until it has applied a fresh barrier from the group's leader (see Recovering after a host crash).
The leader rebuilds it. The leader sees the node answer that it lacks entries it had acknowledged: it commits a voter set without the node, adds it back as a learner, waits until it has replicated the commit index and promotes it. Each step is read off the current membership, so a leader change or another restart of the node converge. Leadership handoffs (the balancer, the commit-stall watchdog) skip the node until then.

While the other voters are up, no operator step is needed. ursulactl wait --node N returns once every group has promoted the node, and ursulactl verify-cluster shows it unready until then. Do not stop another voter before that.

If a majority of a group's voters lost their disks at once, the voters that kept their disks are not a quorum. No leader can prove a quorum to the wiped replicas, so they never obtain a vote floor, and accept-unsynced-loss cannot replace lost voting history. The group stays stopped. Restore the latest verified backup into a fresh cluster (see Backup and disaster recovery). If every voter lost its disk, the bootstrap probe sees only empty voters and initializes the groups again, empty: every acknowledged write is lost, and cold objects written before stay in object storage, unreferenced. Start such a cluster over on a new object-storage prefix, as for any fresh install.

Recovering after a host crash

With the default raft.wal.fsync = "never", a node acknowledges ordinary data appends from the page cache. Membership batches are always fsynced before acknowledgment, so a full-cluster power loss cannot erase the voter configuration. A process crash loses nothing. A host crash or power loss, or a journal I/O failure, can cost the node the unsynced tail of its journals. The node detects it on the next start from its run state (wal_recovery in GET /__ursula/metrics, the ursula.wal.recovering gauge) and keeps the frames that still verify. A node whose run state file is missing while its journals hold records is treated the same way. Every group the node held enters the recovery gate, which is recorded in the node's metadata and survives further restarts:

The node does not campaign, takes no leadership transfer and grants no vote. A node that led the group starts as a follower. It still accepts appends from a leader of its current term or a later one.
It asks the group's leader for a fresh ReadIndex barrier, applies the log up to the barrier's index and then opens the gate. Until then readiness answers 503 with reason recovery_gate_closed, and recovery_gates in GET /__ursula/metrics lists the gated groups.
The leader sees that the node lost entries it had acknowledged and rebuilds it: it removes the voter, adds it back as a learner and promotes it once it caught up. The removal needs a quorum of voters that kept their log and acknowledged the leader within the last 1.5 seconds (the minimum election timeout), counting the leader. Without such a quorum no removal can commit, so the leader resends its own log to the voters that lost entries instead.

Leadership handoffs (draining, balancing, a node that sheds its leaderships, the commit-stall watchdog, the admin transfer endpoint) go only to a voter that kept its log, acknowledged the leader within the last 1.5 seconds and holds every committed entry. A handoff whose target has not taken over within 6 seconds (two maximum election timeouts) is abandoned: the leader steps down and campaigns again.

While at most a minority of a group's voters lost their tail, the other voters hold every acknowledged write and nothing is lost. Wait for every group to open the gate on a node before you stop another voter.

A majority of a group's voters lost their tail at once. If the group's leader kept its log, it stays leader and its barrier opens the gated replicas as above. Otherwise the gated voters refuse every vote, so the group elects no leader and writes answer 503. After 30s without a leader barrier and without applying anything, each gated replica reports its group stalled: readiness reason recovery_stalled, the group under recovery_stalled_groups in GET /__ursula/ready and under recovery_gates.stalled in the metrics, the ursula.raft.recovery_gates{status="stalled"} gauge, and an error log naming the group. Writes acknowledged after the last fsync of the surviving replicas may be gone. Only an operator can accept that, with a recovery plan. The plan elects one replica as the group's leader, and that leader starts a new history of the group: every stream of the group gets a new Stream-Incarnation, and every producer epoch claimed before is closed (see history rewrites). For each stalled group:

ursulactl recover-group --config cluster.toml --group 42 explains the plan it would send for one group and sends nothing: the floor, why it cuts or not, each replica's log and voter sets the plan expects, and the voters that did not answer. With --accept-loss it runs the steps below: it reads every voter, computes the plan, sends it to the plan's leader first and then to every other replica that answered, and waits until the plan is satisfied. It stops with an error when a replica leads the group with a live quorum and no plan was sent. While a replica's gate may still open without a plan, it waits, and once it sent a plan, it waits while a replica leads with a live quorum or follows a live leader it did not reach. It plans again when a plan wedges in two observations in a row (see below), when the plan's leader refuses it, or when replicas refuse it so that the plan's leader and the replicas that took it are no majority, up to --max-plans plans. It never plans again while a replica leads at a term above the last plan's term: that leader was elected under the plan. When the plan's leader refuses for a reason no plan fixes, missing_vote_floor or step_mismatch, it stops with an error. A member that refuses so is left out of the next plan, but the voter sets it reports still count, and a plan that a majority took stands. A leader_not_voter refusal changed nothing: the membership moved since the tool read it, so it reads the replicas again and plans again. Once the plan is satisfied, it names the replicas it reached outside the leader's membership, which keep the replaced history. When the plan's leader does not answer in time, it reads the plan the leader recorded and sends the other replicas that plan term. --accept-timeout-secs bounds one call, which a cut or a demotion restart makes slow (ursulactl). By hand:

On every node that holds a replica of the group, learners included, read GET /__ursula/metrics, and under raft_groups the group's recovery_gate, last_log_term, last_log_index, current_term, committed_index, recovery, leads_live, voter_configs, committed_voter_configs and maintenance.running. voter_configs are the voter sets of the replica's membership, two while a change is joint. committed_voter_configs are the voter sets of the membership it keeps when a plan cuts its log back to its committed entry. Plan without a replica whose maintenance.running is false: its group stopped, and it reports no membership. Keep its last_applied_index for step 2. Read also committed_term and committed_node, which with committed_index name a replica's committed entry, and last_applied_index. Go on only when, in every voter set any replica reports in either field, a majority of its voters answer, only while no replica's gate is awaiting_barrier, catching_up or cutting, and only while no replica reports leads_live: such a replica may still open through a leader's barrier without losing anything, is running a plan's cut, or leads the group. A replica whose own metrics name it leader without leads_live lost its quorum: the plan demotes it.
Compute the plan from the replicas that answered:
- floor: the highest current_term among the replicas whose gate is not open or whose recovery is not null.
- The leader: among the replicas some voter set names, the one with the greatest last log id, highest last_log_term first, then highest last_log_index. An empty log (null) ranks last. Its last_log_index is the plan's point, and cut is false.
- When that replica's voter_configs and committed_voter_configs name more than one voter and its last_log_term is below floor, its log may hold an entry that a lost leader overwrote. When its own voter_configs do not name it, it cannot campaign. In both cases cut is true, the leader is the replica with the greatest committed_index among those whose own committed_voter_configs name them, and its committed_index is the point. Every replica then keeps only the entries it knew were committed.
- When cut is true and a replica, a learner say, reports a committed_index above the point, the voters may still hold that entry in their logs although their markers lie below it. Then the leader is instead the replica with the greatest committed_index among those whose own committed_voter_configs and voter_configs both name them and whose last_log_index reaches that index. The point is that index, and point_entry names the entry: {"term": committed_term, "node_id": committed_node} of the replica that reported it. The leader refuses the plan unless its log holds exactly that entry at the point.
- Stop when any replica that answered, learners and replicas whose group stopped included, reports a committed_index or a last_applied_index above the point: the plan's leader would overwrite that replica's committed entries.
- highest_term: the highest current_term among all the replicas that answered.
Send the plan to the leader first: {"leader", "point", "cut", "step": {"lead": {"highest_term": H}}, "expected": {"last_log_index", "last_log_term", "current_term"}}, where H is highest_term and expected is what you read on the leader. A cut to a reported entry adds "point_entry": {"term", "node_id"}, the same in every request of the plan. Leave it out otherwise. The leader answers outcome leads and leader_term, the plan term.
Then send it to every other replica that answered, with "step": {"follow": {"leader_term": T}} where T is the leader_term the leader answered, and expected as you read it on that replica. Each answers follows. Take T from the leader's answer, never from the metrics. A replica refuses a T below the current_term in its expected with plan_term_below_vote, but it cannot tell a wrong T above it. Send no follow before the leader answered. A replica that takes the plan before its leader can elect the plan's leader at a term below the plan term, before that leader took the plan. The leader then keeps leading below the plan term, and the plan never completes. No replica elects a leader that lacks the plan even then (#643). ursulactl and the script below always send to the leader first.
When the plan cuts, each replica first stops voting and taking replication, restarts its group in the process with the log cut back to its committed entry, and answers once the restarted group holds the plan. A replica that leads or campaigns when it takes a plan restarts its group the same way, so it drops that leadership or campaign. The plan's leader also restarts when its vote term is below the plan term, and starts with the plan term as its vote, so its first election is above the plan term.

The plan term is the higher of highest_term and the leader's own vote term, so it is at least the vote term of every replica that holds the plan. From the moment it takes the plan, the plan's leader campaigns but votes for no other candidate, and while it leads it serves nothing until the plan's history event applies. A replica that holds the plan votes only for the plan's leader, and only in an election at a term above the plan term: an election the leader started after it took the plan. So no replica elects a leader that has not taken the plan, even one that campaigns while the plan reaches the other replicas. It also refuses the plan's leader while the leader's log holds an entry above the point at or below the plan term, and a replica that holds the plan takes replication only from a leader above the plan term. A plan's leader that restarts after it lost entries comes back with a closed gate instead: it votes and campaigns for no one, and the group waits for another plan. A leader elected before the plan at or below the plan term, which may still believe it leads, thus never extends the plan's leader, and none of its entries is applied. A leader at a higher term could only have been elected by replicas that hold no plan: an open replica refuses a leader whose log is behind its own, and a closed one votes for no one, so it cannot get the plan's leader elected over its entries either. Once elected, the leader proposes the group's history event before anything else and answers 503 with Retry-After to writes and reads until the event applies. On each replica the plan is satisfied once it applied the event: history.event_index above the plan's point and recovery back to null in its metrics. The plan is recorded in the node's metadata, so a restart keeps it until then, and the leader serves nothing until its record is gone. A replica that lost power again after taking the plan keeps its vote restricted to the plan's leader across restarts, and once the event applies its gate stays closed until a leader's barrier opens it. Leaders after that hold the event, so a later leader change starts no new history.

If the plan's leader loses entries it had replicated, for example in a crash right after its election, the other replicas' logs are longer than its own and they elect no one. The plan wedges. After 30s in which a replica holding the plan applied nothing, it reports its group stalled like a closed gate does: readiness reason recovery_stalled, and stalled as its recovery_gate. Plan again. The new plan replaces the first, and since the wedged leader's failed elections raised its vote term, it usually cuts every log back to its committed entry. ursulactl recover-group --accept-loss plans again by itself when it sees, in two observations in a row, a replica holding the plan with a longer log than the plan's leader, or reporting the group stalled.

# group 42: one admin URL per node with a replica of the group, learners included, through your forwards
admins=(http://127.0.0.1:5441 http://127.0.0.1:5442 http://127.0.0.1:5443)
for admin in "${admins[@]}"; do
  curl -sS --fail "${admin}/__ursula/metrics" | jq -c --arg admin "${admin}" '
    .process_incarnation as $incarnation
    | .raft_groups[] | select(.raft_group_id == 42)
    | {admin: $admin, process_incarnation: $incarnation, node_id,
       running: .maintenance.running,
       voters: .voter_configs, kept: .committed_voter_configs,
       gate: .recovery_gate, recovery, committed_index, committed_term, committed_node,
       last_applied_index,
       expected: {last_log_index, last_log_term, current_term}}'
done > replicas.jsonl

jq -sc 'def holds($n): any(.[]; . == $n);
  def votes($cut): .node_id as $n | any((if $cut then .kept else .voters end)[]; holds($n));
  . as $all | map(select(.running))
  | (map(.node_id)) as $answered
  | ([.[] | (.voters + .kept)[] | select(length > 0)] | unique) as $sets
  | if all($sets[]; (map(select(. as $n | $answered | holds($n))) | length) * 2 > length) then .
    else error("a reported voter set has no majority of its voters answering") end
  | ($sets | add | unique) as $voters
  | (map(select(.gate != "open" or .recovery != null) | .expected.current_term) | max // 0) as $floor
  | map(select(.node_id as $n | $voters | holds($n))) as $candidates
  | ($candidates | max_by([.expected.last_log_term // -1, .expected.last_log_index // -1, votes(false), .node_id])) as $longest
  | ((($longest.voters + $longest.kept) | add // [] | unique | length) == 1) as $single
  | ((($single | not)) and ((($longest.expected.last_log_term // 0) < $floor) or ($longest | votes(false) | not))) as $cut
  | (if $cut then ($candidates | map(select(votes(true))) | max_by([.committed_index // -1, .node_id])) else $longest end) as $leader
  | if $leader == null then error("no replica the plan may elect is a voter of the membership it would campaign under") else . end
  | {leader: $leader.node_id, cut: $cut,
     point: (if $cut then $leader.committed_index else $leader.expected.last_log_index end // 0)} as $plain
  | ($all | map(select(.committed_index != null and .committed_term != null))
     | max_by([.committed_index, .node_id])) as $known
  | (if $cut and $known != null and $known.committed_index > $plain.point then
       ($candidates | map(select(votes(true) and votes(false)
          and (.expected.last_log_index // -1) >= $known.committed_index))
        | max_by([.committed_index // -1, .node_id])) as $reaches
       | if $reaches == null then $plain else
           {leader: $reaches.node_id, cut: true, point: $known.committed_index,
            point_entry: {term: $known.committed_term, node_id: $known.committed_node}} end
     else $plain end) as $plan
  | if any($all[]; ([.committed_index // 0, .last_applied_index // 0] | max) > $plan.point) then
      error("a replica knows a later committed entry than the plan would keep") else . end
  | sort_by(.node_id != $plan.leader)[] | . + {plan: ($plan + {expected})}' replicas.jsonl > plan.jsonl

# Review plan.jsonl, then send it: the leader first, then every other replica with the
# leader's plan term. Stop at the first refusal.
leader_term=""
while read -r replica; do
  if [ -z "${leader_term}" ]; then
    body=$(jq -c --argjson highest "$(jq -s 'map(.expected.current_term) | max' replicas.jsonl)" \
      '.plan + {step: {lead: {highest_term: $highest}}}' <<<"${replica}")
  else
    body=$(jq -c --argjson term "${leader_term}" '.plan + {step: {follow: {leader_term: $term}}}' <<<"${replica}")
  fi
  answer=$(curl -sS --fail-with-body -H "x-ursula-process-incarnation: $(jq -r .process_incarnation <<<"${replica}")" \
    -H "content-type: application/json" -d "${body}" \
    -X POST "$(jq -r .admin <<<"${replica}")/__ursula/raft/42/recovery/accept-unsynced-loss")
  status=$?
  echo "${answer}"
  [ "${status}" -eq 0 ] || break
  leader_term=${leader_term:-$(jq -r .leader_term <<<"${answer}")}
done < plan.jsonl

Do not plan again while a replica reports itself leader at a term above the last plan's term: it was elected under that plan, and a member that stalled under it is cut off, not wedged. If a live leader elected before a plan keeps a minority of replicas that took the plan, they take no replication from it, since its term is at or below the plan term. Transfer the leadership away from that leader (POST /__ursula/raft/{group}/leader/transfer/{node}). Every transfer raises the term by one: once a leader's term is above the leader_term those replicas show under recovery, they follow it.

A refusal answers 409 with a typed body: {"raft_group_id", "node_id", "refusal": {"reason", ...}, "message"}. If the leader refuses the plan, nothing changed: read the metrics again and plan again. A replica that refuses after the leader took the plan does not undo it. A replica whose reason is follows_leader with the plan's leader as leader refuses because the replicas that took the plan elected it, and it rejoins through it. For any other refusal, read the metrics again and plan again once the group has no leader. A refusal with reason not_stalled means the replica may still open through a leader's barrier: wait until its gate reports stalled before you plan again.

The script plans one group. For many stalled groups, as after a rack power loss, run it or ursulactl recover-group --accept-loss for each, one group at a time.

The call is refused with 409 Conflict, and changes nothing, when:

the replica lost its vote and has not established a durable vote floor. Restore connectivity to an intact quorum so it can obtain a fresh barrier. Accepting unsynced data loss cannot authorize a replica to acknowledge an unknown leader.
the replica leads the group with a quorum that acknowledged it within the last 3 seconds, or follows a leader whose replication reached it within the last 3 seconds, or its gate awaits or applies a leader's barrier or runs a plan's cut. The group has a live leader or may still recover without loss.
the request's step is not the replica's: lead goes to the plan's leader only, and follow to every other replica.
a follow names a plan term below the current_term its expected names (plan_term_below_vote): the term did not come from the plan's leader.
the plan cuts a replica whose voter_configs and committed_voter_configs name only one voter (single_voter_cut): no other voter could have overwritten an entry it holds, so its log is kept whole. The tool decides the cut the same way, from the voter sets of the replica with the longest log.
the replica is taking another plan (plan_in_flight). Wait for its answer and observe again.
the replica is the plan's leader and is no voter of the membership it would campaign under (leader_not_voter): its voter_configs, or its committed_voter_configs when the plan cuts. The membership moved since you read the metrics: read them again and plan again. Only the plan's leader checks membership. Another replica takes the plan whatever its own membership says, since the leader counts its vote under the leader's membership.
the replica no longer holds the log expected names, because it moved since you read the metrics. A replica that campaigns, one with an open gate or the plan's leader, raises its current_term with every election it starts, so only its last log id is compared. Read the metrics again and plan again.
the replica committed or applied an entry above the plan's point. The plan picked the wrong leader: read the metrics again and plan again.
the replica is the plan's leader and its log does not continue from the point: its last log index, or its committed index when the plan cuts. When the plan cuts to a later entry another replica reported committed, the leader must hold exactly that entry at the point.
the plan cuts and the replica knows no committed entry, or no longer holds the committed entry it recorded.
the plan cuts and its cut cannot run when the group restarts, because the log no longer holds the entry it keeps. The replica drops the plan, records its log as recovering so that its gate stays closed across restarts, and answers 409. Wait until the gate reports stalled, then read the metrics again and plan again.

Across a membership change. A membership entry is synced before it is acknowledged, also under never, so a power loss cannot drop it from a replica that holds it. The replicas may still report different voter sets when you plan:

A change that only a minority appended, such as a promotion the leader sent only to the new voter before it was cut off, has three outcomes. A cut removes it. Without a cut, the plan's leader either holds it and commits it, or lacks it and overwrites it.
A cut can also undo a committed change. A replica's commit marker is not synced on its own, so after a crash it can sit below a change the group committed. The replica reports the change in voter_configs but keeps the membership before it, which committed_voter_configs shows. A removed voter can come back, and a promoted voter can go back to being a learner. After a cutting plan, check the group's membership and make the change again.

Count every voter set, as step 1 says, and pick the leader by the membership it campaigns under, as step 2 says. A plan's leader whose cut leaves it no voter of the membership it keeps drops the plan and answers cut_left_no_vote: plan again. Every replica that applies the plan's history event holds the leader's membership. Every replica that holds a plan and does not lead answers client reads of the group (catch-up, live reads, HEAD, snapshots and forwarded reads) with 503 and Retry-After at once until it applied the event, as the plan's leader does after a short wait, so a replica outside the leader's membership, which never receives the event, never serves the history the plan replaced. Its /__ursula/usage and /__ursula/metrics keep answering. If it is a configured voter, the leader adds it back as a learner and it catches up from the plan's leader. If its log committed entries above the plan's point, it cannot follow: rejoin its node as in Rejoining a wiped node, which affects every group on that node. The same holds for a replica that no plan reached while two or more plans ran in a row, or whose committed history a later plan dropped: it refuses the leader's appends and snapshots, since nothing shows that the new history kept its committed entries. It logs the refused snapshot with UnrelatedHistory when two or more plans ran, and with CommittedOverwrite when one plan's history replaced its own below its commit marker. Its own committed entries above the first such plan's point are gone for good: that plan replaced them, and wiping the node only stops it from waiting for them.

Rejoining as a wiped node is an operator action, and it wipes every group on the node. Before you wipe, check every other group the node holds a replica of in /__ursula/metrics. Do not wipe while the node may hold the only copy of another group's acknowledged writes: a group whose other voters are gated, stalled or down, or whose leader is this node. Do not wipe while another group on the node would lose its quorum without this replica: while it is rebuilt, each of its groups needs a quorum of the other voters, and a group whose other voters are gated stalls. Recover or wait for those groups first.

A voter set that names nodes that never come back can block every plan. The tool needs a majority of each set's voters, because it cannot rule out that the set committed, and the plan's leader must win one. When those nodes are gone for good, no plan can prove what the group committed: restore the group from a backup.

A replica that cannot lead knows more is committed. A learner, or a promoted voter whose cut would make it a learner again, can know a later committed entry than every replica that could lead. Commit markers are not synced on their own, so after a power loss the voters can still hold that entry in their logs while their markers lie below it. The tool then plans a cut that keeps every log up to that entry, led by a voter whose log reaches it, and the leader refuses the plan unless it holds exactly that entry. When no voter it reached holds the entry, or a replica reports a committed or applied index above every log that could lead, the tool stops with an error naming that replica: a plan from any voter would overwrite those committed entries. Each replica also refuses an append, or a snapshot install, that would replace entries at or below its commit marker. Waiting does not help, since the voters' logs no longer hold the entry. Bring back a voter that was not reached, if one may hold it, and plan again. Otherwise restore the group from a backup.

A body of the 0.7 form answers 422. A second plan replaces a first one that was never satisfied, as long as floor counts the replicas the first one holds.

A voter that was never gated does not report recovery_stalled. Its readiness answers 503 with raft_replica_unready, and raft_maintenance.group_issues shows leader_unknown for the group.

With fsync = "always", acknowledged appends survive a host crash, but the newest journal segment can still hold bytes written after the last fsync: an unacknowledged batch, or a commit or truncate marker. Recovery refuses damage in sealed segments. In the newest segment it removes an incomplete final frame without gating anything. A complete frame that fails verification, such as a hole, a zero-filled tail or a torn sector inside an unacknowledged batch, looks the same as media corruption of an acknowledged write. Recovery therefore records the recovery gate of every group on that core before it cuts the frame, and those groups rejoin as described above.

A single-voter group has no other replica to recover from, so it needs a plan after every host crash under never, once its gate reports the group stalled, and every such plan gives every stream of the group a new Stream-Incarnation. Stream-Previous-Incarnation and Stream-Inherited-Offset on HEAD then tell clients how much of each stream survived. Use fsync = "always" for single-voter groups whose clients keep state. Under always it needs the call only after a host crash that leaves a complete invalid frame in the newest segment. So does a group whose voters lose power together, for example in a rack power loss, when that gates a majority of them. No acknowledged write is lost in these cases, but only an operator can open the gates.

POST /__ursula/cluster-probe is served on the cluster plane, but it is machinery rather than an operator surface. Nodes send each other probe bodies through it at their peer url to verify cluster-plane egress. The cluster plane runs no ingress admission, so a node under write pressure still answers it.

Memory without retention

Per-stream replicated state, on every replica, is at most about 32 KiB plus bounded producer state, beyond the unflushed payload itself, without retention. It grows neither with the number of messages nor with the length of the cold history. The bounds that make this hold are always on: one receipt per producer with 7-day idle-producer expiry and a 4,096-producer cap, unique stream incarnations with incarnation-scoped cold objects, external payload locators committed first and indexed after, and a compact hot window with no per-message bookkeeping, so cold admission and the flush thresholds charge hot payload only. Snapshot bodies of at least runtime.external_payload_min_size go to the cold tier when a cold backend is configured, up to 1 GiB uploaded in parts. Retention stays available as a storage-cost tool, but it is not the way to bound memory.

The per-group gauges under group_state_gauges in /__ursula/metrics show what each group holds.

Usage export

GET /__ursula/usage returns committed counters and current gauges from local replicated state, aggregated by bucket across this node's Raft groups:

{
  "version": 1,
  "node_id": 1,
  "write_unit_bytes": 10240,
  "buckets": {
    "tenant-a": {
      "committed_append_bytes": 24576,
      "committed_records": 3,
      "committed_write_units": 4,
      "retained_bytes": 24576,
      "stream_count": 1
    }
  },
  "raft_group_count": 1,
  "staged_snapshot_parts": [
    {
      "raft_group_id": 0,
      "term": 7,
      "counting_finished": true,
      "bytes": { "tenant-a": 33554432 }
    }
  ]
}

The raw byte and record counters are pricing-neutral facts. committed_write_units is a convenience derived meter: every committed, non-deduplicated create or append contributes at least one unit and rounds its canonical payload up independently by write_unit_bytes. Consumers must validate both version and write_unit_bytes before interpreting it; the unit is data in the contract, not encoded into the counter name. The same unit is persisted in Raft snapshots, and a build refuses to restore a snapshot carrying a different unit. Changing it therefore requires a new contract version and an explicit state migration, never a configuration edit. Ursula does not attach currency, a price, or an account owner.

Counters are monotonic and survive snapshots, stream deletion, and bucket purge. retained_bytes and stream_count are current-state gauges and may decrease. A node serves local applied state, so replicated consumers should poll every voter and use their own completeness policy rather than treating one response as a cluster-wide linearizable read.

staged_snapshot_parts reports the staged snapshot parts that no commit has used yet, separately from the replicated fields: for each Raft group the answering node leads, the group, the node's term and the bytes of staged parts per bucket. A new term's totals start at the node's first part, commit or discard in the group, or when its Raft metrics show it leads, whichever comes first. The node then lists the group's part directories in the background and reports counting_finished: false until that listing ends. From then on it follows every part it stages, every commit, discard and delete, and every part the orphan sweep reclaims. The bytes are exact except for parts that were still uploading to the previous leader when the term began, parts a failed delete may have removed, and a part the sweep deletes while the seed listing runs, each until its stream is listed again. A collector polls every voter. For each group it keeps the report with the highest term, so a deposed leader that has not yet seen the new term is not counted twice. It treats the total as partial when a kept group has not finished counting, or when no report names one of the raft_group_count groups, for example a group without a leader. ursula_proto::admin::UsageReport::staged_part_bytes applies these rules. node_id is the answering node's id. Ursula bounds staged parts per stream only. A bound per bucket, or a charge for staged parts, is the operator's policy.

Tenant offboarding (bucket purge)

DELETE /__ursula/purge/{bucket} erases one tenant's complete bucket domain: every stream in every Raft group, the bucket, cold chunks and indexes, external payloads, and any uncommitted object-store staging orphan below {bucket}/. The replicated purge also installs a permanent bucket-erasure fence in every group. It retains only the bucket's aggregate usage counters, because deleting a counter before an asynchronous meter observes it would erase committed billable work. Purge is served only on the admin listener (server.admin_listen, loopback 127.0.0.1:4438 by default); the client listener answers 404. On Kubernetes the chart keeps the admin listener on pod loopback, so reach it through kubectl port-forward pod/<voter-pod> 4438:4438 or kubectl exec; any one voter accepts the request for the whole cluster.

The response is a completion report an operator can attest to:

curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X DELETE http://127.0.0.1:4438/__ursula/purge/tenant-a
# {"bucket":"tenant-a","removed_streams":12,"groups_with_streams":[0,3,5],"cold_gc_entries_reclaimed":12,"cold_gc_pending_entries":0,"cold_gc_complete":true,"cold_gc_error":null,"bucket_prefix_absent":true}

Semantics:

Retry on 503. A Raft group whose leader is unknown or changing answers 503 with Retry-After, never a redirect to a peer. Because purge is idempotent, retry the same request.
Idempotent and resumable. Re-purging an erased bucket returns the same report shape with zero removals. A purge interrupted mid-way converges on re-run: the permanent fence and stream removal are replicated commands per group, while cold reclamation and the complete bucket-prefix erasure are repeatable object-store operations.
Permanently fenced. Commands linearized after purge cannot create the bucket or append to its streams. The fence survives Raft snapshots and restore; data-plane access continues to conceal the erased namespace as 404 Not Found instead of recreating it.
Isolated. Other buckets' streams, offsets, checksums, snapshots, and object-store prefixes are untouched. Every cold object, shared packs included, lives under its own bucket's {bucket}/ prefix.
Accounting residue. The retained entry contains only aggregate counters and zero-valued content gauges, not stream data. Ursula does not currently provide an acknowledgement protocol for erasing it after an external meter persists the final delta. Use opaque bucket IDs if retaining the identifier itself would violate an erasure requirement.
Physical absence proof. cold_gc_entries_reclaimed is diagnostic only. After a second replicated purge proves every group's bucket-specific GC queue is empty, Ursula recursively deletes the complete {bucket}/ erasure domain and lists the authoritative object store to prove it contains no files. Only cold_gc_complete=true, cold_gc_pending_entries=0, bucket_prefix_absent=true, and no error constitute the proof. If object storage is unavailable or still lists a file, the response remains incomplete and a later idempotent request retries the proof.
Backups. A purged tenant may persist inside backups taken before the purge until those backups age out; restore replays whatever the manifest contains. Deleting old backup sets is part of completing an offboarding with erasure obligations.
Raft snapshots and logs. Each replica drops the purged streams from its current Raft snapshot and its Raft log within raft.deleted_data_max_age. See Deleted data for what that bound covers.

Deleted data

Deleting a stream and purging a bucket remove the stream from its group's state at once. An expired stream reads as gone at once and leaves its group's state within about a minute, as described below. Copies of its records can stay for a while in each replica's Raft snapshot, its Raft log and the log's files, and in older snapshot objects of an S3 snapshot store. They are cleaned up in the background. raft.deleted_data_max_age (1 hour by default) bounds how long they stay.

Within that time after a replica applies the removal:

The replica's current snapshot no longer holds the stream. That covers the snapshot in memory, the snapshot file in the Raft log directory and, with an S3 snapshot store, the snapshot object that file names.
The replica's Raft log holds no entry with the stream's records.
No file in the replica's Raft log directory holds the stream's records, as long as the journal can copy the entries it keeps fast enough. A purged entry's bytes stay in its journal segment until the segment is deleted. A segment that has held such bytes for a quarter of raft.deleted_data_max_age since the purge is sealed, the few entries other groups still keep in it and in the segments before it are copied into the newest segment at the rate below, and the segments are deleted. At the defaults that takes far less than the rest of the bound. With a short raft.deleted_data_max_age, a slow drive interval and large live logs to copy, it can take longer.
The snapshot store no longer holds the snapshot objects the replica used before, unless another replica still uses them.

Removed bytes count toward a group's next snapshot, so a large removal leaves the snapshot and the log within minutes. A small removal makes its group take a snapshot a quarter of raft.deleted_data_max_age after the replica applied it, and the replica then purges its Raft log through that snapshot. For a large and a small removal alike, the segments that held the purged entries go a quarter of raft.deleted_data_max_age after that purge, plus the time the copies take. Cold chunks, cold indexes and external payloads are deleted by cold GC, usually within seconds, and a bucket purge reports when they are gone.

Each group's leader removes the streams that expired in its group in the background, every minute, or every quarter of raft.deleted_data_max_age if that is shorter. Each time it writes up to 16 small Raft entries per group, and starts no more after 250 ms. Each entry removes the expired stream it names and up to 256 more on every replica, so a group sheds about 4,100 expired streams a minute besides those its own writes remove. A larger backlog takes a few more minutes. The leader removes a stream once its own clock is a second past the stream's expiry, so keep the nodes' clocks in sync. As for a deletion, the time counts from when a replica applies the entry that removed the stream. The removal also queues the stream's cold objects for cold GC and takes the stream out of retained_bytes and stream_count in /__ursula/usage. ttl_expiry_sweeps in /__ursula/metrics counts the entries the node's leaders wrote, and ttl_expiry_sweep_due_groups the groups the latest pass left with expired streams for the next one.

Freeing segments this way copies the live entries of quiet groups forward about once per quarter of raft.deleted_data_max_age, which shows in wal_rewritten_bytes. The snapshot driver drives the copies at most once per storage.snapshot.drive_interval, an eighth of raft.wal.segment_size per core each time, so a mass removal does not saturate the disk. At the defaults that is 8 MiB per second per core with inline snapshots and 1.6 MiB per second with an S3 snapshot store. A group that keeps more than a quarter of a segment there is reported in wal_lagging_groups, and the driver snapshots it first, or purges its log through its snapshot when it applied nothing since. wal_dirty_segments counts the segments that hold purged entries. The first passes after an upgrade or a restart find all the old purged entries at once and copy the live entries of every kept segment forward, up to about twice a core's live log, and snapshot the groups reported lagging, before the journal settles.

The time counts per replica, from when that replica first applies the removal. A replica that is down when the removal commits keeps its copies until it is back. It then applies the removal from the leader's log, or installs a newer snapshot, and the time counts from there. For the snapshot and the log, a restart after the replica applied the removal does not extend the bound: a removal it applies again while it replays its log at startup is due at once, and it purges its log through the snapshot it restored. The log's files follow the purge. Purged entries in a journal segment keep their age across a plain restart, since it is read from the segment's last change, but two cases count from the restart. Entries the replica purges only after it restarts leave the files a quarter of raft.deleted_data_max_age after that purge. A restart that rewrites the journal's segments, after a host crash under raft.wal.fsync = "never" or when the journal ended in a damaged frame, starts their age again. One exception: a replica that loses the removal from its log in a host crash under raft.wal.fsync = "never" learns it again from the leader, and its time counts from then.

The bound and its gauges need the snapshot driver, which runs by default on every node that runs Raft, a single node included. A single node keeps its snapshots inline whatever storage.snapshot.backend says, and never refuses writes for its log's pressure. With storage.snapshot.drive_interval = "0s" nothing enforces the bound, the gauges below stay at 0, and the server logs a warning at startup. Expired streams still leave their group's state without the driver.

While the driver purges a group's log through a snapshot, the group takes no new snapshot until that purge lands, which is usually at once. An operator's POST /__ursula/raft/{group}/snapshot waits for it, and answers 503 with Retry-After when the wait passes 10 seconds. An operator's POST /__ursula/raft/{group}/purge?upto=<index> follows the same rule: while a snapshot of the group builds, OpenRaft has not published the one it built, or an earlier purge of the group has not landed, it answers 503 with Retry-After. Send it again then.

Under write pressure the driver first snapshots the groups that free the most log, and groups that owe a removal wait their turn among the due groups. They lead the driver's passes again once the node's log is below three quarters of its budget.

With an S3 snapshot store, the object a replica used before its current snapshot stays pinned for a quarter of raft.deleted_data_max_age, at most an hour or two raft.install_snapshot_timeouts if that is longer. That covers a peer that started to install it before the replica moved on. Once that time has passed, the replica deletes the object unless another replica still names it as its current snapshot or pins it for an install or its own retirement. It also deletes every other object of the group that is at least that old and that no replica names or pins. An object another replica still names or pins is deleted once that replica moves past it and its own pin expires. Snapshot GC deletes nothing while a configured voter of the group has never stored a snapshot reference, since that voter may use any of the group's objects. The snapshot driver runs these GC passes after its snapshot builds, and holds them back while the node's log is under pressure or the store health gate reports trouble. After a restart the passes come due spread over a further quarter of that time. An object's age is read from its modification time in the store, so the nodes' clocks must agree with the store's to well within that time. snapshot_gc_deleted_objects in /__ursula/metrics counts the objects GC deleted, and snapshot_gc_skipped_unreferenced_voter counts the passes such a voter held back.

The bound does not cover:

Retention advances. Records a stream's retention trims are not a removal and leave the snapshots and the log on the group's usual cadence.
A leader whose follower lags. A leader purges no log entry a follower still needs, so a slow or unreachable follower holds the leader's log through the removal until it catches up or the leader sends it a snapshot.
A group whose Raft is not running on the replica, for example while the replica restarts or after it stopped on a storage error. The time keeps counting, and the driver cleans the group up once its Raft runs again. Until then its live log entries also keep the journal segments they share with other groups.
Journal segments while the snapshot store is in trouble. While the store health gate holds routine snapshots back and the node's log is not under pressure, a group that keeps more than a quarter of such a segment and applied entries since its snapshot is not snapshotted, so the segment stays until the store recovers or the log comes under pressure.
A replica that is down for longer than raft.deleted_data_max_age. It catches up when it is back, and its time counts from when it applies the removal.
Sustained write pressure. While the node's log stays above three quarters of its budget, the driver snapshots the groups that free the most log first and holds snapshot GC back, and the time can run past the bound.
A configured voter that has never stored a snapshot reference. Its group's old snapshot objects stay until it stores one.
A node that left the cluster. Under each group's snapshot prefix it keeps its reference references/node-<id>.json and its pins under references/pins/<id>/, and with them the snapshot objects they name. Delete both, in every group's prefix, but only after the node is stopped and removed from every node's raft.peers. Deleting a running node's pins removes the protection of its fresh builds and installs.
Object versioning. On a bucket with versioning, a deleted snapshot object stays as a noncurrent version until a lifecycle rule expires it.
Backups taken before the removal. Delete them to complete an erasure.
Freed disk blocks and memory. Ursula deletes and replaces files but does not overwrite the blocks they used, and read caches keep blocks of a deleted stream until they are evicted. Use encrypted volumes and buckets where that matters.
A group whose committed application failed. It keeps its whole log for replay until it is repaired.

raft_deleted_data_oldest_age_ms in /__ursula/metrics reports how long ago the oldest removal on the node applied whose stream its current snapshot or its Raft log still holds, and raft_deleted_data_overdue_groups counts the groups whose snapshot still holds one past a quarter of raft.deleted_data_max_age. deleted_data lists each such group with what it still holds: owed or replayed while its snapshot holds the stream, with the bytes and age, and purging while only its log does, with the age and the index the log is purged through. Alert when raft_deleted_data_oldest_age_ms nears half of raft.deleted_data_max_age.

Cleaning S3

Cold data lives under the storage.cold.root prefix. Benchmark and test runs should use a date-stamped root so cleanup is one prefix delete with standard S3 tooling:

aws s3 rm "s3://my-ursula-bucket/ursula-test-20260518T000000Z" --recursive

There is no automatic retention policy. Application checkpoints and explicit stream-retention advances control which cold objects become eligible for GC. After a retention advance, each group's leader deletes the stream's cold objects that lie wholly below the retained offset (chunks and offloaded external payloads, in whole 64 MiB cold-index pages, so up to one page just below the retained offset can remain until retention moves past it) once a 5-minute grace has passed, so reads planned before the advance still complete; its cold-index cursor visits 16 streams per group per minute, so on groups with many streams collection can lag further. The grace restarts after a leader change. Retention is a storage-cost tool only: replica memory does not depend on it (see Memory without retention).

Incomplete multipart uploads

Ursula writes large cold objects with S3 multipart uploads. A crash or shutdown in the middle of a cold write leaves an incomplete upload behind. Ursula never aborts these uploads, and S3 bills for their uploaded parts until they are aborted. Snapshot commits from parts make long cold writes more common, so these leftovers show up more often.

Add an AbortIncompleteMultipartUpload lifecycle rule to every cold bucket, for example after 1 day:

aws s3api put-bucket-lifecycle-configuration --bucket my-ursula-bucket \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "abort-incomplete-multipart-uploads",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 1}
    }]
  }'

This call replaces the bucket's whole lifecycle configuration. If the bucket has other rules, add this rule to them. MinIO and other S3-compatible stores need their own equivalent rule. Check the store's lifecycle documentation for how to expire incomplete multipart uploads.

Backup and disaster recovery

ursulactl provides a verifiable whole-cluster backup and restore workflow:

# Create: one MessagePack snapshot object per raft group + manifest.json,
# into a local directory or s3://bucket/prefix.
ursulactl backup-create --config cluster.yaml --location s3://backups/ursula/2026-07-25

# Verify: manifest completeness, byte sizes, BLAKE3 checksums, and deep
# state-machine validation of every snapshot. Touches no cluster.
ursulactl backup-verify --location s3://backups/ursula/2026-07-25

# Restore: into a FRESH, EMPTY cluster with the same raft group count and the
# source cluster's server.cluster_id.
ursulactl restore --config new-cluster.yaml --location s3://backups/ursula/2026-07-25

An s3:// location uses the AWS SDK credential chain and AWS_REGION. For MinIO or another S3-compatible store, also set AWS_ENDPOINT_URL, for example AWS_ENDPOINT_URL=http://minio.internal:9000.

The recovery contract:

Consistency boundary. Each group export is the deterministic state-machine snapshot the raft snapshot path persists: internally consistent per group while writes continue. Cross-group consistency is not promised. The boundary is per stream. Acknowledged writes in the exporting replica's applied state are included whether or not they were cold-flushed, so RPO is bounded by export time, not by the cold-flush interval. backup-create asks every node and keeps the reply with the highest group commit index.
ursulactl restore is the only supported restore. Do not restore a node or a cluster from volume-level copies, such as disk or EBS snapshots of the Raft WAL or the data directories. A cluster brought back from such copies keeps its history ids, so its streams keep their Stream-Incarnation tokens while their tails go back, and new writes then put other bytes at offsets the old cluster already served under the same tokens. Clients that kept state, and caches that stored immutable reads, take those bytes for the ones they hold. A single rolled-back volume also breaks Raft itself. A restore through ursulactl restore starts a new history, so every token changes.
Keep the cluster id. Give the target cluster the source cluster's server.cluster_id (Helm server.clusterId) before its first start. Clients keep the id with their state and treat a different id as another deployment, so a restore under a new id looks to them like a move to another cluster, not a restore. A target node whose WAL was created with another id refuses to start, so set it before the nodes first run.
Fresh identity on restore. Restore replays each snapshot as one replicated write on the target cluster, which keeps its own raft membership and log identity. Nothing from the source cluster's raft metadata is reused. Groups that already hold buckets refuse the import with 409, and group-count mismatches fail closed before the first import. Group commit indexes and per-stream append counters restart under the new identity. Stream bytes, offsets, close/TTL state, published snapshots, retention floors, and producer dedup states are preserved exactly.
A restore is a new history. Each group's import starts a new history of the group, so every restored stream answers with a new Stream-Incarnation. A client that holds a stream's token from the source cluster gets 412 with the new token, and HEAD names the source token as Stream-Previous-Incarnation with the restored tail as Stream-Inherited-Offset (or, for a stream the source had not written since a rewrite of its own, the token from before that rewrite): what the client derived from bytes below it still holds. Every producer epoch claimed on the source is closed: a producer's next new write of it answers 403 with Producer-Outcome: epoch-closed;by=history and the last seq the backup kept, and the producer claims a new epoch. The import requires x-ursula-backup-blake3, the BLAKE3 digest of the group's backup object, which ursulactl restore sends. A repeat of an import that applied answers 200 with "already": true and starts no second history, so a restore interrupted by a timeout or a leader change can run again.
Cold objects are part of the backup set. Snapshots reference cold-tier chunks, cold-index pages and external payloads by object key. The backup holds the references, not the objects. The restored cluster's cold store must hold the same objects under the same keys: point it at a copy (see Copying cold objects). backup-verify validates snapshots but does not dereference cold objects. restore asks the target cluster, before the first import, to check every group's references against its own cold store (POST /__ursula/backup/group/{group}/cold-check). If any object is missing, it stops with nothing imported and prints how many are missing and the first keys.
Format compatibility. The backup format is the format epoch: format_version 4 in the manifest and in /__ursula/backup/info, and format_epoch 4 in every group object. A backup restores only into a cluster of the format epoch that wrote it. ursulactl refuses a manifest or a target cluster of another version before the first import, and a server refuses to import a group of another format epoch (400).
RTO is dominated by transferring and importing the group snapshots. Each import is a single replicated write per group.

The recovery drill runs in CI (backup_restore_drill_preserves_streams_and_allows_continued_appends): write, snapshot, retain and close, back up, destroy the cluster, restore, verify checksums, offsets and boundaries, and continue appending.

What you can additionally rely on:

Quorum durability. Acknowledged writes survive as long as a majority of voters survives. Three voters across AZs tolerate any single-AZ outage.
Cold-tier durability. Once flushed to S3, a chunk inherits S3-grade durability. The unflushed window is bounded by the flush interval (seconds by default).
Format epoch 4, no migration. The Raft WAL, snapshots, backups, the object-storage prefix and the Raft gRPC protocol carry format epoch 4. A node refuses data, backups and peers of another format epoch and does not migrate them.

Node-level loss: replace the host with the same node_id and cluster config. Each group's leader rebuilds the replacement from the other voters (see Rejoining a wiped node). ursulactl wait --node N returns once the replacement is a voter of every group and has caught up, and ursulactl verify-cluster then requires every configured voter to be present and caught up. ursulactl wait-ready checks the whole cluster: leaders for every group, complete voter sets and 200 from every node's /__ursula/ready. Run it before declaring the recovery done. Total-cluster loss: restore the latest verified backup into a fresh cluster as above.

Copying cold objects

A restored cluster reads cold data under the keys the source cluster wrote, relative to storage.cold.root. Copy the source's objects into the target's root after the target has started and written its URSULA_FORMAT_EPOCH marker, and before restore. Skip two things:

URSULA_FORMAT_EPOCH, the marker. The target's own marker stays.
snapshots/, the source's Raft snapshots and their references. The target writes its own there. With a custom storage.snapshot.s3_prefix inside the cold root, skip that prefix instead.

With the AWS CLI, from the source root to the target root:

aws s3 sync s3://my-bucket/source/cold/ s3://my-bucket/target/cold/ \
  --exclude 'URSULA_FORMAT_EPOCH' --exclude 'snapshots/*'

For MinIO or another S3-compatible store, add --endpoint-url. The copy must match the backup, so the source must not change its cold objects in between. Stopping writes is not enough: an idle cluster still compacts small cold chunks and deletes the ones it replaced a few minutes later. If the source cluster still runs, stop its nodes once the backup verifies, then copy. restore checks that the copy is complete.

Never point two clusters at one prefix, and never point a new cluster at the source's prefix in place. After a restore both clusters hold the same streams under the same keys, but each one deletes cold objects by its own state alone: the cold GC after a delete, a retention advance or a compaction, the orphan sweep for unreferenced objects older than a day, and the pruning of snapshots/ by the cluster's own references. Two clusters on one prefix delete objects the other still reads. A copy also leaves the source cluster and its prefix as they were, so the source stays a way back until clients switch.

Upgrading to 0.8

0.8.0 is a breaking release. It changes the storage and protocol format to format epoch 4. A 0.8 node refuses epoch-3 Raft WAL directories and object-storage prefixes at startup, refuses 0.7 peers, snapshots and backups, and ursulactl restore refuses 0.7 backups. There is no migration, no in-place upgrade, no rolling upgrade and no rollback. If you cannot move your data yet, stay on 0.7.6.

Install fresh. Use empty PVCs and never-used prefixes for the cold store and S3 snapshots. Never reuse a prefix that 0.7, or a failed upgrade attempt, wrote to.
Set the cluster id. Set server.cluster_id on every node of a multi-node cluster, the same on every node and different for every deployment a client can reach. The chart requires server.clusterId for every install, a single replica included.
Choose the network planes before the first install. The chart serves Raft on server.ports.cluster (4439) by default. Do not change server.clusterListen or server.ports.cluster on an initialized cluster. Outside the chart, set server.cluster_listen and client_url together on every node.
Check the inflight body budget. A node with a cold backend needs server.http_inflight_body_size of at least 64 MiB, or it refuses to start with InflightBodyBudgetTooSmall.
Lower the gateway's request limit. A gateway whose --max-request-body-bytes (gateway.maxRequestBodyBytes) is above 33554432 refuses to start. Larger snapshots go up in parts.
Move data by export and re-import. The applications that own the data read it from 0.7 and write it into 0.8. For the SQLite VFS, export each database with 0.7 using VACUUM INTO and import the file with 0.8 (Moving from 0.7).
Update clients and proxies. Error answers have a JSON body, x-ursula-not-applied is gone, a leadership loss after a proposal answers 503 with OUTCOME_UNKNOWN, and Stream-Incarnation changes on a restore or an accepted loss. A proxy in front of Ursula follows the intermediary rules of write outcomes. Also check clients and tooling for these changes:
- A snapshot publish above 32 MiB sent straight to a node answers 413. Send it in parts.
- A failed cold-tier read answers 502 COLD_STORE_FAILED, where 0.7 answered 500, and a write that needs a cold backend the node lacks answers 501 NO_COLD_STORE, where 0.7 answered 503.
- Producer-Seq on 200 and 204 is the epoch's highest applied seq, not the request's.
- A create with producer headers answers 400. Create the stream, then claim an epoch.
- A producer close of a closed stream answers 409. Only a close without producer headers is idempotent there.
- A retention request at or below the retained offset answers 204 and changes nothing, where 0.7 answered 409.
- An SSE session ends on a delete and recreate, since it is pinned to the token its first response announced.
- Read and snapshot ETags have a new form.
- /__ursula/metrics no longer reports receipts, receipt_items or max_receipt_items_per_stream.
- The gateway no longer uses a system proxy from HTTP_PROXY for its upstreams.
- The chart's Helm test pod's name label is now <chart>-test.
- ursulactl reads metrics at a manifest node's client_url, never at http_url. Add client_url to custom manifests.
Update recovery runbooks. accept-unsynced-loss takes a recovery plan, and ursulactl recover-group --accept-loss computes it. A replica whose committed entries a plan dropped waits for its node to be rejoined as a wiped node. See Recovering after a host crash for the checks before a wipe.

Every behavior change, with what to change for each area, is listed in the 0.8.0 release notes.

0.8.1 upgrades 0.8.0 in place with an ordinary rolling restart (helm upgrade), with no fresh install. See the 0.8.1 release notes.

0.8.2 upgrades 0.8.0 and 0.8.1 in place the same way, once your Helm values no longer set the indexer it removes. See the 0.8.2 release notes.

Upgrading to 0.7

0.7.0 is a breaking release. It changes the Raft WAL format (format epoch 3) and removes the memory WAL, so the per-core disk journal is the only Raft WAL. 0.7 cannot read 0.6 data: a node refuses 0.6 Raft WAL directories and object-storage prefixes at startup, ursulactl restore refuses 0.6 backups, and a node will not join a cluster with 0.6 nodes. There is no in-place upgrade, no automatic migration and no rollback.

If you run 0.6 with data you need to keep, stay on 0.6.2. 0.7 provides no migration path from 0.6.
Deploy 0.7 fresh: empty PVCs and a never-used object-storage prefix.
helm upgrade from 0.6 stops before a second voter is replaced. Recover with helm rollback.

Every behavior change is listed in the 0.7.0 release notes.

0.7.6 upgrades 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4 or 0.7.5 in place with an ordinary rolling restart (helm upgrade), with no fresh install. See the 0.7.6 release notes.

To install 0.7:

Install fresh. Install with a new s3.prefix and empty PVCs. With cold storage off and S3 snapshots, also use a new snapshotStore.prefix.
Never reuse a prefix that an earlier release, or a failed upgrade attempt, wrote to.
Update the configuration.
- Remove raft.wal.backend, raft.wal.allow_volatile_multi_peer, raft.memory_bootstrap_marker_dir and raft.rejoin_probe. A config file that still sets one fails to load.
- Set raft.wal.path on every node of a multi-node cluster. 0.6 defaulted to the memory WAL, and a 0.7 node with raft.peers refuses to start without a WAL path.
- Choose raft.wal.fsync (Helm raft.walFsync). The default never acknowledges a write once a quorum of voters holds it in the page cache, and a host crash can cost a replica its unsynced tail (see Recovering after a host crash). Set always to keep the 0.6 disk WAL behavior of one fsync per batch.
- With Helm, remove raft.storageMode, raft.allowVolatileMultiPeer and persistence.enabled. Every pod gets a PVC for its WAL. The values schema refuses unknown keys under raft and persistence.
- Keep runtime.core_count and raft.group_count explicit and unchanged (see WAL routing configuration).
Update runbooks and tooling. ursulactl 0.7 removes the memory-WAL restart verbs prepare-restart, restart-quiesce-capability, finish-prepared-restart, abort-prepared-restart, prepare-amnesiac-restart, repair-restarted-voter, prepare-recovery-handoff and classify-amnesiac. A planned restart is drain, restart, wait and undrain (Restarting a voter), and a voter that lost entries is rebuilt by its group leaders. Custom tooling that calls admin mutations must send x-ursula-process-incarnation (Admin endpoints).

A values file that still sets a removed chart value fails validation, and nothing changes. Otherwise helm upgrade from 0.6 replaces the first pod, with the default RollingUpdate strategy and also with OnDelete plus the graceful rollout. That pod finds the 0.6 data on its volume and exits at startup, so no second voter is replaced. Recover with helm rollback. With OnDelete, also delete the refused pod after the rollback so that it starts from the 0.6 template again. The refused pod wrote nothing, so the old pods are unaffected.

How a 0.7 node refuses data of another release, and what it prints, is listed under Startup failures. Every startup refusal happens before the node writes anything:

Raft WAL directory. {raft.wal.path}/raft-log/FORMAT_EPOCH records the epoch. A directory that holds data without the marker, or a marker of another epoch, is refused. The marker is written after every check passes.
Object storage. URSULA_FORMAT_EPOCH sits at storage.cold.root when the cold backend is S3, otherwise at the S3 snapshot namespace (storage.snapshot.s3_prefix) when snapshots live on S3. A namespace that holds objects without the marker is refused, so it must be used only by this cluster. With a memory cold store and inline snapshots there is no object-storage check.
Peers. Before it writes the markers, a node with configured peers asks each one, in parallel (2 seconds to connect plus 2 seconds for the answer, about 4 seconds at most), whether it speaks Raft protocol 3. A peer that refuses ends startup. An unreachable peer is skipped, because a fresh cluster starts its pods in parallel.
Readiness. Any later Raft protocol mismatch, inbound or outbound, is logged at error level and makes /__ursula/ready answer 503 with "reason": "format_epoch_mismatch" until the node restarts. This also holds for a 0.6 node that a 0.7 node contacts, including through the startup probe. Keep the peer lists of 0.6 and 0.7 nodes apart.
Backups. Backups use format version 3. ursulactl restore and backup-create refuse a cluster that reports another version, and a server refuses to import a group of another format epoch (400).

Bucket IDs snapshots and any ID starting with __ are reserved (400): snapshots is the default raft-snapshot namespace in the object store.

Logs

RUST_LOG=ursula=info,ursula_runtime=info,ursula_raft=info is the baseline. Bump to debug for one crate when chasing a subsystem:

RUST_LOG=ursula_raft=debug ./target/release/ursula server ...

debug is verbose under sustained load, so redirect to a file.

Committed application failures

A stopped group requires corrected-code replay. Retain the original WAL and referenced snapshot objects. Never skip a committed record to recover readiness.