Operations

The first tool to reach for is ursulactl. It covers the raft-aware verbs operators run most often: drain, status, readiness gates. This page covers everything around it: the metrics shape ursulactl reads, the admin endpoints it (and your custom tooling) can call, and the operational policies ursulactl does not encode (backups, log levels). Deployment and process lifecycle belong to the platform, such as Helm.

Tooling map

Surface	When to reach for it
ursulactl	Day-2 cluster ops over HTTP: drain, catch-up waits, status, wait-ready.
/__ursula/metrics and the /__ursula/raft/... admin endpoints	Custom tooling. ursulactl uses these underneath. The surface is small and stable enough to script directly.

There is no Prometheus scrape and no general-purpose orchestrator yet.

Metrics

curl http://127.0.0.1:4437/__ursula/metrics | jq .

The JSON snapshot covers per-core mailbox depth, append and apply counters, cumulative timings (*_ns), each group's leader, term and applied index (raft_groups), hot bytes waiting for cold flush, cold-flush uploads and errors, WAL physical/reclaim/recovery work, WAL disk-pressure admission, SSE counters, live-read waiters, and cold-write backpressure. Start here when triaging slowness, lag, or 503s. Troubleshooting lists jq selectors for the common fields. GET /__ursula/ready provides the local serving signal with a typed reason on refusal, on the client and admin planes alike. Healthy surviving voters can serve during a rebuild while the learner remains unready. Maintenance also requires a complete raft_maintenance report and certain maintenance authority from metrics. ursulactl wait-ready checks both the metrics and this existing readiness endpoint. ursulactl status presents leadership fields across the cluster.

Admin endpoints

These are the primitives ursulactl and any custom operator tooling builds on. They live on the admin plane (server.admin_listen, loopback 127.0.0.1:4438 by default), not the public client plane. Nodes expose no cluster-mutation surface on the network. Reach them over a tunnel to the admin port (see ursulactl's operation providers). The examples below assume a forward from local 4438 to the node's admin plane. Each call is local to one node. To act on every group, loop over the IDs in metrics.

Every admin mutation requires x-ursula-process-incarnation, pinned from GET /__ursula/metrics before the operation. Missing identity returns 428; a restarted target returns 412 before the handler changes state. Keep the same identity for the entire plan and stop on either response. This precondition is public instance identity, not authentication or an exclusive maintenance reservation. Updated ursulactl attaches it automatically and refuses to refresh it during an operation.

# Observe once for this operation; do not refresh after a 412.
process_incarnation=$(curl --fail http://127.0.0.1:4438/__ursula/metrics | jq -er .process_incarnation)

# Force a cold flush for one stream (skip the timer)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/flush-cold/demo/hello

# Trigger a Raft snapshot for one group
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/42/snapshot

# Purge log entries below the last snapshot index for one group
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/42/purge

# Add a learner (non-voting replica) to one group without blocking on catch-up
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/42/learners/4?addr=http://node-4:4437&blocking=false"

# Hand leadership of one group to another voter (used by `ursulactl drain`)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/42/leader/transfer/2

# Replace the voter set of one group (call on the group's current leader)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/42/membership?voters=1,2,3"

# Put this node into maintenance drain / lift it again
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/leadership-shed/maintenance
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X DELETE http://127.0.0.1:4438/__ursula/leadership-shed/maintenance

# Accept the loss of one group's unsynced tail on this node, naming the log its metrics show (see Recovering after a host crash)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d '{"expected_last_log_index": 1234, "expected_current_term": 7}' \
  -X POST http://127.0.0.1:4438/__ursula/raft/42/recovery/accept-unsynced-loss

The snapshot endpoint answers once the group has a snapshot of everything it had applied when the request arrived. It waits while a build of the group runs or other snapshot builds hold all of the node's build permits, and answers 503 with Retry-After: 1 when no build of the group can start within 10 seconds (see Manually trigger a snapshot).

The leader-transfer endpoint refuses with 409 Conflict if the receiving node isn't the current leader and 400 Bad Request if the target isn't a voter. This is why ursulactl is the safer way to chain these calls: it consults metrics first.

The group leader keeps every group at its configured voters: if the membership endpoint drops a configured voter, the leader adds it back as a learner and promotes it again once it has caught up (see Recovering after a host crash). Membership changes are for repair, not for shrinking a group.

The membership endpoint has the same leader-only contract: it answers 409 Conflict (with the current leader's id in the body) when the receiving node does not lead the group, and 400 Bad Request without a non-empty voters query parameter. On success it returns the log index at which the new voter set was committed. It replaces the whole voter set in one call, so include every node that should remain a voter.

Maintenance drain

POST /__ursula/leadership-shed/maintenance marks the node as draining: it stops campaigning and accepting transfers, and the background leadership balancer migrates its current leaderships to eligible peers one transfer per tick until the mark is lifted with DELETE. Both verbs return the node's leadership-shed state so tooling can confirm the transition took effect:

{
  "bits": 1,
  "state": "maintenance-drain",
  "should_accept_transfer": false,
  "should_campaign": false,
  "should_shed_current_leaders": true
}

ursulactl drain wraps exactly this pair: mark first so the node stops re-acquiring groups, then transfer leaderships away explicitly instead of waiting for the balancer. The mark stays set until ursulactl undrain clears it. If a rollout ever dies mid-way, a node can be left refusing leadership. Run ursulactl undrain (or DELETE the mark directly) to recover. Peers read each other's shed state (GET /__ursula/leadership-shed on the cluster-plane listener) when deciding which nodes are eligible to pick up leaderships, so a drained node stops attracting groups cluster-wide, not just locally.

Graceful voter shutdown

On the first SIGTERM or Ctrl-C, the voter marks itself maintenance-drained to prevent re-acquiring leadership. It keeps Raft transport available while attempting to hand off each leadership to an eligible voter (see the handoff rule under Recovering after a host crash). It waits for observed leadership changes for up to five seconds, then closes Raft transport and drains HTTP listeners for up to ten seconds. A connection still open after that, such as one from a peer that lost power, is abandoned. It then stops its Raft groups, fsyncs every journal and records a clean shutdown, so the next start reads the journals strictly and the node holds every entry it acknowledged. The whole shutdown has a deadline of twenty seconds from the first signal. If it has not finished by then, the process exits without recording a clean shutdown, and the next start treats the run as a crash. A second signal also exits at once without recording a clean shutdown. The chart gives voter pods thirty seconds by default. This reduces planned-stop election pauses. It cannot hand off leadership on SIGKILL or OOM.

runtime.node_memory_abort_cap_size is an emergency process-abort threshold, not HTTP admission control. Exceeding it calls abort() without graceful leadership handoff, like an OOM kill. It does not return 503. Leave room below the container memory limit for normal operation and recovery, and use write admission and backpressure to manage load.

Restarting a voter

Restart one voter at a time:

1. ursulactl drain --node N moves every leadership away and keeps the node from taking one back.
2. Restart the process through the platform (Helm, systemd). The graceful shutdown above keeps every journal intact.
3. ursulactl wait --node N returns once the node is a voter of every group again and has caught up.
4. ursulactl undrain --node N lets it hold leaderships again.

Do not stop another voter before wait returns. Two overlapping outages of a three-voter group leave it without a quorum even if the restarts were issued separately. The chart's graceful-rollout hook runs the same sequence for every pod.

WAL routing configuration

A WAL root is bound to runtime.core_count and raft.group_count in <raft.wal.path>/raft-log/topology.bin. Startup holds the WAL lock and checks both values before recovering any group or writing the run state. A mismatch, missing record on an existing WAL, or corrupt record refuses startup.

Keep both values explicit and unchanged when moving a persistent volume or resizing its host. Without explicit values, core_count follows the process's available parallelism and group_count defaults to that parallelism times 16 (resource presets set a fixed group count). Helm defaults to 4 and 64. Changing cores changes the journal directory of some groups. Changing group count changes stream routing. Neither is an online resharding operation.

For a mismatch, configure the stored counts again. The error reports them. Do not manufacture a record from the new configuration or remove the check's files. A replica whose record is missing or corrupt can be rebuilt on an empty WAL directory from a healthy quorum, with the cluster's routing counts. Follow the rejoin procedure below and wait for full catch-up before replacing another replica. A single node has no quorum to rebuild from, so a fresh WAL does not preserve its data.

Rejoining a wiped node

A voter whose WAL directory was wiped or replaced, for example after a lost disk, starts with an empty log for every group. For each group:

• It does not initialize the group again. A per-group membership initializer (raft.init_membership_per_group = true, the chart default) runs Initialize only when every other configured voter answers that it holds nothing for the group either and no leader is seen. A fresh install still bootstraps this way. One voter holding entries or following a leader means the group exists, and the node waits to be replicated to. It waits for every voter, not a quorum, so a fresh install with a voter that never comes up stays uninitialized (it logs "waits for every voter before initializing").
• It refuses replication until the leader proves a quorum without it. The node lost its vote with its disk, so it cannot tell the current leader from a stale one. It refuses appends, heartbeats, snapshots and votes until the group's leader confirms a fresh ReadIndex barrier with a quorum of the other voters. The node then records that leader's vote as its vote floor and accepts replication. In a 3-voter group that quorum needs both other voters up. While the wiped voter waits and another voter is down, the group has no quorum and cannot commit writes.
• It stays behind the recovery gate. It does not campaign and refuses every vote once it knows the group holds entries, until it has applied a fresh barrier from the group's leader (see Recovering after a host crash).
• The leader rebuilds it. The leader sees the node answer that it lacks entries it had acknowledged: it commits a voter set without the node, adds it back as a learner, waits until it has replicated the commit index and promotes it. Each step is read off the current membership, so a leader change or another restart of the node converge. Leadership handoffs (the balancer, the commit-stall watchdog) skip the node until then.

While the other voters are up, no operator step is needed. ursulactl wait --node N returns once every group has promoted the node, and ursulactl verify-cluster shows it unready until then. Do not stop another voter before that.

If a majority of a group's voters lost their disks at once, the voters that kept their disks are not a quorum. No leader can prove a quorum to the wiped replicas, so they never obtain a vote floor, and accept-unsynced-loss cannot replace lost voting history. The group stays stopped. Restore the latest verified backup into a fresh cluster (see Backup and disaster recovery). If every voter lost its disk, the bootstrap probe sees only empty voters and initializes the groups again, empty: every acknowledged write is lost, and cold objects written before stay in object storage, unreferenced. Start such a cluster over on a new object-storage prefix, as for any fresh install.

Recovering after a host crash

With the default raft.wal.fsync = "never", a node acknowledges ordinary data appends from the page cache. Membership batches are always fsynced before acknowledgment, so a full-cluster power loss cannot erase the voter configuration. A process crash loses nothing. A host crash or power loss, or a journal I/O failure, can cost the node the unsynced tail of its journals. The node detects it on the next start from its run state (wal_recovery in GET /__ursula/metrics, the ursula.wal.recovering gauge) and keeps the frames that still verify. A node whose run state file is missing while its journals hold records is treated the same way. Every group the node held enters the recovery gate, which is recorded in the node's metadata and survives further restarts:

• The node does not campaign, takes no leadership transfer and grants no vote. A node that led the group starts as a follower. It still accepts appends from a leader of its current term or a later one.
• It asks the group's leader for a fresh ReadIndex barrier, applies the log up to the barrier's index and then opens the gate. Until then readiness answers 503 with reason recovery_gate_closed, and recovery_gates in GET /__ursula/metrics lists the gated groups.
• The leader sees that the node lost entries it had acknowledged and rebuilds it: it removes the voter, adds it back as a learner and promotes it once it caught up. The removal needs a quorum of voters that kept their log and acknowledged the leader within the last 1.5 seconds (the minimum election timeout), counting the leader. Without such a quorum no removal can commit, so the leader resends its own log to the voters that lost entries instead.

Leadership handoffs (draining, balancing, a node that sheds its leaderships, the commit-stall watchdog, the admin transfer endpoint) go only to a voter that kept its log, acknowledged the leader within the last 1.5 seconds and holds every committed entry. A handoff whose target has not taken over within 6 seconds (two maximum election timeouts) is abandoned: the leader steps down and campaigns again.

While at most a minority of a group's voters lost their tail, the other voters hold every acknowledged write and nothing is lost. Wait for every group to open the gate on a node before you stop another voter.

A majority of a group's voters lost their tail at once. If the group's leader kept its log, it stays leader and its barrier opens the gated replicas as above. Otherwise the gated voters refuse every vote, so the group elects no leader and writes answer 503. After 30s without a leader barrier and without applying anything, each gated replica reports its group stalled: readiness reason recovery_stalled, the group under recovery_stalled_groups in GET /__ursula/ready and under recovery_gates.stalled in the metrics, the ursula.raft.recovery_gates{status="stalled"} gauge, and an error log naming the group. Writes acknowledged after the last fsync of the surviving replicas may be gone. Only an operator can accept that. For each stalled group:

1. On every voter, read the group's gate under recovery_gates.gated in GET /__ursula/metrics (a group missing there is open), and its last_log_term, last_log_index and current_term under raft_groups. The call below succeeds only on a replica that reports the group stalled.
2. Rank the stalled replicas by last log id, as a Raft election does: highest last_log_term first, then highest last_log_index. An empty log (null) ranks last. On the first, run the call below through its admin listener. Its body names the last_log_index and current_term you read on that replica. Repeat down the ranking until the replicas whose gate is open form a majority of the voters. A replica that was not gated counts as open.
3. The group elects a leader among the open replicas, the one with the most up-to-date log, and the remaining gated replicas catch up from it and open their gates.

Stop once the open replicas form a majority, even if the election takes a while. A replica that was not gated may hold the longest log, and then only it can win. Accepting the loss on one more gated replica can let two replicas with shorter logs elect each other and discard the writes only that replica holds.

A voter that was never gated does not report recovery_stalled. Its readiness answers 503 with raft_replica_unready, and raft_maintenance.group_issues shows leader_unknown for the group.

Replicas with the same last log id (last_log_term and last_log_index) hold identical logs, so either one may go first. Raft's log matching property guarantees it: two logs that hold an entry with the same term and index are identical up to that entry.

# group 42 on this node
metrics=$(curl --fail http://127.0.0.1:4438/__ursula/metrics)
process_incarnation=$(jq -er .process_incarnation <<<"${metrics}")
expected=$(jq -ec '.raft_groups[] | select(.raft_group_id == 42)
  | {expected_last_log_index: .last_log_index, expected_current_term: .current_term}' <<<"${metrics}")
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -H "content-type: application/json" \
  -d "${expected}" -X POST http://127.0.0.1:4438/__ursula/raft/42/recovery/accept-unsynced-loss

It answers {"raft_group_id": 42, "node_id": 1, "outcome": "gate_opened", "last_log_index": 1234, "current_term": 7}, or "already_open" when the gate was open. The open gate is recorded, so a restart does not close it again. A replica that led the group before the crash rejoins as a follower, also when it restarts after the call.

The call is refused with 409 Conflict, and changes nothing, when:

• the replica lost its vote and has not established a durable vote floor. Restore connectivity to an intact quorum so it can obtain a fresh barrier. Accepting unsynced data loss cannot authorize a replica to acknowledge an unknown leader.
• the replica's gate is closed but not stalled. It may still open through a leader's barrier without losing anything. Wait for recovery_stalled.
• the replica no longer holds the log the body names, because it moved since you read the metrics. Read the metrics again and decide again.

Many stalled groups at once. A rack power loss can stall many groups together. The script below runs the steps above for every stalled group from one machine, through a forward to each voter's admin plane (see ursulactl's operation providers). It skips a group while one of its voters does not answer or one of its gated replicas is not stalled yet. Review plan.jsonl before the last step, and run the script again until no group is stalled.

# One admin URL per voter, through your forwards.
admins=(http://127.0.0.1:5441 http://127.0.0.1:5442 http://127.0.0.1:5443)

# 1. Observe every replica of every group once.
for admin in "${admins[@]}"; do
  curl -sS --fail "${admin}/__ursula/metrics" | jq -c --arg admin "${admin}" '
    .process_incarnation as $incarnation | (.recovery_gates.gated // {}) as $gated
    | .raft_groups[]
    | {admin: $admin, process_incarnation: $incarnation, raft_group_id, node_id,
       voters: (.voter_ids | length), gate: ($gated[.raft_group_id | tostring] // "open"),
       last_log_term, expected_last_log_index: .last_log_index,
       expected_current_term: .current_term}'
done > replicas.jsonl

# 2. For each group with a stalled replica, plan the stalled replicas to open,
#    highest last log id first (last_log_term, then last_log_index), until the open
#    replicas form a majority.
jq -sc 'group_by(.raft_group_id)[]
  | select(length == .[0].voters and any(.gate == "stalled")
      and all(.gate == "open" or .gate == "stalled"))
  | ((.[0].voters / 2 | floor) + 1 - (map(select(.gate == "open")) | length)) as $needed
  | map(select(.gate == "stalled"))
  | sort_by([.last_log_term // -1, .expected_last_log_index // -1]) | reverse
  | .[:([$needed, 0] | max)][]' replicas.jsonl > plan.jsonl

# 3. Accept the loss on each planned replica. Stop at the first refusal and observe again.
while read -r replica; do
  curl -sS --fail -w '\n' -H "x-ursula-process-incarnation: $(jq -r .process_incarnation <<<"${replica}")" \
    -H "content-type: application/json" \
    -d "$(jq -c '{expected_last_log_index, expected_current_term}' <<<"${replica}")" \
    -X POST "$(jq -r '"\(.admin)/__ursula/raft/\(.raft_group_id)/recovery/accept-unsynced-loss"' <<<"${replica}")" || break
done < plan.jsonl

With fsync = "always", acknowledged appends survive a host crash, but the newest journal segment can still hold bytes written after the last fsync: an unacknowledged batch, or a commit or truncate marker. Recovery refuses damage in sealed segments. In the newest segment it removes an incomplete final frame without gating anything. A complete frame that fails verification, such as a hole, a zero-filled tail or a torn sector inside an unacknowledged batch, looks the same as media corruption of an acknowledged write. Recovery therefore records the recovery gate of every group on that core before it cuts the frame, and those groups rejoin as described above.

A single-voter group has no other replica to recover from, so it needs this call after every host crash under never, once its gate reports the group stalled. Under always it needs the call only after a host crash that leaves a complete invalid frame in the newest segment. So does a group whose voters lose power together, for example in a rack power loss, when that gates a majority of them. No acknowledged write is lost in these cases, but only an operator can open the gates.

POST /__ursula/cluster-probe also exists on the client plane, but it is machinery rather than an operator surface: nodes send each other heartbeat-sized bodies through it to verify cluster-plane egress, and it bypasses ingress admission.

Memory without retention

Per-stream replicated state, on every replica, is at most about 32 KiB plus bounded producer state, beyond the unflushed payload itself, without retention. It grows neither with the number of messages nor with the length of the cold history. The bounds that make this hold are always on: a 1,024-receipt producer receipt window per stream with 7-day idle-producer expiry and a 4,096-producer cap, unique stream incarnations with incarnation-scoped cold objects, external payload locators committed first and indexed after, and a compact hot window with no per-message bookkeeping, so cold admission and the flush thresholds charge hot payload only. Snapshot bodies of at least runtime.external_payload_min_size go to the cold tier when a cold backend is configured (up to 1 GiB). Retention stays available as a storage-cost tool, but it is not the way to bound memory.

The per-group gauges under group_state_gauges in /__ursula/metrics show what each group holds.

Usage export

GET /__ursula/usage returns committed counters and current gauges from local replicated state, aggregated by bucket across this node's Raft groups:

{
  "version": 1,
  "write_unit_bytes": 10240,
  "buckets": {
    "tenant-a": {
      "committed_append_bytes": 24576,
      "committed_records": 3,
      "committed_write_units": 4,
      "retained_bytes": 24576,
      "stream_count": 1
    }
  }
}

The raw byte and record counters are pricing-neutral facts. committed_write_units is a convenience derived meter: every committed, non-deduplicated create or append contributes at least one unit and rounds its canonical payload up independently by write_unit_bytes. Consumers must validate both version and write_unit_bytes before interpreting it; the unit is data in the contract, not encoded into the counter name. The same unit is persisted in Raft snapshots, and a build refuses to restore a snapshot carrying a different unit. Changing it therefore requires a new contract version and an explicit state migration, never a configuration edit. Ursula does not attach currency, a price, or an account owner.

Counters are monotonic and survive snapshots, stream deletion, and bucket purge. retained_bytes and stream_count are current-state gauges and may decrease. A node serves local applied state, so replicated consumers should poll every voter and use their own completeness policy rather than treating one response as a cluster-wide linearizable read.

Tenant offboarding (bucket purge)

DELETE /__ursula/purge/{bucket} erases one tenant's complete bucket domain: every stream in every Raft group, the bucket, cold chunks and indexes, external payloads, and any uncommitted object-store staging orphan below {bucket}/. The replicated purge also installs a permanent bucket-erasure fence in every group. It retains only the bucket's aggregate usage counters, because deleting a counter before an asynchronous meter observes it would erase committed billable work. Purge is served only on the admin listener (server.admin_listen, loopback 127.0.0.1:4438 by default); the client listener answers 404. On Kubernetes the chart keeps the admin listener on pod loopback, so reach it through kubectl port-forward pod/<voter-pod> 4438:4438 or kubectl exec; any one voter accepts the request for the whole cluster.

The response is a completion report an operator can attest to:

curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X DELETE http://127.0.0.1:4438/__ursula/purge/tenant-a
# {"bucket":"tenant-a","removed_streams":12,"groups_with_streams":[0,3,5],"cold_gc_entries_reclaimed":12,"cold_gc_pending_entries":0,"cold_gc_complete":true,"cold_gc_error":null,"bucket_prefix_absent":true}

Semantics:

• Retry on 503. A Raft group whose leader is unknown or changing answers 503 with Retry-After, never a redirect to a peer. Because purge is idempotent, retry the same request.
• Idempotent and resumable. Re-purging an erased bucket returns the same report shape with zero removals. A purge interrupted mid-way converges on re-run: the permanent fence and stream removal are replicated commands per group, while cold reclamation and the complete bucket-prefix erasure are repeatable object-store operations.
• Permanently fenced. Commands linearized after purge cannot create the bucket or append to its streams. The fence survives Raft snapshots and restore; data-plane access continues to conceal the erased namespace as 404 Not Found instead of recreating it.
• Isolated. Other buckets' streams, offsets, checksums, snapshots, and object-store prefixes are untouched. Every cold object, shared packs included, lives under its own bucket's {bucket}/ prefix.
• Accounting residue. The retained entry contains only aggregate counters and zero-valued content gauges, not stream data. Ursula does not currently provide an acknowledgement protocol for erasing it after an external meter persists the final delta. Use opaque bucket IDs if retaining the identifier itself would violate an erasure requirement.
• Physical absence proof. cold_gc_entries_reclaimed is diagnostic only. After a second replicated purge proves every group's bucket-specific GC queue is empty, Ursula recursively deletes the complete {bucket}/ erasure domain and lists the authoritative object store to prove it contains no files. Only cold_gc_complete=true, cold_gc_pending_entries=0, bucket_prefix_absent=true, and no error constitute the proof. If object storage is unavailable or still lists a file, the response remains incomplete and a later idempotent request retries the proof.
• Backups. A purged tenant may persist inside backups taken before the purge until those backups age out; restore replays whatever the manifest contains. Deleting old backup sets is part of completing an offboarding with erasure obligations.

Cleaning S3

Cold data lives under the storage.cold.root prefix. Benchmark and test runs should use a date-stamped root so cleanup is one prefix delete with standard S3 tooling:

aws s3 rm "s3://my-ursula-bucket/ursula-test-20260518T000000Z" --recursive

There is no automatic retention policy. Application checkpoints and explicit stream-retention advances control which cold objects become eligible for GC. After a retention advance, each group's leader deletes the stream's cold objects that lie wholly below the retained offset (chunks and offloaded external payloads, in whole 64 MiB cold-index pages, so up to one page just below the retained offset can remain until retention moves past it) once a 5-minute grace has passed, so reads planned before the advance still complete; its cold-index cursor visits 16 streams per group per minute, so on groups with many streams collection can lag further. The grace restarts after a leader change. Retention is a storage-cost tool only: replica memory does not depend on it (see Memory without retention).

Backup and disaster recovery

ursulactl provides a verifiable whole-cluster backup and restore workflow:

# Create: one MessagePack snapshot object per raft group + manifest.json,
# into a local directory or s3://bucket/prefix.
ursulactl backup-create --config cluster.yaml --location s3://backups/ursula/2026-07-25

# Verify: manifest completeness, byte sizes, BLAKE3 checksums, and deep
# state-machine validation of every snapshot. Touches no cluster.
ursulactl backup-verify --location s3://backups/ursula/2026-07-25

# Restore: into a FRESH, EMPTY cluster with the same raft group count.
ursulactl restore --config new-cluster.yaml --location s3://backups/ursula/2026-07-25

An s3:// location uses the AWS SDK credential chain and AWS_REGION. For MinIO or another S3-compatible store, also set AWS_ENDPOINT_URL, for example AWS_ENDPOINT_URL=http://minio.internal:9000.

The recovery contract:

• Consistency boundary. Each group export is the deterministic state-machine snapshot the raft snapshot path persists: internally consistent per group while writes continue. Cross-group consistency is not promised. The boundary is per stream. Acknowledged writes in the exporting replica's applied state are included whether or not they were cold-flushed, so RPO is bounded by export time, not by the cold-flush interval. backup-create asks every node and keeps the reply with the highest group commit index.
• Fresh identity on restore. Restore replays each snapshot as one replicated write on the target cluster, which keeps its own raft membership and log identity. Nothing from the source cluster's raft metadata is reused. Groups that already hold buckets refuse the import with 409, and group-count mismatches fail closed before the first import. Group commit indexes and per-stream append counters restart under the new identity. Stream bytes, offsets, close/TTL state, published snapshots, retention floors, and producer dedup states are preserved exactly.
• Cold objects are part of the backup set. Snapshots reference cold-tier chunks, cold-index pages and external payloads by object key. The backup holds the references, not the objects. The restored cluster's cold store must hold the same objects under the same keys: point it at a copy (see Copying cold objects). backup-verify validates snapshots but does not dereference cold objects. restore asks the target cluster, before the first import, to check every group's references against its own cold store (POST /__ursula/backup/group/{group}/cold-check). If any object is missing, it stops with nothing imported and prints how many are missing and the first keys.
• Format compatibility. The backup format is the format epoch: format_version 3 in the manifest and in /__ursula/backup/info, and format_epoch 3 in every group object. A backup restores only into a cluster of the format epoch that wrote it. ursulactl refuses a manifest or a target cluster of another version before the first import, and a server refuses to import a group of another format epoch (400).
• RTO is dominated by transferring and importing the group snapshots. Each import is a single replicated write per group.

The recovery drill runs in CI (backup_restore_drill_preserves_streams_and_allows_continued_appends): write, snapshot, retain and close, back up, destroy the cluster, restore, verify checksums, offsets and boundaries, and continue appending.

What you can additionally rely on:

• Quorum durability. Acknowledged writes survive as long as a majority of voters survives. Three voters across AZs tolerate any single-AZ outage.
• Cold-tier durability. Once flushed to S3, a chunk inherits S3-grade durability. The unflushed window is bounded by the flush interval (seconds by default).
• Format epoch 3, no migration. The Raft WAL, snapshots, backups, the object-storage prefix and the Raft gRPC protocol carry format epoch 3. A node refuses data, backups and peers of another format epoch and does not migrate them.

Node-level loss: replace the host with the same node_id and cluster config. Each group's leader rebuilds the replacement from the other voters (see Rejoining a wiped node). ursulactl wait --node N returns once the replacement is a voter of every group and has caught up, and ursulactl verify-cluster then requires every configured voter to be present and caught up. ursulactl wait-ready checks the whole cluster: leaders for every group, complete voter sets and 200 from every node's /__ursula/ready. Run it before declaring the recovery done. Total-cluster loss: restore the latest verified backup into a fresh cluster as above.

Copying cold objects

A restored cluster reads cold data under the keys the source cluster wrote, relative to storage.cold.root. Copy the source's objects into the target's root after the target has started and written its URSULA_FORMAT_EPOCH marker, and before restore. Skip two things:

• URSULA_FORMAT_EPOCH, the marker. The target's own marker stays.
• snapshots/, the source's Raft snapshots and their references. The target writes its own there. With a custom storage.snapshot.s3_prefix inside the cold root, skip that prefix instead.

With the AWS CLI, from the source root to the target root:

aws s3 sync s3://my-bucket/source/cold/ s3://my-bucket/target/cold/ \
  --exclude 'URSULA_FORMAT_EPOCH' --exclude 'snapshots/*'

For MinIO or another S3-compatible store, add --endpoint-url. The copy must match the backup, so the source must not change its cold objects in between. Stopping writes is not enough: an idle cluster still compacts small cold chunks and deletes the ones it replaced a few minutes later. If the source cluster still runs, stop its nodes once the backup verifies, then copy. restore checks that the copy is complete.

Never point two clusters at one prefix, and never point a new cluster at the source's prefix in place. After a restore both clusters hold the same streams under the same keys, but each one deletes cold objects by its own state alone: the cold GC after a delete, a retention advance or a compaction, the orphan sweep for unreferenced objects older than a day, and the pruning of snapshots/ by the cluster's own references. Two clusters on one prefix delete objects the other still reads. A copy also leaves the source cluster and its prefix as they were, so the source stays a way back until clients switch.

Upgrading to 0.7

0.7.0 is a breaking release. It changes the Raft WAL format (format epoch 3) and removes the memory WAL, so the per-core disk journal is the only Raft WAL. 0.7 cannot read 0.6 data: a node refuses 0.6 Raft WAL directories and object-storage prefixes at startup, ursulactl restore refuses 0.6 backups, and a node will not join a cluster with 0.6 nodes. There is no in-place upgrade, no automatic migration and no rollback.

• If you run 0.6 with data you need to keep, stay on 0.6.2. 0.7 provides no migration path from 0.6.
• Deploy 0.7 fresh: empty PVCs and a never-used object-storage prefix.
• helm upgrade from 0.6 stops before a second voter is replaced. Recover with helm rollback.

Every behavior change is listed in the 0.7.0 release notes.

0.7.6 upgrades 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4 or 0.7.5 in place with an ordinary rolling restart (helm upgrade), with no fresh install. See the 0.7.6 release notes.

For a patch upgrade, upgrade gateways before nodes, and roll back nodes before gateways, so that clients keep the benefit of the x-ursula-not-applied marker throughout. The order is a recommendation, not a safety requirement. A client trusts the marker only together with the vouch that gateways from 0.7.3 on add, so whatever order a helm upgrade or helm rollback replaces gateways and voters in, no client counts a committed write as never applied. See refusals that applied nothing.

To install 0.7:

1. Install fresh. Install with a new s3.prefix and empty PVCs. With cold storage off and S3 snapshots, also use a new snapshotStore.prefix. With the event-time indexer, use a new or emptied indexer.s3.prefix and register the indexes again.
2. Never reuse a prefix that an earlier release, or a failed upgrade attempt, wrote to.
3. Update the configuration.
   • Remove raft.wal.backend, raft.wal.allow_volatile_multi_peer, raft.memory_bootstrap_marker_dir and raft.rejoin_probe. A config file that still sets one fails to load.
   • Set raft.wal.path on every node of a multi-node cluster. 0.6 defaulted to the memory WAL, and a 0.7 node with raft.peers refuses to start without a WAL path.
   • Choose raft.wal.fsync (Helm raft.walFsync). The default never acknowledges a write once a quorum of voters holds it in the page cache, and a host crash can cost a replica its unsynced tail (see Recovering after a host crash). Set always to keep the 0.6 disk WAL behavior of one fsync per batch.
   • With Helm, remove raft.storageMode, raft.allowVolatileMultiPeer and persistence.enabled. Every pod gets a PVC for its WAL. The values schema refuses unknown keys under raft and persistence.
   • Keep runtime.core_count and raft.group_count explicit and unchanged (see WAL routing configuration).
4. Update runbooks and tooling. ursulactl 0.7 removes the memory-WAL restart verbs prepare-restart, restart-quiesce-capability, finish-prepared-restart, abort-prepared-restart, prepare-amnesiac-restart, repair-restarted-voter, prepare-recovery-handoff and classify-amnesiac. A planned restart is drain, restart, wait and undrain (Restarting a voter), and a voter that lost entries is rebuilt by its group leaders. Custom tooling that calls admin mutations must send x-ursula-process-incarnation (Admin endpoints).

A values file that still sets a removed chart value fails validation, and nothing changes. Otherwise helm upgrade from 0.6 replaces the first pod, with the default RollingUpdate strategy and also with OnDelete plus the graceful rollout. That pod finds the 0.6 data on its volume and exits at startup, so no second voter is replaced. Recover with helm rollback. With OnDelete, also delete the refused pod after the rollback so that it starts from the 0.6 template again. The refused pod wrote nothing, so the old pods are unaffected.

How a 0.7 node refuses data of another release, and what it prints, is listed under Startup failures. Every startup refusal happens before the node writes anything:

• Raft WAL directory. {raft.wal.path}/raft-log/FORMAT_EPOCH records the epoch. A directory that holds data without the marker, or a marker of another epoch, is refused. The marker is written after every check passes.
• Object storage. URSULA_FORMAT_EPOCH sits at storage.cold.root when the cold backend is S3, otherwise at the S3 snapshot namespace (storage.snapshot.s3_prefix) when snapshots live on S3. A namespace that holds objects without the marker is refused, so it must be used only by this cluster. With a memory cold store and inline snapshots there is no object-storage check.
• Peers. Before it writes the markers, a node with configured peers asks each one, in parallel (2 seconds to connect plus 2 seconds for the answer, about 4 seconds at most), whether it speaks Raft protocol 3. A peer that refuses ends startup. An unreachable peer is skipped, because a fresh cluster starts its pods in parallel.
• Readiness. Any later Raft protocol mismatch, inbound or outbound, is logged at error level and makes /__ursula/ready answer 503 with "reason": "format_epoch_mismatch" until the node restarts. This also holds for a 0.6 node that a 0.7 node contacts, including through the startup probe. Keep the peer lists of 0.6 and 0.7 nodes apart.
• Backups. Backups use format version 3. ursulactl restore and backup-create refuse a cluster that reports another version, and a server refuses to import a group of another format epoch (400).

Bucket IDs snapshots and any ID starting with __ are reserved (400): snapshots is the default raft-snapshot namespace in the object store.

Logs

RUST_LOG=ursula=info,ursula_runtime=info,ursula_raft=info is the baseline. Bump to debug for one crate when chasing a subsystem:

RUST_LOG=ursula_raft=debug ./target/release/ursula server ...

debug is verbose under sustained load, so redirect to a file.

Committed application failures

A stopped group requires corrected-code replay. Retain the original WAL and referenced snapshot objects. Never skip a committed record to recover readiness.