ursulactl

ursulactl manages the logical state of a running cluster over Ursula's admin and metrics HTTP APIs: which node leads which Raft groups, and whether a node is caught up. It executes nothing on hosts. Physical lifecycle belongs to whatever owns the process: Helm and the StatefulSet controller on Kubernetes, systemd on hosts, OpenTofu for the infrastructure underneath. The verbs encode the safety properties an operator otherwise has to remember manually:

• before a node goes down, transfer every Raft group it leads to a healthy successor (drain)
• after it comes back, refuse to move on until last_applied_index has caught up to peers' committed_index (wait)
• abort rather than corner a group with no leader
• change placement only from a view the meta group confirmed (placement, node)

A safe rolling restart wraps the platform's restart in these verbs, one node at a time. On Kubernetes:

ursulactl drain --config manifest.json --node 3
kubectl delete pod ursula-2        # the platform restarts the pod
ursulactl wait --config manifest.json --node 3
ursulactl undrain --config manifest.json --node 3
ursulactl verify-cluster --config manifest.json

On bare metal the restart in the middle is systemctl restart ursula on the host. The drain planning and readiness logic is exercised under deterministic simulation.

Verbs

Verb	Effect
drain --node N	Mark the node draining and transfer away every leadership it holds, the meta group's included. The mark lives in the process: the node attracts no leaderships until undrain or a restart clears it. --dry-run prints the transfer plan
undrain --node N	Clear the drain mark so the node may hold leaderships again
wait --node N	Block until the node is a voter, within --lag-tolerance of peers, in every group the meta leader's placement view places it in and every group its own maintenance report counts it a voter of. Progress-gated: a node that keeps advancing is never timed out
status	Per-node group counts and leadership distribution
wait-ready	Wait for ready maintenance reports and serving health on every node, and every group live on exactly its target voters in the meta leader's placement view. wait-ready, verify-cluster and drain skip the manifest nodes that view shows removed, which may already be stopped
verify-cluster	Require every group to run on exactly its target voters, each caught up, in two consecutive samples before the next rollout step
verify-quorum	Confirm every group of the meta leader's placement view with a fresh ReadIndex round and require every target voter to apply its fixed prefix under the same leader term
verify-survivors	Confirm fresh prefixes on every target voter but one excluded node, while each group keeps its whole target as its voters. Does not certify full redundancy or physical fencing
backup-create	Export every Raft group into a backup at a local directory or s3://bucket/prefix
backup-verify	Check a backup's manifest, sizes, checksums and every group snapshot without contacting a cluster
restore	Restore a verified backup into a fresh, empty cluster with the same group count, once the cluster confirms it holds every cold object the backup references
node replace-disk --node N --expect-root R	Release root R, the WAL disk node N is bound to in meta, once that disk is revoked, so the empty disk N runs on now binds with no restart. It prints N's slot from a view another node confirmed, refuses without --force when N's process still runs on R or reports ready, prints the release rule and sends the release to another node's admin plane. R is 16 lowercase hex digits, as the placement status, the root_refused registration and the first 16 digits of a process incarnation print it. A release cannot be undone. See Rejoining a wiped node
placement status [--node N | --group G] [--confirmed] [--json]	Print a node's placement view: its stage, the meta leader, each node's lifecycle and root, the move limit and the moving groups. For each moving group, or the group --group names, it asks every node the group moves from or to for its replica (GET /__ursula/placement/groups/{group}) and prints the leader's reconcile state, flagging a leader that pursues an older placement than the view's and a group no answering replica leads. --node N lists every group N has a role in. --confirmed reads a view a meta read barrier confirmed, on an admin plane. --json prints the report as JSON
placement move --group G [--group G2 ...] (--to 2,3,4 | --replace OLD=NEW)	Move groups to other voters, as many as each has. --replace OLD=NEW replaces node OLD in each group's target, so a list of groups fills a new node. It decides each group from a confirmed view with the meta state machine's own check: a group already stable at or moving to its target counts as done, so a rerun finishes an interrupted batch, a moving group keeps its target unless --retarget or --expect is given, and a group over the move limit queues until a move ends. Other refusals are reported per group. --expect SINCE (one group) names the placement key the move starts from, term-leader-index as the status prints it, in place of the view's, so a moving group named by its key takes the new target, and a stale key is reported with the current placement. --expect and --retarget exclude each other. A node the move adds must be listed live in the meta leader's report (leader_report of GET /__ursula/placement on the meta leader): it acknowledged the meta leader within 10 seconds and does not shed for its WAL disk. Without a reachable report, or for a node it does not list live, the group is refused unless --allow-not-live is given, and a group moved onto a node that is down then waits until it is back. After a 504 or a lost answer it reads a confirmed view and reports whether the request took effect. --wait waits until every group is stable at its target and prints each leader's state as it changes. --timeout-secs bounds queueing and waiting. It exits 2 when a group was refused or timed out
node cordon --node N [--node M ...], node uncordon --node N [--node M ...]	Cordon nodes, so they keep their replicas and take no new one, or return cordoned or decommissioning nodes to active, which aborts a decommission while the node still has a role. Each change is checked with the meta state machine's own rule on a confirmed view and sent until a confirmed view shows it. It exits 2 when a node was refused or --timeout-secs passed
node decommission --node N [--node M ...] [--wait]	Decommission nodes. It checks every named node first, so a meta voter among them is refused with nothing sent (a meta voter's host is replaced under its id, see Rejoining a wiped node). It cordons every active named node, so no group moves onto a node that leaves in the same scale-down, then decommissions each, and warns that the step is irreversible: the meta leader moves the node's groups away and removes it for good once it has no role. --wait waits until every node is removed, printing how many groups each still has a role in and the groups the meta leader cannot move. --timeout-secs, an hour by default, bounds the command. It exits 2 when a node was refused or the timeout passed
node verify-removed --node N [--node M ...], node verify-joinable --node N [--node M ...], node verify-vacant --node N [--node M ...]	Check against the meta leader's view, read over each node's read URL. verify-removed passes when every named node is removed, vacant or never registered, so it may leave, and fails before the cluster bootstrapped. verify-joinable passes when every named node may start: it never registered, it is vacant, so an empty disk registers as a new node, or it is in a life, so its own disk comes back. verify-vacant passes when every named node is vacant, so a disk left under its id holds a removed life's state. The last two wait up to --timeout-secs while a named node is removed and its removal has not completed, printing what it waits for. The chart's scale-check hook runs them. They exit 2 when a check fails, and verify-vacant exits 3 for a node in a life
placement cancel --group G [--expect SINCE] [--wait]	Move a moving group back to its source. It warns when a node of the source is decommissioning, since the decommission moves the group away again unless the node is uncordoned. It exits 2 when the group was refused or timed out
placement set-move-limit --limit N	Set how many groups may move at once. 0 stops new moves. After a 504 or a lost answer it reads a confirmed view and reports whether the limit took effect. It exits 2 when the limit was refused or did not take effect, and sending it again is safe
recover-group --target meta|G	Recover the meta group first (--target meta), then each data group (--target G): a data group's plan carries the floor the tool reads from a confirmed meta view (GET /__ursula/placement?confirmed=true) on the first node in the manifest that answers one: seeded for a group the view shows seeded, and for a live group certified with the group's converged_at, the membership entry its last convergence certified. Without a confirmed view it refuses and sends nothing. A plan under a certified floor keeps that entry: it is led by a voter of that membership whose log holds it, its last log id reaching the entry's term and index, and it is refused when no such voter answered. Explain the recovery plan for one stalled group and send nothing: the floor, why it cuts or not, each replica's expected log and voter sets, the replicas without a vote floor, and the voters that did not answer. A replica that lost its vote and holds no vote floor (needs_vote_floor in its metrics) refuses every plan, so the tool never counts it and sends it nothing. With --accept-loss, accept the unsynced loss: plan the group from every reachable voter, with a majority of every voter set a replica reports, send the plan to its leader first and then to the other voters with the leader's plan term, and wait until its history event applied on every replica in the leader's membership, naming the replicas outside it. It stops when a replica leads with a live quorum and no plan was sent. It waits while a gate may still open without a plan, and once it sent a plan, while a replica leads with a live quorum or follows a live leader it did not reach. It never plans again while a replica leads above the last plan's term. A plan that wedges in two observations in a row, or that its leader refuses, or that replicas refuse so that it cannot win, is planned again, up to --max-plans. The leader's refusal that no plan fixes stops it. A member that refuses so is left out of the next plan. A leader_not_voter or recovery_point_behind refusal plans again from a new read, and a leader's floor_not_held plans again without that replica. It refuses a plan below a committed or applied index any replica it reached reports, unless a voter's log holds that entry, in which case the plan cuts to it. When the leader's answer is lost, it reads the leader's recorded plan and goes on. --accept-timeout-secs bounds one call. See Recovering after a host crash

Mutating verbs exit 0 on success and 2 on an abort (drain timeout, no safe transfer target, catch-up stall, a placement move or cancel that left a group refused or timed out, a move limit refused or not set, a lifecycle change refused or timed out, a removal check that failed). Anything else is a configuration or transport error with a single-line, machine-greppable message.

Process identity across commands

Each invocation remembers the first server instance it observes on each node. A later observation of another instance on that node fails, and every mutation carries the remembered x-ursula-process-incarnation, so the server refuses it with 412 once the node restarted. A command therefore stops when a node restarts under it. Run each step of a plan as its own invocation, with ursulactl wait between steps. verify-quorum records the incarnation of every process it observed.

The admin plane

Nodes carry no cluster-mutation surface on the network. The mutating operator endpoints (raft snapshot/purge/leader-transfer, placement moves and the move limit, maintenance drain, cold-flush trigger, bucket purge) plus metrics and usage are served on a separate admin plane bound to server.admin_listen, which defaults to loopback (127.0.0.1:4438). The public client plane (:4437) serves only stream traffic and read-only metrics and usage.

Metrics and readiness use an explicit metrics_url when configured, then client_url, then admin_url. The client and admin planes serve them. The Raft http_url is never used for them, since a cluster listener serves neither. Set metrics_url to reach a node's metrics through a separate forward, for example to inspect a replacement before its cluster DNS name resolves. Mutating verbs always use admin_url.

ursulactl is a plain HTTP client and opens no tunnels itself. When the admin plane is not directly reachable, bring your own forward and point the manifest's admin_url at it. On Kubernetes, kubectl port-forward reaches the loopback-bound plane inside each pod:

kubectl port-forward pod/ursula-0 5441:4438 &
kubectl port-forward pod/ursula-1 5442:4438 &
kubectl port-forward pod/ursula-2 5443:4438 &

[[nodes]]
id = 1
admin_url = "http://127.0.0.1:5441"

[[nodes]]
id = 2
admin_url = "http://127.0.0.1:5442"

[[nodes]]
id = 3
admin_url = "http://127.0.0.1:5443"

On bare metal the same shape works with ssh -N -L 5441:127.0.0.1:4438 admin@node1 per node.

When to use ursulactl vs. the other surfaces

Task	Tool
Day-2 logical operations: drain, observe, gate on readiness	ursulactl
Deployment, restarts, upgrades, topology	Helm (systemd on bare metal)
Custom operator tooling	The admin-plane HTTP endpoints, reached over your own tunnel

Install

Build from the workspace alongside the server:

cargo build --release -p ursula-ctl --bin ursulactl

The binary lands at target/release/ursulactl. Drop it on your control machine. It does not need to run on the Ursula hosts themselves.

Manifest format

Every verb accepts --config <path>. The manifest is TOML, JSON, or YAML (chosen by file extension, sniffed when read from stdin with -) and lists the cluster's nodes.

Prefer generating the manifest from whatever already knows the topology instead of writing it by hand. On Kubernetes the Helm chart renders one into its ConfigMap, and the URLs in it are in-cluster DNS, so pipe it to ursulactl running where those names resolve (the server image contains ursulactl):

kubectl get configmap ursula -o jsonpath='{.data.cluster-manifest\.json}' \
  | ursulactl status --config -

For infrastructure provisioned by OpenTofu, emit the manifest as a stack output or a generated file rather than teaching ursulactl to read state files. tofu output -json <name> | ursulactl status --config - composes the same way.

Per-node fields, all optional except id:

• admin_port (default 4438) or an explicit admin_url: the admin plane to reach, directly or through your forward.
• host: address shown in reports. Falls back to the admin URL's host.
• http_url: the node's Raft URL (its [[raft.peers]].url). It is the default host. Nodes take each other's addresses from [[raft.peers]], never from it.
• client_url: the node's client-plane URL (its server.listen), used for metrics and readiness when no metrics_url is set. The chart's manifest sets it.
• metrics_url: optional metrics and readiness endpoint, for example the same tunnel as admin_url. It overrides that access without changing http_url.

Restarting a voter that lost its log

A voter that shut down cleanly comes back with every entry it acknowledged and only catches up on what it missed. A voter that lost entries (a host crash under raft.wal.fsync = "never", a journal I/O error) needs no ursulactl verb either: it stays behind its recovery gate and stays a voter, and each group leader catches it up by itself. A voter on a wiped or replaced disk needs one step first: once the old disk is revoked, node replace-disk releases it, and the node then rejoins the same way (see Rejoining a wiped node). Keep the node drained and run wait until it caught up in every group. A group whose majority lost entries at once waits for an operator instead (see Recovering after a host crash).

The catch-up of a voter that lost its disk installs snapshots for every group and can take 10+ minutes. wait is progress-gated, so no timeout tuning is needed: a catch-up that keeps advancing is never timed out, and --stall-timeout-secs (default 90) only aborts a node that stops making progress.

status

Per-node summary of Raft group count and leadership distribution, sourced from every node's /__ursula/metrics. Nodes whose metrics fail are reported with metrics unavailable — … rather than aborting the report, because status is meant to surface partial cluster health.

ursulactl status --config cluster.json

Sample output:

node 1 (10.0.0.1): groups=4 leaders={1: 2, 2: 2}
node 2 (10.0.0.2): groups=4 leaders={1: 2, 2: 2}
node 3 (10.0.0.3): groups=4 leaders={1: 2, 2: 2}

leaders={…} is the count of groups each node is leading from this reporter's perspective. Healthy clusters report the same distribution from every node.

wait-ready

Wait until every node's raft_maintenance report is ready and every node returns 200 from /__ursula/ready, and until the meta leader's placement view shows the cluster bootstrapped with every Raft group live, no move, and running on exactly its target voters: each target voter's replica reports the target as its voters, no learner, an open gate and a leader among them. The group count and the voters come from the view, so the command takes no group count.

ursulactl wait-ready --config cluster.json

Every check reads a node at the same address as its metrics. The first node that answers GET /__ursula/placement names the meta leader, and the leader's own status, naming itself, gives the view. A meta read barrier (?confirmed=true) is served on the admin listener only, which a chart's test pod does not reach. A view behind the meta log can only name a placement the replicas' memberships still show, and a group whose replicas report other voters keeps the wait going. wait, verify-cluster, verify-quorum and verify-survivors read the same view.

Exits non-zero when the timeout passes, with the refusal of the last poll: a maintenance report, a serving refusal, a view that names no meta leader or is not bootstrapped, or a group that is seeded, moves or does not run on its target voters. Connection failures and 503 answers are retried until the timeout, so the wait can start while a node restarts. Every poll observes the processes afresh, so a node may restart during the wait. Any other failed answer, such as a body that does not decode because ursulactl and the node are of different minors, stops the wait early with its original cause. Healthy survivors can serve while a replica rebuilds, but wait-ready still rejects a second maintenance operation until full redundancy returns.

Underlying HTTP surface

For custom tooling, every verb maps onto a small set of HTTP endpoints on each node:

Verb	Endpoint
status	GET /__ursula/metrics
wait	GET /__ursula/metrics and GET /__ursula/placement
wait-ready	GET /__ursula/metrics, GET /__ursula/ready and GET /__ursula/placement
drain / undrain (maintenance guard)	POST /__ursula/leadership-shed/maintenance, DELETE to clear
drain (transfer step)	POST /__ursula/raft/{target}/leader/transfer/{node_id}, target a group ID or meta
backup-create	GET /__ursula/backup/info, GET /__ursula/backup/group/{raft_group_id}
restore	GET /__ursula/backup/info, POST /__ursula/backup/group/{raft_group_id}/cold-check, POST /__ursula/backup/group/{raft_group_id}/import
placement status	GET /__ursula/placement, GET /__ursula/placement/groups/{group}
placement move, placement cancel	GET /__ursula/placement?confirmed=true, POST /__ursula/placement/groups/{group}/move
placement set-move-limit	POST /__ursula/placement/move-limit
node cordon, uncordon, decommission	GET /__ursula/placement?confirmed=true, POST /__ursula/placement/nodes/{node}/cordon, .../uncordon, .../decommission
node replace-disk	GET /__ursula/placement?confirmed=true, POST /__ursula/placement/nodes/{node}/replace-disk
recover-group	GET /__ursula/metrics, GET /__ursula/placement?confirmed=true, POST /__ursula/raft/{target}/recovery/accept-unsynced-loss

The transfer endpoint returns a typed rejection: 409 Conflict for a changed leader or an unready target, 400 for an invalid target, 404 for an unregistered group, and 500 for a stopped Raft. Drain observes again and plans again after either 409, and other errors stop the operation.

Backup and restore

backup-create, backup-verify and restore implement the backup workflow in Backup and disaster recovery. A backup location is a local directory or s3://bucket/prefix.

ursulactl backup-create --config cluster.json --location s3://backups/ursula/2026-10-07
ursulactl backup-verify --location s3://backups/ursula/2026-10-07
ursulactl restore --config new-cluster.json --location s3://backups/ursula/2026-10-07

• backup-create writes one compact MessagePack snapshot per Raft group and manifest.json. It asks every node for each group and keeps the answer with the highest commit index. --created-unix-ms overrides the manifest timestamp.
• backup-verify checks that the manifest lists every group once, and each object's size, BLAKE3 checksum and snapshot. It reads only the backup.
• restore verifies the backup, refuses a target whose backup format or group count differs, and then asks the target to check every group's cold references against its cold store. Only when nothing is missing does it import the groups, one replicated write each. A target group that already holds data refuses the import. Restore into a fresh, empty cluster.

backup-create and restore take --http-timeout-secs (default 30). The cold check looks up every object a group references, 32 at a time, in one request per group, so a group with tens of thousands of cold objects needs a longer timeout. An s3:// location uses the AWS SDK credential chain and AWS_REGION. For MinIO or another S3-compatible store, also set AWS_ENDPOINT_URL.

The backup format is 5, the format of Ursula 0.9. It changes only when what a backup holds changes, not with every format epoch. ursulactl refuses a backup or a cluster of another backup format, such as a 0.8 backup (format 4).

Each command exits 0 on success and non-zero with an error that names the failed check. A refused restore imports nothing.

verify-quorum

This source provides a bounded observation for recovery measurements and maintenance tooling. Every group and its target voters come from the meta leader's placement view, as for wait-ready:

ursulactl verify-quorum --config cluster.json --timeout-secs 120

The command requires every group to be live with no move, a ready maintenance report from every node it reaches, and on every target voter of each group a replica whose voters are exactly the target, with no learner and an eligible participation report. A target voter that gives no metrics refuses the observation. A node outside a group's target is not asked for that group. It confirms each group's leader through a new outbound ReadIndex probe, then waits for all replicas to apply the captured prefix under that leader's term. Continuous writes can advance without moving this target. One absolute deadline covers metrics requests, RPCs and waits. The JSON output includes the observation start/end times, each group's leader/term/prefix and every replica's applied index.

The prefix is confirmed through the incarnation-bound admin route GET /__ursula/raft/{target}/quorum. The CLI uses shared admin protocol types and has no dependency on ursula-raft. Quorum proof and self-election requests require process identity; servers without the admin protocol are unsupported.

A successful observation holds for the moment it was taken. It is not a continuously refreshed health claim. Disrupt one voter at a time and observe again before the next disruption.

verify-survivors

After one node becomes unavailable, this source can separately observe the others. Keep every node in the manifest and explicitly exclude exactly one of them:

ursulactl verify-survivors --config cluster.json --excluded-node-id 3 --timeout-secs 120

Every group must still have its whole target as its voters, with no learners, and open participation gates on every observed target voter, and the observed target voters of each group must form a quorum of its target. They must agree on a reachable leader and apply each newly confirmed fixed prefix. The command rejects a second missing target voter or an exclusion outside the manifest. It never shrinks a group's target to fit observations.

The output is scoped as surviving_quorum, records the excluded node and the observed nodes, and always reports full_redundancy_restored=false. Excluding a node from observation does not prove its old process or host is stopped. Run the complete verify-quorum check before any subsequent disruption.