ursulactl

ursulactl manages the logical state of a running cluster over Ursula's admin and metrics HTTP APIs: which node leads which Raft groups, and whether a node is caught up. It executes nothing on hosts. Physical lifecycle belongs to whatever owns the process: Helm and the StatefulSet controller on Kubernetes, systemd on hosts, OpenTofu for the infrastructure underneath. The verbs encode the safety properties an operator otherwise has to remember manually:

• before a node goes down, transfer every Raft group it leads to a healthy successor (drain)
• after it comes back, refuse to move on until last_applied_index has caught up to peers' committed_index (wait)
• abort rather than corner a group with no leader

A safe rolling restart wraps the platform's restart in these verbs, one node at a time. On Kubernetes:

ursulactl drain --config manifest.json --node 3
kubectl delete pod ursula-2        # the platform restarts the pod
ursulactl wait --config manifest.json --node 3
ursulactl undrain --config manifest.json --node 3
ursulactl verify-cluster --config manifest.json

On bare metal the restart in the middle is systemctl restart ursula on the host. The drain planning and readiness logic is exercised under deterministic simulation.

Verbs

Verb	Effect
drain --node N	Mark the node draining and transfer away every leadership it holds. The mark persists (the node attracts no leaderships) until undrain. --dry-run prints the transfer plan
undrain --node N	Clear the drain mark so the node may hold leaderships again
wait --node N	Block until the node is a voter in every group and within --lag-tolerance of peers. Progress-gated: a node that keeps advancing is never timed out
pin-incarnations	Save required process identities in a manifest for subsequent commands
status	Per-node group counts and leadership distribution
wait-ready	Wait for complete maintenance evidence and local serving health on every node
verify-cluster	Require every configured voter to be present and caught up in two consecutive samples before the next rollout step
verify-quorum	Confirm every configured group with a fresh ReadIndex round and require every replica to apply its fixed prefix under the same leader term
verify-survivors	Confirm fresh prefixes on two observed survivors while preserving the three-voter configuration; does not certify full redundancy or physical fencing
backup-create	Export every Raft group into a backup at a local directory or s3://bucket/prefix
backup-verify	Check a backup's manifest, sizes, checksums and every group snapshot without contacting a cluster
restore	Restore a verified backup into a fresh, empty cluster with the same group count, once the cluster confirms it holds every cold object the backup references
recover-group --group G	Explain the recovery plan for one stalled group and send nothing: the floor, why it cuts or not, each replica's expected log and voter sets, and the voters that did not answer. With --accept-loss, accept the unsynced loss: plan the group from every reachable voter, with a majority of every voter set a replica reports, send the plan to its leader first and then to the other voters with the leader's plan term, and wait until its history event applied on every replica in the leader's membership, naming the replicas outside it. It stops when a replica leads with a live quorum and no plan was sent. It waits while a gate may still open without a plan, and once it sent a plan, while a replica leads with a live quorum or follows a live leader it did not reach. It never plans again while a replica leads above the last plan's term. A plan that wedges in two observations in a row, or that its leader refuses, or that replicas refuse so that it cannot win, is planned again, up to --max-plans. The leader's refusal that no plan fixes stops it. A member that refuses so is left out of the next plan. A leader_not_voter or recovery_point_behind refusal plans again from a new read. It refuses a plan below a committed or applied index any replica it reached reports, unless a voter's log holds that entry, in which case the plan cuts to it. When the leader's answer is lost, it reads the leader's recorded plan and goes on. --accept-timeout-secs bounds one call. See Recovering after a host crash

Mutating verbs exit 0 on success and 2 on an abort (drain timeout, no safe transfer target, catch-up stall). Anything else is a configuration or transport error with a single-line, machine-greppable message.

Process identity across commands

Each CLI invocation pins the first observed server instance. A process change makes subsequent observations fail, and mutations retain the original x-ursula-process-incarnation header. To preserve the same plan across invocations:

ursulactl pin-incarnations --config cluster.json > pinned-cluster.json
ursulactl drain --config pinned-cluster.json --node 3

The saved nodes carry expected_process_incarnation. After an admitted Pod replacement, pin-incarnations --replace-node 3 can bind that one replacement. Every survivor must still match its saved identity. The Helm rollout persists these pins and the admitted source/replacement Pod UIDs in state schema 3. Resuming the same replacement validates its saved process instead of refreshing it after a container restart. It stops on a missing saved plan, changed survivor, or another unadmitted replacement.

pin-incarnations requires a process incarnation from every node, including an explicitly selected replacement. Nodes that do not report one are refused. verify-quorum records the identity of every observed process, and reservation steps compare that map with the saved plan. Instance identity does not serialize concurrent executors or prove a failed host is physically fenced.

Maintenance executor admission

A reviewed cell reservation consumer can add one immutable maintenance_fence object at the manifest root: reservation_id and executor_id are 32 lowercase hexadecimal characters, and generation is a nonzero cell-wide monotonically increasing integer. Every saved node carries that same token as expected_maintenance_fence; inconsistent or partially populated plans are rejected. The lifecycle commands require saved process identities for every configured voter:

ursulactl activate-maintenance-fence --config admitted-cluster.json
ursulactl drain --config admitted-cluster.json --node 3
# After replacement and reconciliation of its physical lifecycle:
ursulactl retire-maintenance-fence --config admitted-cluster.json
ursulactl verify-quorum --config admitted-cluster.json --expected-groups 256

These commands install or retire the supplied token; they do not acquire a shared reservation, choose generations, authorize Pod deletion, or establish provider-host fencing. A multi-node command may stop after partial acknowledgement. Keep the external operation reserved, resume its same identities, and require all relevant acknowledgements before advancing. Expiry is not authority to select another voter. A new executor uses the higher generation admitted by the external CAS operation; CLI never adopts a token discovered in metrics or refreshes it after 412/428.

Activation waits for admitted mutation handlers and then confirms prior messages were processed by each captured local Raft API queue. Caller cancellation leaves its admitted handler running; abnormal handler termination latches that process closed. Failed or cancelled activation/retirement stays closed until the same pending lifecycle completes or an externally admitted higher generation activates. Retirement retains its generation and cannot be reopened by the same token. This confirms submission ordering, not completion of asynchronous replication, snapshot work, or physical operations. Fresh all-replica prefix certification and external lifecycle reconciliation remain required before releasing the reservation.

New processes start unclaimed, explicitly uncertified until admission. An explicit pin-incarnations --replace-node can bind only that selected unclaimed replacement while preserving the token. Other voters must still match the active saved authority. Once activated, all mutating admin requests require x-ursula-maintenance-fence as reservation_id:generation:executor_id in addition to the process-incarnation header. Metrics report maintenance_fence and maintenance_fence_uncertain. verify-quorum schema 3 records maintenance_executor_certified only when every observed process matches the same caller-supplied active token. A subsequent proof can inspect the same retired token and reports maintenance_executor_retired_certified only when all observed processes have retired it. A mix of active and retired nodes certifies neither state. These flags certify process-local admission or retirement, not the external reservation or physical fencing. Existing chart and Cloud consumers are uncertified until their shared reservation integration is complete.

The admin plane

Nodes carry no cluster-mutation surface on the network. The mutating operator endpoints (raft snapshot/purge/membership/leader-transfer, maintenance drain, cold-flush trigger, bucket purge) plus metrics and usage are served on a separate admin plane bound to server.admin_listen, which defaults to loopback (127.0.0.1:4438). The public client plane (:4437) serves only stream traffic and read-only metrics and usage.

Metrics and readiness use an explicit metrics_url when configured, then client_url, then admin_url. The client and admin planes serve them. The Raft http_url is never used for them, since a cluster listener serves neither. The chart's rollout hook sets metrics_url to its Pod-bound admin tunnel so it can inspect a replacement before cluster DNS becomes reachable. Mutating verbs always use admin_url.

ursulactl is a plain HTTP client and opens no tunnels itself. When the admin plane is not directly reachable, bring your own forward and point the manifest's admin_url at it. On Kubernetes, kubectl port-forward reaches the loopback-bound plane inside each pod:

kubectl port-forward pod/ursula-0 5441:4438 &
kubectl port-forward pod/ursula-1 5442:4438 &
kubectl port-forward pod/ursula-2 5443:4438 &

[[nodes]]
id = 1
admin_url = "http://127.0.0.1:5441"

[[nodes]]
id = 2
admin_url = "http://127.0.0.1:5442"

[[nodes]]
id = 3
admin_url = "http://127.0.0.1:5443"

On bare metal the same shape works with ssh -N -L 5441:127.0.0.1:4438 admin@node1 per node.

When to use ursulactl vs. the other surfaces

Task	Tool
Day-2 logical operations: drain, observe, gate on readiness	ursulactl
Deployment, restarts, upgrades, topology	Helm (systemd on bare metal)
Custom operator tooling	The admin-plane HTTP endpoints, reached over your own tunnel

Install

Build from the workspace alongside the server:

cargo build --release -p ursula-ctl --bin ursulactl

The binary lands at target/release/ursulactl. Drop it on your control machine. It does not need to run on the Ursula hosts themselves.

Manifest format

Every verb accepts --config <path>. The manifest is TOML, JSON, or YAML (chosen by file extension, sniffed when read from stdin with -) and lists the cluster's nodes.

Prefer generating the manifest from whatever already knows the topology instead of writing it by hand. On Kubernetes the Helm chart renders one into its ConfigMap, and the URLs in it are in-cluster DNS, so pipe it to ursulactl running where those names resolve (the server image contains ursulactl):

kubectl get configmap ursula -o jsonpath='{.data.cluster-manifest\.json}' \
  | ursulactl status --config -

For infrastructure provisioned by OpenTofu, emit the manifest as a stack output or a generated file rather than teaching ursulactl to read state files. tofu output -json <name> | ursulactl status --config - composes the same way.

Per-node fields, all optional except id:

• admin_port (default 4438) or an explicit admin_url: the admin plane to reach, directly or through your forward.
• host: address shown in reports. Falls back to the admin URL's host.
• http_url: the node's Raft URL (its [[raft.peers]].url). It is the default host and part of a pinned plan. Nodes take each other's addresses from [[raft.peers]], never from it.
• client_url: the node's client-plane URL (its server.listen), used for metrics and readiness when no metrics_url is set. The chart's manifest sets it.
• metrics_url: optional metrics and readiness endpoint, for example the same tunnel as admin_url. It overrides that access without changing http_url.

Restarting a voter that lost its log

A voter that shut down cleanly comes back with every entry it acknowledged and only catches up on what it missed. A voter that lost entries (a host crash under raft.wal.fsync = "never", a journal I/O error, a wiped or replaced disk) needs no ursulactl verb either: it stays behind its recovery gate and stays a voter, and each group leader catches it up by itself (see Rejoining a wiped node). Keep the node drained and run wait until it caught up in every group. A group whose majority lost entries at once waits for an operator instead (see Recovering after a host crash).

The catch-up of a voter that lost its disk installs snapshots for every group and can take 10+ minutes. wait is progress-gated, so no timeout tuning is needed: a catch-up that keeps advancing is never timed out, and --stall-timeout-secs (default 90) only aborts a node that stops making progress.

status

Per-node summary of Raft group count and leadership distribution, sourced from every node's /__ursula/metrics. Nodes whose metrics fail are reported with metrics unavailable — … rather than aborting the report, because status is meant to surface partial cluster health.

ursulactl status --config cluster.json

Sample output:

node 1 (10.0.0.1): groups=4 leaders={1: 2, 2: 2}
node 2 (10.0.0.2): groups=4 leaders={1: 2, 2: 2}
node 3 (10.0.0.3): groups=4 leaders={1: 2, 2: 2}

leaders={…} is the count of groups each node is leading from this reporter's perspective. Healthy clusters report the same distribution from every node.

wait-ready

Wait until every node reports --expected-groups initialized Raft groups with leaders, a complete uniform voter set in raft_maintenance, and no uncertain maintenance fence. Each node must also return 200 from /__ursula/ready, read from the same address as its metrics.

ursulactl wait-ready --config cluster.json --expected-groups 4

Exits non-zero with the maintenance report or serving refusal when the timeout passes. Connection failures are retried until the timeout, so the wait can start while a node restarts. Every poll observes the processes afresh. Only a node whose process differs from its pinned expected_process_incarnation, or an invalid response, stops the wait early with its original cause. Healthy survivors can serve while a replica rebuilds, but wait-ready still rejects a second maintenance operation until full redundancy returns.

Underlying HTTP surface

For custom tooling, every verb maps onto a small set of HTTP endpoints on each node:

Verb	Endpoint
status, wait	GET /__ursula/metrics
wait-ready	GET /__ursula/metrics and GET /__ursula/ready
drain / undrain (maintenance guard)	POST /__ursula/leadership-shed/maintenance, DELETE to clear
drain (transfer step)	POST /__ursula/raft/{raft_group_id}/leader/transfer/{node_id}
backup-create	GET /__ursula/backup/info, GET /__ursula/backup/group/{raft_group_id}
restore	GET /__ursula/backup/info, POST /__ursula/backup/group/{raft_group_id}/cold-check, POST /__ursula/backup/group/{raft_group_id}/import

The transfer endpoint returns a typed rejection: 409 Conflict for a changed leader or a recovering target, 400 for an invalid target, 404 for an unregistered group, and 500 for a stopped Raft. Drain re-observes and re-plans the two 409 cases; other errors stop the operation.

Backup and restore

backup-create, backup-verify and restore implement the backup workflow in Backup and disaster recovery. A backup location is a local directory or s3://bucket/prefix.

ursulactl backup-create --config cluster.json --location s3://backups/ursula/2026-10-07
ursulactl backup-verify --location s3://backups/ursula/2026-10-07
ursulactl restore --config new-cluster.json --location s3://backups/ursula/2026-10-07

• backup-create writes one compact MessagePack snapshot per Raft group and manifest.json. It asks every node for each group and keeps the answer with the highest commit index. --created-unix-ms overrides the manifest timestamp.
• backup-verify checks that the manifest lists every group once, and each object's size, BLAKE3 checksum and snapshot. It reads only the backup.
• restore verifies the backup, refuses a target whose backup format or group count differs, and then asks the target to check every group's cold references against its cold store. Only when nothing is missing does it import the groups, one replicated write each. A target group that already holds data refuses the import. Restore into a fresh, empty cluster.

backup-create and restore take --http-timeout-secs (default 30). The cold check looks up every object a group references, 32 at a time, in one request per group, so a group with tens of thousands of cold objects needs a longer timeout. An s3:// location uses the AWS SDK credential chain and AWS_REGION. For MinIO or another S3-compatible store, also set AWS_ENDPOINT_URL.

The backup format is 5, the format of Ursula 0.9. It changes only when what a backup holds changes, not with every format epoch. ursulactl refuses a backup or a cluster of another backup format, such as a 0.8 backup (format 4).

Each command exits 0 on success and non-zero with an error that names the failed check. A refused restore imports nothing.

verify-quorum

This source provides a bounded observation for recovery measurements and maintenance tooling. Supply the deployed group count explicitly; the example below uses 256 groups:

ursulactl verify-quorum --config cluster.json --expected-groups 256 --timeout-secs 120

The command requires each configured voter exactly once, every configured group on every voter, complete uniform membership, no learners, and eligible local participation reports. It confirms each group's leader through a new outbound ReadIndex probe, then waits for all replicas to apply the captured prefix under that leader's term. Continuous writes can advance without moving this target. One absolute deadline covers metrics requests, RPCs and waits. The JSON output includes the observation start/end times, each group's leader/term/prefix and every replica's applied index.

The prefix is confirmed through the incarnation-bound admin route GET /__ursula/raft/{group}/quorum. The CLI uses shared admin protocol types and has no dependency on ursula-raft. Quorum proof and self-election requests require process identity; servers without the admin protocol are unsupported.

A successful observation is not a maintenance reservation, a continuously refreshed health claim or physical fencing permission. Independent maintenance workflows still need to serialize disruptions and bind replacement to the actual Pod/process/host incarnation.

verify-survivors

After one voter becomes unavailable, this source can separately observe the two survivors. Keep all three voters in the manifest and explicitly exclude exactly one known node:

ursulactl verify-survivors --config cluster.json --excluded-node-id 3 --expected-groups 256 --timeout-secs 120

Every group must still have the complete uniform three-voter membership, no learners and open participation gates on both observed replicas. Both survivors must agree on a reachable leader and apply each newly confirmed fixed prefix. The command rejects a second missing replica, an unknown exclusion or a changed topology. It never shrinks the configured membership to fit observations.

The output is scoped as surviving_quorum, records the configured and observed voter sets, and always reports full_redundancy_restored=false. Excluding a voter from observation does not prove its old process or host is stopped. Hard fencing and an exclusive maintenance reservation remain prerequisites for physical replacement, and the complete verify-quorum check is required before any subsequent disruption.

Shared reservation proposals

Reservation stores have one schema, version 1, for Pod replacement, host recovery and candidate restaging. Host inventory is optional data, and publishing it keeps the version. ursulactl refuses a store with any other version.

reservation-propose --cell cell.json --snapshot before.json --request request.json produces a whole-ConfigMap CAS proposal without contacting Kubernetes. reservation-acknowledge takes the same inputs plus --response response.json and validates the exact successful update receipt. Conflicts stop without refreshing authority, and missing stores do not initialize themselves.

The policy covers sticky, one-source planned Pod replacement, current-token fresh prefix admission and all-retired completion after the original Pod UID is gone. Its inputs are operational evidence, not authentication. The chart's maintenance-rollout.sh Job drives planned Pod replacement through it. No component drives host recovery or calls a provider to terminate an instance, and this offline interface alone grants no disruption authority. The full protocol is documented in docs/architecture/cluster-membership.md.