ursulactl

ursulactl manages the logical state of a running cluster over Ursula's admin and metrics HTTP APIs: which node leads which Raft groups, whether a node is caught up, and whether a leader will accept an amnesiac node back. It executes nothing on hosts. Physical lifecycle belongs to whatever owns the process: Helm and the StatefulSet controller on Kubernetes, systemd on hosts, OpenTofu for the infrastructure underneath. The verbs encode the safety properties an operator otherwise has to remember manually:

• before a node goes down, transfer every Raft group it leads to a healthy successor (drain)
• after it comes back, refuse to move on until last_applied_index has caught up to peers' committed_index (wait)
• abort rather than corner a group with no leader

A safe rolling restart wraps the platform's restart in these verbs, one node at a time. On Kubernetes:

ursulactl drain --config manifest.json --node 3
ursulactl prepare-restart --config manifest.json --node 3
kubectl delete pod ursula-2        # the platform restarts the pod
ursulactl wait --config manifest.json --node 3
ursulactl verify-cluster --config manifest.json
ursulactl undrain --config manifest.json --node 3

On bare metal the restart in the middle is systemctl restart ursula on the host. The drain planning and readiness logic is exercised under deterministic simulation.

Verbs

Verb	Effect
drain --node N	Mark the node draining and transfer away every leadership it holds. The mark persists (the node attracts no leaderships) until undrain. --dry-run prints the transfer plan
prepare-restart --node N	After drain, detect the cluster WAL backend and arm stable non-target leaders for one empty-log rejoin when memory WAL requires it. Disk-WAL clusters need no rejoin permission
undrain --node N	Clear the drain mark so the node may hold leaderships again
wait --node N	Block until the node is a voter in every group and within --lag-tolerance of peers. Progress-gated: a node that keeps advancing is never timed out
allow-rejoin --node N	Arm one empty-log rejoin per group for a raft-memory node that lost its volatile log. Refused on disk-backed clusters
status	Per-node group counts and leadership distribution
wait-ready	Block until every node reports the expected group count and every group has a leader
verify-cluster	Require every configured voter to be present and caught up in two consecutive samples before the next rollout step

Mutating verbs exit 0 on success and 2 on an abort (drain timeout, no safe transfer target, catch-up stall). Anything else is a configuration or transport error with a single-line, machine-greppable message.

The admin plane

Nodes carry no cluster-mutation surface on the network. The mutating operator endpoints (raft snapshot/purge/membership/learners/leader-transfer/allow-next-revert, maintenance drain, cold-flush trigger) plus metrics are served on a separate admin plane bound to server.admin_listen, which defaults to loopback (127.0.0.1:4438). The public client plane (:4437) serves only stream traffic and read-only metrics.

status and wait-ready prefer each node's http_url because read-only metrics are available on the client plane, and fall back to admin_url. Mutating verbs always use admin_url.

ursulactl is a plain HTTP client and opens no tunnels itself. When the admin plane is not directly reachable, bring your own forward and point the manifest's admin_url at it. On Kubernetes, kubectl port-forward reaches the loopback-bound plane inside each pod:

kubectl port-forward pod/ursula-0 5441:4438 &
kubectl port-forward pod/ursula-1 5442:4438 &
kubectl port-forward pod/ursula-2 5443:4438 &

[[nodes]]
id = 1
admin_url = "http://127.0.0.1:5441"

[[nodes]]
id = 2
admin_url = "http://127.0.0.1:5442"

[[nodes]]
id = 3
admin_url = "http://127.0.0.1:5443"

On bare metal the same shape works with ssh -N -L 5441:127.0.0.1:4438 admin@node1 per node.

When to use ursulactl vs. the other surfaces

Task	Tool
Day-2 logical operations: drain, observe, gate on readiness	ursulactl
Deployment, restarts, upgrades, topology	Helm and OpenTofu (systemd on bare metal)
Custom operator tooling	The admin-plane HTTP endpoints, reached over your own tunnel

Install

Build from the workspace alongside the server:

cargo build --release -p ursula-ctl --bin ursulactl

The binary lands at target/release/ursulactl. Drop it on your control machine. It does not need to run on the Ursula hosts themselves.

Manifest format

Every verb accepts --config <path>. The manifest is TOML, JSON, or YAML (chosen by file extension, sniffed when read from stdin with -) and lists the cluster's nodes.

Prefer generating the manifest from whatever already knows the topology instead of writing it by hand. On Kubernetes the Helm chart renders one into its ConfigMap, and the URLs in it are in-cluster DNS, so pipe it to ursulactl running where those names resolve (the server image contains ursulactl):

kubectl get configmap ursula -o jsonpath='{.data.cluster-manifest\.json}' \
  | ursulactl status --config -

For infrastructure provisioned by OpenTofu, emit the manifest as a stack output or generated file (the same pattern as deploy/eks's generated-values.yaml) rather than teaching ursulactl to read state files. tofu output -json <name> | ursulactl status --config - composes the same way.

Per-node fields, all optional except id:

• admin_port (default 4438) or an explicit admin_url: the admin plane to reach, directly or through your forward.
• host: address shown in reports. Falls back to the admin URL's host.
• http_url: optional client-plane URL used by status and wait-ready for read-only metrics. Metrics fall back to admin_url when omitted.

Restarting raft-memory nodes

On clusters running the volatile raft.wal.backend = "memory", a restarted node rejoins with an empty log, and group leaders refuse that log reversion unless it was explicitly permitted. Run prepare-restart --node N after the drain and before the restart: it detects the reported WAL backend and asks every group's stable leader to accept one empty-log rejoin from the target when memory WAL requires it. It fails closed when nodes omit or disagree on their backend. Disk-WAL clusters report that no permission is needed. allow-rejoin remains the explicit recovery verb, not the normal rolling-restart step.

The permission must land on the node that is leader at that moment, since the endpoint answers 409 Conflict from any other node (ursulactl resolves leadership per group and handles this). A node that is the membership initializer for some groups additionally refuses to start at all after losing its volatile log (the bootstrap-marker guard). That recovery is an explicit operator reset on the host, outside ursulactl's reach.

The rebuild after an amnesiac restart installs snapshots for every group and can take 10+ minutes. wait is progress-gated, so no timeout tuning is needed: a rebuild that keeps advancing is never timed out, and --stall-timeout-secs (default 90) only aborts a node that stops making progress.

status

Per-node summary of Raft group count and leadership distribution, sourced from every node's /__ursula/metrics. Nodes whose metrics fail are reported with metrics unavailable — … rather than aborting the report, because status is meant to surface partial cluster health.

ursulactl status --config cluster.json

Sample output:

node 1 (10.0.0.1): groups=4 leaders={1: 2, 2: 2}
node 2 (10.0.0.2): groups=4 leaders={1: 2, 2: 2}
node 3 (10.0.0.3): groups=4 leaders={1: 2, 2: 2}

leaders={…} is the count of groups each node is leading from this reporter's perspective. Healthy clusters report the same distribution from every node.

wait-ready

Block until every node reports --expected-groups Raft groups, each with a leader. Useful in CI / scripts after a deploy or a config change.

ursulactl wait-ready --config cluster.json --expected-groups 4

Exits non-zero with a one-line reason if the timeout passes (cluster not ready after 120s: node 3 has 1 group(s) without a leader).

Underlying HTTP surface

For custom tooling, every verb maps onto a small set of HTTP endpoints on each node:

Verb	Endpoint
status, wait-ready, wait	GET /__ursula/metrics
drain / undrain (maintenance guard)	POST /__ursula/leadership-shed/maintenance, DELETE to clear
drain (transfer step)	POST /__ursula/raft/{raft_group_id}/leader/transfer/{node_id}
allow-rejoin	POST /__ursula/raft/{raft_group_id}/nodes/{node_id}/allow-next-revert per group, on the group's leader

The transfer endpoint refuses with 409 Conflict if the receiving node isn't the current leader of the group, and 400 if the target node isn't a voter. ursulactl uses this to refuse to attempt a transfer it cannot reason about.