Operations

The first tool to reach for is ursulactl. It covers the raft-aware verbs operators run most often: drain, status, readiness gates. This page covers everything around it: the metrics shape ursulactl reads, the admin endpoints it (and your custom tooling) can call, and the operational policies ursulactl does not encode (backups, log levels). Deployment and process lifecycle belong to the platform, Helm and OpenTofu.

Tooling map

Surface	When to reach for it
ursulactl	Day-2 cluster ops over HTTP: drain, restart, status, wait-ready.
/__ursula/metrics and the /__ursula/raft/... admin endpoints	Custom tooling. ursulactl uses these underneath. The surface is small and stable enough to script directly.

There is no Prometheus scrape and no general-purpose orchestrator yet.

Metrics

curl http://127.0.0.1:4437/__ursula/metrics | jq .

The JSON snapshot covers per-core mailbox depth, append/read counters, latency histograms, per-group leader and last_applied, hot/cold bytes, cold-flush backlog, HTTP status counters, live-read watchers, and cold-write admission state. Start here when triaging slowness, lag, or 503s. ursulactl status is a friendlier read of the leadership-related fields across the whole cluster.

Admin endpoints

These are the primitives ursulactl and any custom operator tooling builds on. They live on the admin plane (server.admin_listen, loopback 127.0.0.1:4438 by default), not the public client plane. Nodes expose no cluster-mutation surface on the network. Reach them over a tunnel to the admin port (see ursulactl's operation providers). The examples below assume a forward from local 4438 to the node's admin plane. Each call is local to one node. To act on every group, loop over the IDs in metrics.

# Force a cold flush for one stream (skip the timer)
curl -X POST http://127.0.0.1:4438/__ursula/flush-cold/demo/hello

# Trigger a Raft snapshot for one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/snapshot

# Purge log entries below the last snapshot index for one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/purge

# Add a learner (non-voting replica) to one group
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/learners/4

# Hand leadership of one group to another voter (used by `ursulactl drain`)
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/leader/transfer/2

# Replace the voter set of one group (call on the group's current leader)
curl -X POST "http://127.0.0.1:4438/__ursula/raft/42/membership?voters=1,2,3"

# Put this node into maintenance drain / lift it again
curl -X POST   http://127.0.0.1:4438/__ursula/leadership-shed/maintenance
curl -X DELETE http://127.0.0.1:4438/__ursula/leadership-shed/maintenance

# Permit the leader to accept one log revert from a wiped, rejoining node
curl -X POST http://127.0.0.1:4438/__ursula/raft/42/nodes/4/allow-next-revert

The leader-transfer endpoint refuses with 409 Conflict if the receiving node isn't the current leader and 400 Bad Request if the target isn't a voter. This is why ursulactl is the safer way to chain these calls: it consults metrics first.

The membership endpoint has the same leader-only contract: it answers 409 Conflict (with the current leader's id in the body) when the receiving node does not lead the group, and 400 Bad Request without a non-empty voters query parameter. On success it returns the log index at which the new voter set was committed. It replaces the whole voter set in one call, so include every node that should remain a voter.

Maintenance drain

POST /__ursula/leadership-shed/maintenance marks the node as draining: it stops campaigning and accepting transfers, and the background leadership balancer migrates its current leaderships to eligible peers one transfer per tick until the mark is lifted with DELETE. Both verbs return the node's leadership-shed state so tooling can confirm the transition took effect:

{
  "bits": 1,
  "state": "maintenance-drain",
  "should_accept_transfer": false,
  "should_campaign": false,
  "should_shed_current_leaders": true
}

ursulactl drain wraps exactly this pair: mark first so the node stops re-acquiring groups, then transfer leaderships away explicitly instead of waiting for the balancer. The mark stays set until ursulactl undrain clears it. If a rollout ever dies mid-way, a node can be left refusing leadership. Run ursulactl undrain (or DELETE the mark directly) to recover. Peers read each other's shed state (GET /__ursula/leadership-shed on the cluster-plane listener) when deciding which nodes are eligible to pick up leaderships, so a drained node stops attracting groups cluster-wide, not just locally.

Rejoining a wiped node

A node that lost its Raft log (disk replaced, data dir wiped) rejoins with a shorter log than the leader has recorded for it, which Raft treats as a fault. POST /__ursula/raft/{group}/nodes/{node}/allow-next-revert tells the group's leader to accept exactly one such log revert from that node and re-replicate from scratch. It is a per-group, one-shot permission: loop over all group IDs when re-admitting a fully wiped node, and expect the leader to answer 500 if the underlying Raft rejects the request.

POST /__ursula/cluster-probe also exists on the client plane, but it is machinery rather than an operator surface: nodes send each other heartbeat-sized bodies through it to verify cluster-plane egress, and it bypasses ingress admission.

Usage export

GET /__ursula/usage returns committed counters and current gauges from local replicated state, aggregated by bucket across this node's Raft groups:

{
  "version": 1,
  "write_unit_bytes": 10240,
  "buckets": {
    "tenant-a": {
      "committed_append_bytes": 24576,
      "committed_records": 3,
      "committed_write_units": 4,
      "retained_bytes": 24576,
      "stream_count": 1
    }
  }
}

The raw byte and record counters are pricing-neutral facts. committed_write_units is a convenience derived meter: every committed, non-deduplicated create or append contributes at least one unit and rounds its canonical payload up independently by write_unit_bytes. Consumers must validate both version and write_unit_bytes before interpreting it; the unit is data in the contract, not encoded into the counter name. The same unit is persisted in Raft snapshots, and a build refuses to restore a snapshot carrying a different unit. Changing it therefore requires a new contract version and an explicit state migration, never a configuration edit. Ursula does not attach currency, a price, or an account owner.

Counters are monotonic and survive snapshots, stream deletion, and bucket purge. retained_bytes and stream_count are current-state gauges and may decrease. A node serves local applied state, so replicated consumers should poll every voter and use their own completeness policy rather than treating one response as a cluster-wide linearizable read.

Tenant offboarding (bucket purge)

DELETE /__ursula/purge/{bucket} (admin plane) erases one tenant's stream content: every stream in every Raft group, the bucket itself, and its quota, followed by one immediate cold-GC pass that reclaims the streams' cold-object prefixes. It retains only the bucket's aggregate usage counters, because deleting a counter before an asynchronous meter observes it would erase committed billable work. The response is a completion report an operator can attest to:

curl -X DELETE http://127.0.0.1:4438/__ursula/purge/tenant-a
# {"bucket":"tenant-a","removed_streams":12,"groups_with_streams":[0,3,5],"cold_gc_entries_reclaimed":12}

Semantics:

• Idempotent and resumable. Purging an absent bucket returns the same report shape with zero counts. A purge interrupted mid-way converges on re-run: stream removal is a replicated command per group, and cold reclamation is a list-then-delete over object prefixes, so nothing depends on in-memory progress.
• Isolated. Other buckets' streams, offsets, checksums, and snapshots are untouched; the purged bucket name conceals as not-found afterwards and may be recreated. A recreated bucket continues the same monotonic usage counters.
• Accounting residue. The retained entry contains only aggregate counters and zero-valued content gauges, not stream data. Ursula does not currently provide an acknowledgement protocol for erasing it after an external meter persists the final delta. Use opaque bucket IDs if retaining the identifier itself would violate an erasure requirement.
• Cold-GC interaction. If the immediate GC pass fails (for example S3 is briefly unavailable), the entries stay queued and the background GC worker finishes reclamation; the purge itself is already durable at that point.
• Backups. A purged tenant may persist inside backups taken before the purge until those backups age out; restore replays whatever the manifest contains. Deleting old backup sets is part of completing an offboarding with erasure obligations.

Cleaning S3

Cold data lives under the storage.cold.root prefix. Benchmark and test runs should use a date-stamped root so cleanup is one prefix delete with standard S3 tooling:

aws s3 rm "s3://my-ursula-bucket/ursula-test-20260518T000000Z" --recursive

There is no automatic retention policy. Application checkpoints and explicit stream-retention advances control which cold objects become eligible for GC.

Backup and disaster recovery

ursulactl provides a verifiable whole-cluster backup and restore workflow:

# Create: one MessagePack snapshot object per raft group + manifest.json,
# into a local directory or s3://bucket/prefix.
ursulactl backup-create --config cluster.yaml --location s3://backups/ursula/2026-07-25

# Verify: manifest completeness, byte sizes, BLAKE3 checksums, and deep
# state-machine validation of every snapshot. Touches no cluster.
ursulactl backup-verify --location s3://backups/ursula/2026-07-25

# Restore: into a FRESH, EMPTY cluster with the same raft group count.
ursulactl restore --config new-cluster.yaml --location s3://backups/ursula/2026-07-25

The recovery contract:

• Consistency boundary. Each group export is the deterministic state-machine snapshot the raft snapshot path persists: internally consistent per group while writes continue. Cross-group consistency is not promised; the boundary is per stream. Acknowledged writes in the exporting replica's applied state are included whether or not they were cold-flushed, so RPO is bounded by export time, not by the cold-flush interval. backup-create asks every node and keeps the reply with the highest group commit index.
• Fresh identity on restore. Restore replays each snapshot as one replicated write on the target cluster, which keeps its own raft membership and log identity — nothing from the source cluster's raft metadata is reused. Groups that already hold buckets refuse the import with 409; group-count mismatches fail closed before the first import. Group commit indexes and per-stream append counters restart under the new identity; stream bytes, offsets, record coordinates, attrs, close/TTL state, published snapshots, retention floors, and producer dedup states are preserved exactly.
• Cold objects are part of the backup set. Snapshots reference cold-tier chunks by object key; point the restored cluster at the same (or a copied) cold-store namespace. backup-verify validates snapshots but does not dereference cold objects.
• Format compatibility. The backup format is versioned (format_version in the manifest) independently from the server binary; tools refuse newer formats and servers refuse imports they cannot validate. Within v0.x, restore into the same minor version is the supported path.
• RTO is dominated by transferring and importing the group snapshots; each import is a single replicated write per group.

The recovery drill — write, snapshot, retain, close; back up; destroy the cluster; restore; verify checksums, coordinates, and boundaries; continue appending — runs in CI (backup_restore_drill_preserves_streams_and_allows_continued_appends).

What you can additionally rely on:

• Quorum durability. Acknowledged writes survive as long as a majority of voters survives. Three voters across AZs tolerate any single-AZ outage.
• Cold-tier durability. Once flushed to S3, a chunk inherits S3-grade durability. The unflushed window is bounded by the flush interval (seconds by default).
• No on-disk migration. v0.x does not promise stable on-disk formats. The runtime won't refuse to start on stale data, but it won't migrate either. Cross-version upgrades currently mean rebuild + replay from external truth.

Node-level loss: replace the host with the same node_id and cluster config, and it rehydrates from peers. Use ursulactl wait-ready afterwards to confirm the replacement is voting and caught up before declaring the recovery done. Total-cluster loss: restore the latest verified backup into a fresh cluster as above.

Logs

RUST_LOG=ursula=info,ursula_runtime=info,ursula_raft=info is the baseline. Bump to debug for one crate when chasing a subsystem:

RUST_LOG=ursula_raft=debug ./target/release/ursula server ...

debug is verbose under sustained load, so redirect to a file.