Troubleshooting

Start at /__ursula/metrics. Most operational symptoms have a clear signal in the JSON snapshot.

Diagnostic surface

curl http://NODE:4437/__ursula/metrics > metrics.json

Useful jq selectors. Arrays named per_core_* are indexed by core, and arrays named per_group_* by Raft group ID.

# Leader, term and applied index of each group. current_leader is null while the group has no leader.
jq '.raft_groups[] | {raft_group_id, current_leader, current_term, last_applied_index}' metrics.json
# Groups without a leader
jq '[.raft_groups[] | select(.current_leader == null) | .raft_group_id]' metrics.json
# Hot bytes waiting for cold flush, per group
jq '.per_group_cold_hot_bytes' metrics.json
# Writes refused by cold-write backpressure, in total and per group
jq '{total: .cold_backpressure_events, per_group: .per_group_cold_backpressure_events}' metrics.json
# Mailbox depth and capacity, per core
jq '{depths: .mailbox_depths, capacities: .mailbox_capacities}' metrics.json
# Live reads waiting for new data, per core
jq '.per_core_live_read_waiters' metrics.json
# SSE streams opened and SSE error events
jq '{opened: .sse_streams_opened, errors: .sse_error_events}' metrics.json
# WAL disk pressure
jq '{available: .wal_available_bytes, minimum: .wal_min_available_bytes, pressure: .wal_disk_pressure}' metrics.json
# WAL reclaim and restart work
jq '{physical: .wal_physical_bytes, segments: .wal_segments, reclaims: .wal_reclaims, lagging: .wal_lagging_groups, recovered_bytes: .wal_recovery_bytes, recovery_ns: .wal_recovery_ns}' metrics.json
# How the WAL opened, and the groups behind a recovery gate
jq '{opening: .wal_recovery, gates: .recovery_gates}' metrics.json
# Local Raft issues that make the node unready or block maintenance
jq '.raft_maintenance | {node_issues, group_issues}' metrics.json
# This node's cluster id, and Raft traffic refused for another one
jq '{cluster_id, refusals: .raft_grpc_peer_cluster_refusals, unidentified: .raft_grpc_unidentified_peer_refusals, peers: .peer_cluster_mismatches}' metrics.json

GET /__ursula/ready is the process readiness probe. It answers 200 when the node may serve, and otherwise 503 with a typed reason. When several apply, it reports the first of:

reason	Meaning
superseded	Another process may answer as this node: the meta group released this process's WAL root or binds its node ID to another root, or a peer saw a newer run of the root. The old disk of a decommissioned node whose ID is vacant reports it too. The process takes part in nothing until it exits. Stop it, and never start that disk again (Rejoining a wiped node, Cordon and decommission)
disk_lease_lost	No probe of the WAL disk synced within raft.wal.lease. The node takes part in no Raft traffic and answers clients 503 until a probe succeeds (The disk lease)
root_refused	The meta group binds this node ID to another WAL root than the new one the node runs on. The node takes part in no group until ursulactl node replace-disk releases the old root (Rejoining a wiped node)
removal_pending	This node ID is decommissioned and its removal has not completed, and the node runs on a new disk. The node takes part in no group and binds with no restart once the meta leader marks the ID vacant. leader_report.removal_pending on the meta leader names what the removal waits for, often a group without a leader or the old process still running (Cordon and decommission)
predecessor_wait	A process of an earlier run of this root may still run, after a host crash, a reboot, a remount or a lost run state, or the root is a new disk that just joined. The node grants no vote and opens no recovery gate for 12 seconds
format_epoch_mismatch	The node exchanged a Raft RPC with a peer of another format epoch. It stays unready until it restarts
wal_disk_pressure	The WAL filesystem is below raft.wal.min_available_size, or its free-space check failed
recovery_stalled	A group's recovery gate on this node is stalled on a replica that counts as a voter. The groups are listed under recovery_stalled_groups
recovery_gate_closed	A group's recovery gate on this node is closed on a replica that counts as a voter, until the replica has applied a fresh barrier from the group's leader
raft_replica_unready	A local replica is not a healthy voter, or the node does not know its placement. raft_maintenance.node_issues and group_issues name the issue (Maintenance report)

Ursula does not expose /healthz, /readyz, Prometheus /metrics, or /cluster/status. Use the JSON snapshot for detailed diagnosis and ursulactl wait-ready for cluster-wide Raft readiness.

Maintenance report

raft_maintenance (schema version 2) checks this node's replicas against its placement view and their effective memberships. placement names the view's stage, the applied meta entry, the node's meta role and the meta leader it knows, its lifecycle and the last registration answer. expected_groups lists the groups in which the bootstrapped view gives the node a role, with the role, the placement key, the target voters and whether the replica counts as a voter. unplaced lists the replicas the node holds without a role, with whether each counts as a voter and last_answer, what a live leader of the group last answered this node's membership asker about it (voter, learner or not_member, null before an answer), and stage, how far the node's teardown of the replica went (ask while it asks the group whether the node is still a member, fence from the first not_member answer on, while the replica closes and the node asks again before it retires the log, null while no teardown runs). A node with no group is ready.

serving_ready decides /__ursula/ready, and ready decides whether a second disruption is safe (ursulactl wait, wait-ready and the quorum checks). A replica that does not count as a voter, such as an incoming learner or a replica its group answered is no voter, is checked only for raft_stopped and apply_stopped. Neither blocks anything while expected_groups or unplaced lists the replica. Before a bootstrapped view and while meta is damaged the report lists no replica, so a stopped replica blocks as a voter's would: raft_stopped blocks both, and apply_stopped blocks maintenance. Node issues:

Issue	Serving	Maintenance	Meaning
placement_unknown	blocks	blocks	The node does not know whether it is a voter: it confirmed no view in this run, holds no replica, its meta replica holds no bootstrapped state and no registration was answered. A wiped voter reports it until meta answers. A Raft node without a meta runtime reports it too
placement_not_bootstrapped		blocks	The node holds replicas and its view has not bootstrapped. The replicas keep their voter checks
genesis_mismatch		blocks	A peer's genesis claim differs from this node's
meta_unavailable			The node's meta replica has known no leader for 30 s. Restarts need no meta
meta_damaged			The node runs no meta replica in this run. It replaces every issue the view gives, and the replicas keep their voter checks
group_count_mismatch	blocks	blocks	raft.group_count differs from the cluster's
replication_factor_mismatch		blocks	raft.placement.replication_factor, capped at the [[raft.peers]] count, differs from the cluster's. A cluster that formed with fewer nodes than the factor recorded its node count: set the factor to that on every node before it grows (Adding a node)
node_removed			The view shows the node removed, or the meta group refused its registration as removed. It replaces every other node issue. The node has no expected group, so each replica it still runs is unplaced_replica and keeps its voter checks. Stop the node (Cordon and decommission)

Group issues against the placement: missing_group (a voter or incoming role and no replica, blocking serving for a voter), unplaced_replica (a replica held without a role), incomplete_voter_set (the membership is not the target voters with no learner), unplaced_member, move_in_progress (the view shows a move of the group, reported for every role in it, a replica that does not count as a voter included) and move_stalled (on the leader of a moving group whose reconciler made no progress for five minutes) block maintenance. The voter checks against the effective membership, such as local_replica_not_voter, recovery_barrier, membership_not_applied or apply_lag, block both. So does leader_not_acknowledged: since it opened in this run, the replica has not accepted an append that took its log to its leader's commit index, and a leader's append reached it in the last 3 seconds or it opened less than 10 seconds ago. Once no leader contacts it for that long its group has no leader that could count it, and the issue ends. The checks that a group's quorum on other nodes decides, leader_unknown, leader_outside_voters and membership_uncommitted (the replica's newest membership entry is not committed), block maintenance only, so a meta outage or a group that lost its quorum elsewhere never makes a node unready.

Serving while a replica catches up

raft_replica_unready means local Raft participation is not safe for serving. Inspect raft_maintenance.group_issues for local_replica_not_voter, membership_not_applied, not_applied, apply_lag or leader_not_acknowledged. A restarted node reports leader_not_acknowledged until each of its replicas has caught up with its group's leader, so a rolling restart stops the next pod only once the restarted one counts toward its groups' quorums again. A wiped replica stays 503 while its recovery gate is closed and it catches up. It stays a voter throughout. Healthy surviving voters can return 200 during the repair. A voter leaving a joint membership also stays unready once any configuration excludes it. A replica whose own gate is open reports leader_unknown for a group that has no leader, for example a group stalled because a majority of its voters are gated, and the node stays ready.

A 200 response does not certify full redundancy. Keep maintenance blocked until ursulactl wait-ready succeeds. Alert separately on incomplete voter sets in the maintenance report, as described in Observability.

503 with APPLY_STOPPED from one group

A committed command failed to apply and stopped that group on this replica. The node stays ready and serves its other groups. Its maintenance report names the group with apply_stopped, so ursulactl wait-ready refuses disruption until the group recovers. Follow Recovering stopped Raft groups and never skip the committed record.

503 with GROUP_RESTARTING from one group

The group's engine on this node did not open. There are two causes.

The snapshot store did not answer the group's restore. With an S3 snapshot store, opening a group reads the snapshot object its record names. When the store does not answer that read, the open fails after the whole restore attempt, and the node logs the persisted snapshot object does not restore at error level with the group and the object's key, and the snapshot store did not answer the group's restore at warn level with how many opens failed in a row. The node then holds the group closed for a wait that doubles per failed open, from 1 second up to 30 seconds, and opens the group again by itself. During the wait and during that reopen, it answers the group's requests with this code at once. open_backoff_groups in /__ursula/metrics counts the groups that wait. An open runs in a task of the core that owns the group, so the core's other groups keep serving. A request that reaches the group during its first open waits for the open, which the store can hold until the S3 client gives up, up to 22 seconds for each request with the default storage.snapshot.s3_timeout and s3_max_retries. Fix the node's reachability of the snapshot store: its endpoint, credentials and network path. The group opens at the first open after the store answers. A node whose store does not answer at startup starts anyway and keeps those groups closed until then.

The previous engine is still stopping. The group's engine on this node is stopping or restarting, and the previous engine's Raft work has not ended yet. The node opens the group only once that work ends, because it could still change the group's state. A request that arrives meanwhile is refused at once, so the code shows up briefly during any restart of the group. Retry the request. If the answer persists for more than 30 seconds, the previous engine is stuck. Its shutdown logged EngineWorkOutlivedShutdown at error level, with the group and how many of the engine's state-machine workers, snapshot builders, log readers, network clients and snapshot installs still ran. Collect the node's logs and restart the node.

Startup failures

WAL topology mismatch or missing topology

topology.bin binds a WAL root to runtime.core_count and raft.group_count. For a mismatch, restore the stored counts reported in the error. A missing record on an existing WAL or a corrupt record also refuses startup. Do not replace it with the current configuration. See WAL routing configuration for rebuilding a replica.

raft.wal.path is required when raft.peers lists another node

Every node of a cluster keeps its Raft WAL in raft.wal.path. Set it to a directory on a persistent volume. Only a single node may omit it, and then its WAL lives in a temporary directory that is removed on clean shutdown.

server.cluster_id is required when raft.wal.path is set

Every node with raft.wal.path, a single node included, names its deployment with server.cluster_id. Set the same value on every node, and a different one for every deployment a client can reach. With the Helm chart, set server.clusterId. The chart refuses to render without it. See Configuration.

the WAL at … belongs to cluster …

The node's WAL records the cluster id of its first start, and the configured server.cluster_id differs. A disk does not move between clusters. Set server.cluster_id to the stored id the error names, or start the node on an empty WAL and let it rejoin as a wiped node.

raft.node_id must be non-zero

Every Raft node needs a unique non-zero raft.node_id in the config file or a --node-id override.

raft.peers must include this node id

The local raft.node_id must appear in [[raft.peers]]. The peer list always includes self, and every node with raft.wal.path lists itself, a single node included.

raft.group_count is required when raft.peers lists another node

Every node of a cluster runs the same groups, so a node that lists another peer sets raft.group_count instead of the default of its own host. Set the same value on every node. A value above 4,096 is refused with raft.group_count (…) must be at most 4096.

raft.meta_voters names node N, which is not in raft.peers

The meta group's voters are nodes of [[raft.peers]]. List the node there, or leave raft.meta_voters unset for the smallest three node IDs.

raft.wal.lease (…) must be from 2s to 10s

Set raft.wal.lease within the range, or leave it unset for 5s.

the WAL at … belongs to node N, but raft.node_id is M

A WAL root opens only under the node ID that created it. The volume was moved to another node, or --node-id changed. Start node N on it, or start node M on an empty WAL. A node ID the cluster already knows on an empty WAL then waits for ursulactl node replace-disk (Rejoining a wiped node).

the WAL identity file is missing at … but prior WAL state exists

identity.bin counts the runs of a WAL root, so a root with prior state and no identity.bin cannot tell this process from an earlier one. Start the node on an empty WAL, and release its old root with ursulactl node replace-disk once the node reports root_refused. the WAL at … counted N runs names a root whose run counter is exhausted and is resolved the same way.

storage.cold.s3.bucket is required when cold backend is 's3'

Set storage.cold.s3.bucket, or set storage.cold.backend = "none" if you don't want cold storage.

Port already in use

server.listen, server.cluster_listen or server.admin_listen is taken. The client plane and inter-node gRPC share server.listen unless server.cluster_listen is set.

A node at a new address stays unready

Every node dials its peers at the url its own [[raft.peers]] lists. A node that moved, or a node whose peers still list its old address, gets no Raft traffic from the peers that list the wrong one. Its groups catch up on nothing, and ursulactl wait does not return. Check that [[raft.peers]] is the same on every node and names each node where it listens now, then restart the nodes whose list was wrong. A move of a listener, such as a new cluster port, needs every voter restarted together (Changing a peer address).

unplaced_member

The maintenance report names unplaced_member for a group whose Raft membership names a node that has no role in the group's placement, as a voter or a learner. No node dials a member that [[raft.peers]] does not list, and each node logs that once:

raft.peers lists no node 4, a member of a Raft group: not dialing it; list the node in raft.peers or remove it from the group's membership

The node keeps serving, but maintenance readiness fails, so ursulactl drain and wait stop. The group's leader removes such a learner once the group is seeded or its voters equal its placement, and such a voter only by a move or by leaving a joint configuration (Admin endpoints). List the node in [[raft.peers]] again if it should stay.

raft WAL directory … holds data without a format marker

The data directory was written by Ursula 0.5.x or earlier (format epoch 1). This release reads format epoch 7 only and has no in-place upgrade. Start on an empty raft.wal.path and a new object-storage prefix, or run the release that wrote it.

raft WAL directory … is format epoch N

The directory's FORMAT_EPOCH marker names another epoch. This release reads format epoch 7 only. Format epoch 6 is Ursula 0.10 and format epoch 5 is Ursula 0.9, whose data moves to this release by backup and restore (Upgrading to 0.11). Format epoch 4 is Ursula 0.8. Run the release that wrote it, or start on an empty directory.

object storage s3://… is not empty and has no URSULA_FORMAT_EPOCH marker

The cold root (or, with cold storage off, the S3 snapshot namespace) holds objects from Ursula 0.5.x or from another application. Point storage.cold.root (Helm: s3.prefix or coldStorage.prefix), or storage.snapshot.s3_prefix when only snapshots use S3, at an empty prefix used only by this cluster.

object storage s3://… is format epoch N

The prefix's URSULA_FORMAT_EPOCH marker names another epoch. Use a new prefix and never reuse one across epochs. The data of a 0.10 or a 0.9 cluster (format epoch 6 or 5) moves to this release by backup and restore, with its cold objects copied into the new prefix without the marker (Upgrading to 0.11).

journal … uses Ursula WAL version N

A Raft WAL journal of another format epoch: version 1 is Ursula 0.5.x, version 2 is 0.6, version 3 is 0.7, version 4 is 0.8, version 5 is 0.9, version 6 is 0.10. This release reads format epoch 7 only. The directory marker normally refuses the directory first, and the journal check is the backstop. Start on an empty raft.wal.path.

peer N (…) refused Raft protocol 7

A configured peer runs a release of another format epoch (Ursula 0.10 speaks protocol 6, 0.9 protocol 5, 0.8 protocol 4, 0.7 protocol 3, 0.6 protocol 2, 0.5.x protocol 1). A node cannot join a cluster of another format epoch. Install this release as a new cluster.

/__ursula/ready answers 503 with format_epoch_mismatch

The node exchanged a Raft RPC with a peer on another format epoch: a configured peer that was unreachable during the startup probe, or a node of this cluster on another release that sent this node a Raft RPC. The node stays not ready until it restarts. Remove the mismatched node from the peer lists and restart. A [[raft.peers]] url that points at a listener serving no Raft gRPC, such as a peer's client listener while that peer sets server.cluster_listen, answers like an older release and gives this reason too: point it at the peer's cluster listener. A sender that names no cluster id, such as a 0.7 node of another deployment, does not make this node unready (see Peers of another cluster).

refusing Raft traffic with a peer of another cluster

The node and a peer have different server.cluster_id values: one of them is mistyped, or a peer URL points at a node of another deployment. Each refuses the other's Raft traffic. The log line names the peer's node id, cluster id and address. /__ursula/ready and /__ursula/metrics list a configured peer under peer_cluster_mismatches with the id it reported, and raft_grpc_peer_cluster_refusals counts the refusals. The refusals never change readiness: an unready node's reason is its own cause. Find the node whose id differs from the rest of the deployment and correct its server.cluster_id or the peer URL. A node whose WAL already holds the wrong id refuses to start with the corrected one, so restart it on an empty WAL and let it rejoin as a wiped node. The healthy nodes stop listing the peer a minute after its last refusal. See Peers of another cluster.

refusing Raft traffic from a sender that names no cluster id

A node before format epoch 4, or a client that is not an Ursula node, sent a Raft RPC to this node, usually because a peer list of another deployment names this node's address. The sender is refused and gets a format-epoch refusal. This node stays ready and counts it in raft_grpc_unidentified_peer_refusals. Remove this node's address from the other deployment's peer list.

/__ursula/ready answers 503 with recovery_gate_closed

The node's logs may be missing entries it acknowledged: it started after a host crash or power loss under raft.wal.fsync = "never", after a journal I/O error, or without its run state file, or with an emptied WAL directory. Under raft.wal.fsync = "always", a host crash gates the groups of a core whose newest journal segment ended in a complete invalid frame, although no acknowledged write was lost. Each gated group waits until it has applied a fresh barrier from the group's leader, and its leader catches the replica up meanwhile, which stays a voter. recovery_gates in /__ursula/metrics lists the gated groups. This normally clears in seconds once a leader is reachable. A gate of a replica that does not count as a voter, such as an incoming learner, never makes the node unready, closed or stalled. See Recovering after a host crash.

/__ursula/ready answers 503 with recovery_stalled

A gated group on this node has seen no leader barrier and applied nothing for 30s. A majority of the group's voters are probably gated, so the group has no leader and refuses writes. The groups are listed under recovery_stalled_groups. Restore connectivity to a healthy quorum first. If the stalled replicas retain their voting history, an operator can accept the loss of writes acknowledged after the last fsync with a recovery plan, sent through POST /__ursula/raft/{target}/recovery/accept-unsynced-loss to the plan's leader first and then to every other replica that answers, with the leader's plan term (ursulactl recover-group --accept-loss does both). Recover the meta group first (--target meta): a data group's plan needs a confirmed meta view. A stalled meta gate shows as "meta" under recovery_gates.stalled in the metrics and in the ursula.raft.recovery_gates gauge, never in recovery_stalled_groups or readiness. The plan starts a new history of the group, so every stream of the group gets a new Stream-Incarnation. A voter of the group that was never gated does not report recovery_stalled. It stays ready and reports leader_unknown for the group. See Recovering after a host crash.

accept-unsynced-loss answers 409 Conflict

Nothing changed. A replica that lost its voting history must first obtain a durable vote floor from a fresh quorum proof, and the call cannot replace that proof. Restore connectivity to intact voters.

The call also refuses when the replica leads or follows a live leader, when its gate awaits or applies a leader's barrier (it may still open without losing anything) or runs a plan's cut, when the request's step is not the replica's (lead for the plan's leader, follow for every other replica), when the plan's leader is not a voter, when the replica no longer holds the log the plan's expected names, when it committed or applied an entry above the plan's point, when it is the plan's leader and its log does not continue from the point or, with a point_entry, does not hold exactly that entry there, or when the plan's cut cannot run because the log no longer holds the entry it keeps. The answer's body names the refusal by its reason. Read /__ursula/metrics on every replica again and plan again with what it shows, once the group has no leader and, after a not_stalled refusal, once the gate reports stalled. Send each member the leader_term from the leader's answer, never a term read from the metrics. A body of the 0.7 form answers 422.

Writes and reads answer 503 after a recovery plan

The plan's leader serves nothing until the group's new history event applies. It proposes the event once it is elected and has applied its first entry, so this normally clears within a second. If it persists, check that the plan's leader is running and that every other replica that holds the plan can reach it: a replica with a plan votes only for the plan's leader. If that leader lost entries it had replicated, the members refuse it and the group stalls again: plan again with the replicas' current metrics.

LogBelowSnapshot: a group's log contradicts its restored snapshot record

The replica found an entry, or a commit marker, at or after its snapshot record's index from an older term, and keeps the group closed. The group's other replicas keep serving it. Rejoin the node as a wiped node after checking its other groups. See A log below its snapshot record.

I/O error on raft.wal.path

The directory must be writable and owned by exactly one Ursula process. Each core journal holds an advisory journal.lock containing the owner PID. Do not remove it while that process is alive; stop the conflicting process or fix the volume mount. A stale lock file is harmless because ownership is enforced by the OS lock, not the file's existence.

Backup and restore

backup manifest format_version N or target cluster speaks backup format N

The backup or the cluster uses another backup format. This release reads and writes backup format 5, written by Ursula 0.9, 0.10 and 0.11, so a 0.10 or a 0.9 backup restores into it. A backup of another format, such as a 0.8 backup (format 4), cannot be restored, and a cluster that reports another backup format cannot be backed up or restored into.

… cold objects this backup references are missing from the target cluster's cold store

The backup references cold objects that the target's storage.cold.root does not hold. restore stopped before the first import, so the target is still empty. Copy the source cluster's cold objects into the target's root (see Copying cold objects) and run restore again. The error names the first missing keys. A key under cold-index/ is an index page, so everything that page lists is unchecked until the page is there.

Bootstrap and cluster join

A fresh cluster never elects leaders

/__ursula/metrics on each node should show a non-null current_leader and a stable current_term for every group in raft_groups. A fresh cluster's groups form only after its meta group formed and bootstrapped. GET /__ursula/placement shows how far it got:

• meta_genesis: the meta group has not formed. unanswered lists the raft.peers nodes that did not answer empty yet, and genesis_mismatch the nodes that claim other values of raft.meta_voters, raft.peers, raft.group_count or the replication factor. Start every raft.peers node with the same values.
• registering: the meta group formed, and waiting_for lists the nodes that have not registered. Bootstrap waits for every raft.peers node. last_bootstrap_refusal names why a bootstrap was refused.
• bootstrapped: every group is placed. Every seed voter of a group must be up: the group's initializer waits for all of them to report the group empty.

Every peer URL must be reachable from every node.

Replacing or restarting a node leaves it leaderless

Restart with the same raft.node_id and peer list. A fresh raft.wal.path reports root_refused until ursulactl node replace-disk releases the old disk, and then rehydrates from peers, which can be slow on cold-storage-only history (see Rejoining a wiped node).

A new node holds no group

A node added to raft.peers after the cluster formed registers and holds nothing until groups move onto it with ursulactl placement move (Adding a node).

Write path

503 with Retry-After on append

Inspect the error body's error.code. WAL_DISK_PRESSURE means the WAL filesystem is below raft.wal.min_available_size (or its free-space stat failed); the node also becomes unready and attempts to yield its leaders. Free space or expand the volume. Admission resumes only at raft.wal.resume_available_size, preventing probe flapping.

RAFT_LOG_PRESSURE means the node's unsnapshotted Raft log is over twice raft.snapshot_log_budget. The node refuses writes with a body until snapshots bring the log below 1.5 times the budget. Before it gets there, the snapshot driver runs more snapshot builds at once the further the log is past the budget, up to raft.snapshot_build_max_concurrency. When writes still reach the limit, the node's metrics show how far the log got (raft_log_bytes_high_watermark), whether the driver used every build (raft_snapshot_builds_in_flight_max), and how long a build's upload takes (raft_snapshot_build_ns over raft_snapshot_builds). If every build was in use, raise raft.snapshot_build_max_concurrency (the chart's raft.snapshotBuildMaxConcurrency) or the budget. Each build holds its group's captured state and, while it uploads, the compressed snapshot, so raise the builds only as far as the node's memory holds that many builds of its largest groups. Set it to 1 to keep one build at a time, as before 0.7.4. Every Raft node answers RAFT_LOG_PRESSURE, a single node included.

INGRESS_BODY_BYTES_LIMIT_REACHED means the write bodies in flight on the node leave too little of server.http_inflight_body_size for this one.

SNAPSHOT_COMMITS_BUSY means that the node copies as many snapshot commits from parts as it may, two at once. SNAPSHOT_COMMIT_IN_PROGRESS means that the node copies this same commit now: repeat it, and it answers once the commit ends. snapshot_commits_busy in /__ursula/metrics counts the first.

WAL_DISK_PRESSURE, RAFT_LOG_PRESSURE and INGRESS_BODY_BYTES_LIMIT_REACHED are write admission refusals.

Other 503s come from the stream's Raft group, and their code names the cause. COLD_BACKPRESSURE means that the group's hot ring would exceed storage.cold.max_hot_size_per_group: cold flush isn't keeping up, or the cold backend is slow or erroring out. Raise the per-group ceiling, lower storage.cold.flush_interval, raise storage.cold.flush_max_concurrency, or fix the cold backend. RAFT_UNCOMMITTED_BACKPRESSURE means that the group's write bytes that are proposed but not yet applied would exceed raft.max_uncommitted_size_per_group.

UPLOAD_BUDGET_EXHAUSTED comes from ursulagw: its request body budget (--body-budget-bytes) stayed spent for --body-budget-wait-ms, so it sent nothing on. Bodies of at most --small-body-max-bytes have headroom of their own, so this hits large bodies under heavy upload load. Writes resume once the uploads under way end. The gateway's upload_refusals_budget counter shows these refusals. See the gateway's request body budget.

A 503 also answers a write while the group has no known leader, for example during an election, and on a node that does not host the stream's group and knows no address of a node that does. A node whose replica of the group is no voter and knows no leader redirects (307) instead, as a node that does not host the group does.

A client resolves a 503 like any unknown outcome: it repeats the write with its producer key. See write outcomes.

408 with BODY_TOO_SLOW from the gateway

No byte of the body reached ursulagw for --body-stall-timeout, or less than --body-min-bytes-per-sec arrived over one such window, so the gateway cut the body and sent nothing on. With the defaults (30 s, 1 KiB/s) this does not happen to a SQLite commit, since the extension gives up on a commit after 10 s. Over HTTP/1.1 the gateway closes the connection with the 408, so a client still sending usually meets the closed connection instead. Keep --body-stall-timeout above 10 s, or a commit still being sent can meet that closed connection on every attempt. The gateway's upload_refusals_too_slow counter shows the cuts.

409 Conflict with producer-expected-seq / producer-received-seq

Out-of-order producer headers. Response tells you what was expected:

• producer-expected-seq: N is the next allowed sequence
• producer-received-seq: M is what the client sent

Common causes:

• Producer restart without bumping Producer-Epoch. Bump every restart.
• Two writers share the same Producer-Id. Use distinct IDs.
• A retry skipped a sequence number. Producer sequences must be contiguous within an epoch.

409 Conflict from Stream-Seq

The supplied Stream-Seq is not bytewise greater than the last value accepted on this stream: another writer committed first, or the values are not fixed-width ("10" sorts before "9"). The response's Stream-Next-Offset is the tail. Read what you missed, re-derive the value and retry. See Conditional writes.

404 Not Found on POST /{bucket}/{stream}

The stream hasn't been created. Create with PUT /{bucket}/{stream} first. There is no implicit creation on POST.

400 Bad Request on JSON appends

JSON streams are validated on the server. Malformed JSON, invalid UTF-8, messages nested deeper than 127 levels, and empty arrays without allow_empty_array are rejected. Send valid JSON or switch to application/octet-stream.

Read path

No Stream-Up-To-Date on a read

The response is partial: cut by max_bytes or the 8 MiB cap, or served by a follower, which never sends it with data. Read on from Stream-Next-Offset. Not an error.

SSE drops after 30-60 s idle

A proxy or load balancer is closing idle TCP. Raise its idle timeout or enable TCP keepalive on the proxy-to-Ursula leg. Ursula doesn't emit SSE keepalive comments.

SSE on a binary stream returns base64 text

Expected. SSE wire format is text-only, so binary stream data events carry raw base64 text and the Stream-Sse-Data-Encoding: base64 header signals it. See binary SSE.

Reads stall for hundreds of ms for cold offsets

The first read of a cold offset triggers an S3 GetObject range read. It runs off the actor turn so it doesn't block other commands, but the request still pays S3 latency.

Replication and consensus

A group has no leader (current_leader: null)

Either an election is in progress or quorum is lost for that group. Only streams hashed to that group are affected. A follower names a leader only while that leader's appends reach it, within the last 3 s, so a follower cut off from its leader shows null even while the other nodes still name one.

• Multiple nodes briefly claim leader: election is flapping. Check peer reachability and CPU pressure.
• No node claims leader: fewer than n/2+1 voters are reachable, or a majority of the group's voters are behind their recovery gates. Restore reachability. For gated voters, see Recovering after a host crash.

One follower lags behind

State-machine apply runs on the follower's owner core. If CPU is pinned, last_applied_index trails committed_index. Check:

• Per-core mailbox depth (mailbox_depths). Sustained queueing means saturation.
• Disk pressure on the WAL volume (fsync latency, in particular with raft.wal.fsync = "always").
• A pathological hot stream forcing constant cold flushes (per_group_cold_hot_bytes).

Manually trigger a snapshot

These are admin-plane endpoints (server.admin_listen, loopback :4438 by default), reached over a tunnel (see ursulactl's operation providers). The examples assume a forward from local 4438 to the node's admin plane.

process_incarnation=$(curl --fail http://127.0.0.1:4438/__ursula/metrics | jq -er .process_incarnation)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/{group_id}/snapshot
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/{group_id}/purge?upto={index}"

The snapshot request answers once the group has a snapshot of everything it had applied when the request arrived. It waits while a build of the group runs and while other snapshot builds hold all of the node's build permits, and a snapshot that a build begun since the request made answers it too. When no build of the group can start within 10 seconds, it answers 503 with Retry-After: 1. Send it again then. Unless the group's published snapshot already covers the request, it then waits up to 10 seconds for the group's running build to end and up to 10 more for the snapshot to show, and answers 500 when it does not.

The purge request drops log entries through upto, at most through the group's published snapshot. While a snapshot of the group builds, OpenRaft has not published the one it built, or an earlier purge of the group has not landed, it answers 503 with Retry-After: 1. Send it again then.

Purge only after the snapshot replicates. Otherwise a slow follower may need log entries you just dropped.

Cold flush issues

Writes succeed but data never reaches S3

• storage.cold.backend is none, or set to memory when you expected s3.
• IAM permissions missing. Required: s3:GetObject, s3:PutObject, s3:ListBucket, s3:DeleteObject.
• storage.cold.s3.endpoint unreachable.
• Cold flush failing or falling behind (cold_flush_write_errors or cold_backpressure_events climbing, cold_hot_bytes growing).

If hot bytes grow unbounded, you'll eventually hit storage.cold.max_hot_size_per_group and writes start returning 503. Fix the backend before that.

SQLite VFS

These entries cover the sqlite-ursula-vfs extension and @tonbo/sqlite-ursula 0.8. See SQLite on Ursula.

synced() does not resolve

synced() waits until the stream holds every commit made before the call. Commits keep working meanwhile. db.status().replication.condition says why it waits:

condition	Cause	What to do
retrying	No definite answer yet: the stream is unreachable, answers 5xx, 429 or 408, or the connection dropped. The append is resent unchanged after backoff. A 429 with the code UPLOADS_EXHAUSTED or UPLOAD_BYTES_EXHAUSTED means the bucket's large bodies under way reached its max_concurrent_uploads or max_upload_bytes in the gateway's --quota-policy	Check that the stream URL's host resolves and that the gateway or the server behind it is up. Nothing is lost while it retries. For the upload refusals, raise those keys, or set URSULA_VFS_MAX_REQUEST_BYTES at or below the gateway's --small-body-max-bytes (1 MiB by default), whose bodies never count. Small commits alone do not stay below it, since an append packs commits up to 4 MiB
unauthorized	Two 401s in a row, the second after reading the token again	Fix the token passed to open or the file URSULA_VFS_TOKEN_FILE names. A changed file is read again at once
forbidden	An authorizer in front of the stream answered 403	Allow the write in the authorizer
payment_required	A layer in front of the stream answered 402	Settle what the layer asks for
request_too_large	A layer in front of the stream allows request bodies below 4 KiB	Raise the layer's limit
redirected	The stream URL answers with a redirect, which the extension never follows	Point the URL at a gateway or at a single node
wrong_cluster	The stream URL now reaches another deployment (another Ursula-Cluster-Id)	Fix the routing. Replication resumes by itself within 30 s
protocol_mismatch	A HEAD the session sent to confirm an answer such as a 412 or a 404 answered as a server of another release line, such as a cluster moved to another minor	Run the extension of the server's release, or point the URL back at a server of the extension's line. Stop every SQLite session before you move a cluster to another release line, since a session checks the line only at such a HEAD
no_server_identity	A HEAD answer carries no Ursula-Protocol and Ursula-Cluster-Id: a layer that answers HEAD itself, such as a gateway whose authorizer conceals the stream	Check the token, and send HEAD through to the server
rejected	A status the extension does not expect, such as a layer that rewrites bodies	Look at the layers between the client and the gateway

A blocked condition is retried every 30 s, and commits continue until commits.pending_bytes reaches maxPendingBytes. A synced() with a signal rejects with the signal's reason and leaves the commits replicating.

A write fails with error code 7178

SQLITE_IOERR_AUTH (errcode 7178 in node:sqlite, UrsulaWriteRefusedError from the package) means the write was refused before anything of it was committed. The database stays usable.

• backlog_full: compressed commits waiting for the stream reached maxPendingBytes (64 MiB by default). Await db.synced() and run the transaction again. If it stays full, see synced() does not resolve.
• unsupported: the transaction ran in EXCLUSIVE locking mode, or its commit leaves WAL mode or 4096-byte pages. Change the transaction. Other transactions go on.
• too_large: the commit's pages take more than URSULA_VFS_MAX_COMMIT_BYTES (128 MiB by default). The message names the commit's size and the limit. Split the transaction, or raise the limit (at most 1 GiB) on every host that may own the database, with memory for about three times the limit. A VACUUM of a large database hits it: see Errors for the ways around it.

Statements fail with error code 266 or 778

The database was reset. Every statement throws an UrsulaResetError (errcode 266 for a read, 778 for a write), and db.status().reset names the kind:

• fenced (stop): another process opened the stream and owns it now. Find that owner before you open the stream again, since opening it takes the stream back.
• stale (reopen): the stream moved on while this process could not see it, for example after it was idle past producer expiry or opened provisionally during an outage. Open the stream again.
• rewound (reopen): a restore or an accepted loss removed commits the stream had acknowledged. Open the stream again: the open moves the cache aside into salvage/ (see below) and builds from the stream.
• deleted (fix): the stream was deleted. Open it with recreate: true to create it again from the cache's data, or open another stream.
• local_damage (reopen): a local fsync failed, a WAL page read back other bytes, or SQLite ended a commit without publishing it. Open the stream again: the log replays every commit. Check the disk if it repeats.
• stream_state (fix): another writer appended to the stream, the stream was closed, or the server lost acknowledged state. The stream no longer matches the cache. Look at who writes to the stream.
• internal (reopen): a bug inside the extension. A panic line on stderr names it. Open the stream again, and report it.

open fails

open throws an UrsulaSetupError or, when its signal fires first, an UrsulaUnavailableError. Every refusal leaves the cache as it was and sends nothing to the stream.

kind	Cause	What to do
protocol_mismatch	The server is of another release line than the extension: its Ursula-Protocol is not the extension's line (11 for 0.11.x). A 0.8 to 0.10 server sends 2	Run the extension of the server's release
no_server_identity	The stream's HEAD answer lacks Ursula-Protocol and Ursula-Cluster-Id: a server before 0.8, or a layer that answers HEAD itself, such as a gateway whose authorizer conceals the stream, whose 404 carries neither header	Check the token, send HEAD through to the server, and run a server of the extension's release
server_too_old	Reported by a 0.8 to 0.10 extension when the server is of another release line, newer ones included: those extensions accept only the Ursula-Protocol: 2 that 0.8 to 0.10 servers send	Upgrade the extension with the server, and stop its sessions before the move
stream_format	The stream holds 0.7 data, or another content type	Move the database with an import, see Moving from 0.7
wrong_cluster	The cache was built on another deployment than the one the URL reaches now	Fix the routing, or use another dir for the other deployment
not_found	The stream is missing and the cache holds data	Open with recreate: true, or open another stream
in_use	Another process holds the cache	Stop that process, or use another dir
bound_elsewhere	The stream is open in this process under another dir	Close that database, or pass its dir
stream_state	The stream is closed, or an import into it is running or failed (the stream up to N is the claim of an import)	Open a closed stream read-only. Wait for a running import. After a failed import, delete or recreate the stream and import again
unreachable, unavailable	The signal fired while the stream could not be reached or did not answer	Check the network and the cluster. With a cache that claimed the stream before, open resolves provisionally instead

A cache appears in salvage/

status().salvage lists caches the extension moved aside, in salvage/ in the stream's cache directory: after a history change that lost commits the cache had, after a history change while the cache held commits it had no acknowledgement for, when the db file was not the cache's own (replaced), and with recreate (recreated). Each holds a plain SQLite database db, its log and a manifest.json with the tokens and the count of unacknowledged commits. Open db with any SQLite, compare it with the stream's database, and write again what you need. The extension never deletes a salvage.

snapshot.error is set

status().snapshot.error says why snapshots are not published, and snapshot.failures counts the attempts. A database whose compressed snapshot is above the request limit (32 MiB) needs a cold backend on the server, which takes it in parts, up to 1 GiB. Until a snapshot publishes, the log grows and every fresh host reads all of it. Failed attempts wait from 100 ms up to five minutes, or an hour when the body is too large.

file is not a database

A cache's db file holds a marker in place of SQLite's header magic, and its WAL is db-uwal. SQLite outside the extension (the sqlite3 shell, node:sqlite without it, a connection opened before it loaded, ?vfs=unix) fails with SQLITE_NOTADB and changes nothing. Read the database through open, or export it with VACUUM INTO from a connection open returned.

Closing a Pi session hangs during an outage

On Pi Durable 1.3, Session.close waits for the storage calls in flight, and through openUrsulaPiStorage each waits for synced(), which waits for the stream. So closeTimeoutMs applies only once the session reaches the storage's close. Bound the shutdown in your host when it must end during an outage. The log keeps every commit for the next open.

Still stuck?

Open a GitHub issue with:

• /__ursula/metrics from every node (with timestamps)
• Config file and CLI overrides per node (redact credentials)
• Last ~200 log lines at RUST_LOG=ursula_runtime=debug,ursula_raft=debug,ursula=info