Troubleshooting

Start at /__ursula/metrics. Most operational symptoms have a clear signal in the JSON snapshot.

Diagnostic surface

curl http://NODE:4437/__ursula/metrics > metrics.json

Useful jq selectors. Arrays named per_core_* are indexed by core, and arrays named per_group_* by Raft group ID.

# Leader, term and applied index of each group. current_leader is null while the group has no leader.
jq '.raft_groups[] | {raft_group_id, current_leader, current_term, last_applied_index}' metrics.json
# Groups without a leader
jq '[.raft_groups[] | select(.current_leader == null) | .raft_group_id]' metrics.json
# Hot bytes waiting for cold flush, per group
jq '.per_group_cold_hot_bytes' metrics.json
# Writes refused by cold-write backpressure, in total and per group
jq '{total: .cold_backpressure_events, per_group: .per_group_cold_backpressure_events}' metrics.json
# Mailbox depth and capacity, per core
jq '{depths: .mailbox_depths, capacities: .mailbox_capacities}' metrics.json
# Live reads waiting for new data, per core
jq '.per_core_live_read_waiters' metrics.json
# SSE streams opened and SSE error events
jq '{opened: .sse_streams_opened, errors: .sse_error_events}' metrics.json
# WAL disk pressure
jq '{available: .wal_available_bytes, minimum: .wal_min_available_bytes, pressure: .wal_disk_pressure}' metrics.json
# WAL reclaim and restart work
jq '{physical: .wal_physical_bytes, segments: .wal_segments, reclaims: .wal_reclaims, lagging: .wal_lagging_groups, recovered_bytes: .wal_recovery_bytes, recovery_ns: .wal_recovery_ns}' metrics.json
# How the WAL opened, and the groups behind a recovery gate
jq '{opening: .wal_recovery, gates: .recovery_gates}' metrics.json
# Local Raft issues that make the node unready or block maintenance
jq '.raft_maintenance | {node_issues, group_issues}' metrics.json
# This node's cluster id, and Raft traffic refused for another one
jq '{cluster_id, refusals: .raft_grpc_peer_cluster_refusals, unidentified: .raft_grpc_unidentified_peer_refusals, peers: .peer_cluster_mismatches}' metrics.json

GET /__ursula/ready is the process readiness probe. It answers 200 when the node may serve, and otherwise 503 with a typed reason. When several apply, it reports the first of:

reason	Meaning
format_epoch_mismatch	The node exchanged a Raft RPC with a peer of another format epoch. It stays unready until it restarts
wal_disk_pressure	The WAL filesystem is below raft.wal.min_available_size, or its free-space check failed
recovery_stalled	A group's recovery gate on this node is stalled. The groups are listed under recovery_stalled_groups
recovery_gate_closed	A group's recovery gate on this node is closed until the replica has applied a fresh barrier from the group's leader
raft_replica_unready	A local replica is not a healthy voter. raft_maintenance.group_issues names the issue

Ursula does not expose /healthz, /readyz, Prometheus /metrics, or /cluster/status. Use the JSON snapshot for detailed diagnosis and ursulactl wait-ready for cluster-wide Raft readiness.

Serving during a replica rebuild

raft_replica_unready means local Raft participation is not safe for serving. Inspect raft_maintenance.group_issues for local_replica_not_voter, membership_not_applied, not_applied, apply_lag or leader_unknown. A wiped replica stays 503 while its recovery gate is closed and while it is a learner. Healthy surviving voters can return 200 during the repair. A voter leaving a joint membership also stays unready once any configuration excludes it. A replica whose own gate is open reports leader_unknown for a group that has no leader, for example a group stalled because a majority of its voters are gated.

A 200 response does not certify full redundancy. Keep maintenance blocked until ursulactl wait-ready succeeds. Alert separately on incomplete voter sets in the maintenance report, as described in Observability.

503 with APPLY_STOPPED from one group

A committed command failed to apply and stopped that group on this replica. The node stays ready and serves its other groups. Its maintenance report names the group with apply_stopped, so ursulactl wait-ready refuses disruption until the group recovers. Follow Recovering stopped Raft groups and never skip the committed record.

Startup failures

WAL topology mismatch or missing topology

topology.bin binds a WAL root to runtime.core_count and raft.group_count. For a mismatch, restore the stored counts reported in the error. A missing record on an existing WAL or a corrupt record also refuses startup. Do not replace it with the current configuration. See WAL routing configuration for rebuilding a replica.

raft.wal.path is required when raft.peers is set

Every node of a static cluster keeps its Raft WAL in raft.wal.path. Set it to a directory on a persistent volume. Only a single node without raft.peers may omit it, and then its WAL lives in a temporary directory that is removed on clean shutdown.

server.cluster_id is required when raft.peers is set

Every node of a static cluster names its deployment with server.cluster_id. Set the same value on every node, and a different one for every deployment a client can reach. With the Helm chart, set server.clusterId. The chart refuses to render without it. See Configuration.

the WAL at … belongs to cluster …

The node's WAL records the cluster id of its first start, and the configured server.cluster_id differs. A disk does not move between clusters. Set server.cluster_id to the stored id the error names, or start the node on an empty WAL and let it rejoin as a wiped node.

raft.node_id must be non-zero

Static Raft mode requires a unique raft.node_id in the config file or a --node-id override.

raft.peers must include this node id

The local raft.node_id must appear in [[raft.peers]]. The peer list always includes self.

storage.cold.s3.bucket is required when cold backend is 's3'

Set storage.cold.s3.bucket, or set storage.cold.backend = "none" if you don't want cold storage.

Port already in use

server.listen, server.cluster_listen or server.admin_listen is taken. The client plane and inter-node gRPC share server.listen unless server.cluster_listen is set.

raft group … records node … at …, but raft.peers configures …

The node exits at startup with:

error: raft group 3 records node 2 at http://ursula-1.ursula-headless.ursula.svc.cluster.local:4437, but raft.peers configures http://ursula-1.ursula-headless.ursula.svc.cluster.local:4439: a node's Raft url (and server.cluster_listen) cannot change after its groups are initialized

This is PeerAddressChanged. Each group records every member's Raft address when it first initializes, and peers dial that recorded address from then on. The node's own [[raft.peers]].url no longer matches it. The host, the port or both changed, for example because server.cluster_listen moved the cluster plane to another port, or because the chart's server.clusterListen, server.ports.cluster or global.clusterDomain changed after the first install. A node serving at the new address would get no Raft traffic, so it refuses to start.

Restore the recorded address the error names and roll the node again. With the Helm chart, set server.clusterListen, server.ports.cluster and global.clusterDomain back to the values of the first install. Outside the chart, restore the url and the server.cluster_listen port. Every node of the cluster needs the same peer list, so revert the change on all of them.

Ursula 0.8 cannot change a recorded peer address in place. To move a cluster to a new address, install a fresh cluster at that address. Issue #633 tracks migrating a recorded address without a fresh install.

raft WAL directory … holds data without a format marker

The data directory was written by Ursula 0.5.x or earlier (format epoch 1). This release reads format epoch 4 only and has no in-place upgrade. Start on an empty raft.wal.path and a new object-storage prefix, or run the release that wrote it.

raft WAL directory … is format epoch N

The directory's FORMAT_EPOCH marker names another epoch. This release reads format epoch 4 only. Run the release that wrote it, or start on an empty directory.

object storage s3://… is not empty and has no URSULA_FORMAT_EPOCH marker

The cold root (or, with cold storage off, the S3 snapshot namespace) holds objects from Ursula 0.5.x or from another application. Point storage.cold.root (Helm: s3.prefix or coldStorage.prefix), or storage.snapshot.s3_prefix when only snapshots use S3, at an empty prefix used only by this cluster.

object storage s3://… is format epoch N

The prefix's URSULA_FORMAT_EPOCH marker names another epoch. Use a new prefix and never reuse one across epochs.

journal … uses Ursula WAL version N

A Raft WAL journal of another format epoch: version 1 is Ursula 0.5.x, version 2 is 0.6, version 3 is 0.7. This release reads format epoch 4 only. The directory marker normally refuses the directory first, and the journal check is the backstop. Start on an empty raft.wal.path.

peer N (…) refused Raft protocol 4

A configured peer runs a release of another format epoch (Ursula 0.7 speaks protocol 3, 0.6 protocol 2, 0.5.x protocol 1). A node cannot join a cluster of another format epoch. Install this release as a new cluster.

/__ursula/ready answers 503 with format_epoch_mismatch

The node exchanged a Raft RPC with a peer on another format epoch: a configured peer that was unreachable during the startup probe, or a node of this cluster on another release that sent this node a Raft RPC. The node stays not ready until it restarts. Remove the mismatched node from the peer lists and restart. A sender that names no cluster id, such as a 0.7 node of another deployment, does not make this node unready (see Peers of another cluster).

refusing Raft traffic with a peer of another cluster

The node and a peer have different server.cluster_id values: one of them is mistyped, or a peer URL points at a node of another deployment. Each refuses the other's Raft traffic. The log line names the peer's node id, cluster id and address. /__ursula/ready and /__ursula/metrics list a configured peer under peer_cluster_mismatches with the id it reported, and raft_grpc_peer_cluster_refusals counts the refusals. The refusals never change readiness: an unready node's reason is its own cause. Find the node whose id differs from the rest of the deployment and correct its server.cluster_id or the peer URL. A node whose WAL already holds the wrong id refuses to start with the corrected one, so restart it on an empty WAL and let it rejoin as a wiped node. The healthy nodes stop listing the peer a minute after its last refusal. See Peers of another cluster.

refusing Raft traffic from a sender that names no cluster id

A node before format epoch 4, or a client that is not an Ursula node, sent a Raft RPC to this node, usually because a peer list of another deployment names this node's address. The sender is refused and gets a format-epoch refusal. This node stays ready and counts it in raft_grpc_unidentified_peer_refusals. Remove this node's address from the other deployment's peer list.

/__ursula/ready answers 503 with recovery_gate_closed

The node's logs may be missing entries it acknowledged: it started after a host crash or power loss under raft.wal.fsync = "never", after a journal I/O error, or without its run state file, or with an emptied WAL directory. Under raft.wal.fsync = "always", a host crash gates the groups of a core whose newest journal segment ended in a complete invalid frame, although no acknowledged write was lost. Each gated group waits until it has applied a fresh barrier from the group's leader, and its leader rebuilds the replica meanwhile. recovery_gates in /__ursula/metrics lists the gated groups. This normally clears in seconds once a leader is reachable. See Recovering after a host crash.

/__ursula/ready answers 503 with recovery_stalled

A gated group on this node has seen no leader barrier and applied nothing for 30s. A majority of the group's voters are probably gated, so the group has no leader and refuses writes. The groups are listed under recovery_stalled_groups. Restore connectivity to a healthy quorum first. If the stalled replicas retain their voting history, an operator can accept the loss of writes acknowledged after the last fsync with a recovery plan, sent through POST /__ursula/raft/{group}/recovery/accept-unsynced-loss to the plan's leader first and then to every other replica that answers, with the leader's plan term (ursulactl recover-group --accept-loss does both). The plan starts a new history of the group, so every stream of the group gets a new Stream-Incarnation. A voter of the group that was never gated does not report recovery_stalled. It answers raft_replica_unready with leader_unknown for the group. See Recovering after a host crash.

accept-unsynced-loss answers 409 Conflict

Nothing changed. A replica that lost its voting history must first obtain a durable vote floor from a fresh quorum proof, and the call cannot replace that proof. Restore connectivity to intact voters.

The call also refuses when the replica leads or follows a live leader, when its gate awaits or applies a leader's barrier (it may still open without losing anything) or runs a plan's cut, when the request's step is not the replica's (lead for the plan's leader, follow for every other replica), when the plan's leader is not a voter, when the replica no longer holds the log the plan's expected names, when it committed or applied an entry above the plan's point, when it is the plan's leader and its log does not continue from the point or, with a point_entry, does not hold exactly that entry there, or when the plan's cut cannot run because the log no longer holds the entry it keeps. The answer's body names the refusal by its reason. Read /__ursula/metrics on every replica again and plan again with what it shows, once the group has no leader and, after a not_stalled refusal, once the gate reports stalled. Send each member the leader_term from the leader's answer, never a term read from the metrics. A body of the 0.7 form answers 422.

Writes and reads answer 503 after a recovery plan

The plan's leader serves nothing until the group's new history event applies. It proposes the event once it is elected and has applied its first entry, so this normally clears within a second. If it persists, check that the plan's leader is running and that every other replica that holds the plan can reach it: a replica with a plan votes only for the plan's leader. If that leader lost entries it had replicated, the members refuse it and the group stalls again: plan again with the replicas' current metrics.

I/O error on raft.wal.path

The directory must be writable and owned by exactly one Ursula process. Each core journal holds an advisory journal.lock containing the owner PID. Do not remove it while that process is alive; stop the conflicting process or fix the volume mount. A stale lock file is harmless because ownership is enforced by the OS lock, not the file's existence.

Backup and restore

backup manifest format_version N or target cluster speaks backup format N

The backup or the cluster is of another format epoch. This release reads and writes backup format 4 only, so a backup of another epoch cannot be restored, and a cluster of another epoch cannot be backed up or restored into.

… cold objects this backup references are missing from the target cluster's cold store

The backup references cold objects that the target's storage.cold.root does not hold. restore stopped before the first import, so the target is still empty. Copy the source cluster's cold objects into the target's root (see Copying cold objects) and run restore again. The error names the first missing keys. A key under cold-index/ is an index page, so everything that page lists is unchecked until the page is there.

Bootstrap and cluster join

A fresh cluster never elects leaders

/__ursula/metrics on each node should show a non-null current_leader and a stable current_term for every group in raft_groups. If groups stay leaderless:

• First start must include raft.init_membership_per_group = true (or raft.init_membership = true) on every voting node.
• Every peer URL must be reachable from every node.

Replacing or restarting a node leaves it leaderless

Restart with the same raft.node_id and peer list. A fresh raft.wal.path means rehydrating from peers, which can be slow on cold-storage-only history (see Rejoining a wiped node).

Write path

503 with Retry-After on append

Inspect the error body's error.code. WAL_DISK_PRESSURE means the WAL filesystem is below raft.wal.min_available_size (or its free-space stat failed); the node also becomes unready and attempts to yield its leaders. Free space or expand the volume. Admission resumes only at raft.wal.resume_available_size, preventing probe flapping.

RAFT_LOG_PRESSURE means the node's unsnapshotted Raft log is over twice raft.snapshot_log_budget. The node refuses writes with a body until snapshots bring the log below 1.5 times the budget. Before it gets there, the snapshot driver runs more snapshot builds at once the further the log is past the budget, up to raft.snapshot_build_max_concurrency. When writes still reach the limit, the node's metrics show how far the log got (raft_log_bytes_high_watermark), whether the driver used every build (raft_snapshot_builds_in_flight_max), and how long a build's upload takes (raft_snapshot_build_ns over raft_snapshot_builds). If every build was in use, raise raft.snapshot_build_max_concurrency (the chart's raft.snapshotBuildMaxConcurrency) or the budget. Each build holds its group's captured state and, while it uploads, the compressed snapshot, so raise the builds only as far as the node's memory holds that many builds of its largest groups. Set it to 1 to keep one build at a time, as before 0.7.4. Only a cluster node answers RAFT_LOG_PRESSURE. A node without raft.peers never refuses writes for its log in 0.8.x.

INGRESS_BODY_BYTES_LIMIT_REACHED means the write bodies in flight on the node leave too little of server.http_inflight_body_size for this one.

SNAPSHOT_COMMITS_BUSY means that the node copies as many snapshot commits from parts as it may, two at once. SNAPSHOT_COMMIT_IN_PROGRESS means that the node copies this same commit now: repeat it, and it answers once the commit ends. snapshot_commits_busy in /__ursula/metrics counts the first.

WAL_DISK_PRESSURE, RAFT_LOG_PRESSURE and INGRESS_BODY_BYTES_LIMIT_REACHED are write admission refusals.

Other 503s come from the stream's Raft group, and their code names the cause. COLD_BACKPRESSURE means that the group's hot ring would exceed storage.cold.max_hot_size_per_group: cold flush isn't keeping up, or the cold backend is slow or erroring out. Raise the per-group ceiling, lower storage.cold.flush_interval, raise storage.cold.flush_max_concurrency, or fix the cold backend. RAFT_UNCOMMITTED_BACKPRESSURE means that the group's write bytes that are proposed but not yet applied would exceed raft.max_uncommitted_size_per_group.

UPLOAD_BUDGET_EXHAUSTED comes from ursulagw: its request body budget (--body-budget-bytes) stayed spent for --body-budget-wait-ms, so it sent nothing on. Bodies of at most --small-body-max-bytes have headroom of their own, so this hits large bodies under heavy upload load. Writes resume once the uploads under way end. The gateway's upload_refusals_budget counter shows these refusals. See the gateway's request body budget.

A 503 also answers a write while the group has no known leader, for example during an election, and on a node that does not host the stream's group and knows no node that does.

A client resolves a 503 like any unknown outcome: it repeats the write with its producer key. See write outcomes.

408 with BODY_TOO_SLOW from the gateway

No byte of the body reached ursulagw for --body-stall-timeout, or less than --body-min-bytes-per-sec arrived over one such window, so the gateway cut the body and sent nothing on. With the defaults (30 s, 1 KiB/s) this does not happen to a SQLite commit, since the extension gives up on a commit after 10 s. Over HTTP/1.1 the gateway closes the connection with the 408, so a client still sending usually meets the closed connection instead. Keep --body-stall-timeout above 10 s, or a commit still being sent can meet that closed connection on every attempt. The gateway's upload_refusals_too_slow counter shows the cuts.

409 Conflict with producer-expected-seq / producer-received-seq

Out-of-order producer headers. Response tells you what was expected:

• producer-expected-seq: N is the next allowed sequence
• producer-received-seq: M is what the client sent

Common causes:

• Producer restart without bumping Producer-Epoch. Bump every restart.
• Two writers share the same Producer-Id. Use distinct IDs.
• A retry skipped a sequence number. Producer sequences must be contiguous within an epoch.

409 Conflict from Stream-Seq

The supplied Stream-Seq is not bytewise greater than the last value accepted on this stream: another writer committed first, or the values are not fixed-width ("10" sorts before "9"). The response's Stream-Next-Offset is the tail. Read what you missed, re-derive the value and retry. See Conditional writes.

404 Not Found on POST /{bucket}/{stream}

The stream hasn't been created. Create with PUT /{bucket}/{stream} first. There is no implicit creation on POST.

400 Bad Request on JSON appends

JSON streams are validated on the server. Malformed JSON, invalid UTF-8, messages nested deeper than 127 levels, and empty arrays without allow_empty_array are rejected. Send valid JSON or switch to application/octet-stream.

Read path

No Stream-Up-To-Date on a read

The response is partial: cut by max_bytes or the 8 MiB cap, or served by a follower, which never sends it with data. Read on from Stream-Next-Offset. Not an error.

SSE drops after 30-60 s idle

A proxy or load balancer is closing idle TCP. Raise its idle timeout or enable TCP keepalive on the proxy-to-Ursula leg. Ursula doesn't emit SSE keepalive comments.

SSE on a binary stream returns base64 text

Expected. SSE wire format is text-only, so binary stream data events carry raw base64 text and the Stream-Sse-Data-Encoding: base64 header signals it. See binary SSE.

Reads stall for hundreds of ms for cold offsets

The first read of a cold offset triggers an S3 GetObject range read. It runs off the actor turn so it doesn't block other commands, but the request still pays S3 latency.

Replication and consensus

A group has no leader (current_leader: null)

Either an election is in progress or quorum is lost for that group. Only streams hashed to that group are affected.

• Multiple nodes briefly claim leader: election is flapping. Check peer reachability and CPU pressure.
• No node claims leader: fewer than n/2+1 voters are reachable, or a majority of the group's voters are behind their recovery gates. Restore reachability. For gated voters, see Recovering after a host crash.

One follower lags behind

State-machine apply runs on the follower's owner core. If CPU is pinned, last_applied_index trails committed_index. Check:

• Per-core mailbox depth (mailbox_depths). Sustained queueing means saturation.
• Disk pressure on the WAL volume (fsync latency, in particular with raft.wal.fsync = "always").
• A pathological hot stream forcing constant cold flushes (per_group_cold_hot_bytes).

Manually trigger a snapshot

These are admin-plane endpoints (server.admin_listen, loopback :4438 by default), reached over a tunnel (see ursulactl's operation providers). The examples assume a forward from local 4438 to the node's admin plane.

process_incarnation=$(curl --fail http://127.0.0.1:4438/__ursula/metrics | jq -er .process_incarnation)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/{group_id}/snapshot
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST "http://127.0.0.1:4438/__ursula/raft/{group_id}/purge?upto={index}"

The snapshot request answers once the group has a snapshot of everything it had applied when the request arrived. It waits while a build of the group runs and while other snapshot builds hold all of the node's build permits, and a snapshot that a build begun since the request made answers it too. When no build of the group can start within 10 seconds, it answers 503 with Retry-After: 1. Send it again then. Unless the group's published snapshot already covers the request, it then waits up to 10 seconds for the group's running build to end and up to 10 more for the snapshot to show, and answers 500 when it does not.

The purge request drops log entries through upto, at most through the group's published snapshot. While a snapshot of the group builds, OpenRaft has not published the one it built, or an earlier purge of the group has not landed, it answers 503 with Retry-After: 1. Send it again then.

Purge only after the snapshot replicates. Otherwise a slow follower may need log entries you just dropped.

Cold flush issues

Writes succeed but data never reaches S3

• storage.cold.backend is none, or set to memory when you expected s3.
• IAM permissions missing. Required: s3:GetObject, s3:PutObject, s3:ListBucket, s3:DeleteObject.
• storage.cold.s3.endpoint unreachable.
• Cold flush failing or falling behind (cold_flush_write_errors or cold_backpressure_events climbing, cold_hot_bytes growing).

If hot bytes grow unbounded, you'll eventually hit storage.cold.max_hot_size_per_group and writes start returning 503. Fix the backend before that.

SQLite VFS

These entries cover the sqlite-ursula-vfs extension and @tonbo/sqlite-ursula 0.8. See SQLite on Ursula.

synced() does not resolve

synced() waits until the stream holds every commit made before the call. Commits keep working meanwhile. db.status().replication.condition says why it waits:

condition	Cause	What to do
retrying	No definite answer yet: the stream is unreachable, answers 5xx, 429 or 408, or the connection dropped. The append is resent unchanged after backoff. A 429 with the code UPLOADS_EXHAUSTED or UPLOAD_BYTES_EXHAUSTED means the bucket's large bodies under way reached its max_concurrent_uploads or max_upload_bytes in the gateway's --quota-policy	Check that the stream URL's host resolves and that the gateway or the server behind it is up. Nothing is lost while it retries. For the upload refusals, raise those keys, or set URSULA_VFS_MAX_REQUEST_BYTES at or below the gateway's --small-body-max-bytes (1 MiB by default), whose bodies never count. Small commits alone do not stay below it, since an append packs commits up to 4 MiB
unauthorized	Two 401s in a row, the second after reading the token again	Fix the token passed to open or the file URSULA_VFS_TOKEN_FILE names. A changed file is read again at once
forbidden	An authorizer in front of the stream answered 403	Allow the write in the authorizer
payment_required	A layer in front of the stream answered 402	Settle what the layer asks for
request_too_large	A layer in front of the stream allows request bodies below 4 KiB	Raise the layer's limit
redirected	The stream URL answers with a redirect, which the extension never follows	Point the URL at a gateway or at a single node
wrong_cluster	The stream URL now reaches another deployment (another Ursula-Cluster-Id)	Fix the routing. Replication resumes by itself within 30 s
rejected	A status the extension does not expect, such as a layer that rewrites bodies	Look at the layers between the client and the gateway

A blocked condition is retried every 30 s, and commits continue until commits.pending_bytes reaches maxPendingBytes. A synced() with a signal rejects with the signal's reason and leaves the commits replicating.

A write fails with error code 7178

SQLITE_IOERR_AUTH (errcode 7178 in node:sqlite, UrsulaWriteRefusedError from the package) means the write was refused before anything of it was committed. The database stays usable.

• backlog_full: compressed commits waiting for the stream reached maxPendingBytes (64 MiB by default). Await db.synced() and run the transaction again. If it stays full, see synced() does not resolve.
• unsupported: the transaction ran in EXCLUSIVE locking mode, or its commit leaves WAL mode or 4096-byte pages. Change the transaction. Other transactions go on.
• too_large: the commit's pages take more than URSULA_VFS_MAX_COMMIT_BYTES (128 MiB by default). The message names the commit's size and the limit. Split the transaction, or raise the limit (at most 1 GiB) on every host that may own the database, with memory for about three times the limit. A VACUUM of a large database hits it: see Errors for the ways around it.

Statements fail with error code 266 or 778

The database was reset. Every statement throws an UrsulaResetError (errcode 266 for a read, 778 for a write), and db.status().reset names the kind:

• fenced (stop): another process opened the stream and owns it now. Find that owner before you open the stream again, since opening it takes the stream back.
• stale (reopen): the stream moved on while this process could not see it, for example after it was idle past producer expiry or opened provisionally during an outage. Open the stream again.
• rewound (reopen): a restore or an accepted loss removed commits the stream had acknowledged. Open the stream again: the open moves the cache aside into salvage/ (see below) and builds from the stream.
• deleted (fix): the stream was deleted. Open it with recreate: true to create it again from the cache's data, or open another stream.
• local_damage (reopen): a local fsync failed, a WAL page read back other bytes, or SQLite ended a commit without publishing it. Open the stream again: the log replays every commit. Check the disk if it repeats.
• stream_state (fix): another writer appended to the stream, the stream was closed, or the server lost acknowledged state. The stream no longer matches the cache. Look at who writes to the stream.
• internal (reopen): a bug inside the extension. A panic line on stderr names it. Open the stream again, and report it.

open fails

open throws an UrsulaSetupError or, when its signal fires first, an UrsulaUnavailableError. Every refusal leaves the cache as it was and sends nothing to the stream.

kind	Cause	What to do
server_too_old	The stream's HEAD lacks Ursula-Protocol: 2 or Ursula-Cluster-Id: a 0.7 server, or a layer that answers HEAD itself, such as a gateway whose authorizer conceals the stream, whose 404 carries neither header	Run a 0.8 server, check the token, and send HEAD through to the server
stream_format	The stream holds 0.7 data, or another content type	Move the database with an import, see Moving from 0.7
wrong_cluster	The cache was built on another deployment than the one the URL reaches now	Fix the routing, or use another dir for the other deployment
not_found	The stream is missing and the cache holds data	Open with recreate: true, or open another stream
in_use	Another process holds the cache	Stop that process, or use another dir
bound_elsewhere	The stream is open in this process under another dir	Close that database, or pass its dir
stream_state	The stream is closed, or an import into it is running or failed (the stream up to N is the claim of an import)	Open a closed stream read-only. Wait for a running import. After a failed import, delete or recreate the stream and import again
unreachable, unavailable	The signal fired while the stream could not be reached or did not answer	Check the network and the cluster. With a cache that claimed the stream before, open resolves provisionally instead

A cache appears in salvage/

status().salvage lists caches the extension moved aside, in salvage/ in the stream's cache directory: after a history change that lost commits the cache had, after a history change while the cache held commits it had no acknowledgement for, when the db file was not the cache's own (replaced), and with recreate (recreated). Each holds a plain SQLite database db, its log and a manifest.json with the tokens and the count of unacknowledged commits. Open db with any SQLite, compare it with the stream's database, and write again what you need. The extension never deletes a salvage.

snapshot.error is set

status().snapshot.error says why snapshots are not published, and snapshot.failures counts the attempts. A database whose compressed snapshot is above the request limit (32 MiB) needs a cold backend on the server, which takes it in parts, up to 1 GiB. Until a snapshot publishes, the log grows and every fresh host reads all of it. Failed attempts wait from 100 ms up to five minutes, or an hour when the body is too large.

file is not a database

A cache's db file holds a marker in place of SQLite's header magic, and its WAL is db-uwal. SQLite outside the extension (the sqlite3 shell, node:sqlite without it, a connection opened before it loaded, ?vfs=unix) fails with SQLITE_NOTADB and changes nothing. Read the database through open, or export it with VACUUM INTO from a connection open returned.

Closing a Pi session hangs during an outage

On Pi Durable 1.3, Session.close waits for the storage calls in flight, and through openUrsulaPiStorage each waits for synced(), which waits for the stream. So closeTimeoutMs applies only once the session reaches the storage's close. Bound the shutdown in your host when it must end during an outage. The log keeps every commit for the next open.

Still stuck?

Open a GitHub issue with:

• /__ursula/metrics from every node (with timestamps)
• Config file and CLI overrides per node (redact credentials)
• Last ~200 log lines at RUST_LOG=ursula_runtime=debug,ursula_raft=debug,ursula=info