Troubleshooting

Start at /__ursula/metrics. Most operational symptoms have a clear signal in the JSON snapshot.

Diagnostic surface

curl http://NODE:4437/__ursula/metrics > metrics.json

Useful jq selectors. Arrays named per_core_* are indexed by core, and arrays named per_group_* by Raft group ID.

# Leader, term and applied index of each group. current_leader is null while the group has no leader.
jq '.raft_groups[] | {raft_group_id, current_leader, current_term, last_applied_index}' metrics.json
# Groups without a leader
jq '[.raft_groups[] | select(.current_leader == null) | .raft_group_id]' metrics.json
# Hot bytes waiting for cold flush, per group
jq '.per_group_cold_hot_bytes' metrics.json
# Writes refused by cold-write backpressure, in total and per group
jq '{total: .cold_backpressure_events, per_group: .per_group_cold_backpressure_events}' metrics.json
# Mailbox depth and capacity, per core
jq '{depths: .mailbox_depths, capacities: .mailbox_capacities}' metrics.json
# Live reads waiting for new data, per core
jq '.per_core_live_read_waiters' metrics.json
# SSE streams opened and SSE error events
jq '{opened: .sse_streams_opened, errors: .sse_error_events}' metrics.json
# WAL disk pressure
jq '{available: .wal_available_bytes, minimum: .wal_min_available_bytes, pressure: .wal_disk_pressure}' metrics.json
# WAL reclaim and restart work
jq '{physical: .wal_physical_bytes, segments: .wal_segments, reclaims: .wal_reclaims, lagging: .wal_lagging_groups, recovered_bytes: .wal_recovery_bytes, recovery_ns: .wal_recovery_ns}' metrics.json
# How the WAL opened, and the groups behind a recovery gate
jq '{opening: .wal_recovery, gates: .recovery_gates}' metrics.json
# Local Raft issues that make the node unready or block maintenance
jq '.raft_maintenance | {node_issues, group_issues}' metrics.json

GET /__ursula/ready is the process readiness probe. It answers 200 when the node may serve, and otherwise 503 with a typed reason. When several apply, it reports the first of:

reason	Meaning
format_epoch_mismatch	The node exchanged a Raft RPC with a peer of another format epoch. It stays unready until it restarts
wal_disk_pressure	The WAL filesystem is below raft.wal.min_available_size, or its free-space check failed
recovery_stalled	A group's recovery gate on this node is stalled. The groups are listed under recovery_stalled_groups
recovery_gate_closed	A group's recovery gate on this node is closed until the replica has applied a fresh barrier from the group's leader
raft_replica_unready	A local replica is not a healthy voter. raft_maintenance.group_issues names the issue

Ursula does not expose /healthz, /readyz, Prometheus /metrics, or /cluster/status. Use the JSON snapshot for detailed diagnosis and ursulactl wait-ready for cluster-wide Raft readiness.

Serving during a replica rebuild

raft_replica_unready means local Raft participation is not safe for serving. Inspect raft_maintenance.group_issues for local_replica_not_voter, membership_not_applied, not_applied, apply_lag or leader_unknown. A wiped replica stays 503 while its recovery gate is closed and while it is a learner. Healthy surviving voters can return 200 during the repair. A voter leaving a joint membership also stays unready once any configuration excludes it. A replica whose own gate is open reports leader_unknown for a group that has no leader, for example a group stalled because a majority of its voters are gated.

A 200 response does not certify full redundancy. Keep maintenance blocked until ursulactl wait-ready succeeds. Alert separately on incomplete voter sets in the maintenance report, as described in Observability.

503 with ApplyStopped from one group

A committed command failed to apply and stopped that group on this replica. The node stays ready and serves its other groups. Its maintenance report names the group with apply_stopped, so ursulactl wait-ready refuses disruption until the group recovers. Follow Recovering stopped Raft groups and never skip the committed record.

Startup failures

WAL topology mismatch or missing topology

topology.bin binds a WAL root to runtime.core_count and raft.group_count. For a mismatch, restore the stored counts reported in the error. A missing record on an existing WAL or a corrupt record also refuses startup. Do not replace it with the current configuration. See WAL routing configuration for rebuilding a replica.

raft.wal.path is required when raft.peers is set

Every node of a static cluster keeps its Raft WAL in raft.wal.path. Set it to a directory on a persistent volume. Only a single node without raft.peers may omit it, and then its WAL lives in a temporary directory that is removed on clean shutdown.

raft.node_id must be non-zero

Static Raft mode requires a unique raft.node_id in the config file or a --node-id override.

raft.peers must include this node id

The local raft.node_id must appear in [[raft.peers]]. The peer list always includes self.

storage.cold.s3.bucket is required when cold backend is 's3'

Set storage.cold.s3.bucket, or set storage.cold.backend = "none" if you don't want cold storage.

Port already in use

server.listen is taken. Public HTTP and inter-node gRPC share the same port unless server.cluster_listen is configured separately.

raft WAL directory … holds data without a format marker

The data directory was written by Ursula 0.5.x or earlier (format epoch 1). This release reads format epoch 3 only and has no in-place upgrade. Start on an empty raft.wal.path and a new object-storage prefix, or run the release that wrote it.

raft WAL directory … is format epoch N

The directory's FORMAT_EPOCH marker names another epoch. This release reads format epoch 3 only. Run the release that wrote it, or start on an empty directory.

object storage s3://… is not empty and has no URSULA_FORMAT_EPOCH marker

The cold root (or, with cold storage off, the S3 snapshot namespace) holds objects from Ursula 0.5.x or from another application. Point storage.cold.root (Helm: s3.prefix or coldStorage.prefix), or storage.snapshot.s3_prefix when only snapshots use S3, at an empty prefix used only by this cluster.

object storage s3://… is format epoch N

The prefix's URSULA_FORMAT_EPOCH marker names another epoch. Use a new prefix and never reuse one across epochs.

journal … uses Ursula WAL version N

A Raft WAL journal of another format epoch: version 1 is Ursula 0.5.x, version 2 is 0.6. This release reads format epoch 3 only. The directory marker normally refuses the directory first, and the journal check is the backstop. Start on an empty raft.wal.path.

peer N (…) refused Raft protocol 3

A configured peer runs a release of another format epoch (Ursula 0.6 speaks protocol 2, 0.5.x protocol 1). A node cannot join a cluster of another format epoch. Install this release as a new cluster.

/__ursula/ready answers 503 with format_epoch_mismatch

The node exchanged a Raft RPC with a peer on another format epoch: a peer that was unreachable during the startup probe, or a node of another release that has this node in its peer list. The node stays not ready until it restarts. Remove the mismatched node from the peer lists and restart.

/__ursula/ready answers 503 with recovery_gate_closed

The node's logs may be missing entries it acknowledged: it started after a host crash or power loss under raft.wal.fsync = "never", after a journal I/O error, or without its run state file, or with an emptied WAL directory. Under raft.wal.fsync = "always", a host crash gates the groups of a core whose newest journal segment ended in a complete invalid frame, although no acknowledged write was lost. Each gated group waits until it has applied a fresh barrier from the group's leader, and its leader rebuilds the replica meanwhile. recovery_gates in /__ursula/metrics lists the gated groups. This normally clears in seconds once a leader is reachable. See Recovering after a host crash.

/__ursula/ready answers 503 with recovery_stalled

A gated group on this node has seen no leader barrier and applied nothing for 30s. A majority of the group's voters are probably gated, so the group has no leader and refuses writes. The groups are listed under recovery_stalled_groups. Restore connectivity to a healthy quorum first. If the stalled replicas retain their voting history, an operator can accept the loss of writes acknowledged after the last fsync, with POST /__ursula/raft/{group}/recovery/accept-unsynced-loss on the gated replicas with the most up-to-date last log id (highest last_log_term, then highest last_log_index), until the open replicas form a majority. A replica without a durable vote floor refuses this call and still needs a fresh quorum proof. The request body names the replica's last_log_index and current_term from /__ursula/metrics. A voter of the group that was never gated does not report recovery_stalled. It answers raft_replica_unready with leader_unknown for the group. See Recovering after a host crash.

accept-unsynced-loss answers 409 Conflict

Nothing changed. A replica that lost its voting history must first obtain a durable vote floor from a fresh quorum proof, and the call cannot replace that proof. Restore connectivity to intact voters.

The call also refuses when the replica's recovery gate is not stalled, so it may still open through a leader's barrier without losing anything, or the replica no longer holds the last_log_index and current_term the request names. Wait for recovery_stalled, read /__ursula/metrics on the replica again, and decide again with what it shows.

I/O error on raft.wal.path

The directory must be writable and owned by exactly one Ursula process. Each core journal holds an advisory journal.lock containing the owner PID. Do not remove it while that process is alive; stop the conflicting process or fix the volume mount. A stale lock file is harmless because ownership is enforced by the OS lock, not the file's existence.

Backup and restore

backup manifest format_version N or target cluster speaks backup format N

The backup or the cluster is of another format epoch. This release reads and writes backup format 3 only, so a backup of another epoch cannot be restored, and a cluster of another epoch cannot be backed up or restored into.

… cold objects this backup references are missing from the target cluster's cold store

The backup references cold objects that the target's storage.cold.root does not hold. restore stopped before the first import, so the target is still empty. Copy the source cluster's cold objects into the target's root (see Copying cold objects) and run restore again. The error names the first missing keys. A key under cold-index/ is an index page, so everything that page lists is unchecked until the page is there.

Bootstrap and cluster join

A fresh cluster never elects leaders

/__ursula/metrics on each node should show a non-null current_leader and a stable current_term for every group in raft_groups. If groups stay leaderless:

• First start must include raft.init_membership_per_group = true (or raft.init_membership = true) on every voting node.
• Every peer URL must be reachable from every node.

Replacing or restarting a node leaves it leaderless

Restart with the same raft.node_id and peer list. A fresh raft.wal.path means rehydrating from peers, which can be slow on cold-storage-only history (see Rejoining a wiped node).

Write path

503 with Retry-After on append

Inspect the JSON error. WalDiskPressure means the WAL filesystem is below raft.wal.min_available_size (or its free-space stat failed); the node also becomes unready and attempts to yield its leaders. Free space or expand the volume. Admission resumes only at raft.wal.resume_available_size, preventing probe flapping.

RaftLogPressure means the node's unsnapshotted Raft log is over twice raft.snapshot_log_budget. The node refuses writes with a body until snapshots bring the log below 1.5 times the budget. Before it gets there, the snapshot driver runs more snapshot builds at once the further the log is past the budget, up to raft.snapshot_build_max_concurrency. When writes still reach the limit, the node's metrics show how far the log got (raft_log_bytes_high_watermark), whether the driver used every build (raft_snapshot_builds_in_flight_max), and how long a build's upload takes (raft_snapshot_build_ns over raft_snapshot_builds). If every build was in use, raise raft.snapshot_build_max_concurrency (the chart's raft.snapshotBuildMaxConcurrency) or the budget. Each build holds its group's captured state and, while it uploads, the compressed snapshot, so raise the builds only as far as the node's memory holds that many builds of its largest groups. Set it to 1 to keep one build at a time, as before 0.7.4.

IngressBodyBytesLimitReached means the write bodies in flight on the node leave too little of server.http_inflight_body_size for this one.

WalDiskPressure, RaftLogPressure and IngressBodyBytesLimitReached are write admission refusals. Their responses also carry the code in x-ursula-not-applied: the node refused the request before it proposed anything of it. A client can count the attempt as never applied only when the response also carries x-ursula-not-applied-checked, which a gateway at 0.7.3 or newer adds. See refusals that applied nothing.

Other 503s come from the stream's Raft group, and their text names the cause. ColdBackpressure means that the group's hot ring would exceed storage.cold.max_hot_size_per_group: cold flush isn't keeping up, or the cold backend is slow or erroring out. Raise the per-group ceiling, lower storage.cold.flush_interval, raise storage.cold.flush_max_concurrency, or fix the cold backend. RaftUncommittedBackpressure means that the group's write bytes that are proposed but not yet applied would exceed raft.max_uncommitted_size_per_group.

The node refuses both before it proposes the append, so from 0.7.3 their responses also carry the name in x-ursula-not-applied, like the admission refusals above. A 503 while the group has no known leader, for example during an election, carries NotLeader when the node found that before it proposed the append. A 503 from a node that does not host the stream's group and knows no node that does carries GroupNotHosted: the node refuses before any engine runs.

409 Conflict with producer-expected-seq / producer-received-seq

Out-of-order producer headers. Response tells you what was expected:

• producer-expected-seq: N is the next allowed sequence
• producer-received-seq: M is what the client sent

Common causes:

• Producer restart without bumping Producer-Epoch. Bump every restart.
• Two writers share the same Producer-Id. Use distinct IDs.
• A retry skipped a sequence number. Producer sequences must be contiguous within an epoch.

409 Conflict from Stream-Seq

The supplied Stream-Seq is not bytewise greater than the last value accepted on this stream: another writer committed first, or the values are not fixed-width ("10" sorts before "9"). The response's Stream-Next-Offset is the tail. Read what you missed, re-derive the value and retry. See Conditional writes.

404 Not Found on POST /{bucket}/{stream}

The stream hasn't been created. Create with PUT /{bucket}/{stream} first. There is no implicit creation on POST.

400 Bad Request on JSON appends

JSON streams are validated on the server. Malformed JSON, invalid UTF-8, messages nested deeper than 127 levels, and empty arrays without allow_empty_array are rejected. Send valid JSON or switch to application/octet-stream.

Read path

No Stream-Up-To-Date on a read

The response is partial: cut by max_bytes or the 8 MiB cap, or served by a follower, which never sends it with data. Read on from Stream-Next-Offset. Not an error.

SSE drops after 30-60 s idle

A proxy or load balancer is closing idle TCP. Raise its idle timeout or enable TCP keepalive on the proxy-to-Ursula leg. Ursula doesn't emit SSE keepalive comments.

SSE on a binary stream returns base64 text

Expected. SSE wire format is text-only, so binary stream data events carry raw base64 text and the Stream-Sse-Data-Encoding: base64 header signals it. See binary SSE.

Reads stall for hundreds of ms for cold offsets

The first read of a cold offset triggers an S3 GetObject range read. It runs off the actor turn so it doesn't block other commands, but the request still pays S3 latency.

Replication and consensus

A group has no leader (current_leader: null)

Either an election is in progress or quorum is lost for that group. Only streams hashed to that group are affected.

• Multiple nodes briefly claim leader: election is flapping. Check peer reachability and CPU pressure.
• No node claims leader: fewer than n/2+1 voters are reachable, or a majority of the group's voters are behind their recovery gates. Restore reachability. For gated voters, see Recovering after a host crash.

One follower lags behind

State-machine apply runs on the follower's owner core. If CPU is pinned, last_applied_index trails committed_index. Check:

• Per-core mailbox depth (mailbox_depths). Sustained queueing means saturation.
• Disk pressure on the WAL volume (fsync latency, in particular with raft.wal.fsync = "always").
• A pathological hot stream forcing constant cold flushes (per_group_cold_hot_bytes).

Manually trigger a snapshot

These are admin-plane endpoints (server.admin_listen, loopback :4438 by default), reached over a tunnel (see ursulactl's operation providers). The examples assume a forward from local 4438 to the node's admin plane.

process_incarnation=$(curl --fail http://127.0.0.1:4438/__ursula/metrics | jq -er .process_incarnation)
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/{group_id}/snapshot
curl --fail -H "x-ursula-process-incarnation: ${process_incarnation}" -X POST http://127.0.0.1:4438/__ursula/raft/{group_id}/purge

The snapshot request answers once the group has a snapshot of everything it had applied when the request arrived. It waits while a build of the group runs and while other snapshot builds hold all of the node's build permits, and a snapshot that a build begun since the request made answers it too. When no build of the group can start within 10 seconds, it answers 503 with Retry-After: 1. Send it again then. Unless the group's published snapshot already covers the request, it then waits up to 10 seconds for the group's running build to end and up to 10 more for the snapshot to show, and answers 500 when it does not.

Purge only after the snapshot replicates. Otherwise a slow follower may need log entries you just dropped.

Cold flush issues

Writes succeed but data never reaches S3

• storage.cold.backend is none, or set to memory when you expected s3.
• IAM permissions missing. Required: s3:GetObject, s3:PutObject, s3:ListBucket, s3:DeleteObject.
• storage.cold.s3.endpoint unreachable.
• Cold flush failing or falling behind (cold_flush_write_errors or cold_backpressure_events climbing, cold_hot_bytes growing).

If hot bytes grow unbounded, you'll eventually hit storage.cold.max_hot_size_per_group and writes start returning 503. Fix the backend before that.

SQLite VFS

These entries cover the sqlite-ursula-vfs extension and @tonbo/sqlite-ursula. See SQLite on Ursula.

A commit or attach is refused with error code 7178

SQLITE_IOERR_AUTH (errcode 7178 in node:sqlite, UrsulaWriteRefusedError from open, its database, attach and openUrsulaPiStorage) means that a write was refused before it reached the stream, so nothing of it was applied. A refused commit was rolled back, but the file is not poisoned. Reads go on, and the next commit succeeds once the refusals stop, with no new attach. A refused attach claimed nothing. Attach again later, or attach read-only to read the database meanwhile. An attach whose claim a 402 refuses fails with SQLITE_AUTH (errcode 23) instead, and open and attach throw UrsulaPaymentRequiredError. For a refused commit, status(file) (ursula_status) reports writes_refused: true and names the refusal:

refused_kind	Refusal	What to do
payment_required	A layer in front of the stream answered 402	Settle what the layer asks for. refused_reason carries its explanation
unauthorized	The token was refused (401) until URSULA_VFS_RETRY_MS ran out	Set a fresh token with setToken, or fix the file URSULA_VFS_TOKEN_FILE names
forbidden	The gateway's authorizer denied the write with a 403	Allow the write in the authorizer
rate_limited	429 until URSULA_VFS_RETRY_MS ran out	With refused_reason tenant request rate limit exceeded, lower the write rate or raise the bucket's requests_per_sec in the gateway's --quota-policy. With tenant concurrent upload limit exceeded or tenant upload bytes limit exceeded, the bucket's large bodies under way reached its max_concurrent_uploads or max_upload_bytes there: raise them, or keep commits at or below the gateway's --small-body-max-bytes (1 MiB by default), which never count
not_applied	Any other refusal marked x-ursula-not-applied, vouched for by ursula gateway 0.7.3 or later	See refused_reason below
too_large	The transaction's commit frame is over URSULA_VFS_MAX_FRAME_BYTES (32 MiB by default), so the extension never sent it. Or a lower limit in front of the stream, such as the gateway's --max-request-body-bytes or a bucket's body limit, refused it with a 413 that carries both headers	Split the transaction. Other commits succeed as usual, and the next one that does clears the refusal. See Rules
unreachable	From 0.7.4: until URSULA_VFS_RETRY_MS ran out, every attempt failed before any byte of it went out, or was refused as above, and the last one failed before it went out. The stream URL's host did not resolve, or no connection to it was made: refused, timed out, or a failed TLS handshake. Nothing of the commit was sent	Check that the stream URL's host resolves from the client and that the gateway or the server behind it is up and reachable, through the proxy in HTTPS_PROXY, HTTP_PROXY or ALL_PROXY when one is set (NO_PROXY excludes hosts). A failed TLS handshake shows in refused_reason as rustls reports it. io: invalid peer certificate: UnknownIssuer means the client does not trust the endpoint's certificate authority: point URSULA_VFS_CA_FILE at it. io: invalid peer certificate: certificate not valid for name ... means the stream URL names a host the certificate does not cover: use one it covers. io: received corrupt message of type InvalidContentType means an https:// URL points at a port that speaks plain HTTP. io: received fatal alert: ... means the endpoint refused the handshake, for example over its TLS versions or cipher suites. Commits succeed again once the endpoint is reachable

refused_reason is the refusing layer's response body, or the value of x-ursula-not-applied when the body is empty, and always that value for not_applied. For too_large it is the frame's size and the limit, such as 41943104-byte frame over the 33554432-byte frame limit, or the value of the 413's x-ursula-not-applied. For unreachable it is the last attempt's failure, such as timeout: resolve or io: Connection refused (os error 111). The values Ursula sends for not_applied:

refused_reason	Meaning
RaftLogPressure	The server's Raft log is over its hard limit under sustained write load. Writes resume once snapshots catch up
WalDiskPressure	The server's WAL filesystem is below raft.wal.min_available_size. Free space or expand the volume
IngressBodyBytesLimitReached	The server holds as many request bytes in flight as it admits
RaftUncommittedBackpressure	The stream's Raft group holds more proposed but unapplied write bytes than raft.max_uncommitted_size_per_group allows
ColdBackpressure	The stream's Raft group holds more hot bytes than storage.cold.max_hot_size_per_group allows, because cold flush is behind or the cold backend is slow
NotLeader	A 503: the server does not lead the stream's Raft group and knows no leader yet, for example during an election. For the same reason, the redirects it sends before it proposes a write carry the same marker, and the gateway follows them. Writes resume once the group has a leader
GroupNotHosted	A 503: the server does not host the stream's Raft group and knows no server that does. Its redirects to a server that does carry the same marker, and the gateway follows them
RequestBodyUnreadable	The gateway could not read the request body, so it sent nothing on
UploadBudgetExhausted	A 503: the gateway's request body budget (--body-budget-bytes) stayed spent for --body-budget-wait-ms, so it sent nothing on. Commits of at most --small-body-max-bytes have headroom of their own, so this hits large commits under heavy upload load. Writes resume once uploads under way end. The gateway's upload_refusals_budget counter shows these refusals
RequestBodyTooSlow	A 408: no byte of the body reached the gateway for --body-stall-timeout, or less than --body-min-bytes-per-sec arrived over one such window, so the gateway cut it and sent nothing on. With the defaults (30 s, 1 KiB/s) this does not happen to a commit, since the extension gives up on a commit after 10 s. Over HTTP/1.1 the gateway closes the connection with the 408, so a client still sending usually meets the closed connection instead. Keep --body-stall-timeout above 10 s, or a commit still being sent can meet that closed connection on every attempt and poison the database. The gateway's upload_refusals_too_slow counter shows the cuts
NotFound	The gateway's access policy does not serve the stream to these credentials: no token, or one the policy does not allow. The cause is the credentials or the policy, not load
AuthenticationUnavailable, AuthorizationUnavailable	The gateway could not verify the token or reach a decision, for example while its token issuer is down
NoUpstream, UpstreamUnreachable	The gateway could not reach a server, so it sent nothing on. After redirects, the nodes it reached before refused the write as not their own to propose
LeaderNotFound	The gateway found no node that leads the stream's group among its upstreams within three hops and 5 s, and every node it reached refused the write before proposing it. Writes go through once the group has a leader. If it persists, check that --upstream reaches every voter, by its peer URL where possible

Refusals that applied nothing lists every value. A commit fails with 7178 only when every attempt of it was refused this way, or failed before any byte of it went out. One attempt that went out and got no answer, a 307, or a 5xx without both headers, poisons the file instead, because that attempt may have been applied.

The file is poisoned at the commit after a refused one

status(file) reports poisoned: true with a reason that starts with a commit refused as never applied may have been applied after all. A commit failed with 7178, refused with an answer that says nothing was applied (a 401, 402 or 429, or a refusal marked x-ursula-not-applied with the gateway's vouch). The next commit went out with the same producer sequence, and the stream answered it as a duplicate. So the stream holds a commit at that sequence that the file lacks: either the refused commit, applied after all, or the next commit, sent again by a layer in front of the stream. The answer does not tell which, so the extension poisoned the file instead of acknowledging a commit that may never have reached the stream.

Attach the file again. The attach replays whichever commit the stream holds. Check which of the two transactions the database then holds before you run either again. When it holds the refused one, although its commit failed, a layer broke the rule in Writes refused before they reach the stream. Find it:

• A proxy or service mesh that retries writes. Istio before 1.24 retries a 503 twice by default, POSTs included, and Istio 1.24 to 1.26 does the same when istiod runs with EXCLUDE_UNSAFE_503_FROM_DEFAULT_RETRY=false, which compatibilityVersion=1.23 sets. Any version retries on a route whose retryOn includes 503, 5xx, gateway-error or reset. Set retries: attempts: 0 on the routes that carry stream writes, to the gateway and from the gateway to the servers.
• A proxy that follows redirects for writes.
• A layer between ursula gateway and the servers that answers 401, 402 or 429.
• A layer that refuses a write after it passed the write on, such as metering that answers 402 after the stream answered.

When the database holds the next transaction instead, a gateway sent that commit again to a new leader, after the old leader proposed it and redirected the gateway. No layer broke the rule, and nothing is lost.

Reads of a fenced file fail with error code 266

From 0.7.5, once an owner knows that another owner claimed its stream (its commit failed, and status(file) reports poisoned: true and fenced: true, or error.kind fenced), every new read transaction on the file fails with SQLITE_IOERR_READ (errcode 266 in node:sqlite, disk I/O error). open's database and openUrsulaPiStorage report these reads, those inside a transaction included, as UrsulaFencedError (an UrsulaReplicationError with fenced: true, whose action is stop) with a message that starts with read refused. The file no longer holds what the stream does, so it serves nothing. A read transaction already open when the fence landed keeps its snapshot until it ends. Find the other owner. Attaching the file again takes the stream back from it, so do that only for a deliberate takeover.

A file opens with "unable to open database file" after a failed attach

From 0.7.5 a file whose attach failed in the process (SQLITE_CANTOPEN, errcode 14) opens only after an attach of it succeeds, whether or not it has a sidecar <db>-ursula, so that no commit bypasses the stream. status(file) throws an error that names the failure: ursula_status: attach_failed: <kind>: <file> is not attached: its last attach failed (<reason>). The kind says what to do, as in ursula_attach's own error: invalid_name or invalid_url (fix the stream URL), unreachable or unavailable (attach again later), local_setup (the directory refuses the file: fix its permissions, its mount or its free space), outcome_unknown (the attach failed after its claim, which may have fenced the earlier owner: attach again), fenced (another owner claimed the stream during the attach). A failed attach that wrote nothing into a new file removes the sidecar it created. An attach refused because of the file itself, before it changed anything, records no failed attach and changes nothing. It fails with in_use while the process has the file, its -wal, -shm or -journal open outside the extension (a connection opened before the extension was loaded, or through another VFS, read-only ones included), or another process has the file open: close those connections, or that process, and attach again. The process's own descriptors are checked on Linux with /proc mounted and on macOS only. It fails with unreplicated_file when the file holds data but no sidecar: the file still opens, so copy its rows into a newly attached file, or move it and its -wal aside to attach the path. An attach that meets another process holding the file only after it changed the local files is a failed attach (attach_failed: in_use).

Still stuck?

Open a GitHub issue with:

• /__ursula/metrics from every node (with timestamps)
• Config file and CLI overrides per node (redact credentials)
• Last ~200 log lines at RUST_LOG=ursula_runtime=debug,ursula_raft=debug,ursula=info