Recovering stopped Raft groups

A panic or invariant violation while applying a committed command stops that local Raft group. Other groups remain running. Isolation relies on Rust panic unwinding, so a process abort, OOM kill or machine failure is outside it. The failed command receives no successful application response, and the applied index does not advance past it. A normal stream rejection (for example, deleting a missing stream) remains a deterministic application result and does not stop the group.

/__ursula/metrics reports raft_groups[].apply_failure with the log term, index, failure kind and diagnostic message. The affected group's maintenance.running is false. The maintenance report names the group with apply_stopped, so ursulactl wait-ready and rollouts refuse another disruption. The node's /__ursula/ready stays 200, its other groups keep serving, and planned leadership handoffs of those groups continue. Requests to the stopped group answer a typed 503. Invariant violations can have partially modified memory: that state must not be read, flushed, or made into a snapshot. A previously captured snapshot only contains its earlier successful prefix.

Supported recovery: corrected-code replay

This procedure requires the original durable WAL and snapshot/cold objects. It does not repair physical corruption or recover data lost beyond the chosen WAL fsync policy. Do not truncate a committed record, edit its bytes, reset its applied index, copy a snapshot made from failed memory, or use an unsynced-loss operation to bypass a poison record.

1. Record the group, failing log term/index, binary revision and diagnostics on every replica. Stop writes to affected streams. A failed request may already be committed, so retain its producer sequence and idempotency information.
2. Preserve the WAL directories, snapshot pointers and referenced cold objects. Have a corrected binary ready before stopping the affected server to take a consistent filesystem backup. Keep node IDs, membership and storage paths unchanged. Do not delete the failed group's data to make the pod healthy. A restart with the faulty binary keeps the stopped group stopped and the other groups serving, so it is safe but does not repair anything.
3. Correct the deterministic application bug, keeping command encoding and semantics compatible with the retained history. Reproduce the failing record on an isolated copy. Verify the last acknowledged records and the failing command are replayed exactly once into fresh in-memory state. A deployment of the same faulty code is expected to stop at the same record.
4. Deploy the corrected binary to the affected replicas using their original volumes. A group with a live quorum can resume by normal replay and catch-up. If every replica stopped, bring back enough corrected replicas for quorum. Never create a replacement cluster or silently skip a committed command.
5. Before restoring traffic, require no apply_failure, running/recovery-ready group metrics, a current leader and applied progress through the recorded failing index. Read and compare previously acknowledged payloads and the failed request using its original identity before deciding whether to retry.

If the failure occurs during initial WAL replay, the owner retains an ApplyStopped terminal group instead of a live Raft handle. Other groups can start and serve. Metrics retain the failing committed term and index and the last successfully applied prefix. Maintenance readiness refuses disruption while the node stays ready. Requests to the stopped group return ApplyStopped without repeatedly opening its WAL. A corrected compatible binary must restart and replay the original history before that group can serve again. Ordinary configuration, WAL-open and snapshot-restore errors still fail startup. They are not application failures and are not silently converted to isolated groups.

There is no online skip/quarantine command in this recovery contract. Repair requires a corrected compatible binary. If none is available, keep the group stopped and preserve its data. Rolling a snapshot or command format to an incompatible binary is not a recovery method.

Local drill

Run cargo test -p ursula-raft --lib apply_failure_tests. It uses real disk WALs with fsync=always, a partial-mutation panic and an invariant violation, and a three-replica in-process Raft network. It verifies stopped-group isolation, unchanged applied boundary, retained records, same-code replay failure with healthy-group reads and writes on the same owner thread, and payload preservation after corrected-code replay. The three-replica drill waits for a surviving follower to elect a new leader and encounter the committed poison through normal Raft replication.

The deterministic simulation a_poison_record_isolates_its_group_until_corrected_replay (RUSTFLAGS="--cfg madsim" cargo test -p ursula-sim --lib disk_tests) runs the same poison on every voter of one group while another group shares each node's core journal. It restarts a node with the faulty code and then replays every journal with corrected code. These are application-failure drills, not power-loss or Kubernetes Chaos qualifications.

WAL retention cost

A stopped group is excluded from snapshot selection and the lagging-snapshot queue. Reclaim moves each retained payload once into an immutable frozen-*.seg file beside the journal segments, using the existing journal encoding and disk interface. The shared journal keeps checksummed references. Later reclaim passes copy only those references, so the stopped group does not hold healthy groups' segments or repeatedly copy its payloads.

The archive is synced and its directory entry made durable before the journal reference is synced. Only then can reclaim discard the original records. Recovery verifies every referenced archive and the exact retained log IDs. Missing or damaged archives fail startup. Segment continuity checks still apply. Preserve the entire WAL directory, including the frozen files, during repair.

wal_stopped_rewritten_bytes counts payload bytes moved into archives. wal_frozen_reference_rewritten_bytes counts later reference copying. wal_physical_bytes includes retained archive files. Alert on ursula.raft.apply_stopped and monitor retained disk space while repairing the binary. The payload counter should stop growing once that group's retained records have been archived, even while healthy groups keep writing.

Corrected code replays the same committed entries through the references. Archives are collected after no remaining physical journal record references them and the old segments' deletion is durable. Cleanup failures are retried without blocking healthy journal reclamation. During an unverified-tail repair run, cleanup waits until the affected segment has been durably reclaimed. A later clean startup validates the durably repaired journal before collecting orphans.

A binary that predates frozen references rejects them on replay, including in the newest segment. Once references have been written, repair with compatible code rather than downgrading or deleting the archives.