Configuration

Everything configurable in Ursula lives on this page. Ursula is configured by a TOML config file, an optional resource preset, and a small set of CLI overrides, merged in this order:

built-in defaults < preset defaults < config file < CLI overrides

The normal startup command is:

ursula --config /etc/ursula/ursula.toml --node-id 1

With no config file, ursula falls back to the default preset, which starts a single-node in-memory runtime without Raft, and nothing survives a restart. That's what Install uses. Every other preset, and a config file loaded without --preset default, runs Raft on the disk WAL described under Raft.

CLI flags

Flag	Default	Notes
--config FILE	searches ./ursula.toml, /etc/ursula/ursula.toml, then user config paths	TOML config file
--preset PRESET	default when no config file is found, otherwise none	Resource preset: default, dev, tiny, small, standard, or large
--node-id ID	config file or preset	Overrides raft.node_id

CLI --node-id intentionally overrides any raft.node_id value in the config file so the same file can be mounted on every node. Helm derives it from the StatefulSet ordinal.

Resource presets

--preset supplies a base configuration before the config file is merged, and config-file values win. Pick by node size:

Preset	Intended use	Main defaults
default, dev	Local development	Single-node in-memory runtime without Raft
tiny	Memory-bound tests or very small nodes	64 groups, 64 MiB cold cache, 4 MiB cold flushes, 2 cold writes, 8 MiB hot/admission caps, 64 MiB HTTP in-flight body budget
small	Small nodes or cost-sensitive tests	128 groups, 64 MiB cold cache, 4 MiB cold flushes, 2 cold writes, 16 MiB hot/admission caps, 64 MiB HTTP in-flight body budget
standard	Production baseline	256 groups, 256 MiB cold cache, 8 MiB cold flushes, 4 cold writes, 64 MiB hot/admission caps, 256 MiB HTTP in-flight body budget
large	Larger nodes with more cache and write headroom	512 groups, 512 MiB cold cache, 16 MiB cold flushes, 8 cold writes, 128 MiB hot/admission caps, 512 MiB HTTP in-flight body budget

Server and runtime

Config key	Default	Purpose
server.listen	127.0.0.1:4437	Client-plane listener: the stream API, metrics, readiness and usage
server.cluster_listen	none	Separate cluster-plane listener for Raft gRPC between voters. Unset, the cluster plane shares server.listen. Set, every [[raft.peers]] entry needs a client_url
server.admin_listen	127.0.0.1:4438	Admin-plane listener for the mutating operator routes, plus metrics, readiness and usage. Keep it on loopback and reach it through a tunnel
server.cluster_id	none	The deployment's id, sent as Ursula-Cluster-Id on every HEAD: 1 to 64 characters from A-Z, a-z, 0-9, ., _ and -. Required when raft.wal.path is set, the same on every node and different for every deployment a client can reach. A single node without raft.wal.path runs under a generated one, u followed by 32 hex digits, new on every start. The WAL records the id, and a node whose configured id differs refuses to start. Peers check it: every Raft gRPC request carries the sender's id, and a node refuses a peer with another id (see Peers of another cluster)
server.http_inflight_body_size	256MiB (64MiB in the tiny and small presets)	Write body bytes the node holds at once. A write that would exceed it answers 503 with INGRESS_BODY_BYTES_LIMIT_REACHED. A snapshot commit's copy of parts is charged against it too, from 32 MiB to 56 MiB depending on the budget, and starts only while 32 MiB stays free beside it. With a cold backend it must be at least 64MiB, or the node refuses to start
runtime.cpu_affinity	automatic	On Linux, true forces pinning and false disables it; omitted pins only when cpuset size equals core_count
runtime.core_count	available_parallelism	Owner worker threads; Linux automatically pins only when the allowed cpuset matches this count. Journal writers and the main Tokio runtime use additional threads
runtime.live_read_max_waiters_per_core	65536	SSE waiter cap per core. 0 disables the limit
observability.tokio_console	false	Enables Tokio console when the binary is built with the feature

A node refuses a request body over 32 MiB with 413 and names that limit in Ursula-Max-Request-Bytes. The limit is the protocol's and is fixed. The gateway's --max-request-body-bytes (the chart's gateway.maxRequestBodyBytes) is at most 32 MiB, and a gateway or a chart set above it refuses to start, since the node refuses such a body before reading it and the gateway would answer 502 instead of a typed 413. With a cold backend, a larger snapshot goes up in parts instead.

Without server.cluster_listen, server.listen serves both the client and the cluster plane, and each [[raft.peers]].url points at that address. Leader redirects then name the same URL, so clients must be able to reach it too.

With server.cluster_listen set, server.listen serves only the client plane. Each [[raft.peers]].url points at the peer's cluster listener, and each [[raft.peers]].client_url at its client listener. A follower redirects a client to the leader's client_url, so only peers need to reach the cluster listener. A node refuses to start when a peer lacks client_url. ursula gateway follows a redirect to an upstream directly when its --upstream equals the leader's client_url, and searches its upstreams for the leader otherwise. See Security for what each plane serves.

[server]
listen = "0.0.0.0:4437"
cluster_listen = "0.0.0.0:4439"

[[raft.peers]]
node_id = 1
url = "http://10.0.0.1:4439"
client_url = "http://10.0.0.1:4437"

Raft

Config key	Default	Purpose
raft.node_id	none	This node's stable ID, usually overridden per host with --node-id
raft.append_transport_budget_bytes	134217728	Node-wide charged append bytes per direction, shared across cores and peers; excludes tonic/socket buffers
raft.group_count	the available parallelism × 16, at most 4,096	Total data groups, at most 4,096: a larger value is refused at startup, and the default is capped there. The meta group's snapshot holds one placement per group. Higher values improve stream-to-group hash spread. Required once raft.peers lists another node, since every node of a cluster runs the same groups. Every node of a fresh cluster claims it at genesis, and a node whose value differs from the cluster's reports group_count_mismatch
raft.wal.path	none	Directory of the Raft WAL. Required when raft.peers lists another node. With it, raft.peers must list this node and server.cluster_id must be set, on a single node too. Raft logs live under PATH/raft-log, and removing the directory wipes state cleanly. Without it, a single node runs its WAL in a temporary directory that is removed on clean shutdown
raft.wal.fsync	always with one or two voters per group, else never	When journal appends reach stable storage. Unset, it takes always when min(raft.placement.replication_factor, N) over the N nodes of raft.peers is 1 or 2, and never otherwise. never acknowledges ordinary data appends from the page cache. Membership batches are always fsynced before acknowledgment. always acknowledges a batch after fsync. Votes, each group's log state and the node's run state are always written with fsync, and the meta group's journal in raft-log/meta/ always runs always
raft.wal.segment_size	64MiB	Size at which each core's journal starts a new segment. Purge deletes whole segments, so this is also how much disk space comes back at a time. At least 4KiB
raft.wal.cache_size	256MiB	Recent Raft entries kept in memory across the node, split evenly between its groups. Older entries, such as a lagging follower's catch-up, are read from disk
raft.wal.min_available_size	512MiB	Reject writes and mark /__ursula/ready unavailable below this much free WAL-filesystem space. 0 disables the guard
raft.wal.resume_available_size	1GiB	Free space required before disk-pressure admission clears. Must exceed the minimum
raft.wal.lease	5s	The disk lease. With no synced probe of the WAL disk for this long, the node stops all Raft traffic, leads no group, sends no object writes and answers clients 503 until a probe succeeds. From 2s to 10s. Raise it on slow disks
raft.peers	empty	Static gRPC peer list, with one [[raft.peers]] entry per node, including this node. Every node with raft.wal.path lists itself, a single node included. A single node without raft.wal.path and without peers runs with an entry of itself, at server.cluster_listen, else server.listen. Each entry has node_id, url (the address peers dial for the cluster plane) and client_url (the client-plane address leader redirects name, required with server.cluster_listen and url by default). It is the only source of peer addresses: keep it the same on every node, and change it there to move a node (Changing a peer address)
raft.placement.replication_factor	3	Voters of each data group, capped at the node count. When the cluster first forms, group g is placed on that many consecutive raft.peers nodes in ascending node ID, from index g mod N, and the first of them initializes it once every voter reports the group empty and no leader. A replica that ever held its group never initializes it again. Read at genesis only. A move keeps a group's voter count, and a node added to raft.peers later hosts no group until groups move onto it (Placement and the meta group)
raft.placement.max_concurrent_moves	4	How many groups may move at once, recorded when the cluster bootstraps. 0 stops every move. Change it on a running cluster with ursulactl placement set-move-limit
raft.meta_voters	the smallest min(3, N) IDs of raft.peers	The meta group's voters, as a list of raft.peers node IDs, for example [1, 2, 3]. Read at genesis only: the meta group's voters never change afterwards, and a meta voter is never decommissioned. An empty list is refused, a repeated ID counts once, and an ID outside raft.peers is refused. Every other node is a meta learner

The Raft WAL is always the per-core disk journal. A node whose raft.peers lists another node refuses to start without raft.wal.path. A single node without a path creates a fresh ursula-wal-* directory under the system temporary directory, logs its location at startup, and removes it on a clean shutdown (SIGTERM or Ctrl-C). A failed start, or a shutdown that cannot close the WAL, leaves the directory in place. Its streams survive no restart. Set raft.wal.path to keep them.

With fsync = "never", a process crash loses nothing because the page cache survives it, and a graceful shutdown fsyncs every journal before it exits. A host crash or power loss can drop the unsynced tail of that replica's journal. The replica restarts from the frames that still verify and reports that it is recovering. Each of its groups stays behind a recovery gate: it does not campaign or vote until it has applied a fresh barrier from the group's leader, so it never helps elect a leader that lacks a write it acknowledged. While at most a minority of a group's voters lost their tail, the other voters hold every acknowledged write and the group keeps serving. If a majority lost it at once, the group has no leader and refuses writes until an operator accepts the loss (see Operations).

With fsync = "always", a host crash loses no acknowledged write. Every batch waits for an fsync, which costs write latency. A host crash can still leave a complete invalid frame after the last fsync in a journal's newest segment. Recovery cannot tell it from media corruption, so it gates that core's groups as above. A group gated on a majority of its voters, or a single-voter group, then waits for an operator although nothing was lost (see Operations).

Cold storage

Ursula keeps recent data in an in-memory hot ring on every replica and flushes older segments and snapshot blobs to a cold backend. Multi-node clusters should use S3 or an S3-compatible object store so every replica can read chunks flushed by any leader. All replicas must point at the same shared bucket.

[storage.cold]
backend = "s3"
root = "ursula-prod-20260518"

[storage.cold.s3]
bucket = "my-ursula-bucket"
region = "us-east-1"
# endpoint = "http://127.0.0.1:9000"
# access_key_id = "AKIA..."
# secret_access_key = "..."
# session_token = "..."
# server_side_encryption = "aes256"   # default; "aws-kms" or "none"
# kms_key_id = "arn:aws:kms:..."      # customer managed key for "aws-kms"

When access_key_id and secret_access_key are omitted, Ursula uses the standard AWS SDK credential chain (instance profile, environment, profile, and so on). Prefer that over static keys. Set endpoint for S3-compatible stores such as MinIO, R2, or TOS.

Every cold-tier object write requests server-side encryption by default (server_side_encryption = "aes256", i.e. SSE-S3) — free on AWS S3, and Raft snapshot uploads inherit the same setting. aws-kms switches to SSE-KMS, using the AWS managed key unless kms_key_id names a customer managed key. MinIO honors SSE headers only when a KMS/KES is configured; for a MinIO deployment without one, set server_side_encryption = "none" explicitly or writes fail with an SSE error.

Config key	Default	Purpose
storage.cold.backend	none	none, memory, or s3
storage.cold.root	none	Prefix prepended to every cold key. Use a date-stamped value for benchmark runs so cleanup can't touch production data
storage.cold.s3.bucket	none	Required when backend = "s3"
storage.cold.s3.server_side_encryption	aes256	SSE mode for every object write: aes256 (SSE-S3), aws-kms, or none
storage.cold.s3.kms_key_id	none	Customer managed KMS key; only valid with aws-kms
storage.cold.s3.probe_timeout	5s	Deadline of the snapshot-store health probe, retries included. A probe whose first attempt fails fast still has time for one retry, while an attempt that hangs runs into this deadline. Must be non-zero
storage.cold.s3.unhealthy_ticks	3	Bad health ticks, with no good tick between them, before the node sheds its leaderships. Must be non-zero
storage.cold.s3.heal_ticks	2	Good health ticks in a row before a node that shed may lead again. Must be non-zero
storage.cold.flush_interval	1s	Background flush worker tick interval
storage.cold.flush_size	8MiB	Target bytes flushed per group per tick
storage.cold.flush_max_concurrency	4	Parallel cold writes in flight
storage.cold.max_hot_size_per_group	64MiB	Backpressure ceiling: when a group's hot bytes exceed this, new writes get HTTP 503 until the flush catches up. Explicit 0 disables the cap
storage.cold.flush_max_hot_age	5m	A stream's hot tail older than this is flushed whole even when its group is below the flush threshold. 0 disables
storage.cold.compaction_enabled	true	Background compaction: merges small chunks that flushes recorded as compaction debt (no bucket listing) and rewrites packed slices once a stream holds 64 of them
storage.snapshot.backend	auto	auto (S3 snapshots when storage.cold.backend = "s3", inline otherwise), inline, or s3. Every Raft node, a single node included, uses it
storage.snapshot.s3_timeout	10s	Per-attempt timeout of an S3 snapshot-store request. The snapshot store has its own timeout and retries, apart from storage.cold.s3.timeout and max_retries. The timeout covers one attempt, and a PUT sends the whole object in one attempt, so it must also cover a healthy GET or PUT of the largest snapshot. A follower that installs a snapshot sends up to four requests one after another within raft.install_snapshot_timeout: a GET of the leader's object, a PUT of its copy, a second PUT under a new name when the first one's name is taken, and a HEAD that checks the copy. Each request can take every attempt to its timeout and every backoff before a retry, 22 seconds with the defaults, so the copy takes at most 88 seconds of the default 120. With S3 snapshots, the server refuses to start when the four requests can take longer than raft.install_snapshot_timeout, with SnapshotCopyOutlastsInstall. Must be non-zero
storage.snapshot.s3_max_retries	1	Retries of an S3 snapshot-store request. The first retry waits up to 2 seconds, the second up to 4, each later one twice as long up to 120 seconds. Counts toward the same bound as storage.snapshot.s3_timeout
raft.snapshot_log_budget	1GiB	Node Raft-log budget of the snapshot driver. A group snapshots once its log since the last snapshot reaches the larger of this budget over twice the group count (at most 16MiB) and twice its last snapshot's size. Above three quarters of the budget a pressure pass snapshots the groups that hold the most log first, one at a time, until the node would fall to half the budget. Past the budget it runs more builds at once, in even steps up to raft.snapshot_build_max_concurrency before twice the budget. With the default of 2 that is two builds past the budget. With 4 it is two past the budget, three past 1.33 times it and four past 1.67 times it. Over twice the budget the node, a single node included, refuses writes with a body until its log is below 1.5 times the budget
raft.snapshot_backstop_logs	100000	Far entry backstop of the snapshot driver. A group snapshots once it applied this many entries since its last snapshot, whatever its log bytes. Entry count bounds a group's log index in memory and its replay at restart, so lower it for groups of many small entries. It is the same on a single node as on a cluster node. 0.8 kept a single node's groups within 5,000 entries, and 5000 keeps that bound
raft.snapshot_build_max_concurrency	2	Snapshot builds the node runs at once. Each build holds its group's captured state (the hot window, at most storage.cold.max_hot_size_per_group, and the streams' metadata) and, while it uploads, its compressed snapshot, about as large again, and it sends its snapshot-store requests one after another. The snapshot driver starts a build only while fewer run than the node's log calls for: one below raft.snapshot_log_budget and more past it. Builds it started past the budget finish after the log falls below it. A snapshot an operator triggers while every build slot is taken is deferred. Set 1 to keep one build at a time, as before 0.7.4. Raise it, to 4 for example, when the node keeps reaching twice the budget with every build in use (raft_snapshot_builds_in_flight_max at this value) and its memory holds that many builds of its largest groups. The server refuses to start with 0 or more than 16
raft.snapshot_pressure_max_groups_per_tick	16	Groups one snapshot driver pass starts while the node's log is below raft.snapshot_log_budget. Past the budget a pass starts this many for each build it keeps in flight. It also caps the snapshot GC passes one driver tick runs
raft.deleted_data_max_age	the larger of 1h and eight times raft.install_snapshot_timeout	How long a replica keeps a deleted, purged or expired stream in its current Raft snapshot and its Raft log after it applies the removal. The snapshot driver snapshots a group a quarter of this after its oldest removal that no snapshot covers, sooner when the removed bytes reach the group's snapshot threshold, and then purges the group's log through that snapshot. See Deleted data for what the bound covers. With an S3 snapshot store, snapshot GC deletes the snapshot object a replica replaced once no send of it is in flight, within one raft.install_snapshot_timeout and a drive interval. A set value must be at least eight times raft.install_snapshot_timeout (16 minutes at the default), or the server refuses to start, which leaves that room with a margin of several installs

Hot bytes here are payload bytes: the hot window keeps no per-message bookkeeping, so admission and flush thresholds charge exactly the unflushed payload. Flush defaults are conservative, so tune them under load. A typical benchmark profile drops the interval to 200ms, raises concurrency to 32, and bumps the per-group ceiling. Advanced deployments can split the flush threshold and batch size with storage.cold.flush_min_hot_size and storage.cold.flush_max_size, but the single storage.cold.flush_size knob is the normal path. Once a group's accumulated hot payload reaches the threshold, Ursula packs eligible slices from that group into one immutable object up to the configured maximum; single-stream slices of 1 MiB or more keep the direct-object layout, and smaller ones are packed even when they are alone.

The snapshot driver takes every Raft snapshot and always runs. It ticks every storage.snapshot.drive_interval, 5s by default with S3 snapshots and 1s with inline ones, and the server refuses to start with 0s. With S3 snapshots, a health gate checks the store on each of its ticks. Each tick probes the store and reads the node's cold-flush writes. A tick is bad when the probe fails or misses its deadline, or when a write failed within the flush window and no write has succeeded since. The window is as long as the S3 client can take to give up on a write: storage.cold.s3.timeout for each of its max_retries plus one attempts, the backoff between them, and the flush worker's pause between passes. That is 55 seconds with the defaults. A tick in which writes both failed and succeeded neither ends a bad streak nor adds to it. One failed probe is not enough on its own, and once a later write succeeds a failed write weighs only on its own tick. A store that refuses writes but answers the probe is still caught. After storage.cold.s3.unhealthy_ticks bad ticks with no good tick between them the node stops campaigning and hands its leaderships to its peers. With the defaults that is 10 to 20 seconds into an outage when the probes fail, about 10 to 30 seconds when only writes fail (sooner when the store refuses them outright), and about a minute when every write attempt hangs until its timeout. While a tick is bad the node also holds routine snapshots back, though a log-pressure pass still runs. After storage.cold.s3.heal_ticks good ticks in a row it may lead again. For a node shed for failed writes, the good ticks start with the first write that succeeds after the last failed one. A node that has handed off every leadership flushes nothing, so its good ticks start only once its last failed write leaves the window, about a minute after that write with the defaults.

A shedding node hands off at most four leaderships per tick, the groups with the most unflushed bytes first, and only to a peer whose replica of the group may take the leadership. A peer that is draining, waiting at its recovery gate, or shedding for its own snapshot store, cluster egress or WAL disk is passed over, and so is a replica that has not acknowledged its leader since its node started. The gate asks each peer for its leadership-shed state and waits one minimum election timeout (1.5 s) for the answer, as the leadership balancer, the shutdown handoff and a group leader asking a learner for its recovery gate do. A peer that does not answer in time is treated as unable to campaign. A handoff is done once another node leads the group, and one that Raft abandons is handed off again on a later tick. A node with N leaderships therefore evacuates in about ceil(N / 4) ticks, for example 22 ticks or 110 seconds for 86 leaderships at the default interval. The cap is per gate. The cold-health gate (governance.cold_health, every 2s by default) drains the same way and independently, so a node that sheds for both moves up to four leaderships per 2 seconds plus four per 5 seconds. The leadership balancer (governance.leadership_balance) also keeps moving leaderships off a node that holds more than its fair share, up to max_per_tick per balancer tick. The fair share is the cluster's data groups divided among the nodes that may campaign, are serving-ready and hold a role in some group, rounded up, and each peer reports how many groups it leads. The balancer moves a leadership only to a serving-ready node, so it pulls none onto a node that restarted before each of its replicas acknowledged its leader again. A leadership moves only to a voter below its share that ends up leading fewer groups than the node it leaves, so the balancer settles, also on clusters with more nodes than the replication factor. No leadership goes to a replica a move takes out. The cluster-egress gate (governance.cluster_probe) sheds only while the fault is the node's own: its own probe fails to reach a quorum of some group, and either a voter of such a group answers that its own last probe was clean, or fewer peers answer at all than a quorum of each of its groups needs, as for a node cut off from its peers or on the minority side of a partition. A node that hears a quorum of one of its groups keeps campaigning, because its elections stop in every group at once, and that group would be left without a candidate. A delay or loss on every node's path fails every node's probe while every peer still answers, and no node sheds, since handing leaderships to nodes no better off would only leave groups without a leader. The node logs a warning once when its probe fails and it keeps its leaderships for that reason. Each tick asks the peers for their answers alongside its probes, and a failed tick waits up to 1.5 s for them. A node that hears too few peers for a quorum sheds on its first such tick, at most one interval plus 1.5 s after the cut (2 s by default). One that a peer does better than sheds after unhealthy_ticks failed ticks. Before its shed, a cut-off node raises no group's term: an election starts with a pre-vote, and no peer answers it. Once it sheds, the gate hands every leadership off at once, because a node that cannot reach a quorum cannot replicate, and it plans again on every tick for the ones still pending. Like the other gates, it hands a leadership only to a peer that answered that its replica of the group may take the leadership.

Raft snapshotting does not force a partial cold flush. Payload below the cold threshold stays in the bounded hot tail and is included in the next group snapshot, so a short snapshot interval does not turn small tails into small cold objects.

Helm and the EC2 scripts map their deployment inputs onto these same keys, so this table applies to every deployment method.

Complete examples

Disk-backed single node (streams survive a restart). It lists itself in raft.peers and names its deployment:

[server]
listen = "127.0.0.1:4437"
cluster_id = "dev-local"

[raft]
node_id = 1
group_count = 16

[raft.wal]
path = "./data"

[[raft.peers]]
node_id = 1
url = "http://127.0.0.1:4437"

ursula --config ./ursula.toml

Three-node durable cluster on a single port per node (same file on each host, override the node ID per host). Deploy a Cluster shows the same cluster with the cluster plane on its own port, which keeps Raft off the client port:

[server]
listen = "0.0.0.0:4437"
cluster_id = "prod-us-east-1"

[runtime]
core_count = 16

[raft]
node_id = 1
group_count = 256

[raft.wal]
path = "/var/lib/ursula"

[storage.cold]
backend = "s3"
root = "ursula-prod-20260518"

[storage.cold.s3]
bucket = "my-ursula-bucket"
region = "us-east-1"

[[raft.peers]]
node_id = 1
url = "http://10.0.0.1:4437"

[[raft.peers]]
node_id = 2
url = "http://10.0.0.2:4437"

[[raft.peers]]
node_id = 3
url = "http://10.0.0.3:4437"

ursula --config /etc/ursula/ursula.toml --node-id 1

See Deploy a Cluster for the full deployment walk-through.