ursuladurable streams
v0.11.3 · apache-2.0

Deploy a Cluster

Production Ursula is a Raft cluster of three or more nodes across availability zones, each with a durable Raft log, and a shared S3 bucket for the cold tier. Its meta group places every data group on three of the nodes by default and moves groups between them on request (Placement and the meta group).

The recommended way to run it is Kubernetes via the Helm chart. Bring a Kubernetes cluster with a StorageClass for the voters' volumes and an S3 bucket for the cold tier. The chart installs Ursula into it. If you are not running Kubernetes, see bare metal at the end.

Helm

The chart and images are published to GHCR on every release. One command starts a three-voter cluster:

helm install ursula oci://ghcr.io/tonbo-io/charts/ursula --version 0.11.3 \
  --set server.clusterId=my-cluster

The chart defaults to the ghcr.io/tonbo-io/ursula image pinned to the chart's appVersion. The default install runs three voters and 64 Raft groups, with durable per-pod Raft logs on PVCs, a headless peer Service, a client Service, and a quorum-protecting PodDisruptionBudget.

Each voter listens on two ports. The client port (server.ports.client, 4437) serves the stream API, and the client Service exposes only that port. The cluster port (server.ports.cluster, 4439) carries Raft between voters, and only the headless peer Service names it. The chart renders each peer's url on the cluster port and its client_url on the client port, so leader redirects send clients to a client port. server.clusterListen=false serves both planes on the client port instead. The admin plane stays on pod loopback (127.0.0.1:4438).

For production, add shared S3 cold storage and workload identity:

s3:
  bucket: my-ursula-bucket
  region: us-east-1
  prefix: ursula-prod

coldStorage:
  enabled: true

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ursula-s3

With cold storage enabled the chart also turns on cold-chunk compaction and lets Ursula pick S3 for Raft snapshots (snapshotStore.backend: auto). Set coldStorage.compaction.enabled=false or snapshotStore.backend=inline to opt out.

For MinIO or another S3-compatible backend, set s3.endpoint. Prefer workload identity over static S3 credentials. To use a registry mirror, set global.image.repository, global.image.tag, and optionally global.imagePullSecrets.

server.replicaCount is the server pod count: 1, or 3 and more. The chart refuses 2, which tolerates no node loss. Each Raft group has at most three voters whatever the pod count, so production should use 3. Raising it on a cluster of three or more pods adds pods that register and hold no group, and ursulactl placement move moves groups onto them, which stays manual. Node ids are the pod ordinal plus one. To lower it, first run ursulactl node decommission --node <id> --wait for every id above the new count (never a meta voter, ids 1 to 3), then lower the count right away: a dropped pod that restarts in between, after a removal pass ran while it was down, comes back superseded, unready and holding the disruption budget. The dropped pods' volumes stay. Once the meta leader completed a removal the id is vacant, and a later raise starts those ids again on empty volumes as new nodes. A single pod never grows under the chart: a growing single node needs raft.meta_voters and raft.placement.replication_factor set (Adding a node), and the chart sets neither.

Before every upgrade and rollback the chart's scale-check hook checks a change of the count. It refuses a raise from 1 and a lower count whose dropped ids are not removed. On a raise it waits up to 200 s for each started id whose removal has not completed, then deletes the volume of each started id that is vacant, which holds a removed life's disk, and keeps the volume of an id that is in a life, so a scale-down that skipped the hook comes back on its volumes. A volume under an id that is neither refuses the raise. The hook runs inside Helm's default 5 minute --timeout. A raise binds the new ids, so the hook refuses a helm rollback to the lower count until they are decommissioned again.

On AWS (EKS)

The chart's examples/production-eks.yaml is a three-zone starting point: three voters on gp3 volumes, S3 cold storage and snapshots, and three gateways. It needs:

  • an EKS cluster with nodes in three availability zones,
  • the EBS CSI driver and a StorageClass named gp3,
  • an S3 bucket,
  • an IAM role for the Ursula service account (IRSA or EKS Pod Identity), limited to its S3 prefix.

The chart README lists the details. Replace the bucket and the role ARN in the example, then install it:

helm install ursula oci://ghcr.io/tonbo-io/charts/ursula --version 0.11.3 \
  --namespace ursula --create-namespace -f production-eks.yaml
helm test ursula --namespace ursula

Verify

helm test ursula

Gate on this test, not on helm install --wait, which waits only for each pod's own readiness probe. The test mounts the chart-generated cluster manifest and runs ursulactl wait-ready. It requires a ready maintenance report and a 200 response from every node's /__ursula/ready probe, and, in the meta leader's placement view, every group live and running on exactly its target voters with a leader. To query a node directly:

kubectl port-forward svc/ursula 4437:4437
curl http://127.0.0.1:4437/__ursula/metrics

Production notes

  • Run three voters across three availability zones, each with its own zonal persistent volume for the Raft log. Never use ephemeral voter data in production. The chart always gives each voter pod a raft-data volume claim.
  • Shared S3 is required for any multi-node cluster: replicas must be able to read chunks flushed by any leader.
  • Add an AbortIncompleteMultipartUpload lifecycle rule to every cold bucket, for example after 1 day. Ursula never aborts the multipart upload of a cold write that a crash or shutdown interrupted, and S3 bills its parts until they are aborted. See Incomplete multipart uploads.
  • Set server.clusterId, which the chart renders as server.cluster_id, the id every HEAD answer carries as Ursula-Cluster-Id. The chart refuses to render without it, for any replica count. Name the environment and the region, such as prod-us-east-1, and give every deployment one client can reach its own id, since two Kubernetes clusters can run the same namespace and release name. Keep the id when a restore replaces a deployment. Each node's WAL records the id, and a node whose configured id differs refuses to start.
  • Put stateless gateway replicas behind authenticated TLS ingress and keep voter and peer Services private. Ursula itself has no TLS or auth (see Security).
  • Set server.networkPolicy.enabled=true on a CNI that enforces NetworkPolicy. The policy admits only the release's voters to the cluster port, and the sources in server.networkPolicy.clientIngressFrom to the client port (every source when the list is empty) plus the release's gateway and Helm test pods. See Kubernetes NetworkPolicy.
  • The StatefulSet updates by RollingUpdate, one pod at a time, each waiting for the pod's readiness probe, and every voter hands off its leaderships on SIGTERM. A pod the controller recreates during the roll, after a crash or an eviction, takes the new revision. On Kubernetes 1.37 and later, the controller first replaces every pod that is already unready on the old revision, all at once, and replaces a ready pod only while every other pod is ready. Start an upgrade only when ursulactl verify-cluster passes. To restart one voter outside an upgrade, ursulactl drain the node, delete the pod, then ursulactl wait --node N and undrain before moving to the next one. See Operations for day-2 work.
  • The StatefulSet readiness probe calls GET /__ursula/ready and removes a pod from service during WAL disk pressure. Cluster-wide Raft readiness still comes from helm test or ursulactl wait-ready. The mutating admin surface is bound to pod loopback (127.0.0.1:4438) and is reached with kubectl port-forward.
  • Clusters that serve the SQLite VFS take its appends of at most 4 MiB at any preset. Its snapshots of up to 32 MiB go in one request, so keep per-group admission caps of at least 64 MiB (standard or large preset) for them to be admitted. Larger snapshots go up in parts and need a cold backend.

Without Kubernetes (bare metal / VMs)

The same cluster works by hand. Every node runs the same config file, and only the node ID differs:

[server]
listen = "0.0.0.0:4437"
cluster_listen = "0.0.0.0:4439"
cluster_id = "prod-us-east-1"

[raft]
group_count = 256

[raft.wal]
path = "/var/lib/ursula"

[storage.cold]
backend = "s3"
root = "ursula-prod"

[storage.cold.s3]
bucket = "my-ursula-bucket"
region = "us-east-1"

[[raft.peers]]
node_id = 1
url = "http://10.0.0.1:4439"
client_url = "http://10.0.0.1:4437"

[[raft.peers]]
node_id = 2
url = "http://10.0.0.2:4439"
client_url = "http://10.0.0.2:4437"

[[raft.peers]]
node_id = 3
url = "http://10.0.0.3:4439"
client_url = "http://10.0.0.3:4437"

Start each node with its own ID:

ursula --config /etc/ursula/ursula.toml --node-id 1   # node 2 and 3 likewise

Peers dial each other's url on the cluster port (4439), and a follower redirects clients to the leader's client_url on the client port (4437). Let only the cluster's nodes reach port 4439, with a security group or a firewall. Without server.cluster_listen, both planes share server.listen and each peer's url names that port (see Network planes).

server.cluster_id is required with raft.wal.path. Use the same value on every node and a different one for every deployment a client can reach, such as the environment and the region. Start every node of a fresh cluster: the meta group forms once every raft.peers node answered, over the meta voters raft.meta_voters, by default nodes 1 to 3. At bootstrap the meta group seeds each group on three consecutive nodes in node ID order (every node in a cluster of three), and the first of them initializes it on the first start of a fresh cluster, only when every voter of the group reports it empty and no leader. A replica that ever held its group never initializes it again (see Rejoining a wiped node). The placement lives in the meta group from then on: a node added to raft.peers later registers and holds no group until groups move onto it (Adding a node), and a node leaves only by ursulactl node decommission. A removed node ID never holds a group again. Verify the cluster with a one-file manifest:

cat > cluster-manifest.json <<'JSON'
{
  "nodes": [
    {"id": 1, "http_url": "http://10.0.0.1:4439", "client_url": "http://10.0.0.1:4437", "host": "10.0.0.1"},
    {"id": 2, "http_url": "http://10.0.0.2:4439", "client_url": "http://10.0.0.2:4437", "host": "10.0.0.2"},
    {"id": 3, "http_url": "http://10.0.0.3:4439", "client_url": "http://10.0.0.3:4437", "host": "10.0.0.3"}
  ]
}
JSON

ursulactl wait-ready --config cluster-manifest.json
ursulactl status      --config cluster-manifest.json

http_url is the node's Raft address, so it names the cluster port. client_url is where ursulactl reads metrics and readiness on the client plane.

Every config key (peers, listeners, WAL, cold-tier tuning) is documented in Configuration.