ursuladurable streams
v0.9.0 · apache-2.0

Deploy a Cluster

Production Ursula is a static-membership Raft cluster: three voting nodes across availability zones, a durable Raft log per node, and a shared S3 bucket for the cold tier.

The recommended way to run it is Kubernetes via the Helm chart. Bring a Kubernetes cluster with a StorageClass for the voters' volumes and an S3 bucket for the cold tier. The chart installs Ursula into it. If you are not running Kubernetes, see bare metal at the end.

Helm

The chart and images are published to GHCR on every release. One command starts a three-voter cluster:

helm install ursula oci://ghcr.io/tonbo-io/charts/ursula --version 0.9.0 \
  --set server.clusterId=my-cluster

The chart defaults to the ghcr.io/tonbo-io/ursula image pinned to the chart's appVersion. The default install runs three voters and 64 Raft groups, with durable per-pod Raft logs on PVCs, a headless peer Service, a client Service, and a quorum-protecting PodDisruptionBudget.

Each voter listens on two ports. The client port (server.ports.client, 4437) serves the stream API, and the client Service exposes only that port. The cluster port (server.ports.cluster, 4439) carries Raft between voters, and only the headless peer Service names it. The chart renders each peer's url on the cluster port and its client_url on the client port, so leader redirects send clients to a client port. server.clusterListen=false serves both planes on the client port instead. The admin plane stays on pod loopback (127.0.0.1:4438).

For production, add shared S3 cold storage and workload identity:

s3:
  bucket: my-ursula-bucket
  region: us-east-1
  prefix: ursula-prod

coldStorage:
  enabled: true

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ursula-s3

With cold storage enabled the chart also turns on cold-chunk compaction and lets Ursula pick S3 for Raft snapshots (snapshotStore.backend: auto). Set coldStorage.compaction.enabled=false or snapshotStore.backend=inline to opt out.

For MinIO or another S3-compatible backend, set s3.endpoint. Prefer workload identity over static S3 credentials. To use a registry mirror, set global.image.repository, global.image.tag, and optionally global.imagePullSecrets.

server.replicaCount controls the voter set for a fresh cluster and supports 1, 3, and 5. Production should use 3, or 5 only when tolerating two simultaneous voter failures justifies the larger write quorum. Changing it on an initialized cluster is not safe Raft voter reconfiguration. Safe scaling needs the future operator workflow.

On AWS (EKS)

The chart's examples/production-eks.yaml is a three-zone starting point: three voters on gp3 volumes, S3 cold storage and snapshots, and three gateways. It needs:

  • an EKS cluster with nodes in three availability zones,
  • the EBS CSI driver and a StorageClass named gp3,
  • an S3 bucket,
  • an IAM role for the Ursula service account (IRSA or EKS Pod Identity), limited to its S3 prefix.

The chart README lists the details. Replace the bucket and the role ARN in the example, then install it:

helm install ursula oci://ghcr.io/tonbo-io/charts/ursula --version 0.9.0 \
  --namespace ursula --create-namespace -f production-eks.yaml
helm test ursula --namespace ursula

Verify

helm test ursula

Gate on this test, not on helm install --wait. Helm does not wait for a StatefulSet with the OnDelete update strategy, so --wait can report the release deployed before any voter is ready. The test mounts the chart-generated cluster manifest and runs ursulactl wait-ready. It requires the expected group inventory and leaders, a complete uniform voter set in each maintenance report, no uncertain maintenance fence, and a 200 response from every node's /__ursula/ready probe. To query a node directly:

kubectl port-forward svc/ursula 4437:4437
curl http://127.0.0.1:4437/__ursula/metrics

Production notes

  • Run three voters across three availability zones, each with its own zonal persistent volume for the Raft log. Never use ephemeral voter data in production. The chart always gives each voter pod a raft-data volume claim.
  • Shared S3 is required for any multi-node cluster: replicas must be able to read chunks flushed by any leader.
  • Add an AbortIncompleteMultipartUpload lifecycle rule to every cold bucket, for example after 1 day. Ursula never aborts the multipart upload of a cold write that a crash or shutdown interrupted, and S3 bills its parts until they are aborted. See Incomplete multipart uploads.
  • Set server.clusterId, which the chart renders as server.cluster_id, the id every HEAD answer carries as Ursula-Cluster-Id. The chart refuses to render without it, for any replica count. Name the environment and the region, such as prod-us-east-1, and give every deployment one client can reach its own id, since two Kubernetes clusters can run the same namespace and release name. Keep the id when a restore replaces a deployment. Each node's WAL records the id, and a node whose configured id differs refuses to start.
  • Put stateless gateway replicas behind authenticated TLS ingress and keep voter and peer Services private. Ursula itself has no TLS or auth (see Security).
  • Set server.networkPolicy.enabled=true on a CNI that enforces NetworkPolicy. The policy admits only the release's voters to the cluster port, and the sources in server.networkPolicy.clientIngressFrom to the client port (every source when the list is empty) plus the release's gateway and Helm test pods. See Kubernetes NetworkPolicy.
  • Kubernetes rolling updates do not transfer Raft leaders on their own. For an operationally safe restart, wrap the platform's restart in ursulactl's maintenance verbs: ursulactl drain the node, let Kubernetes restart the pod, then ursulactl wait --node N and undrain before moving to the next one. See Operations for day-2 work.
  • The StatefulSet readiness probe calls GET /__ursula/ready and removes a pod from service during WAL disk pressure. Cluster-wide Raft readiness still comes from helm test or ursulactl wait-ready. The mutating admin surface is bound to pod loopback (127.0.0.1:4438) and is reached with kubectl port-forward.
  • helm uninstall leaves the graceful-rollout state ConfigMap (<statefulset>-rollout-state, for example ursula-rollout-state) behind. Delete it before you install again under the same name, so the rollout hook does not resume a rollout of the old release.
  • Clusters that serve the SQLite VFS take its appends of at most 4 MiB at any preset. Its snapshots of up to 32 MiB go in one request, so keep per-group admission caps of at least 64 MiB (standard or large preset) for them to be admitted. Larger snapshots go up in parts and need a cold backend.

Without Kubernetes (bare metal / VMs)

The same static-membership model works by hand. Every node runs the same config file, and only the node ID differs:

[server]
listen = "0.0.0.0:4437"
cluster_listen = "0.0.0.0:4439"
cluster_id = "prod-us-east-1"

[raft]
group_count = 256
init_membership_per_group = true

[raft.wal]
path = "/var/lib/ursula"

[storage.cold]
backend = "s3"
root = "ursula-prod"

[storage.cold.s3]
bucket = "my-ursula-bucket"
region = "us-east-1"

[[raft.peers]]
node_id = 1
url = "http://10.0.0.1:4439"
client_url = "http://10.0.0.1:4437"

[[raft.peers]]
node_id = 2
url = "http://10.0.0.2:4439"
client_url = "http://10.0.0.2:4437"

[[raft.peers]]
node_id = 3
url = "http://10.0.0.3:4439"
client_url = "http://10.0.0.3:4437"

Start each node with its own ID:

ursula --config /etc/ursula/ursula.toml --node-id 1   # node 2 and 3 likewise

Peers dial each other's url on the cluster port (4439), and a follower redirects clients to the leader's client_url on the client port (4437). Let only the voters reach port 4439, with a security group or a firewall. Without server.cluster_listen, both planes share server.listen and each peer's url names that port (see Network planes).

server.cluster_id is required with raft.peers. Use the same value on every node and a different one for every deployment a client can reach, such as the environment and the region. init_membership_per_group = true bootstraps each group on the first start of a fresh cluster. Keep it on every node across restarts: a replica that ever held its group never initializes it again, and an empty replica initializes a group only when every configured voter reports it empty and no leader (see Rejoining a wiped node). Verify the cluster with a one-file manifest:

cat > cluster-manifest.json <<'JSON'
{
  "nodes": [
    {"id": 1, "http_url": "http://10.0.0.1:4439", "client_url": "http://10.0.0.1:4437", "host": "10.0.0.1"},
    {"id": 2, "http_url": "http://10.0.0.2:4439", "client_url": "http://10.0.0.2:4437", "host": "10.0.0.2"},
    {"id": 3, "http_url": "http://10.0.0.3:4439", "client_url": "http://10.0.0.3:4437", "host": "10.0.0.3"}
  ]
}
JSON

ursulactl wait-ready --config cluster-manifest.json --expected-groups 256
ursulactl status      --config cluster-manifest.json

http_url is the node's Raft address, so it names the cluster port. client_url is where ursulactl reads metrics and readiness on the client plane.

Every config key (peers, listeners, WAL, cold-tier tuning) is documented in Configuration.