ursuladurable streams
v0.7.6 · apache-2.0

Deploy a Cluster

Production Ursula is a static-membership Raft cluster: three voting nodes across availability zones, a durable Raft log per node, and a shared S3 bucket for the cold tier.

The recommended way to run it is Kubernetes via the Helm chart. Bring a Kubernetes cluster with a StorageClass for the voters' volumes and an S3 bucket for the cold tier. The chart installs Ursula into it. If you are not running Kubernetes, see bare metal at the end.

Helm

The chart and images are published to GHCR on every release. One command starts a three-voter cluster:

helm install ursula oci://ghcr.io/tonbo-io/charts/ursula --version 0.7.6

The chart defaults to the ghcr.io/tonbo-io/ursula image pinned to the chart's appVersion. The default install runs three voters and 64 Raft groups, with durable per-pod Raft logs on PVCs, a headless peer Service, a client Service, and a quorum-protecting PodDisruptionBudget.

For production, add shared S3 cold storage and workload identity:

s3:
  bucket: my-ursula-bucket
  region: us-east-1
  prefix: ursula-prod

coldStorage:
  enabled: true

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ursula-s3

With cold storage enabled the chart also turns on cold-chunk compaction and lets Ursula pick S3 for Raft snapshots (snapshotStore.backend: auto). Set coldStorage.compaction.enabled=false or snapshotStore.backend=inline to opt out.

For MinIO or another S3-compatible backend, set s3.endpoint. Prefer workload identity over static S3 credentials. To use a registry mirror, set global.image.repository, global.image.tag, and optionally global.imagePullSecrets.

server.replicaCount controls the voter set for a fresh cluster and supports 1, 3, and 5. Production should use 3, or 5 only when tolerating two simultaneous voter failures justifies the larger write quorum. Changing it on an initialized cluster is not safe Raft voter reconfiguration. Safe scaling needs the future operator workflow.

On AWS (EKS)

The chart's examples/production-eks.yaml is a three-zone starting point: three voters on gp3 volumes, S3 cold storage and snapshots, three gateways, and a two-replica indexer pool with its own workload identity. It needs:

  • an EKS cluster with nodes in three availability zones,
  • the EBS CSI driver and a StorageClass named gp3,
  • an S3 bucket,
  • IAM roles for the Ursula and indexer service accounts (IRSA or EKS Pod Identity), each limited to its own S3 prefix.

The chart README lists the details. Replace the bucket and the role ARNs in the example, then install it:

helm install ursula oci://ghcr.io/tonbo-io/charts/ursula --version 0.7.6 \
  --namespace ursula --create-namespace -f production-eks.yaml
helm test ursula --namespace ursula

Verify

helm test ursula

Gate on this test, not on helm install --wait. Helm does not wait for a StatefulSet with the OnDelete update strategy, so --wait can report the release deployed before any voter is ready. The test mounts the chart-generated cluster manifest and runs ursulactl wait-ready. It requires the expected group inventory and leaders, a complete uniform voter set in each maintenance report, no uncertain maintenance fence, and a 200 response from every node's /__ursula/ready probe. To query a node directly:

kubectl port-forward svc/ursula 4437:4437
curl http://127.0.0.1:4437/__ursula/metrics

Production notes

  • Run three voters across three availability zones, each with its own zonal persistent volume for the Raft log. Never use ephemeral voter data in production. The chart always gives each voter pod a raft-data volume claim.
  • Shared S3 is required for any multi-node cluster: replicas must be able to read chunks flushed by any leader.
  • Put stateless gateway replicas behind authenticated TLS ingress and keep voter and peer Services private. Ursula itself has no TLS or auth (see Security).
  • Kubernetes rolling updates do not transfer Raft leaders on their own. For an operationally safe restart, wrap the platform's restart in ursulactl's maintenance verbs: ursulactl drain the node, let Kubernetes restart the pod, then ursulactl wait --node N and undrain before moving to the next one. See Operations for day-2 work.
  • The StatefulSet readiness probe calls GET /__ursula/ready and removes a pod from service during WAL disk pressure. Cluster-wide Raft readiness still comes from helm test or ursulactl wait-ready. The mutating admin surface is bound to pod loopback (127.0.0.1:4438) and is reached with kubectl port-forward.
  • helm uninstall leaves the graceful-rollout state ConfigMap (<statefulset>-rollout-state, for example ursula-rollout-state) behind. Delete it before you install again under the same name, so the rollout hook does not resume a rollout of the old release.
  • Clusters that serve the SQLite VFS need per-group admission caps of at least 64 MiB (standard or large preset) so a 32 MiB commit can be admitted.

Optional: event-time indexer (experimental)

ursula-indexer is an optional worker pool that builds queryable event-time indexes from application/json and application/x-ndjson streams, kept outside the voter processes so query and compaction work never touches commit latency. It is experimental: its API and S3 format may change in a minor release, which then needs a new or emptied indexer.s3.prefix and re-registration.

indexer:
  enabled: true
  replicaCount: 2
  s3:
    prefix: indexes
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ursula-index

Register streams dynamically with PUT /v1/indexes/{id} on the internal indexer Service; see Browser Telemetry. No Helm upgrade is needed. The Service is an internal ClusterIP. If remote applications need read access, front it with an authenticated, path-aware proxy and never expose the registration or administration routes.

Without Kubernetes (bare metal / VMs)

The same static-membership model works by hand. Every node runs the same config file, and only the node ID differs:

[server]
listen = "0.0.0.0:4437"

[raft]
group_count = 256
init_membership_per_group = true

[raft.wal]
path = "/var/lib/ursula"

[storage.cold]
backend = "s3"
root = "ursula-prod"

[storage.cold.s3]
bucket = "my-ursula-bucket"
region = "us-east-1"

[[raft.peers]]
node_id = 1
url = "http://10.0.0.1:4437"

[[raft.peers]]
node_id = 2
url = "http://10.0.0.2:4437"

[[raft.peers]]
node_id = 3
url = "http://10.0.0.3:4437"

Start each node with its own ID:

ursula --config /etc/ursula/ursula.toml --node-id 1   # node 2 and 3 likewise

init_membership_per_group = true bootstraps each group on the first start of a fresh cluster. Keep it on every node across restarts: a replica that ever held its group never initializes it again, and an empty replica initializes a group only when every configured voter reports it empty and no leader (see Rejoining a wiped node). Verify the cluster with a one-file manifest:

cat > cluster-manifest.json <<'JSON'
{
  "nodes": [
    {"id": 1, "http_url": "http://10.0.0.1:4437", "host": "10.0.0.1"},
    {"id": 2, "http_url": "http://10.0.0.2:4437", "host": "10.0.0.2"},
    {"id": 3, "http_url": "http://10.0.0.3:4437", "host": "10.0.0.3"}
  ]
}
JSON

ursulactl wait-ready --config cluster-manifest.json --expected-groups 256
ursulactl status      --config cluster-manifest.json

Every config key (peers, listeners, WAL, cold-tier tuning) is documented in Configuration.