Security

The current v0.x security model is deliberately narrow. Ursula is built to slot behind your existing edge layer, not to be one.

What Ursula does

• Quorum-acknowledged writes. An append is acknowledged only after a majority of voters has replicated it.
• Per-group backpressure. When a group's hot ring exceeds storage.cold.max_hot_size_per_group, appends to that group return 503 with Retry-After until cold flush catches up. Per-group, not global or per-client.
• Stream-level isolation. Streams hash to disjoint Raft groups and disjoint owner cores. A hot stream on one group cannot starve writes on a different group on a different core.

Network planes

A node serves three planes, each on its own listener:

Plane	Listener	Routes	Who connects
Client	server.listen	The Durable Streams protocol, plus read-only /__ursula/metrics, /__ursula/ready and /__ursula/usage	Applications, the gateway, and clients following a leader redirect
Cluster	server.cluster_listen, or server.listen when it is unset	Raft gRPC (append entries, votes, snapshots, leader-read checks, group reads and writes one node forwards to another, leadership transfer), /__ursula/leadership-shed and /__ursula/cluster-probe	Other voters and learners, at the url of their [[raft.peers]] entry
Admin	server.admin_listen, loopback 127.0.0.1:4438 by default	The mutating operator routes (Raft group operations, maintenance fences and drains, cold flush, bucket purge, backups) plus metrics, readiness and usage	Operators and ursulactl through a tunnel such as kubectl port-forward

None of the three authenticates its caller. The admin plane is protected by its loopback bind, not by a credential, so bind it to a reachable address only for a trusted in-cluster operator.

Set server.cluster_listen so the client plane serves no Raft route. Each [[raft.peers]] entry then names the peer's cluster listener in url and its client listener in client_url, which is where a follower redirects a client. A client that can reach only the client port then cannot send Raft traffic, as long as the network also keeps it off the cluster port. The Helm chart sets server.cluster_listen by default (see Kubernetes NetworkPolicy).

Every node of a deployment carries the same server.cluster_id. A check that makes peers compare it on every Raft RPC and refuse a peer with another id is planned for 0.8 (#601). It stops two deployments wired to each other by mistake from exchanging Raft traffic. It is not authentication: the id is not a secret, and anyone who can reach a cluster port can still send it.

What Ursula does not do

Handle the following outside Ursula:

• TLS / HTTPS. The public listener serves plain HTTP. No built-in rustls.
• Inter-node encryption and authentication. Peer gRPC (Raft heartbeats, append-entries, snapshots, and leader-read checks) runs over h2c on the cluster plane, and a node accepts it from any caller. Peers must share a private network. A client's write to a non-leader gets a 307 redirect to the group leader's client URL rather than being forwarded over gRPC. Only node-internal calls are forwarded between nodes.
• API authentication on nodes. Ursula nodes themselves accept any caller with network reach. Bearer-token validation is available as an opt-in feature of the gateway (see below); node listeners must stay on a private network either way.
• Authorization / multi-tenancy on nodes. Nodes enforce no per-user, per-bucket, or per-scope ACLs. The gateway's opt-in access control provides a bucket-level tenant boundary; anything finer stays upstream.
• Admin authentication. The admin plane has no auth gate. Its loopback bind is the only thing that keeps it private.
• Per-client rate limiting. A single noisy client can saturate a core's mailbox or a group's hot ring.
• Health/readiness endpoints. GET /__ursula/ready is unauthenticated on the client plane and reports only readiness plus WAL free-space guard state. /__ursula/metrics on the same plane contains much more operational detail. Keep the client listener private.
• At-rest encryption beyond the cold tier. Cold-tier S3 writes (including Raft snapshots) request SSE-S3 by default (storage.cold.s3.server_side_encryption, switchable to aws-kms or none). The hot ring is in memory; WAL and Raft log directories live on disk in plaintext — use full-disk encryption at the host level. Per-tenant KMS keys and client-side encryption are out of scope.

CORS is permissive (Access-Control-Allow-Origin: *). Restrict at the proxy for browser traffic.

Tenant offboarding has a first-class erasure path: the admin-plane bucket purge endpoint removes a tenant's streams, bucket, and cold objects idempotently, leaving other tenants untouched. It deliberately retains aggregate monotonic usage counters so asynchronous accounting cannot miss committed work; see the operations note for the remaining identifier-erasure limitation.

Gateway access control (opt-in)

A shared or internet-facing deployment can enable OAuth resource-server checks on ursula gateway. The feature is off by default; without the flags the gateway keeps its original trusted pass-through behavior.

ursula gateway \
  --upstream http://ursula-0:4437 \
  --auth-issuer https://issuer.example \
  --auth-audience https://streams.example \
  --auth-policy /etc/ursula/policy.toml

• Authentication. Bearer credentials are validated as RFC 9068 JWT access tokens: the header must declare typ: at+jwt (OIDC ID tokens are rejected), the signature must verify against the issuer's JWKS, and iss, aud, sub, client_id, iat, exp, and jti must all be present and valid. The JWKS location comes from --auth-jwks-url or RFC 8414 metadata discovery; keys are cached by kid and refetched on rotation.
• Tenant boundary. The bucket is the top-level namespace and logical tenant boundary. The policy file declares each bucket's owners (issuer-qualified subjects) and whether anonymous reads are allowed:

[[bucket]]
id = "tenant-a"
public_read = true
owners = [{ issuer = "https://issuer.example", subject = "user-1" }]

• Concealment. Unknown buckets, private buckets probed by strangers, and write attempts without ownership all answer the same 404 a missing resource would, so a private stream's existence is not observable.
• Internal paths. Every /__ursula/* path answers 404, the gateway's own metrics at /__ursula/gateway/metrics included. The counters hold no tenant data, but they show traffic volume. --auth-public-metrics serves them to every caller, with or without a token, as a gateway without access control does. The gateway then does not check a presented token on that path, so an invalid or expired one gets the counters too. Set the flag only where untrusted clients cannot reach the gateway, or where a proxy in front of it denies /__ursula/*. Gateways before 0.7.4 served the metrics to every caller, with or without access control.
• Credential termination. The gateway strips Authorization before forwarding; upstream nodes never see end-user credentials and must remain on a private network.
• Anonymous public reads. public_read grants exactly the read-only actions (read, head, tail, snapshot read) to unauthenticated callers — never writes, deletes, or bucket administration.
• Subscriptions do not outlive their credential. A live tail is one long-lived request, so a single admission check at connection time would turn a short-lived token into an unbounded read. The gateway ends an SSE subscription at the credential's exp with a final event: credential-expired frame, so a client can tell expiry from end-of-stream and re-authenticate. Anonymous reads on public_read buckets have no credential and so no deadline. Note that this bounds duration, not revocation: a token revoked before its exp keeps an open subscription until then.

An access-controlled gateway can additionally enable per-tenant admission limits and usage accounting:

ursula gateway ... \
  --quota-policy /etc/ursula/quotas.toml \
  --usage-log /var/log/ursula/usage.jsonl

• Quotas (--quota-policy): per-bucket request rate (429 with Retry-After), concurrent live-read connections, and request body size. Limits are gateway-process-local; a horizontally scaled deployment multiplies effective limits by replica count. Ursula's own 503 backpressure semantics are unchanged. Neither Ursula nor this gateway enforces per-bucket stream-count or retained-bytes limits; enforce them in your control plane from the aggregated /__ursula/usage counters.
• Usage (--usage-log): per-tenant request, ingress, and egress byte counters aggregated by (bucket, principal, action class) and appended as sequence-numbered JSONL batches on --usage-flush-secs intervals. --usage-chunk-bytes adds a chunks counter alongside them: each append is counted as ceil(bytes / unit), never below one, summed per request. It exists because that sum cannot be recovered afterwards — two appends of 5 KiB and 25 KiB and two of 10 KiB and 20 KiB agree on both request count and byte total while owing four units and three — so a deployment that charges per write unit has to be handed the sum rather than the ingredients. The unit size is a pricing choice and Ursula does not pick one; 10 KB and 25 KB are both in use. A batched append costs proportionally less than the same records sent individually, which is the intended incentive: batching is cheaper to serve. A failing sink delays reporting (batches queue and merge) but never blocks requests or drops counts. Egress is counted from actually streamed bytes, including SSE bodies. Committed-truth counters (append bytes surviving retries, retained bytes) come from Ursula's replicated state and are a separate, complementary ledger.

Cross-origin reads (opt-in)

public_read is only nominally public until the origin answers CORS: without it, browser JavaScript cannot read a public stream cross-origin. Allowed origins are deployment policy, so nothing is sent unless you list them.

ursula gateway ... \
  --cors-allowed-origin https://app.example.com \
  --cors-allowed-origin https://studio.example.com

Pass * instead to allow any origin.

Three properties are deliberate:

• Credentials are never allowed. Ursula authenticates from an Authorization header the caller sets explicitly, which CORS does not treat as credentials, and no cookies are involved. So * grants no ambient access — a cross-origin page must still present its own bearer token.
• Access-Control-Expose-Headers: *. A read carries its continuation in response headers (stream-next-offset, stream-cursor), and a browser cannot see those without exposure — an unexposed client can read one page and never advance. The wildcard is only honoured while credentials stay disallowed, which is the second reason they are.
• Preflight never consults the resource. OPTIONS arrives without Authorization, so a per-bucket answer would tell an unauthenticated caller whether a private bucket exists. The preflight reply is identical for every path and is produced before authorization runs, which keeps concealment intact.

Recommended deployment

                    Untrusted internet
                            │
                            v
                  ┌─────────────────────┐
                  │   Reverse proxy     │  TLS, authn, per-client
                  │ (nginx / Envoy / …) │  rate limiting, CORS
                  └──────────┬──────────┘
                             │ plain HTTP, private network
                ┌────────────┼────────────┐
                v            v            v      client port (4437)
           ┌────────┐   ┌────────┐   ┌────────┐
           │ Ursula │   │ Ursula │   │ Ursula │
           │  node  │   │  node  │   │  node  │
           └────────┘   └────────┘   └────────┘
                ↕ gRPC h2c between voters only, cluster port (4439)
                          (Raft replication)

Checklist

• Bind to the private interface. Set server.listen = "10.0.0.X:4437" or use a security group / firewall so the listener is unreachable from public networks.
• Split the cluster plane off. Set server.cluster_listen and give every peer a client_url, then let only the voters reach the cluster port.
• Keep the admin plane on loopback. Leave server.admin_listen at its loopback default and reach it through a tunnel.
• Terminate TLS at the proxy. Ursula stays plain HTTP on the internal side.
• Authenticate at the proxy. Validate the caller (OAuth2, mTLS, signed requests) and reject unauthenticated traffic before it reaches Ursula.
• Block internal paths from public traffic. Deny /__ursula/* at the proxy. The client plane serves metrics, readiness and usage there. A gateway with access control answers 404 for them.
• Use IAM roles for S3. Omit static storage.cold.s3.access_key_id / storage.cold.s3.secret_access_key values and let the AWS SDK credential chain discover credentials.
• Encrypt data volumes. Apply full-disk encryption to raft.wal.path.
• Keep peer traffic private. Never route gRPC peer traffic across the public internet.

Kubernetes NetworkPolicy

The Helm chart serves the cluster plane on its own port by default (server.clusterListen: true, server.ports.cluster: 4439). Peers dial that port through the headless peer Service. The client Service exposes only the client port, and the admin plane stays on pod loopback. Without a NetworkPolicy, any pod can still dial a voter's pod IP on the cluster port.

A Service only decides what gets a stable name. Any pod in the cluster can still dial a pod IP on any port. Set server.networkPolicy.enabled: true to render a NetworkPolicy for the server pods, on a CNI that enforces NetworkPolicy:

• The client port admits the sources in server.networkPolicy.clientIngressFrom, a list of NetworkPolicy peers. Empty, the default, admits every source. The release's gateway and Helm test pods are always admitted when they are enabled. List every other client: applications, anything that polls /__ursula/metrics, and an ipBlock for traffic that arrives through a LoadBalancer or NodePort client Service.
• The cluster port admits only the release's own server pods.
• The admin port needs no rule, because it listens on loopback. The chart refuses to render the policy when server.adminListen is not on loopback.

server:
  networkPolicy:
    enabled: true
    clientIngressFrom:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: apps

The policy selects only ingress, so egress stays open. Kubelet probes reach the client port from the node, which most CNIs admit whatever the policy says. Check yours before you restrict clientIngressFrom.

Reporting vulnerabilities

Open a GitHub Security Advisory on tonbo-io/ursula rather than a public issue.