Observability

Ursula emits OpenTelemetry traces and metrics over OTLP, and keeps a built-in JSON metrics endpoint for quick inspection. Telemetry export is off by default: with no collector configured the server runs exactly as before and the hot read/write path pays nothing.

Enabling OTLP export

Set the standard OpenTelemetry OTLP exporter settings before starting ursula:

# Send traces and metrics to a collector (Tempo, Jaeger, the OTel Collector, …).
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4318"

# Optional: head-sampling ratio for new root traces (0.0–1.0, default 1.0).
export OTEL_TRACES_SAMPLER_ARG="0.1"

cat > /tmp/ursula-otel.toml <<'TOML'
[server]
listen = "0.0.0.0:8080"
TOML

ursula --preset default --config /tmp/ursula-otel.toml

The exporter uses OTLP over HTTP/protobuf (port 4318), reusing the server's HTTP stack rather than opening a second gRPC tree alongside Raft. When the endpoint is unset, the process stays on the stderr fmt logger driven by RUST_LOG (default info).

Spans batch-export continuously and metrics export on a fixed interval (OTEL_METRIC_EXPORT_INTERVAL, milliseconds). A final flush on clean shutdown is only a backstop. Don't rely on it to observe data.

Identifying the node

Every span and metric carries the process's OpenTelemetry resource. The server sets service.name=ursula and, when running with a Raft node id, service.instance.id=<node_id> so a cluster's nodes are distinguishable. Add more resource attributes (e.g. host.name, region, AZ) with the standard variable. They stay bounded by cluster size, so this is a resource dimension, not a high-cardinality label:

export OTEL_RESOURCE_ATTRIBUTES="host.name=$(hostname),deployment.environment=prod"

Traces

A single request produces one boundary span (http.append, http.read, or http.head) tagged with bucket and stream, never the payload. That span is propagated across the internal actor mailboxes, so on-core work (core.append, core.read, …) nests underneath it. When a follower forwards a read to the leader, the W3C traceparent rides the Raft gRPC request, so the follower and leader appear in the same trace end to end.

Trace context travels only in gRPC transport metadata. It is never written to the Raft log, so state-machine determinism and replay are unaffected.

Sampling and the hot path

Request-boundary spans (http.*) are info level, so production traces always carry one span per request. The on-core spans below them (core.append, core.read, …) are debug level: filtered out at the default info and compiled out of release builds entirely (release-max-info, on by default), so the hot read/write path pays nothing for them. Measured cost:

Operation	Cost
info boundary span (export off, fmt only)	~285 ns
debug span filtered at info	~1 ns
cross-mailbox context capture	~5 ns

For high-throughput deployments, lower OTEL_TRACES_SAMPLER_ARG to bound export volume.

Seeing on-core detail

The core.* spans only materialize in a build without release-max-info (e.g. a debug build), and only when their target is enabled. Use a crate-scoped filter, never a global debug:

# Good: only Ursula's own crates emit debug spans.
export RUST_LOG="info,ursula_runtime=debug,ursula_raft=debug"

A global RUST_LOG=debug turns on debug spans/logs from OpenRaft, the OpenTelemetry SDK, reqwest, and every other dependency, which floods the collector and bloats the OTLP payload. Scope the filter to the ursula_* crates instead.

tokio-console needs tokio's TRACE-level instrumentation, which release-max-info compiles out. Build with --no-default-features --features tokio-console when using it.

Metrics

Two surfaces expose the same counters:

• OTLP: when an endpoint is configured, runtime metrics are exported periodically as OpenTelemetry instruments (ursula.appends.accepted, ursula.mutations.applied, ursula.raft_apply.ns, ursula.wal.records, ursula.group_mailbox.depth, …), ready for Prometheus/Grafana via a collector.
• Cluster id: ursula.cluster.info is 1 with the node's server.cluster_id as its cluster_id label, so dashboards and alerts tell deployments apart. ursula.raft.peer_cluster_refusals counts the Raft RPCs refused because the peer is in another cluster, both ways, and ursula.raft.unidentified_peer_refusals those refused because they named no cluster id. Alert on any increase (see Peers of another cluster).
• JSON: GET /__ursula/metrics returns the full per-core/per-group snapshot for quick inspection and is what ursulactl status reads.

For redundancy alerts, inspect the JSON metric raft_maintenance.group_issues. Alert on incomplete_voter_set, unconfigured_member or joint_membership, and treat a missing maintenance report as unavailable evidence. Also alert on nonempty raft_maintenance.node_issues. These fields remain strict when healthy survivors return 200 from /__ursula/ready while a replica catches up. They are JSON report fields, not new OTLP instrument names.

curl --fail http://NODE:4437/__ursula/metrics | jq '
  .raft_maintenance | {node_issues, group_issues}'

Staged snapshot parts are reported separately: the JSON field staged_snapshot_part_bytes and the OTLP gauge ursula.snapshot_parts.staged_bytes are the bytes of unused parts in the groups the node leads. For a cluster total, read /__ursula/usage from every voter and keep the highest term per group (see usage export). Bounding or billing staged parts beyond Ursula's per-stream bound is the operator's policy.

Metrics are gathered on the hot path with lock-free per-core counters. The OTLP layer is an export-time bridge: it reads a snapshot at the collection interval rather than calling the metrics SDK per record, so enabling export adds nothing to the append/read path.

For WAL operations, the JSON snapshot includes per core the journal's size and segments (wal_physical_bytes, wal_segments), rotations (wal_rotations), segments deleted and their bytes (wal_reclaims, wal_reclaimed_bytes), the time reclaim took and the passes that stopped on an error (wal_reclaim_ns, wal_reclaim_failures), live entry bytes copied out of old segments (wal_rewritten_bytes), and the groups reported lagging to the snapshot driver with the segments kept only for them (wal_lagging_groups, wal_pinned_segments), and the segments that hold purged or otherwise dead entries (wal_dirty_segments), which the journal frees within a quarter of raft.deleted_data_max_age. A core that keeps writing always has some, so wal_dirty_segments is rarely 0 there and is no signal to alert on. Reads report entries served from the cache and read from disk (wal_cache_hits, wal_cache_misses) and the frames and bytes read (wal_disk_reads, wal_disk_read_bytes), per core and per group. Each group reports its cached bytes and indexed entries (wal_cache_bytes, wal_indexed_entries). Startup scan work is under wal_recovery_ns, wal_recovery_records, wal_recovery_bytes, and wal_recovery_live_entries. Free-space admission is reported separately as wal_available_bytes, its two watermarks, wal_disk_pressure, and wal_disk_stat_errors. These are bounded per-node, per-core and per-group series.

For the snapshot driver, the JSON snapshot includes the node's unsnapshotted Raft log and the most it has held (raft_log_bytes, raft_log_bytes_high_watermark), its pressure level against raft.snapshot_log_budget (raft_snapshot_pressure_level: 0 below three quarters of the budget, 1 up to the budget, 2 past it), the snapshot builds a pass keeps in flight at that level (raft_snapshot_pressure_builds), and the builds that run now and the most that ran at once (raft_snapshot_builds_in_flight, raft_snapshot_builds_in_flight_max). Its pressure passes that planned a build or changed the groups held out (a group whose trigger went unclaimed is held out for 0.5 s, doubling for each such trigger in a row up to 16 s, until a trigger of it is claimed) add up in raft_snapshot_pressure_passes, the builds they started in raft_snapshot_pressure_groups, and their time in raft_snapshot_pressure_pass_ns.

For deleted data (see Deleted data), the JSON snapshot reports how long ago the oldest removal applied whose stream a snapshot or a Raft log on the node still holds (raft_deleted_data_oldest_age_ms, 0 for none) and the groups whose snapshot holds one past a quarter of raft.deleted_data_max_age (raft_deleted_data_overdue_groups). deleted_data lists each group whose replica still holds removed streams, with their state, bytes and age. Both gauges come from the snapshot driver, so they stay at 0 in the in-memory mode, which runs no Raft. Alert when raft_deleted_data_oldest_age_ms nears half of raft.deleted_data_max_age.

Each group's leader removes expired streams in the background (see Deleted data). ttl_expiry_sweeps counts the ExpireStreams entries the node's leaders committed, each of which removes up to 256 expired streams, and ttl_expiry_swept_streams the expired streams they removed. ttl_expiry_sweep_due_groups reports the groups the latest pass left with expired streams. A group the pass stopped at its 16 entries or its 250 ms is swept again a second later, with 50 ms per group, so the gauge returns to 0 once its backlog drains. A value that stays above 0 across many passes means a group expires streams faster than its re-runs remove them, that its commits are slow enough to reach the sweep's time per group, or that the sweep stopped on an error, such as a group whose history is pending or whose apply stopped. A sweep an error stopped runs again only at the usual interval. All three work in the in-memory mode too.

With an S3 snapshot store, snapshot_gc_deleted_objects counts the snapshot objects the node's snapshot GC deleted from its own directory. snapshot_gc_failed_passes counts the GC passes that failed, for example because the store refuses deletes, and a failed pass runs again later, backed off up to 15 minutes. Alert when it keeps rising: while GC fails, old snapshot objects that hold deleted streams stay in the store.

cold_gc_foreign_paths_refused counts the refusals of cold GC. Cold GC refuses a path of a queued entry that is not an exclusive chunk, a pack or an external payload of the entry's bucket that the group owns, and an entry for a stream that is not of the entry's bucket or is placed in another group (see Cleaning S3). A pass that runs an entry again counts its refusals again, so the value is not a number of distinct paths. It stays at 0 unless a corrupt or hand-edited backup was restored. Alert on any increase.

open_backoff_groups counts the groups whose engine did not open because the snapshot store did not answer its restore, and per_group_open_backoff is 1 for each of them. Such a group answers GROUP_RESTARTING until its core opens it again, after a wait of 1 to 30 seconds. Alert when it stays above 0: the node cannot read its snapshots.

The OTLP collection interval defaults to 60s. For a quick check, shorten it so readings arrive without waiting (and without relying on the shutdown flush):

export OTEL_METRIC_EXPORT_INTERVAL=1000  # milliseconds