Observability

Ursula emits OpenTelemetry traces and metrics over OTLP, and keeps a built-in JSON metrics endpoint for quick inspection. Telemetry export is off by default: with no collector configured the server runs exactly as before and the hot read/write path pays nothing.

Enabling OTLP export

Set the standard OpenTelemetry OTLP exporter settings before starting ursula:

# Send traces and metrics to a collector (Tempo, Jaeger, the OTel Collector, …).
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4318"

# Optional: head-sampling ratio for new root traces (0.0–1.0, default 1.0).
export OTEL_TRACES_SAMPLER_ARG="0.1"

cat > /tmp/ursula-otel.toml <<'TOML'
[server]
listen = "0.0.0.0:8080"
TOML

ursula --preset default --config /tmp/ursula-otel.toml

The exporter uses OTLP over HTTP/protobuf (port 4318), reusing the server's HTTP stack rather than opening a second gRPC tree alongside Raft. When the endpoint is unset, the process stays on the stderr fmt logger driven by RUST_LOG (default info).

Spans batch-export continuously and metrics export on a fixed interval (OTEL_METRIC_EXPORT_INTERVAL, milliseconds). A final flush on clean shutdown is only a backstop. Don't rely on it to observe data.

Identifying the node

Every span and metric carries the process's OpenTelemetry resource. The server sets service.name=ursula and, when running with a Raft node id, service.instance.id=<node_id> so a cluster's nodes are distinguishable. Add more resource attributes (e.g. host.name, region, AZ) with the standard variable. They stay bounded by cluster size, so this is a resource dimension, not a high-cardinality label:

export OTEL_RESOURCE_ATTRIBUTES="host.name=$(hostname),deployment.environment=prod"

Traces

A single request produces one boundary span (http.append, http.read, or http.head) tagged with bucket and stream, never the payload. That span is propagated across the internal actor mailboxes, so on-core work (core.append, core.read, …) nests underneath it. When a follower forwards a read to the leader, the W3C traceparent rides the Raft gRPC request, so the follower and leader appear in the same trace end to end.

Trace context travels only in gRPC transport metadata. It is never written to the Raft log, so state-machine determinism and replay are unaffected.

Sampling and the hot path

Request-boundary spans (http.*) are info level, so production traces always carry one span per request. The on-core spans below them (core.append, core.read, …) are debug level: filtered out at the default info and compiled out of release builds entirely (release-max-info, on by default), so the hot read/write path pays nothing for them. Measured cost:

Operation	Cost
`info` boundary span (export off, fmt only)	~285 ns
`debug` span filtered at `info`	~1 ns
cross-mailbox context capture	~5 ns

For high-throughput deployments, lower OTEL_TRACES_SAMPLER_ARG to bound export volume.

Seeing on-core detail

The core.* spans only materialize in a build without release-max-info (e.g. a debug build), and only when their target is enabled. Use a crate-scoped filter, never a global debug:

# Good: only Ursula's own crates emit debug spans.
export RUST_LOG="info,ursula_runtime=debug,ursula_raft=debug"

A global RUST_LOG=debug turns on debug spans/logs from OpenRaft, the OpenTelemetry SDK, reqwest, and every other dependency, which floods the collector and bloats the OTLP payload. Scope the filter to the ursula_* crates instead.

tokio-console needs tokio's TRACE-level instrumentation, which release-max-info compiles out. Build with --no-default-features --features tokio-console when using it.

Metrics

Two surfaces expose the same counters:

OTLP: when an endpoint is configured, runtime metrics are exported periodically as OpenTelemetry instruments (ursula.appends.accepted, ursula.mutations.applied, ursula.raft_apply.ns, ursula.wal.records, ursula.group_mailbox.depth, …), ready for Prometheus/Grafana via a collector.
JSON: GET /__ursula/metrics returns the full per-core/per-group snapshot for quick inspection and is what ursulactl status reads.

For redundancy alerts, inspect the JSON metric raft_maintenance.group_issues. Alert on incomplete_voter_set or joint_membership, and treat a missing maintenance report as unavailable evidence. Also alert on nonempty raft_maintenance.node_issues. These fields remain strict when healthy survivors return 200 from /__ursula/ready during a replica rebuild. They are JSON report fields, not new OTLP instrument names.

curl --fail http://NODE:4437/__ursula/metrics | jq '
  .raft_maintenance | {node_issues, group_issues}'

Metrics are gathered on the hot path with lock-free per-core counters. The OTLP layer is an export-time bridge: it reads a snapshot at the collection interval rather than calling the metrics SDK per record, so enabling export adds nothing to the append/read path.

For WAL operations, the JSON snapshot includes per core the journal's size and segments (wal_physical_bytes, wal_segments), rotations (wal_rotations), segments deleted and their bytes (wal_reclaims, wal_reclaimed_bytes), the time reclaim took and the passes that stopped on an error (wal_reclaim_ns, wal_reclaim_failures), live entry bytes copied out of old segments (wal_rewritten_bytes), and the groups reported lagging to the snapshot driver with the segments kept only for them (wal_lagging_groups, wal_pinned_segments). Reads report entries served from the cache and read from disk (wal_cache_hits, wal_cache_misses) and the frames and bytes read (wal_disk_reads, wal_disk_read_bytes), per core and per group. Each group reports its cached bytes and indexed entries (wal_cache_bytes, wal_indexed_entries). Startup scan work is under wal_recovery_ns, wal_recovery_records, wal_recovery_bytes, and wal_recovery_live_entries. Free-space admission is reported separately as wal_available_bytes, its two watermarks, wal_disk_pressure, and wal_disk_stat_errors. These are bounded per-node, per-core and per-group series.

For the snapshot driver, the JSON snapshot includes the node's unsnapshotted Raft log and the most it has held (raft_log_bytes, raft_log_bytes_high_watermark), its pressure level against raft.snapshot_log_budget (raft_snapshot_pressure_level: 0 below three quarters of the budget, 1 up to the budget, 2 past it), the snapshot builds a pass keeps in flight at that level (raft_snapshot_pressure_builds), and the builds that run now and the most that ran at once (raft_snapshot_builds_in_flight, raft_snapshot_builds_in_flight_max). Its pressure passes that planned a build or changed the groups held out (a group whose trigger went unclaimed is held out for 0.5 s, doubling for each such trigger in a row up to 16 s, until a trigger of it is claimed) add up in raft_snapshot_pressure_passes, the builds they started in raft_snapshot_pressure_groups, and their time in raft_snapshot_pressure_pass_ns.

The OTLP collection interval defaults to 60s. For a quick check, shorten it so readings arrive without waiting (and without relying on the shutdown flush):

export OTEL_METRIC_EXPORT_INTERVAL=1000  # milliseconds