ursuladurable streams
v0.3.22 · apache-2.0

Resumable AI Stream

Stream LLM tokens through a durable timeline so clients can refresh, reconnect, and resume without losing a token.

Instead of piping a model's output straight into one HTTP response, the generating worker appends tokens to a stream, and clients replay and tail that stream. A client that reconnects mid-generation replays the answer so far and continues from the same position, without losing or duplicating tokens. The upstream chat-aisdk example uses the same pattern.

One stream per run

Create a text stream per generation, with a TTL so finished runs clean themselves up:

curl -X PUT http://127.0.0.1:4437/agent/run-42 \
  -H 'Content-Type: text/plain' \
  -H 'Stream-TTL: 86400'

Producer: append tokens, close at the end

The worker that talks to the model appends each chunk. Producer headers make retries safe: a chunk re-sent after a network error is deduplicated instead of appended twice:

const base = "http://127.0.0.1:4437";
let seq = 0;

for await (const chunk of modelResponse) {
  await fetch(`${base}/agent/run-42`, {
    method: "POST",
    headers: {
      "Content-Type": "text/plain",
      "Producer-Id": "worker-1",
      "Producer-Epoch": "1",
      "Producer-Seq": String(seq++),
    },
    body: chunk.text,
  });
}

// Seal the stream: readers get an EOF signal, further appends return 409.
await fetch(`${base}/agent/run-42`, {
  method: "POST",
  headers: { "Stream-Closed": "true" },
});

If the worker crashes and restarts, bump Producer-Epoch and start Producer-Seq back at 0 (see Exactly-once writes).

Consumer: replay, then tail

offset=-1 replays everything appended so far, then stays live for new tokens. Reopening the page mid-generation therefore shows the partial answer immediately and continues streaming:

const es = new EventSource(`${base}/agent/run-42?offset=-1&live=sse`);

es.addEventListener("data", (e) => {
  answerEl.textContent += e.data;
});

es.addEventListener("control", (e) => {
  if (JSON.parse(e.data).streamClosed) es.close(); // generation finished
});

Or watch it from a terminal:

curl -N 'http://127.0.0.1:4437/agent/run-42?offset=-1&live=sse'

The stream is durable and readable by any number of clients, so the same transcript can be tailed from a second device, read after the fact, or consumed by another process. The producer does not need to know about its readers.

Next