Streaming responses

Set stream to true in the request body. Olyx reads the provider stream and prints each update as a JSON object on its own line (NDJSON). Updates are written as they arrive.

Request streaming output

Save this as streaming-request.json:

{
  "model": "fast",
  "stream": true,
  "messages": [{"role": "user", "content": "Explain local trace storage."}]
}

olyx route --input streaming-request.json

Each update has type snapshot and contains the cumulative response. The final event has type completed, includes trace_id, and contains the full response in output.

Consumers should parse one JSON object per line, replace their current response with each accumulated snapshot, and treat the completed event as successful termination. Do not concatenate snapshots as if they were text deltas. Compatible responses and plugin outputs can include tool calls or other supported non-text fields.

Store and inspect streamed work

A completed response is recorded in the local trace database. Use the completed event’s trace ID:

olyx debug TRACE_ID
olyx replay TRACE_ID

Replay sends the recorded request again and returns the complete response. To compare aliases, configure both names first, then pass --compare ALIAS1,ALIAS2.

Stream to an application

olyx serve returns incremental Server-Sent Events (SSE), rather than cumulative terminal snapshots. Start it with your existing configuration:

export OLYX_SERVER_API_KEY=your-application-secret
olyx serve --local --config olyx.json

From a second terminal, export the same application key and request a stream:

export OLYX_SERVER_API_KEY=your-application-secret
curl -N http://127.0.0.1:8787/v1/chat/completions \
  -H "Authorization: Bearer $OLYX_SERVER_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"fast","stream":true,"messages":[{"role":"user","content":"Explain local trace storage."}]}'

An application should read each data: event and accumulate choices[].delta. A successful HTTP stream ends with data: [DONE]. The response header X-Olyx-Trace-Id identifies its local trace.

Interface	Updates	Successful completion
olyx route	NDJSON snapshot with cumulative output	completed event and exit status 0
olyx serve	SSE chunks with incremental choices[].delta	[DONE] marker

A stream error can arrive after HTTP headers were sent. Treat an SSE error or a closed connection without [DONE] as incomplete even when the initial HTTP status was 200.

Interrupted output

Press Ctrl-C during olyx route or olyx replay to cancel the active request. The CLI saves its cancellation diagnostic before exiting with status 130. Use the trace ID from an earlier snapshot with olyx debug TRACE_ID. If no snapshot was emitted before interruption, list recent traces with olyx debug and inspect the cancelled entry. The provider may already have processed part of the request. Forced termination, such as SIGKILL, prevents Olyx from saving its cancellation details.

A malformed or interrupted provider stream fails the trace and returns a nonzero exit status. Olyx stops the route rather than starting another provider after partial output. A failed step is excluded from replay.

When scripting, require both a successful exit status and a completed event before accepting a streamed result. Partial output is useful for diagnosis but is not a completed model response. See troubleshooting for error handling.

Provider plugins

A streaming plugin must declare chat and streaming and advertise both in its version 1 handshake. It returns cumulative snapshot events followed by a final result; the last snapshot must match that result. Olyx converts these into the usual terminal NDJSON or application SSE. The provider plugin does not write SSE to its stdout. A timeout, invalid event, or failed child process fails the trace; partial output is retained and does not trigger fallback.

See the plugin contract and account-free verification kit.