Testing Olyx and Provider Plugins

This guide explains how to test Olyx without creating accounts with model providers. It covers the core runtime, the provider-plugin contract, and optional live checks.

Prerequisites

For the full suite, use a checkout of the Olyx CLI repository with Rust 1.88 or newer, a C compiler, and Python 3. Run commands from the repository root. Downloaded binary archives do not include the full development test suite. The initial Cargo build may download packages; the local test suite does not call model providers or require Docker.

To test a separately installed plugin with an existing CLI binary, you need Python 3 and that adapter’s executable and runtime. You do not need the Rust compiler unless you build Olyx from source.

What each test checks

Olyx owns configured routing, request validation, streaming to clients, SQLite traces, and replay. An HTTP endpoint or external executable supplies model output. Tests for these two responsibilities establish different facts:

Check	What it establishes	What it does not establish
Core tests	Routing, fallback, validation, storage, and cancellation behavior	Model quality or availability
Compatible HTTP fixtures	Request/response handling, SSE, tools, and embedding validation	Acceptance by a particular hosted account
Plugin contract tests	Version negotiation, capabilities, process lifecycle, and normalized output	Correct translation of every native vendor API
Live checks	Selected operations worked with a particular model, account, and binary at that time	Support for all models, regions, options, or future versions

An echo test program returns predictable responses without running a model. It can test requests passing through Olyx, but not answer quality. Passing these tests does not prove that every provider accepts the same requests.

Run the account-free suite

From the CLI repository root, run:

./scripts/test-providers

The command builds the debug binary, runs Rust unit tests, and executes local integration suites for compatible HTTP, plugins, application serving, embeddings, diagnostics, adversarial inputs, SQLite persistence, and standalone setup. It also runs the Python verification tests, including public-package and release-archive checks.

The summary reports each suite and local contract coverage for openai and plugin. The command never opts in live checks merely because credentials are present in your shell. Local test subprocesses receive a restricted environment.

A failed suite does not prevent the remaining suites from running. The command exits nonzero if a check fails, times out, or cannot start. To investigate, run the named smoke script directly; for example:

python3 scripts/smoke-plugins.py target/debug/olyx

The fixture checks version/capability errors, tools, streaming, embeddings, replay immutability, child failures and timeouts, credential isolation, ordered fallback, and cancellation when an HTTP client disconnects.

The storage suite checks concurrent CLI writers, exact payload persistence, process restarts, backup/restore, retention, locked-write failures, and recovery following forced termination. It uses disposable databases rather than your application’s traces. To run it independently:

python3 scripts/smoke-storage.py target/debug/olyx

Using an existing binary

To skip compilation and Rust unit tests, use an existing Olyx binary:

python3 scripts/test-providers \
  --binary target/release/olyx \
  --report provider-tests.json

On Windows, use the corresponding .exe path. The report path must be a new file; an existing report is not overwritten. The JSON records suite status and contract coverage, while excluding raw subprocess output, credentials, and local paths.

Check a plugin independently

The test-plugin script tests a single configured adapter through actual CLI and HTTP operations. Use the adapter’s fixture mode to avoid contacting its service. The script executes trusted local code; it does not simulate a plugin for you.

For a reproducible starting point, the CLI checkout includes examples/plugins/echo.py. It echoes text, emits synthetic vectors and tool calls, and never makes a provider request. Create its configuration from the repository root:

python3 - <<'PY'
import json
from pathlib import Path
import sys

configuration = {
    "models": {
        "fixture": [{
            "protocol": "plugin",
            "model": "fixture-model",
            "plugin": {
                "command": sys.executable,
                "args": [str(Path("examples/plugins/echo.py").resolve())],
                "capabilities": ["chat", "streaming", "tools", "embeddings"]
            }
        }]
    }
}
with Path("plugin-fixture.json").open("x") as output:
    json.dump(configuration, output, indent=2)
PY

The exclusive file creation prevents replacing an existing configuration. The absolute interpreter/script paths make invocation independent of Olyx’s working directory. The fixture needs no credential variable.

Run its conformance checks:

python3 scripts/test-plugin \
  --binary target/debug/olyx \
  --config plugin-fixture.json \
  --alias fixture \
  --report plugin-contract.json

For this fixture, the checks cover configuration, chat with trace/replay, streaming, tools, and HTTP embeddings with trace/replay. A plugin declaring fewer capabilities is tested only for the operations it declares. The report is labeled plugin-contract-only; a pass does not certify a real model service.

To develop your own adapter, follow Creating a Provider Plugin. Keep its service-specific translation tests and fixture mode in the adapter’s own repository. A plugin does not require a source change in Olyx.

Opt in to live verification

Live checks are explicit. Supply a private configuration, exactly one candidate per selected alias, and the credentials that configuration references. For Hugging Face, replace the model placeholder with a served model and set HF_TOKEN in the environment before running:

python3 scripts/test-providers \
  --binary target/release/olyx \
  --live-config /path/to/private-provider-config.json \
  --alias huggingface \
  --require-live \
  --report provider-tests-live.json

Local checks run first. If they fail, no live check is attempted. The live runner checks compatible HTTP chat, streaming, trace persistence, and replay. Each selected route makes three inference calls with a 128-token output limit and 90-second command timeout; provider usage still applies.

Missing credentials or invalid selections produce SKIP. --require-live makes any selected skip fail the run. Repeat --alias to opt in another route. A fallback chain is not accepted for certification because its successful result could come from a different provider.

Native plugins are verified independently using their own translation and live service tests. Hugging Face chat access does not establish Bedrock access or OpenAI-compatible embedding support.

Record and share test results

Keep contract coverage separate from live verification in release notes and adapter documentation. Record the tested binary, model scope, date, and provider; only share public model or account details when appropriate.

The compatible HTTP certification report includes timestamp and binary/suite fingerprints. It excludes aliases, account URLs, deployment IDs, prompts, responses, trace IDs, and raw provider errors. The plugin contract report records contract version and check results. Neither report is independent attestation.

Review reports before sharing them. Do not attach a private configuration, provider tokens, SQLite database, or raw production recording. A contributor can run live checks using their own account and share the sanitized evidence without sharing credentials with Olyx maintainers.