Providers & failover

The provider set

Provider	Registers when	Notes
OpenAI	OPENAI_API_KEY	never free; frontier rates — set deliberately
Gemini	GEMINI_API_KEY_1..16	each numbered key becomes an independently metered pool slot gemini_1..N; callers ask for logical gemini
OpenRouter	OPEN_ROUTER_API_KEY	note the underscore — it fails silently if guessed
Ollama	OLLAMA_MODEL	local, $0.00; no default model — you must name one
Groq / Cerebras / NVIDIA / GitHub Models	their keys	supported by code, out of the default order

A provider registers only when its key is set. After editing .env, check GET /v1/providers — a misspelled variable produces a silently missing provider, and the first symptom is unknown provider on a call that should have worked.

Each provider is one module under costhelm/providers/; most are thin subclasses of the OpenAI-compatible workhorse, which also normalises per-model quirks: reasoning-effort dialects, fixed-temperature models, vision content blocks, prompted-tool fallback for local models without native tool calling.

Picking a candidate

For each request the router walks the candidate ring and skips providers that:

• lack a required capability — tools, vision, reasoning, structured output are inferred from the request;
• can’t hold the prompt (max_ctx vs the token estimate);
• are out of rate budget — RPM / RPD / TPM windows and per-provider cooldowns, tracked per instance (each Gemini key slot meters independently);
• are benched — either by config (routing.yaml unavailable:) or by a failure-class cooldown.

Every skip is recorded in the response’s attempted list with its reason.

Failure-class cooldowns

When a provider fails, the error is classified and the provider benched for a class-specific time (routing.yaml backoff:): queue-full 15s, RPM 60s, rate-limited 30s, upstream 5xx 20s, auth 10min — and timeout 10 minutes, deliberately much longer than the 180s it takes to notice, because a dead endpoint that comes straight back as a candidate burns another 180s on the next request.

Limits are data

Per-provider pacing (rpm/rpd/tpm/cooldown/max_ctx) lives in routing.yaml’s limits: block, merged over built-in defaults at boot — partial entries allowed, so raising one provider’s rpm is a one-line edit. The shipped values are deliberate under-estimates of paid tiers: too low only throttles locally, too high buys 429s.