Inference you can put in front of customers
Relay routes every request to the model that answers it best, then shows you exactly what it cost and how long it took. Streaming, structured output, and per-request tracing on one endpoint.
1M tokens free a month. No card until you pass it.
Summarise this support thread and flag anything that needs a human.
What the endpoint does
Six things, measured where measuring is possible.
One endpoint, every model
Send a request without naming a model. Relay picks based on the task, the context length, and what you have told it to optimise for. Change the policy, not your code.
14models routed
Structured output that validates
Pass a JSON schema and get back something that matches it, or an error that tells you why it could not. No parsing prose, no retry loops written by hand.
99.2% schema conformance
Streaming by default
Tokens arrive as they are generated over SSE. First token typically lands before a spinner would have finished its first rotation.
180ms to first token
Every request is traceable
Each call returns a trace ID that resolves to the full request, the route decision, token counts, and cost. When something goes wrong at 3am, the answer is already recorded.
Your data stays yours
Requests are not retained beyond the trace window you set, and are never used for training. Zero-retention mode is one header.
Fails predictably
When a provider degrades, Relay reroutes mid-flight and tells you it did. You get a slower answer instead of a 503.
Models on the router
Pin one by name, or let the router choose and read which it picked in the response headers.
| Model | Context | First token | Price | Best for |
|---|---|---|---|---|
| relay-router | auto | 180ms | varies | Default. Picks per request. |
| frontier-lg | 200K | 420ms | $3.00 / Mtok | Long documents, hard reasoning |
| frontier-sm | 128K | 190ms | $0.80 / Mtok | Chat, summarisation |
| swift | 32K | 90ms | $0.15 / Mtok | Classification, extraction |
| embed-3 | 8K | 40ms | $0.02 / Mtok | Search and retrieval |
Pricing
Token prices are the provider's list price. You are paying for routing, tracing, and the failover, not a markup per token.
Build
$0to start
For prototypes and side projects.
- 1M tokens a month
- All models
- 7-day trace retention
- Community support
Production
$400a month
For products with customers in front of them.
- 50M tokens included, then usage
- Custom routing policies
- 90-day trace retention
- Zero-retention mode
- 99.9% uptime commitment
Scale
Talk to us
For teams with compliance requirements or volume commitments.
- Everything in Production
- Dedicated capacity
- SSO and audit log
- Custom retention and residency
- Support with an SLA
Questions worth asking first
What happens when a model provider goes down?
Relay reroutes to the next model that satisfies your policy, mid-request where the stream allows it. The response includes a header naming the model that actually served it, so your logs stay honest about what happened.
Do you train on our requests?
No. Requests are retained only for the trace window you configure, and are excluded from training by contract with every upstream provider. Setting the zero-retention header drops them immediately after the response is sent.
How is routing decided?
By a policy you write: optimise for cost, for latency, or for a quality floor measured against your own evals. Absent a policy, Relay uses task type and context length. The route decision is recorded in the trace, so it is auditable rather than magic.
Can we pin a specific model?
Yes. Name a model and Relay calls it directly, skipping routing. Most teams start pinned and move to routing once they have evals they trust.
What does this cost compared to calling providers directly?
Token prices are the provider's list price. Relay charges for the platform, not a per-token markup. Teams usually spend less overall because routing sends cheap requests to cheap models.
Put it in front of a real request
The free tier needs a card only when you exceed it. Most prototypes never do.