Change one line. Keep your stack.
from openai import OpenAI client = OpenAI( base_url="https://api.deepinference.co/v1", # the only change api_key=os.environ["DEEPINFERENCE_API_KEY"], ) stream = client.chat.completions.create( model="qwen3-235b-a22b", messages=[{"role": "user", "content": "..."}], stream=True, )
OpenAI-compatible, end to end. Same SDKs, same request shape, same streaming. Your retries, tracing and evals keep working.
Behind the URL is a dedicated endpoint: your model, your SLOs, no shared queue.
Browse the models →Nothing is ever done optimizing.
Traffic shifts, models update, new silicon lands. The loop below runs against your production traffic for as long as the endpoint exists.
Your live traffic
Prompt and output lengths, prefix hit rates, arrival curves, p50/p99 against the SLOs you set.
The full stack
Serving engine (vLLM, SGLang, TensorRT-LLM), precision and quantization recipes, kernels, speculative decoding, batching, prefill/decode split, KV-cache tiering, routing, topology.
Correctness first
Every candidate replays held-out traffic. Accuracy deltas and SLOs must hold before anything ships.
Shadow, canary, full
The winning config rolls out behind the same URL and becomes the baseline the next search has to beat.
Agents find it. Every endpoint gets it.
Our agents profile every serious open-weight release within hours of the weights dropping. What one endpoint’s search finds — a kernel, a bit-width, a batching recipe — is encoded into the search space and applied to every endpoint and every architecture.
Search agents
Retune every endpoint against its own live traffic, around the clock, scored on your SLOs.
- precision & quantization recipes
- batching, routing, KV-cache tiering
- speculative-decode tuning
- hardware topology & placement
Kernel agents
Write, profile and verify kernels the reference recipes do not have yet, then hand them to the search.
- custom kernels per architecture (CUDA, HIP, Triton)
- bit-widths past the reference recipe
- new-silicon bring-up
- day-zero support for new models
The right silicon is an empirical question.
One workload, measured across accelerators on our stack. Performance per dollar decides — and the answer changes every hardware generation.
Llama-4-Maverick-400B, 4k tokens in / 800 out, aggregate tok/s per GPU at 64 concurrent streams, measured on our serving stack on partner-cloud GPUs. GPU-hour is the public on-demand rate. We will tell you honestly where your workload should run — including when it is not the hardware you were about to buy.
Three ways to run it. One is ready today.
Start building →Your cloud
available nowThe full serving stack in your VPC, on your GPU quota. Agents tune it there. Weights and prompts never leave your network.
Dedicated endpoints
coming soonYour model on partner-cloud capacity that we operate for you, behind a single URL.
Reserved capacity
coming soonCommitted throughput at a committed price, for workloads with a floor you can predict.
Point your base_url at us.
Leave your details and our sales team will get back to you within a day.