Isoquant · October 2026
We evaluate how well AI models improve inference performance.
6 models inspect code, make changes and submit solutions across 15 tasks spanning GPU optimization, complete serving and CPU optimization. We measure their submissions against supplied baselines and check that they meet each task’s requirements. Results summarize three independent engineering attempts per model and task.
GPU optimization
9 tasks · geometric mean of valid results
Why this benchmark
Tokenizers, kernels, caches and schedulers decide how much compute each request uses and how long people wait for an answer. Making this code faster without degrading the model’s output quality is at the core of inference engineering.
This benchmark measures how much of that work AI agents can do today. Every task starts from software that already works, so the only credit is for making it faster. A submission that breaks any requirement of its task earns no speedup at all, however fast it runs.
Each column is a separate measurement. A bar shows how much faster a model’s submissions ran than the task’s supplied baseline: 1× is unchanged, and 2× does the same work in half the time.
| Model | |||||
|---|---|---|---|---|---|
| Gemini 4 Argon | |||||
| GPT-6 Astra | |||||
| Muse Wonder 1.0 | |||||
| Claude Opus 5.5 | |||||
| Claude Fable 5.1 | |||||
| GPT-6.1 Sol |
The agent receives a working implementation, the task’s objective, tools and a time limit.
It reads the code, makes changes and runs its own tests and measurements, as an engineer would.
When it finishes, its submission is frozen. Nothing is changed after that point.
We measure the submission and the supplied baseline on the same machine and check every requirement of the task.
Each model makes three independent attempts on every task. On task cards, means two of three attempts were valid.
Nine components found inside modern inference engines. Each starts from a working baseline and must stay correct as batch sizes, shapes and routes change. Four are shown below: normalization, a mixture-of-experts block, recurrent attention and expert dispatch across two GPUs. The other five are measured the same way and count toward the GPU mean.
Compute a scaled residual and its normalization faster across changing batches.
Execute a complete routed mixture-of-experts block faster.
Run matrix-state attention through prefill and continuing decode faster.
Execute sharded experts faster across two GPUs and combine their outputs.
+ 5 more GPU tasksMeasured the same way and counted in the GPU mean.
These three tasks move from a single component to a complete model service. Each serves Qwen3-8B on H100 GPUs, starting from a vLLM deployment, and is scored on end-to-end request latency under mixed traffic. A gain has to hold for the whole service, not just one part of it.
Reduce complete Qwen3-8B request latency under mixed traffic.
Reduce complete request latency with both target and draft models executing.
Reduce request latency across two model workers sharing cached prefixes.
Not all inference work happens on the GPU. Prompts must be tokenized before the model sees them, and streamed output must be parsed into text, reasoning and tool calls as it arrives. These three tasks cover that work, plus quantized projections served from the CPU. Each is measured on ARM64 with a two-CPU limit.
Convert UTF-8 prompts into exact token IDs and byte offsets faster.
Parse interleaved model output into text, reasoning and tool-call events faster.
Serve decode and prefill projections from changing packed weights faster.
Choose any two models. Each row places their speedups for one task on a shared scale, so you can see where one pulls ahead and where they finish level.
Evaluate your model