CASE STUDY / Multi-step order resolution
From 15.8% to 98.7% task success.
Isoquant identified an agent configuration that completed 497 more order workflows correctly, preserving every baseline success.
The task: resolve the order, not just answer the question.
These agents must retrieve case details, read every page of line items, apply policy, make refund or replacement decisions, update records and close the case. Evaluation checks the resulting business state and tool effects, not just the final answer.
The original GPT-5.4 mini configuration completed 95 of 600 workflows. Isoquant explored different model and instruction configurations against the same executable tasks.
A dramatic quality gain from the right configuration.
A public Qwen3.6-35B-A3B configuration with improved execution instructions completed 592 of 600 workflows. It fixed 497 failures and retained all 95 original successes, with eight failures remaining.
No specialist weights were trained. The improvement came from model selection and an execution strategy suited to the workflow: checking sources, following policy and completing the required actions.
Make the tradeoff explicit. Choose the outcome that matters.
This featured result is the high-quality alternative discovered during a cost-focused search. Its 98.7% success rate came with 9% higher estimated inference cost: $9.37 rather than $8.60 per 1,000 workflows. Mean completion rose from 6.07 to 15.28 seconds.
The saved cost-first recommendation was a different configuration: GPT-5.4 mini with revised instructions, at 21.7% success and 30.3% lower cost. Isoquant retained both options so the team could choose the balance of quality, cost and latency.
By the numbers.
| Metric | Baseline | With Isoquant |
|---|---|---|
| Task success | 15.8% | 98.7% |
| Correct / total | 95 / 600 | 592 / 600 |
| Inference cost / 1,000 tasks | $8.60 | $9.37 |
| Mean completion latency | 6.07 s | 15.28 s |
More than one way to improve.
The saved cost-first recommendation is shown separately from the featured quality alternative.
GPT-5.4 mini + Isoquant optimization, cost-first recommendation
- Task success
- 21.7% (130 / 600)
- Cost / 1,000 tasks
- $5.99
- Mean latency
- 6.00 s