Results & accuracy
55.8% fewer total model tokens in our current paired test.
| Condition | Total tokens | Correct answers |
|---|---|---|
| Full context | 194,420 | 15 / 36 |
| Optimise hosted API, guarded 5× target | 85,935 | 19 / 36 |
The count includes all model turns, three retrievals, four full-source fallbacks and one validation retry. Five baseline failures improved; one previously correct answer failed exact matching (“three measures” instead of “three”).
What we tested
72 runs across 36 previously evaluated public-document questions on 21 October 2026. Same local Qwen3-4B-Instruct-2507 Q4_K_M model, prompts, seed 41 and temperature 0.2. The agent chose its reads, retrievals and answers. No execution errors in the completed run.
What this means for you
This is one model and seed, using previously evaluated documents. Accuracy was 19/36; it is not a guarantee of complete evidence or correct answers. Token reduction is not a measured invoice saving. Other models, caching and retries can change the result.
Run your own comparison
- Choose unseen questions with known answers.
- Run full-context and Optimise versions with the same model and prompts.
- Count all input, output and retry tokens.
- Compare correct answers, especially previously correct answers that become wrong.
- Apply your provider’s actual prices and caching discounts.
Share non-sensitive findings through feedback. Do not submit customer documents or secrets.