Context
While writing the cost engineering article, I wanted to test whether the intuitions I'd formed from real usage held up under structured evaluation. The thesis: model cost and model performance are less correlated than most engineers assume.
Experimental Design
- Models: DeepSeek Flash, GLM 5.2, MiMo V2.5, MiniMax M3, DeepSeek V4 Pro
- Task types: Build (Calculator), Bug Fix (Budget Tracker), Refactor (Report Generator)
- Total evaluations: 15 (5 models × 3 tasks), all succeeded
- Judge: DeepSeek V4 Flash (not a contestant)
- Metrics: Correctness (pytest), Prompt Adherence (AST checklist), Code Quality (judge score 1–5 across 9 dimensions)
- Total experiment cost: $0.39 — 1.2% of my OpenCode spend that week
Results
Composite scores (avg across 9 rubric dimensions, 0–5 scale):
| Model | Calculator | Bug Fix | Refactor | Overall |
|---|---|---|---|---|
| DeepSeek Flash | 4.22 | 4.44 | 3.67 | 4.11 |
| MiniMax M3 | 4.67 | 2.67 | 3.89 | 3.74 |
| GLM 5.2 | 3.56 | 4.33 | 2.33 | 3.41 |
| DeepSeek V4 Pro | 4.00 | 4.11 | 1.89 | 3.33 |
| MiMo V2.5 | 4.56 | 1.78 | 3.33 | 3.22 |
Cost vs quality:
| Model | Cost/Run | Cost per Quality Point |
|---|---|---|
| MiMo V2.5 | $0.007 | $0.002 |
| DeepSeek Flash | $0.008 | $0.002 |
| MiniMax M3 | $0.025 | $0.007 |
| DeepSeek V4 Pro | $0.042 | $0.013 |
| GLM 5.2 | $0.047 | $0.014 |
Key Findings
- DeepSeek Flash won overall — most consistent across all three tasks and cheapest cost per quality point. Reliability beat raw capability.
- GLM 5.2 was the most expensive and scored lowest overall — it made up 65.6% of my real-world OpenCode spend with no performance advantage to show for it.
- MiMo V2.5 ($0.007/run) outperformed GLM 5.2 ($0.047/run) on the Calculator task — the cheapest model beat the most expensive one at nearly 7× the price difference.
- Refactoring was the hardest task for everyone — all models averaged lower on refactor (3.02) vs build (4.20) and bug fix (3.47). The task that most resembles real maintenance work is where model capability gaps matter least.
- High variance is a risk — MiniMax M3 scored 4.67 on Calculator and 2.67 on Bug Fix. A model that's excellent on one task type can be unreliable on another.
Takeaway
The expensive model is not the safe choice. Consistency across task types matters more than peak performance on a single one — and the cheapest model per quality point was also the most predictable. That's the combination worth optimising for.