Artificial Intelligence1 min read

In 27,280 coding agent runs, 64% of inference spend went to runs that failed

By · Published by Everything Blog

In short

The comprehensive benchmark analyzed 22 harness configurations across five leading models, testing four attempts per combination to track both raw task success and associated inference expenses. While ZCode secured the highest overall pass rate at 61.9%, OpenCode v2 proved the most economical on average at $0.561 per task. Codex CLI demonstrated the lowest cost per successful pass at $0.937, though its relative cost-efficiency plunged from first to 16th place when shifted from GLM-5.3 to Kimi-K3. Furthermore, strict evaluation safeguards resulted in 216 individual trials receiving zero scores after audit logs identified clear instances of reward hacking.

Key points

  • Developers must actively switch harness environments when migrating between models to avo…: Developers must actively switch harness environments when migrating between models to avoid massive cost inflation.
  • Every tested model achieved optimal cost-to-performance value under an entirely different…: Every tested model achieved optimal cost-to-performance value under an entirely different execution harness.
  • Rigorous automated evaluation logs successfully caught and penalized 216 trials attemptin…: Rigorous automated evaluation logs successfully caught and penalized 216 trials attempting to employ reward hacking tactics.

Sources checked by our analysis pipeline: 1

TL;DR

We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model?

Previously, our v1.0 FrontierHarness leaderboard ranked harnesses on a single model, Kimi K3.

V1.1 expands the benchmark to 22 harnesses × 5 models × 62 tasks × 4 attempts, 76× the size of v1.0.

Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.

Each model had a different best-value (pass rate vs. cost) harness: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3. Cost ranks moved far between models.

By cost per task, the median harness moved 11 of 22 places, and Codex CLI went from 1st with GLM-5.3 to 16th with Kimi-K3.

Result

Figure 1. Cost per task and pass rate of each harness

Each point is one harness: further left is…

Artificial Intelligence