Artificial Intelligence1 min read

How the Claude Code team thinks about verification

By · Published by Everything Blog

In short

Relying on agents to independently verify their code with proof enables software engineering teams to scale their development throughput far beyond single-operator limits. To validate system integrity, the team utilizes property tests that generate thousands of random inputs alongside end-to-end and differential testing suites. Internal testing revealed that introducing an independent grader to evaluate outputs against a rubric improved structured file generation success by up to 10 points. By feeding testing insights back into the environment through hooks and updated rules, teams can ensure agents avoid repeating past mistakes.

Key points

  • Scaling agent operations to dozens or hundreds running in parallel requires automated loo…: Scaling agent operations to dozens or hundreds running in parallel requires automated loops where agents verify their own work with evidence.
  • Flaky and slow tests degrade model performance by encouraging repeated runs instead of ac…: Flaky and slow tests degrade model performance by encouraging repeated runs instead of actual fixes and reducing testing frequency.
  • Engineering teams can permanently eliminate recurring errors by updating rules or establi…: Engineering teams can permanently eliminate recurring errors by updating rules or establishing hooks based on failed verification runs.

Sources checked by our analysis pipeline: 1

How end-to-end, property and differential tests let coding agents verify their work with evidence, and what the Claude Code team does to keep its verifiers honest.

On the Claude Code team, the vast majority of our code (~90%) is written and reviewed by Claude. At this scale, how we equip our agents to verify their work with evidence becomes extremely important.

Verification loops are the building block that lets you delegate more work to Claude. Without them, you can really only run one or two Claudes at a time, because you have to check everything yourself. With them, you can run dozens or hundreds in parallel.

That’s because giving agents a way to verify their output is an effective way to improve quality. In internal testing around structured file generation, having a separate grader check Claude’s work against a rubric improved task success by up to 10 points over a standard prompting loop, with the largest gains on the hardest problems.

What was once expensive to verify (tests, benchmarks, specs, real end-to-end runs) is now cheap to request from an…

Artificial Intelligence