The runner, the models, and the faults that mattered.

The runner

A purpose-built runner drives Claude Code headless (claude -p with stream-json output) against frozen task banks. For each run it builds an isolated configuration directory from the arm's definition, provisions a workspace from the task's fixture, records the full transcript, hook events and session log, scores the final workspace on a fresh copy, and checkpoints so that an interrupted campaign resumes where it stopped. It pins the CLI binary, refuses to resume a campaign whose configuration changed, and stops itself at a share of the usage window. It has 356 automated tests.

Models outside the agent

Labels, grades and drafted tasks came from models of other families than the agent under test; harvested tasks come from real commits. Claude appears only as the agent, in the recall probe that checks it has not memorised a task, inside one review metric, and as the difficulty classifier in one probe. The table shows the final assignment: each model called through its own maker's API, every non-GPT model at its highest reasoning setting. Earlier, GPT-6.1-sol and Gemini 3.1 Pro labelled review findings, and GPT-6.1-sol wrote the first build slices and the activation prompts.

Measurement faults found during the study

The faults in our own instrument that mattered most, found while the study ran, with how each surfaced. About half would have changed a reported number had they gone unnoticed. Anyone running a similar comparison is likely to meet the same kinds of fault.