Data · Arms
107 configurations specified.
Every candidate change found in the research was written up as an arm before anything ran: a hypothesis, a control, a treatment, a metric, a predicted effect and a falsifier. They were ranked by what a resolved result would settle, the chance of resolving it, and its cost.
- Measured: 2: the forced skill-evaluation hook on its own, in two versions. The full setup, which bundles many of the others, was measured as a whole and is not in this list.
- Specified, not run: 73. Most needed a task bank on which the control fails often enough to show a gain, and that bank could not be built.
- Dropped: 27, with the review campaign, when the review bank proved too easy and the question moved to building software.
- Set aside or removed: 5.
Stages: AMB ambient context · RIG rigging the harness · CON contract · PLN plan · IMP implement · REV review · RTN routines · BEN benchmark instruments.
| ID | Arm | Stage | Task bank | State |
|---|---|---|---|---|
| 7 | Forced skill evaluation (UserPromptSubmit hook) vs native discovery | AMB | TB-A v1 (Spence's 41 positive + 5 negative prompts, harvested; Svelte skills as the … | measured |
| 7b | The forcing hook in the build's wording, on the build's own skills (TB-A v2) | AMB | TB-A v2 (frozen 2026-10-10: 63 positives on the build's seven model-invocable skills, 14 … | measured |
| 1 | A1 − A0: what the user-scope rationale CLAUDE.md buys | AMB | TB-B | specified, not run |
| 2 | Scaffolding necessity by model tier (bitter-lessoned vs artisanal) | AMB | TB-B; phuryn's battery shape (floor tasks + planted-key tasks) for the floor items | specified, not run |
| 3a | Current obra/superpowers as a system (the premise falsifier) | AMB | TB-B (C2 also runs TB-R 100 × 1 as a secondary) | specified, not run |
| 4 | Before/after shrink exemplars as CLAUDE.md-audit calibration | AMB | not stated | specified, not run |
| 5a | Project glossary of terms of art | AMB | TB-B items with domain terms | specified, not run |
| 5b | The preference block: the four missing daily-lane instructions | AMB | TB-B (multi-part slices) | specified, not run |
| 5c | Perishable/permanent split with dates | AMB | TB-B | specified, not run |
| 6 | Block-at-submit hook (pass-file on git commit) vs the Stop-hook bound | RIG | TB-B | specified, not run |
| 8 | Bram-style worklist gate (PreToolUse: an edit must map to an approved item) | AMB | field trial (a week of our real use), not a TB | specified, not run |
| 9 | jj snapshot-per-edit hook vs git worktree + Stop hook (optional) | RIG | none defined (no bank has bad-edit recovery items) | specified, not run |
| 10 | Contract trio (what would have to be true / falsifiers / base rate) as a human-run checklist | CON | field trial: our own tasks, logged | specified, not run |
| 11 | Pre-mortem skill at Plan | PLN | TB-P | specified, not run |
| 12 | knowledge-work-plugins templates (system-design, write-spec, tech-debt) and a thin Shape-Up design skill | PLN | TB-P | specified, not run |
| 13 | idea-generation "Key Risks: what would make this wrong / next steps" template at Contract | CON | TB-S | specified, not run |
| 14 | Lightweight ADRs with a Chesterton's-fence field vs without | PLN | TB-M (items whose session 2 is tempted to remove a load-bearing constraint) | specified, not run |
| 15a | compound-engineering ce-plan (full) vs native planning | PLN | TB-P 30 × 2; TB-B 20 × 1 downstream | specified, not run |
| 15b | ce-plan with model elevation off | PLN | TB-P | specified, not run |
| 16a | Session opener: "First run the tests" | IMP | TB-B | specified, not run |
| 16b | Session opener / in-loop instruction: red/green TDD | IMP | TB-B (TS and Python slices with Stryker / mutmut) | specified, not run |
| 17a | Test authorship: isolated test-writer → implementer vs single agent + fresh-context reviewer | IMP | TB-B | specified, not run |
| 17b | tdd-guard hard gate (optional) | IMP | TB-B | specified, not run |
| 18 | Approved Scenarios for behaviour-heavy slices | IMP | TB-B behaviour-heavy slices | specified, not run |
| 19 | Conformance suite / differential oracle for port and refactor tasks | IMP | TB-B port/refactor items (none planned yet) | specified, not run |
| 20 | gstack /qa browser routine vs the official webapp-testing skill | IMP | TB-B web slices with a seeded UI regression | specified, not run |
| 22 | Elements of Style as a reference for documentation generation | IMP | documentation tasks (none in the five banks) | specified, not run |
| 25c | Fail-closed Stop-gate review vs deliberate review invocation | REV | TB-B (implementation sessions with seeded defects reachable at stop) | specified, not run |
| 25e | Oracle-style escalation on stuck tasks | REV | TB-B items where the agent stalls | specified, not run |
| 27 | Evidence ledger + review freshness (gstack Step 16) vs the Stop hook + /code-review | REV | TB-B | specified, not run |
| 28 | Compound / lessons loop (ce-compound → docs/solutions/) vs dated lessons notes | RTN | field trial: 30 days of our work | specified, not run |
| 29 | Log-mining routine + Cherny-style maintenance routines vs none | RTN | field trial | specified, not run |
| 30 | Ralph loop (as shipped, with an iteration cap) vs /goal + Stop hook for autonomous batch runs | RTN | TB-B long-horizon items (multi-slice backlogs) | specified, not run |
| 31 | Browser-artifact evidence (shot-scraper video or Playwright run) for web features | IMP | TB-B web slices with a seeded UI regression | specified, not run |
| 32 | Trajectory-shape sensor: verify-steps per edit (nilenso) | BEN | every campaign | specified, not run |
| 33 | Rules as encyclopedia articles vs rules as commands | AMB | TB-B | specified, not run |
| 37 | type-design-analyzer's 4-dimension rubric as a Haiku grader on type designs | BEN | hand-labelled type designs (none yet) | specified, not run |
| 40 | Mutation testing as a merge gate (Stryker on TS) | IMP | TB-B TS slices | specified, not run |
| 43 | Coverage-drop: hard gate vs a rationale line | IMP | TB-B items with a spec-over-tests temptation | specified, not run |
| 44 | Approved-fixtures review cost | IMP | field trial: our own reviews | specified, not run |
| 45 | Binding / mock-leak sweep | IMP | Python slices | specified, not run |
| 46 | Maintenance-proxy gate (Shore) | RTN | field: our repos | specified, not run |
| 47 | ADR Y-statement index (auto-drafted at Plan) vs auto-memory only | PLN | TB-M | specified, not run |
| 48 | Index-then-retrieve (3–5 Y-statements) vs the full ADR corpus always on | PLN | TB-M items with ≥ 10 prior ADRs | specified, not run |
| 49 | GCC-style versioned scratchpad vs the ADR index | PLN | TB-M | specified, not run |
| 50 | Visible-vs-held-out test-pass gap as a reward-hacking sensor | BEN | TB-B | specified, not run |
| 52 | Delegation: plan big, execute small (coordinator + cheap workers) vs rigor-matched solo | IMP | TB-B | specified, not run |
| 55 | Lint-feedback phrasing: coaching guide vs bare linter message vs "improve this code" | RIG | LiinaSuoniemi/prompt-vs-metric-eval cases (18 functions; clone, harvested) | specified, not run |
| 56 | Haiku self-preference rerun | BEN | 24 code-shaped prompts (the card's spec) | specified, not run |
| 57b | HumanLayer commands after trimming | PLN | not stated | specified, not run |
| 58 | anti-slop lint preset on vs off (TypeScript lane) | RIG | TB-B TypeScript slices | specified, not run |
| 59 | Phrasing: positive recipe vs prohibition vs no guidance | AMB | the spec's composition fixture (superpowers repo) | specified, not run |
| 60 | Rationalization-table ablation (TDD skill) | AMB | TB-B, or superpowers-evals' own tasks if obtained | specified, not run |
| 61 | Round-trip-loss check (syntaqlite) | BEN | port/refactor items | specified, not run |
| 62 | episodic-memory's "50–100× context saving" | RTN | our session archive | specified, not run |
| 63 | Verification hook set: PostToolUse lint/typecheck + Stop-hook test gate | RIG | TB-B | specified, not run |
| 64 | Hooks + Playwright browser check on web slices | RIG | TB-B web subset | specified, not run |
| 65 | A-sham for superpowers (load vs content) | AMB | TB-B | specified, not run |
| 66 | Verification-nag lines in CLAUDE.md | AMB | TB-B | specified, not run |
| 73 | Native planning + an Explore research subagent | PLN | TB-P | specified, not run |
| 74 | beads vs the ADR index | PLN | TB-M | specified, not run |
| 75 | Bundle confirmation: A1′ + every adopted helper | BEN | TB-B 40 × 2; TB-R 60 × 2 | specified, not run |
| 76 | Cadence: continuous build then evaluator vs evaluator per slice | IMP | TB-B multi-slice items | specified, not run |
| 77 | Held-out scenario evals with satisfaction scores alongside boolean tests | REV | TB-B | specified, not run |
| 78 | Vertical-slice instruction | IMP | TB-B multi-layer slices | specified, not run |
| 79 | Modifier handles (YAGNI, subtractive, make-the-change-easy, WET) | IMP | TB-B | specified, not run |
| 80 | /goal grader: Haiku rubric vs Opus judgment | IMP | TB-B items with an unmet acceptance criterion at first stop | specified, not run |
| 81 | Hook placement: full-suite PostToolUse gate vs Stop-hook only | RIG | TB-B | specified, not run |
| 82 | Circuit-breaker calibration | RIG | TB-B long-horizon items | specified, not run |
| 83 | JSONL-vs-markdown overwrite heuristic | AMB | TB-M two-session items with a state file | specified, not run |
| 84 | Stock vs --bare (the bundle question) | BEN | standard set v1 (12 tasks), run on the Opus 5.5 lane via --task-set-from claude-sonnet-5 | specified, not run |
| 89 | Opus 5.5 on the verbose prompt (CLAUDE_CODE_SIMPLE_SYSTEM_PROMPT=0) | AMB | TB-B | specified, not run |
| 91 | Daily effort: Opus 5.5 at high (this machine's setting) vs medium (D7) | IMP | TB-B | specified, not run |
| 92 | Subagent routing: CLAUDE_CODE_SUBAGENT_MODEL=sonnet vs unset | AMB | not stated (TB-B is where delegation happens) | specified, not run |
| 93 | Claude Mods prompt.section: trimming the verbose lane | RIG | not stated | specified, not run |
| 23 | The build's combined reviewer vs a free-form "review this PR" | REV | TB-R | dropped |
| 24a | Reviewer + question-bank items (EL Example 1; B6 §7c) | REV | TB-R | dropped |
| 24b | Reviewer + inversion / via-negativa stance | REV | TB-R silent-fallback items | dropped |
| 24c | Reviewer + competitive nudge ("whoever finds more real issues") | REV | TB-R | dropped |
| 25a | Thin codex review seam (shimo4228) vs same-family fresh-context review | REV | TB-R | dropped |
| 25b | codex-plugin-cc /codex:adversarial-review vs the shimo4228 seam | REV | TB-R | dropped |
| 25d | pal-mcp consensus + clink vs the seam | REV | TB-R | dropped |
| 26 | Cross-provider peer agreement vs a single cross-model call | REV | TB-R | dropped |
| 34 | /code-review --fix vs /simplify | REV | TB-R (incl. clean diffs) | dropped |
| 35 | Single generalist reviewer vs the five-evidence panel at the ≥ 80 gate | REV | TB-R | dropped |
| 36 | pr-test-analyzer's behavioural-coverage rubric vs our test-gap heuristic | REV | not stated | dropped |
| 38 | Grader-agreement calibration pre-run | BEN | 100–120 labeled traces per dimension, ~50/50 (validate-evaluator:25), split 15/42/43, … | dropped |
| 42 | Cross-model transfer of harness self-optimization (Superpowers-6 style) | RTN | TB-R | dropped |
| 51 | Fresh-context single-pass reviewer vs same-session review | REV | TB-R items replayed as session2 (implement then review) | dropped |
| 53 | Thin-template reviewer (knowledge-work-plugins code-review/SKILL.md, 118 lines, 0 imperatives) | REV | TB-R | dropped |
| 54 | Anchored vs continuous confidence scores | BEN | TB-R | dropped |
| 67 | Official code-review (same-family fresh context, compute-matched) vs no review pass | REV | TB-R | dropped |
| 68 | Persona: generic role ("You are a senior backend engineer reviewing for correctness") | REV | TB-R, 1 run | dropped |
| 69 | Persona: real name only ("Review as Peter Steinberger would") | REV | TB-R, 1 run | dropped |
| 70 | Persona: real name + explicit heuristics | REV | TB-R, 1 run | dropped |
| 71 | Persona: explicit heuristics only, rationale style, no name | REV | TB-R, 1 run | dropped |
| 72 | Persona: heuristics-only through the cross-model seam | REV | TB-R, 1 run | dropped |
| 85 | Find-everything lane: Sonnet 5.5 at max vs Opus 5.5 at max (and Fable 5.1 at max) | REV | TB-R (120 defects / 60 diffs, 15 clean) | dropped |
| 86 | Find-everything effort ladder: Opus 5.5 at max (and xhigh) vs medium | REV | TB-R | dropped |
| 87 | Sonnet 5.5 at max with Anthropic's no-reviewer-subagents paragraph | REV | TB-R | dropped |
| 88 | /code-review --max-findings all vs the default limit | REV | TB-R | dropped |
| 90 | Haiku grader on the lean vs the verbose prompt | BEN | a hand-labelled set | dropped |
| 3b | MOASP one-shot probe | AMB | TB-B, one run per item (exploratory) | set aside |
| 41 | Mandatory TDD system prompt on Fable 5.1 (Kent Beck) vs none | IMP | TB-B | set aside |
| 21 | AC-first overnight pipeline (opslane/verify) | IMP | — | removed |
| 39 | Refactoring-to-reduce-tokens (per-step token deltas) | PLN | our own codebases (a protocol, not a TB) | removed |
| 57a | mattpocock architecture skills (codebase-design + improve-codebase-architecture) vs nothing | PLN | our own codebases (no TB) | removed |