Data · Task banks
The task banks.
Three kinds of task were built. TB-A (skill activation): prompts that call for a specific skill, and near-misses that call for none. TB-R (review): real commits that introduced a bug, scored by whether the review finds it. TB-B (build): features re-implemented from their pull-request description inside the repository's parent commit, and greenfield slices from a written spec, scored by hidden tests.
Each bank is frozen as an immutable version with a checksum, so that a result always names the exact tasks it was measured on.
| Bank | Version | Tasks | Checksum | Frozen | |
|---|---|---|---|---|---|
| TB-A | v1 | 46 | 90bedbabdcf7 | 2026-10-05 | |
| TB-A | v2 | 77 | 21f9afbdc6ec | 2026-10-10 | |
| TB-R | v1 | 69 | 9f27c2f16be8 | 2026-10-05 | retired |
| TB-R | v2 | 69 | 6834e3ad75af | 2026-10-05 | retired |
| TB-R | v3 | 55 | 7a7fe2b20aaf | 2026-10-08 | |
| TB-R | v4 | 55 | 4f0111805448 | 2026-10-08 | |
| TB-R | v5 | 55 | f1d14dfacec6 | 2026-10-08 | |
| TB-R | v6 | 55 | 0563c21a6b39 | 2026-10-08 | |
| TB-B | v1 | 40 | c583b39a4928 | 2026-10-09 | |
| TB-B | v2 | 24 | 848be42af1ee | 2026-10-09 | |
| TB-B | v3 | 40 | f8f406ca72e5 | 2026-10-09 | |
| TB-B | v4 | 38 | b9d393cfb5e7 | 2026-10-09 | |
| TB-B | v5 | 40 | 669ee6241db3 | 2026-10-09 | |
| TB-B | v6 | 39 | 937df8f98020 | 2026-10-09 | |
| TB-B | v7 | 26 | 7383d7eb9ddf | 2026-10-11 | |
| TB-B | v8 | 8 | e781741c5a0d | 2026-10-11 | |
| TB-B | v9 | 69 | 7e3d3df0fd15 | 2026-10-12 |
The 69 tasks of the harm check
Scores are the mean share of hidden tests passed, stock and with the setup.
| Task | Source | Stack | Kind | Hidden tests | Stock | Setup |
|---|---|---|---|---|---|---|
Lightning-AI__pytorch-lightning-eea74433ce | Lightning-AI/pytorch-lightning | py | lib | 16 | 0.75 | 0.69 |
PyCQA__pyflakes-d539458de5 | PyCQA/pyflakes | py | cli | 10 | 1.00 | 1.00 |
aio-libs__yarl-09e500de4b | aio-libs/yarl | py | lib | 12 | 1.00 | 1.00 |
alecthomas__kong-4e565e8d77 | alecthomas/kong | go | cli | 1 | 1.00 | 1.00 |
gdamore__tcell-d57374f097 | gdamore/tcell | go | lib | 2 | 1.00 | 1.00 |
go-resty__resty-29010be3b2 | go-resty/resty | go | lib | 1 | 1.00 | 1.00 |
golang-jwt__jwt-73c870b18e | golang-jwt/jwt | go | lib | 1 | 1.00 | 1.00 |
h3js__rou3-01bb393e2b | h3js/rou3 | ts | lib | 4 | 1.00 | 1.00 |
h3js__rou3-79480e642f | h3js/rou3 | ts | web | 1 | 1.00 | 1.00 |
h3js__rou3-835332aca0 | h3js/rou3 | ts | web | 12 | 1.00 | 1.00 |
h3js__srvx-555a7932ae | h3js/srvx | ts | web | 3 | 1.00 | 1.00 |
h3js__srvx-6b3b2910ae | h3js/srvx | ts | web | 46 | 0.89 | 0.89 |
h3js__srvx-e0934021d4 | h3js/srvx | ts | web | 18 | 1.00 | 1.00 |
hgrecco__pint-bbc6b4c600 | hgrecco/pint | py | lib | 1 | 0.00 | 0.00 |
joke2k__faker-4ca9116ec2 | joke2k/faker | py | lib | 2 | 1.00 | 1.00 |
junegunn__fzf-f779e6a4df | junegunn/fzf | go | lib | 2 | 1.00 | 1.00 |
labstack__echo-abbb92edbb | labstack/echo | go | web | 2 | 1.00 | 1.00 |
mark3labs__mcp-go-e9a7badcce | mark3labs/mcp-go | go | web | 7 | 1.00 | 0.50 |
ml-explore__mlx-lm-e9308d7b08 | ml-explore/mlx-lm | py | lib | 5 | 1.00 | 1.00 |
modelcontextprotocol__go-sdk-3f43bcf012 | modelcontextprotocol/go-sdk | go | lib | 1 | 1.00 | 1.00 |
mswjs__interceptors-4c01ff252e | mswjs/interceptors | ts | lib | 6 | 1.00 | 1.00 |
mswjs__interceptors-7f1b37ca80 | mswjs/interceptors | ts | lib | 6 | 0.67 | 0.67 |
openai__openai-go-7e4e1b1912 | openai/openai-go | go | app | 2 | 0.50 | 0.50 |
openai__openai-go-a29869f512 | openai/openai-go | go | app | 10 | 0.60 | 0.60 |
pypa__packaging-23669b1b69 | pypa/packaging | py | lib | 2 | 0.50 | 0.50 |
pypa__packaging-58c6cd700b | pypa/packaging | py | lib | 54 | 0.89 | 0.83 |
python-attrs__attrs-5aa76a4450 | python-attrs/attrs | py | lib | 3 | 1.00 | 1.00 |
slice-go-cli-request-latency-report | greenfield slice | go | cli | 13 | 1.00 | 1.00 |
slice-go-cli-sample-crate-planner | greenfield slice | go | cli | 40 | 1.00 | 1.00 |
slice-go-web-gallery-cue-catalog | greenfield slice | go | web | 35 | 1.00 | 1.00 |
slice-go-web-signed-event-gateway | greenfield slice | go | web | 12 | 1.00 | 1.00 |
slice-go-worker-digest-packet-worker | greenfield slice | go | app | 36 | 1.00 | 1.00 |
slice-go-worker-revision-rebuild-worker | greenfield slice | go | app | 11 | 1.00 | 1.00 |
slice-py-cli-inventory-journal-fold | greenfield slice | py | cli | 12 | 1.00 | 1.00 |
slice-py-cli-prefix-policy-forge | greenfield slice | py | cli | 40 | 0.95 | 0.95 |
slice-py-web-buffered-dock-board | greenfield slice | py | web | 40 | 1.00 | 1.00 |
slice-py-web-gallery-caption-press | greenfield slice | py | web | 40 | 0.97 | 0.97 |
slice-py-web-harbor-berth-ledger | greenfield slice | py | web | 38 | 1.00 | 1.00 |
slice-py-web-incident-timeline-pages | greenfield slice | py | web | 12 | 1.00 | 1.00 |
slice-py-worker-checkpointed-maintenance-worker | greenfield slice | py | app | 12 | 1.00 | 0.92 |
slice-py-worker-porchlight-courier | greenfield slice | py | app | 49 | 1.00 | 1.00 |
slice-ts-cli-asset-revision-scout | greenfield slice | ts | cli | 38 | 1.00 | 0.95 |
slice-ts-cli-catalog-placeholder-audit | greenfield slice | ts | cli | 12 | 1.00 | 1.00 |
slice-ts-cli-deploy-wave-planner | greenfield slice | ts | cli | 12 | 1.00 | 1.00 |
slice-ts-cli-env-overlay-plan | greenfield slice | ts | cli | 12 | 1.00 | 1.00 |
slice-ts-cli-redirect-chain-auditor | greenfield slice | ts | cli | 10 | 1.00 | 1.00 |
slice-ts-cli-support-log-scrubber | greenfield slice | ts | cli | 12 | 1.00 | 1.00 |
slice-ts-web-apiary-inspection-api | greenfield slice | ts | web | 50 | 1.00 | 1.00 |
slice-ts-web-artifact-manifest-registry | greenfield slice | ts | web | 13 | 1.00 | 1.00 |
slice-ts-web-atomic-lane-layout | greenfield slice | ts | web | 12 | 1.00 | 1.00 |
slice-ts-web-durable-dispatch-desk | greenfield slice | ts | web | 40 | 1.00 | 1.00 |
slice-ts-web-leased-review-queue | greenfield slice | ts | web | 12 | 1.00 | 1.00 |
slice-ts-web-metered-usage-ledger | greenfield slice | ts | web | 42 | 1.00 | 1.00 |
slice-ts-web-restoration-queue-api | greenfield slice | ts | web | 35 | 1.00 | 1.00 |
slice-ts-web-transient-note-vault | greenfield slice | ts | web | 41 | 1.00 | 1.00 |
slice-ts-web-weekly-availability-api | greenfield slice | ts | web | 12 | 1.00 | 1.00 |
slice-ts-worker-attachment-scan-worker | greenfield slice | ts | app | 12 | 1.00 | 1.00 |
slice-ts-worker-caption-proof-conveyor | greenfield slice | ts | app | 40 | 1.00 | 1.00 |
slice-ts-worker-mirror-outbox-worker | greenfield slice | ts | app | 12 | 1.00 | 1.00 |
slice-ts-worker-receipt-gated-workflows | greenfield slice | ts | app | 12 | 1.00 | 1.00 |
slice-ts-worker-tenant-export-dispatcher | greenfield slice | ts | app | 13 | 1.00 | 1.00 |
stanfordnlp__dspy-33aaa19e03 | stanfordnlp/dspy | py | lib | 8 | 0.88 | 0.81 |
stanfordnlp__dspy-d58c3007eb | stanfordnlp/dspy | py | lib | 12 | 0.33 | 0.33 |
stanfordnlp__dspy-e8ec009c6b | stanfordnlp/dspy | py | lib | 31 | 0.24 | 0.34 |
tobymao__sqlglot-da9a9c9664 | tobymao/sqlglot | py | lib | 18 | 0.67 | 0.64 |
tobymao__sqlglot-e3ed74860f | tobymao/sqlglot | py | lib | 2 | 0.50 | 0.50 |
tox-dev__filelock-126cab7456 | tox-dev/filelock | py | lib | 31 | 0.74 | 0.74 |
tox-dev__platformdirs-dfaeabf921 | tox-dev/platformdirs | py | lib | 7 | 0.86 | 0.86 |
yuin__goldmark-ee62f0eaf9 | yuin/goldmark | go | lib | 2 | 0.00 | 0.50 |