Today backcheck can tell you the suite passed and that nobody disabled a test to get there.
It cannot tell you the passing suite actually exercises the code that changed — the most
common way a green bar means nothing.
An agent edits src/billing.py, runs the suite, everything passes. If no test touches the new
branch, the green is real and meaningless.
What would close the gap
Changed-line coverage. The agent edited these files; the coverage report says these lines
ran. Were any of the changed lines covered?
✗ unsupported tests pass
`pytest` passed, but none of the 14 lines changed in src/billing.py were executed
by the suite (coverage.xml)
This is tractable because coverage output is a file on disk with a stable format, and
backcheck already knows exactly which lines were edited — the Edit payloads in the
transcript carry oldString/newString and a structuredPatch.
Mutation signal (harder, later). A test that runs a line but asserts nothing about it still
passes when the line is broken. Scoped mutation testing on changed lines only is the rigorous
version of "was this green earned", and is
discussed
as the real answer to agents gaming test suites. It is slow and invasive, so it would have to be
opt-in (backcheck --mutate) and probably delegated to mutmut/cargo-mutants/stryker
rather than reimplemented.
Design constraint
Coverage parsing keeps the no-model-calls rule intact and stays deterministic, which is why it
is the right first step. Anything that needs to run code (mutation) must be explicitly opt-in
and must never happen inside the Stop hook — the hook has to stay fast enough to be invisible.
Start with one coverage format end-to-end rather than a general framework.
Today
backcheckcan tell you the suite passed and that nobody disabled a test to get there.It cannot tell you the passing suite actually exercises the code that changed — the most
common way a green bar means nothing.
An agent edits
src/billing.py, runs the suite, everything passes. If no test touches the newbranch, the green is real and meaningless.
What would close the gap
Changed-line coverage. The agent edited these files; the coverage report says these lines
ran. Were any of the changed lines covered?
This is tractable because coverage output is a file on disk with a stable format, and
backcheckalready knows exactly which lines were edited — theEditpayloads in thetranscript carry
oldString/newStringand astructuredPatch.coverage.xml/.coverage(Python)lcov.info(JS, Rust via tarpaulin/llvm-cov)cobertura.xml(widely emitted)Mutation signal (harder, later). A test that runs a line but asserts nothing about it still
passes when the line is broken. Scoped mutation testing on changed lines only is the rigorous
version of "was this green earned", and is
discussed
as the real answer to agents gaming test suites. It is slow and invasive, so it would have to be
opt-in (
backcheck --mutate) and probably delegated tomutmut/cargo-mutants/strykerrather than reimplemented.
Design constraint
Coverage parsing keeps the no-model-calls rule intact and stays deterministic, which is why it
is the right first step. Anything that needs to run code (mutation) must be explicitly opt-in
and must never happen inside the Stop hook — the hook has to stay fast enough to be invisible.
Start with one coverage format end-to-end rather than a general framework.