Locally, backcheck runs as a Stop hook and catches an unsupported claim the moment it is made.
That only protects the person who installed it. The claims that do the most damage are the ones
in a pull request description that a reviewer takes at face value.
The idea
- uses: VectorInstitute/backcheck@v1
with:
transcript: .backcheck/session.jsonl # committed or uploaded as an artifact
fail-on: unsupported # unsupported | contradicted | never
comment: true # post findings as a PR review comment
The action would post something a reviewer can act on:
backcheck — 1 claim not fully supported
- tests pass:
pytest passed, but only a subset ran (tests/test_billing.py)
- test integrity: a skip marker was added to
tests/test_billing.py
Why it needs design first
The hard part is not the action, it is getting the transcript to CI. Options, none obviously
right:
- A
SessionEnd hook writes a redacted summary into the repo, committed with the work.
- The agent uploads the transcript as a workflow artifact.
backcheck emits a signed, minimal "evidence receipt" (claims + verdicts + the commands
that ran, no file contents) small enough to live in a commit trailer or PR body.
Option 3 is the interesting one and the most privacy-preserving — a transcript is far too
sensitive to commit, but the conclusions are not.
Worth discussing the shape here before anyone writes code. backcheck --json already produces
a stable schema that could be the basis for the receipt.
Smaller pieces anyone can pick up
Locally,
backcheckruns as a Stop hook and catches an unsupported claim the moment it is made.That only protects the person who installed it. The claims that do the most damage are the ones
in a pull request description that a reviewer takes at face value.
The idea
The action would post something a reviewer can act on:
Why it needs design first
The hard part is not the action, it is getting the transcript to CI. Options, none obviously
right:
SessionEndhook writes a redacted summary into the repo, committed with the work.backcheckemits a signed, minimal "evidence receipt" (claims + verdicts + the commandsthat ran, no file contents) small enough to live in a commit trailer or PR body.
Option 3 is the interesting one and the most privacy-preserving — a transcript is far too
sensitive to commit, but the conclusions are not.
Worth discussing the shape here before anyone writes code.
backcheck --jsonalready producesa stable schema that could be the basis for the receipt.
Smaller pieces anyone can pick up
--format githuboutput mode producing workflow annotations (::warning file=...)--format markdownmode suitable for a PR commentfail-onbehaviour