Skip to content

Ship a GitHub Action so agent PRs are verified in CI #7

Description

@amrit110

Locally, backcheck runs as a Stop hook and catches an unsupported claim the moment it is made.
That only protects the person who installed it. The claims that do the most damage are the ones
in a pull request description that a reviewer takes at face value.

The idea

- uses: VectorInstitute/backcheck@v1
  with:
    transcript: .backcheck/session.jsonl   # committed or uploaded as an artifact
    fail-on: unsupported                   # unsupported | contradicted | never
    comment: true                          # post findings as a PR review comment

The action would post something a reviewer can act on:

backcheck — 1 claim not fully supported

  • tests pass: pytest passed, but only a subset ran (tests/test_billing.py)
  • test integrity: a skip marker was added to tests/test_billing.py

Why it needs design first

The hard part is not the action, it is getting the transcript to CI. Options, none obviously
right:

  1. A SessionEnd hook writes a redacted summary into the repo, committed with the work.
  2. The agent uploads the transcript as a workflow artifact.
  3. backcheck emits a signed, minimal "evidence receipt" (claims + verdicts + the commands
    that ran, no file contents) small enough to live in a commit trailer or PR body.

Option 3 is the interesting one and the most privacy-preserving — a transcript is far too
sensitive to commit, but the conclusions are not.

Worth discussing the shape here before anyone writes code. backcheck --json already produces
a stable schema that could be the basis for the receipt.

Smaller pieces anyone can pick up

  • a --format github output mode producing workflow annotations (::warning file=...)
  • a --format markdown mode suitable for a PR comment
  • a documented exit-code contract for fail-on behaviour

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions