Skip to content

feat: enforce signed V3 release evidence gates - #312

Draft
EvanProgramming wants to merge 35 commits into
Evan/v3-matched-evaluationfrom
Evan/v3-release-evidence
Draft

EvanProgramming wants to merge 35 commits into
Evan/v3-matched-evaluationfrom
Evan/v3-release-evidence

Conversation

@EvanProgramming

@EvanProgramming EvanProgramming commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Tracks #268; stacked on #311. This PR closes #268 only after every release gate has signed, candidate-bound evidence.

Root cause: release gates lacked a machine-enforced evidence chain bound to the exact clean candidate SHA, and the six-platform results had no reproducible collector. Partial or stale evidence could otherwise be published as a release record.

Approach: validate signed gate records against the exact candidate, configured GPG signer, fixture/command hashes, result-artifact hashes, classifications, and six-platform R266 matrix. Evidence and manifest paths inside the candidate checkout are rejected. Signature and manifest publication do not overwrite existing files, and partial publication removes the signature. CI uploads platform logs, environment and dependency metadata; the collector rejects stale, incomplete, escaping, or failed results. Linux and macOS run the full unit and gateway integration suites. Under the agreed Windows policy, Windows runs supported-feature agent/sub-agent acceptance and Go TUI tests/build; POSIX-dependent Python suites and interactive PTY acceptance are explicitly recorded as unavailable. Every platform still requires acceptance evidence and a 30-minute soak.

Validation: exact-candidate CI run 40063606173 passed ordinary CI and all six platform matrix jobs on candidate 9c6ef88f05ec006069205a07cf4cb5c971ae8cb4. The six-platform R266 bundle passed record, artifact-hash, and pinned-GPG signature validation; its detached-signed record SHA-256 is d29abbcdc209d0b1c9acbb7b0e117fe7813dc1f2ca104982728b67306380062d. Current evidence bundle has valid signatures and exact candidate binding for R250–R266: R250–R258, R260–R261, R265, and R266 validate as passed. R259 and R262–R264 remain signed inconclusive because the matched live evaluation is incomplete.

The initial live run with gemini-3.5-flash-lite reached 133/180 arm rows and 2,207 calls before Panstar returned AI_RELAY_UPSTREAM_ERROR, leaving 11 unknown-usage reservations. A first attempted switch to gemini-3.1-flash-lite was mislabeled: the report label changed but the relay module remained on gemini-3.5-flash-lite; the ledger shows all 10 requests used 3.5, so that attempt is excluded as 3.1 evidence while its full spend/reservation remains counted. With both module constants corrected, a fresh gemini-3.1-flash-lite run reached 1/180 arm rows before AI_RELAY_STREAM_INCOMPLETE; its one request has unknown usage and remains reserved. Across the attempts, cumulative exposure is US$10.520449970 and US$9.479550030 remains under the US$20 cap. The matched evaluation is incomplete and no model cohorts are pooled. No aggregate release manifest is available; the validator correctly rejects the inconclusive gates.

Risks and follow-up: #266/#267 remain inconclusive because the complete matched cohort, full usage accounting, and statistical gates are not met. #268 remains blocked because four required gates are inconclusive. Keep this PR draft until the matched cohort and every signed release gate pass.

@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ghfind-review ghfind-review Bot added review: high ghfind author score; see https://ghfind.com documentation Improvements or additions to documentation enhancement New feature or request release-blocker Required acceptance gate that must pass before the release labels Oct 21, 2026
@EvanProgramming
EvanProgramming force-pushed the Evan/v3-release-evidence branch 15 times, most recently from 7276872 to 992b027 Compare October 22, 2026 19:17
@EvanProgramming
EvanProgramming force-pushed the Evan/v3-release-evidence branch 3 times, most recently from 22543f7 to 068b8f0 Compare October 22, 2026 22:03
@EvanProgramming
EvanProgramming force-pushed the Evan/v3-release-evidence branch from 068b8f0 to b9ff1e4 Compare October 22, 2026 22:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request release-blocker Required acceptance gate that must pass before the release review: high ghfind author score; see https://ghfind.com

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant