CI: interactive PR control panel comment

[joseph-isaacs]

Rationale for this change

Benchmarks are triggered today by adding an action/benchmark* label, which the bot removes once the run starts. That works for a handful of switches, but the combinations (suite × runner × preset × flags) have outgrown what a flat label list can express or display — you cannot see the current configuration anywhere, and every new knob is another label.

This PR proves out an alternative UX: a comment posted unconditionally on every PR that behaves like a form. It is deliberately not wired to any real benchmark — the goal is to validate the interaction model before moving the benchmark plumbing onto it.

Does GitHub actually support this?

Yes, with one primitive: the markdown task-list checkbox is the only clickable control GitHub renders in a comment. Ticking one rewrites the comment body and delivers issue_comment: edited. Everything else is built on top of that:

user ticks a box
      │
      ▼
issue_comment: edited ──► pr-panel.yml (apply)
                              │  parse checkboxes, diff against the embedded state blob
                              │  canonicalize (radios, momentary buttons)
                              ├─► PATCH the comment  ──► panel redraws
                              └─► workflow_dispatch  ──► pr-panel-run.yml
                                                            renders the state back to the PR

The comment body is the entire storage layer. Each control is a tagged task-list line, and a trailing HTML comment records the state as of the last render:

- [x] Random access <!--c:suite.random_access-->
...
<!--vortex-pr-panel:state:{"controls":{...},"rev":3,"v":1}-->

Diffing the checkboxes against that blob identifies which control was clicked, which is what makes controls richer than a bare checkbox expressible:

Control	Behaviour
Toggle	Persistent on/off.
Radio	Pick-one — checking an option unchecks its siblings on redraw; unchecking the selected one restores it, so a group is never empty.
Button	Momentary — a tick dispatches an action and is cleared on redraw, mirroring today's add-label/remove-label pattern.
Section	Grouping; a closed section renders as a folded <details> block.

Properties that fall out of the design:

• Authorization is GitHub's own. Only users with write access can tick a checkbox on someone else's comment, so there is no separate permission check.
• No feedback loop. The redraw is written with GITHUB_TOKEN, and edits made with that token do not trigger workflows.
• Races are serialized. A per-PR concurrency group that never cancels, plus re-reading the body from the API instead of trusting the (possibly stale) event payload.
• Hand edits self-heal, because the comment is redrawn from canonical state.
• Redraws are skipped when they would be a no-op — ticking a toggle already leaves the comment correct, so only radios and buttons force a write-back.

What changes are included in this PR?

• .github/scripts/pr_panel/ — the panel itself. spec.py is a declarative control tree (adding a knob is a few lines there and nothing else); panel.py holds render, parse, and click semantics; __main__.py is the CLI used by the workflows, plus a demo subcommand that renders the panel and simulates clicks locally.
• .github/workflows/pr-panel.yml — posts the panel on PR open/reopen, and applies clicks on issue_comment: edited.
• .github/workflows/pr-panel-run.yml — the dispatched downstream workflow. It renders the received state as a table into the job summary and a PR comment, which is what demonstrates that the settings cross the workflow boundary intact. This is where a benchmark run would eventually be launched from.
• .github/scripts/upsert-comment.sh — marker-based find-or-create so repeated runs edit one comment instead of piling up.
• ci.yml — a pr-panel job running the 24 unit tests (pure stdlib, well under a second).
• .github/scripts/pyproject.toml — pins packages = ["fuzz_report"], so that adding a sibling package here does not break pip install -e .github/scripts via setuptools flat-layout auto-discovery.

Two things to validate on a live PR

Both are load-bearing assumptions that unit tests cannot cover, and neither takes effect until this is merged, because issue_comment and pull_request_target workflows always run from the default branch:

1. That a checkbox toggle inside a collapsed <details> block stays clickable. The Runner and Options sections are deliberately collapsed to exercise this; if it does not hold, the fallback is rendering every section as a plain open heading, which is a one-line change in spec.py.
2. End-to-end latency of the click → redraw loop, which determines whether the interaction feels responsive enough to keep.

What APIs are changed? Are there any user-facing changes?

No library or public API changes — this is CI tooling only. It adds no behaviour to existing workflows: the current action/benchmark* labels and bench-dispatch.yml are untouched, and the panel dispatches nothing but its own no-op reporting workflow.

The user-facing change is a new bot comment on every PR once merged. .github/scripts/pr_panel/README.md documents the design and how to iterate locally.

Checks run

• uv run --no-project --with pytest pytest .github/scripts/tests/test_pr_panel.py — 24 passed
• yamllint --strict -c .yamllint.yaml .github/ — clean
• ruff check --config .github/scripts/pyproject.toml pr_panel tests — clean
• reuse lint — compliant
• bash -n .github/scripts/upsert-comment.sh
• Verified uv pip install --no-deps -e .github/scripts still resolves fuzz_report after the setuptools pin

Not run: Rust checks (cargo fmt/clippy/nextest), as no Rust, feature flags, or generated files are touched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T8xYqGxvVBTBjXbN8qa6Vw

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will improve performance by 26.88%

⚡ 2 improved benchmarks
✅ 1840 untouched benchmarks
⏩ 55 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	take_map[(0.1, 1.0)]	2.2 ms	1.7 ms	+33.77%
⚡	Simulation	take_map[(0.1, 0.5)]	1,182.5 µs	982.5 µs	+20.35%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/github-actions-bot-comments-qkpasq (e17bced) with develop (3226d32)}

Footnotes

1. 55 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

Benchmark Control Panel

Configure a benchmark run by clicking the checkboxes below, then press Run to dispatch it. Your click is applied by a workflow, so the panel takes a few seconds to redraw.

Important

Preview window closed. Push to this branch to reopen it.

Note

Nothing dispatched yet.

Suites

Which benchmark suites to include in the run.

• Random access
• Compression
• SQL (TPC-H / Clickbench)
• GPU compression

Runner

Where the run executes.

Machine — pick one:

• c6id.metal (bare metal, default)
• c7i.8xlarge
• g5.xlarge (GPU)

Options

Extra knobs applied to every selected suite.

Matrix preset — pick one:

• pr — quick subset

• pr-full — everything

• Capture a continuous profile

• Enable unstable encodings

Actions

Tick one of these to press it. GitHub strips <button> from comments, so a checkbox is the only clickable control available — these clear themselves once the click is handled.

•  ▶️  RUN BENCHMARKS — with the settings above
•  ♻️  RESET — restore the default settings

_{Checkboxes are editable by anyone with write access. Every click is applied by the PR Control Panel workflow, which redraws this comment; the buttons clear themselves once handled.}

[github-actions]

Panel run report

Dispatched by: run

Rendered by this workflow run.

Control	Value
Suites / suite.random_access	✅ on
Suites / suite.compression	✅ on
Suites / suite.sql	⬜ off
Suites / suite.gpu_compression	⬜ off
Runner / runner.machine	c6id.metal
Options / options.preset	pr
Options / options.profile	⬜ off
Options / options.unstable_encodings	⬜ off

_{Panel revision 3.}

[AdamGS]

where's the polarsignals comment?

[joseph-isaacs]

no idea what do you think of the UX?