Skip to content

refactor(bench): retire the duckdb arm, and the docs it was holding up - #253

Merged
FBumann merged 1 commit into
mainfrom
docs/lean
Jul 28, 2026
Merged

refactor(bench): retire the duckdb arm, and the docs it was holding up#253
FBumann merged 1 commit into
mainfrom
docs/lean

Conversation

@FBumann

@FBumann FBumann commented Jul 28, 2026

Copy link
Copy Markdown
Owner

duckdb stopped being a dependency when #189 landed. The arm that measured it
was there to answer one question — was replacing it worth doing — and that
question is closed, so what is left is a column nobody can re-run without
keeping a foreign checkout synced indefinitely. The comparison itself survives
in #189 and in git, which is where a decided decision belongs.

Harness. FOREIGN_ARMS, --duckdb-root, --duckdb-limits, the
foreign-interpreter plumbing in _child, and --memory-limit in _run_case
all go. _child is now four lines of subprocess with no notion of running
another checkout's code under another checkout's python. The report drops to
two arms, which takes the ladder tables from fourteen columns to ten.

docs/benchmarks.md: 649 -> 419 lines. It had become the largest file in the
repo — larger than SPEC.md — and most of the excess was argument for the
engine choice rather than measurement of the engine we have. The tables are the
same measurements with the duckdb columns dropped, not a re-run.

What that argument was hiding is worth keeping, and is now stated plainly: at
120M variables the peak ratio is 1.07x. Past a certain size both lanes hold
the same model and the representation stops deciding anything, so the memory
headroom this lane has is a small-and-sparse-model property rather than a
scaling one. The old section reached the opposite impression by reading duckdb's
budgeted column as an engine measurement.

The chart page loses its third and fourth series with the arm. The band
section changes question with it — from "is the ordering between three engines
robust to model shape" to "does the advantage over linopy hold whatever the
model looks like" — which the two-arm data answers directly: clear of 1.0 at
every rung but profiled/l on time, straddling it on peak with a threefold
spread between sector and dispatch. Palette re-validated at two slots
(worst CVD dE 24.7 light / 26.8 dark, no contrast warning).

SPEC.md still justified refusing a stray dim with "under a memory budget
the first quietly builds a bigger model" — there is no budget now, and the
clause is stronger without it.

423 passed / 1 xfailed, ruff and pyrefly clean.

🤖 Generated with Claude Code

duckdb stopped being a dependency when #189 landed. The arm that measured it
was there to answer one question — was replacing it worth doing — and that
question is closed, so what is left is a column nobody can re-run without
keeping a foreign checkout synced indefinitely. The comparison itself survives
in #189 and in git, which is where a decided decision belongs.

**Harness.** `FOREIGN_ARMS`, `--duckdb-root`, `--duckdb-limits`, the
foreign-interpreter plumbing in `_child`, and `--memory-limit` in `_run_case`
all go. `_child` is now four lines of `subprocess` with no notion of running
another checkout's code under another checkout's python. The report drops to
two arms, which takes the ladder tables from fourteen columns to ten.

**docs/benchmarks.md: 649 -> 419 lines.** It had become the largest file in the
repo — larger than SPEC.md — and most of the excess was argument for the
engine choice rather than measurement of the engine we have. The tables are the
same measurements with the duckdb columns dropped, not a re-run.

What that argument was hiding is worth keeping, and is now stated plainly: at
120M variables the peak ratio is **1.07x**. Past a certain size both lanes hold
the same model and the representation stops deciding anything, so the memory
headroom this lane has is a small-and-sparse-model property rather than a
scaling one. The old section reached the opposite impression by reading duckdb's
budgeted column as an engine measurement.

**The chart page** loses its third and fourth series with the arm. The band
section changes question with it — from "is the ordering between three engines
robust to model shape" to "does the advantage over linopy hold whatever the
model looks like" — which the two-arm data answers directly: clear of 1.0 at
every rung but `profiled/l` on time, straddling it on peak with a threefold
spread between `sector` and `dispatch`. Palette re-validated at two slots
(worst CVD dE 24.7 light / 26.8 dark, no contrast warning).

**SPEC.md** still justified refusing a stray dim with "under a memory budget
the first quietly builds a bigger model" — there is no budget now, and the
clause is stronger without it.

423 passed / 1 xfailed, ruff and pyrefly clean.
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@FBumann, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 27 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f79a5eec-4396-47cb-a990-80e613eb5ac1

📥 Commits

Reviewing files that changed from the base of the PR and between de4794d and 6ee4338.

📒 Files selected for processing (8)
  • SPEC.md
  • bench/README.md
  • bench/_run_case.py
  • bench/plot.py
  • bench/report.py
  • bench/run.py
  • docs/benchmarks-scaling.html
  • docs/benchmarks.md
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/lean

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@FBumann
FBumann merged commit 6eaef1e into main Jul 28, 2026
3 checks passed
@FBumann
FBumann deleted the docs/lean branch July 31, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant