Skip to content

docs(bench): re-run all three engines on the current base - #243

Merged
FBumann merged 1 commit into
polars-enginefrom
bench/refresh-ladder
Jul 28, 2026
Merged

docs(bench): re-run all three engines on the current base#243
FBumann merged 1 commit into
polars-enginefrom
bench/refresh-ladder

Conversation

@FBumann

@FBumann FBumann commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Every number in docs/benchmarks.md predated #233, #234 and #238, and was measured against linopy 0.9.0 rather than the v1-semantics build this branch now pins. Re-taken in one run: five cases, two sinks, three arms, best of three, plus the marginal-cost pass. 408 timings and 72 loop records, no failures.

Ahead on both axes on every case, through the hand-off

l rung polars linopy duckdb wall peak
dispatch 10M 0.68 s 1.92 s 2.75 s 0.35x 0.95x
nodal 3M 0.34 s 1.07 s 1.78 s 0.32x 0.84x
sector 1M 0.31 s 1.23 s 0.25x 0.32x
transport 9.8M 1.12 s 2.49 s 3.59 s 0.45x 0.76x
profiled 12M 2.15 s 2.49 s 6.34 s 0.86x 0.88x

profiled was the one exception until #238 stopped the duplicate-coordinate check grouping 12M rows to answer a yes/no question.

The LP file stays the weaker route — 0.71x to 1.35x on wall, transport 1.69x on peak — and structurally so: most of an LP write is float-to-text, which neither lane avoids, so that ratio compresses toward 1.00 however fast the build gets.

Against duckdb: 2.2-5.2x faster on every case and both sinks, and duckdb 1.2-2.9x lighter.

The parity gate now proves something

It agrees across all three arms at 0.0e+00 on every shared case — which is also what shows this branch's v1 semantics (#234) and main's (#239) build the same model, rather than each merely claiming to.

Provenance

Names a commit per arm rather than a version, because two of the three have no meaningful version string: the duckdb arm is a checkout, and the polars arm is an editable install that reports its sync point rather than the tree that ran.

arm engine commit
farkas polars engine, polars 1.43.0 45902ec
linopy 0.8.0.post1.dev140+g346943317 (PyPSA/linopy#717) shim at the same commit
duckdb duckdb 1.5.5 4a13d38 on main

Also corrects the shim-overhead claim: it is a constant ~2.3 ms (1.26x at xs, 1.15x at m), not the "~15%" this file carried from one measurement at one rung.

417 passed / 1 xfailed, ruff and pyrefly clean.

🤖 Generated with Claude Code

Every number in this file predated #233, #234 and #238, and was measured
against linopy 0.9.0 rather than the v1-semantics build this branch now pins.
Re-taken in one run: five cases, two sinks, three arms, best of three, plus
the marginal-cost pass. 408 timings and 72 loop records, no failures.

**Ahead on both axes on every case through the hand-off**, which is the sink
most callers use: wall 0.35x, 0.32x, 0.25x, 0.45x, 0.86x and peak 0.95x,
0.84x, 0.32x, 0.76x, 0.88x. `profiled` was the one exception until #238
stopped the duplicate-coordinate check grouping 12M rows to answer a yes/no
question; it is a win now like the rest.

The LP file is the weaker route and stays that way — 0.71x to 1.35x on wall,
and `transport` 1.61x on peak — because most of an LP write is float-to-text,
work neither lane avoids.

Against duckdb: 2.2-5.2x faster on every case and both sinks, and duckdb
1.2-2.9x lighter. duckdb is slower than the eager lane everywhere here.

The parity gate agrees across all three arms at 0.0e+00 on every shared case,
which is also what proves this branch's v1 semantics (#234) and `main`'s
(#239) build the same model rather than merely claiming to.

Provenance now names a commit per arm rather than a version, since two of the
three arms have no meaningful version string — the duckdb arm is a checkout
and the polars arm is an editable install that reports its sync point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 23a0ced6-8daa-4bc6-abdd-1baa5b3a9774

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch bench/refresh-ladder

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@FBumann
FBumann merged commit 1913fd9 into polars-engine Jul 28, 2026
3 checks passed
FBumann added a commit that referenced this pull request Jul 28, 2026
Full three-arm ladder on `polars-engine@3fb95d6`, which now carries #238, #243
and #245. Five cases, two sinks, three arms, best of three, plus the
marginal-cost pass. 408 timings and 72 loop records, no failures, parity gate
exact across all three engines.

Two numbers moved, both for a stated reason:

  transport LP peak   1.69x -> 1.19x   (#245, chunked constraint emission)
  dispatch  LP peak   1.01x -> 0.95x   (same)

To a loaded solver we are ahead on both axes on all five cases: wall 0.35x,
0.32x, 0.25x, 0.44x, 0.81x and peak 0.95x, 0.84x, 0.33x, 0.77x, 0.88x.

On the LP route only two numbers are still against us — `profiled` 1.34x on
wall and `transport` 1.19x on peak — and the section now says why that route
is structurally weaker: most of an LP write is float-to-text, which neither
lane avoids, so the ratio compresses toward 1.00 however fast the build gets.

Against duckdb: 2.2-5.3x faster on every case and both sinks, duckdb 1.2-2.8x
lighter.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@FBumann
FBumann deleted the bench/refresh-ladder branch July 31, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant