docs(bench): re-run all three engines on the current base - #243
Merged
Conversation
Every number in this file predated #233, #234 and #238, and was measured against linopy 0.9.0 rather than the v1-semantics build this branch now pins. Re-taken in one run: five cases, two sinks, three arms, best of three, plus the marginal-cost pass. 408 timings and 72 loop records, no failures. **Ahead on both axes on every case through the hand-off**, which is the sink most callers use: wall 0.35x, 0.32x, 0.25x, 0.45x, 0.86x and peak 0.95x, 0.84x, 0.32x, 0.76x, 0.88x. `profiled` was the one exception until #238 stopped the duplicate-coordinate check grouping 12M rows to answer a yes/no question; it is a win now like the rest. The LP file is the weaker route and stays that way — 0.71x to 1.35x on wall, and `transport` 1.61x on peak — because most of an LP write is float-to-text, work neither lane avoids. Against duckdb: 2.2-5.2x faster on every case and both sinks, and duckdb 1.2-2.9x lighter. duckdb is slower than the eager lane everywhere here. The parity gate agrees across all three arms at 0.0e+00 on every shared case, which is also what proves this branch's v1 semantics (#234) and `main`'s (#239) build the same model rather than merely claiming to. Provenance now names a commit per arm rather than a version, since two of the three arms have no meaningful version string — the duckdb arm is a checkout and the polars arm is an editable install that reports its sync point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FBumann
added a commit
that referenced
this pull request
Jul 28, 2026
Full three-arm ladder on `polars-engine@3fb95d6`, which now carries #238, #243 and #245. Five cases, two sinks, three arms, best of three, plus the marginal-cost pass. 408 timings and 72 loop records, no failures, parity gate exact across all three engines. Two numbers moved, both for a stated reason: transport LP peak 1.69x -> 1.19x (#245, chunked constraint emission) dispatch LP peak 1.01x -> 0.95x (same) To a loaded solver we are ahead on both axes on all five cases: wall 0.35x, 0.32x, 0.25x, 0.44x, 0.81x and peak 0.95x, 0.84x, 0.33x, 0.77x, 0.88x. On the LP route only two numbers are still against us — `profiled` 1.34x on wall and `transport` 1.19x on peak — and the section now says why that route is structurally weaker: most of an LP write is float-to-text, which neither lane avoids, so the ratio compresses toward 1.00 however fast the build gets. Against duckdb: 2.2-5.3x faster on every case and both sinks, duckdb 1.2-2.8x lighter. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every number in
docs/benchmarks.mdpredated #233, #234 and #238, and was measured against linopy 0.9.0 rather than the v1-semantics build this branch now pins. Re-taken in one run: five cases, two sinks, three arms, best of three, plus the marginal-cost pass. 408 timings and 72 loop records, no failures.Ahead on both axes on every case, through the hand-off
lrungprofiledwas the one exception until #238 stopped the duplicate-coordinate check grouping 12M rows to answer a yes/no question.The LP file stays the weaker route — 0.71x to 1.35x on wall,
transport1.69x on peak — and structurally so: most of an LP write is float-to-text, which neither lane avoids, so that ratio compresses toward 1.00 however fast the build gets.Against duckdb: 2.2-5.2x faster on every case and both sinks, and duckdb 1.2-2.9x lighter.
The parity gate now proves something
It agrees across all three arms at 0.0e+00 on every shared case — which is also what shows this branch's v1 semantics (#234) and
main's (#239) build the same model, rather than each merely claiming to.Provenance
Names a commit per arm rather than a version, because two of the three have no meaningful version string: the duckdb arm is a checkout, and the polars arm is an editable install that reports its sync point rather than the tree that ran.
farkas45902eclinopyduckdb4a13d38onmainAlso corrects the shim-overhead claim: it is a constant ~2.3 ms (1.26x at
xs, 1.15x atm), not the "~15%" this file carried from one measurement at one rung.417 passed / 1 xfailed, ruff and pyrefly clean.
🤖 Generated with Claude Code