bench: restore the duckdb arm, and price a return to it - #391
Conversation
#189 replaced the duckdb engine with polars and deleted the harness machinery that had measured the two against each other. Pricing that decision again needs the engine it replaced, and the only honest version of that number comes from the engine as it shipped rather than from a fresh port written to lose. So `duckdb` is a foreign arm again: this tree's harness, models, ladder and parquet cache, run under another checkout's interpreter. Only the engine package comes from there, which is what keeps `fleet`, `sector` and the `xl`/`2xl` rungs — all added since — covered rather than silently skipped. Two gaps had opened since the arm was deleted. The package was renamed (#336), so `_engine()` absorbs `farkas` under its old name; and the `expression:` surface replaced `equations:`, so today's models are a validation error to that engine. `_dialect` rewrites the declaration on the way in, before the clock starts — a rewrite rather than a second copy of each model, because copies drift and the drift stays invisible until it changes an optimum. The parity gate is what makes that checkable: it agrees to 0.0e+00 relative on all six cases, including the two the duckdb checkout has never seen. `bench.report --arms SUBJECT AGAINST` renders the comparison through the same tool as the published ladder, rather than a second renderer that drifts. `duckdb-spike.md` carries the costing: ~2,300 of the engine's 3,317 lines rewritten, ~6,300 above the plan untouched, `primal()`'s return type the one public break. `sql_tax.py` measures what the hardest operators cost in SQL, against the polars compiler on live data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six cases, xs..xl, both sinks, both engines, best of two. The duckdb arm ran at its own default `memory_limit='1GB'` — not a choice made here, `fk.build` defaults to it, so this is the engine as it shipped against the engine as it ships. That default is also the whole difference in kind: one arm has a ceiling and spills, the other's peak tracks the model. The write path is the column #189 never published, because its headline compares to a loaded solver where the build is ~9% of peak and any build-side difference is diluted. Measured directly it is not small: duckdb is 2.69x lighter at dispatch/l, 3.15x at dispatch/xl and 4.20x at profiled/xl, and the gap *widens* up the ladder (1.11 / 1.25 / 1.63 / 2.69 / 3.15). The wall gap does not widen with it — 3.27x at l but 1.57x at xl — because polars does more work per byte once the model stops fitting comfortably. `fleet/xl` failed on the duckdb arm, OutOfMemoryException at 953 MiB, twice. That is a 1 GB budget being too tight for 48M variables rather than the engine breaking, and it is the behaviour a ceiling exists for: it fails instead of taking the machine with it. Recorded rather than dropped — a failure is a result here. On the solver path the engines converge to 1.15-1.35x, and at profiled/l polars is *lighter* (0.95x). Kept in the table rather than smoothed: the gap is a consequence of how much of the build survives to the hand-off, not a law. So the trade is not "slower and lighter, plus a knob" — it is a bounded build worth 1.6-2.4x wall clock on the write path and almost nothing on the solve path. That makes it a product question about `lps.write`, not an engine question, and §9 now says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Superseded by #399. The valuable half moved there; the rest was made obsolete by the engine landing in-tree. Kept, in #399: Dropped: the foreign-arm machinery ( The committed results are still the old shipped engine measured that way, and the doc now says so rather than quietly re-baselining against the port. |
1 participant
Draft. This half is meant to merge — the engine port that answers it is stacked on top in #392 and is not.
Why
The polars/duckdb decision has already flipped once (#189, three days ago), and the reasoning did not survive on its own: I had a stored note from the day before #189 landed that still said "keep duckdb". A decision that reverses on a 24-hour boundary needs provenance rather than memory, and this repo already has the machinery for that — #189's harness had a
duckdbforeign arm, and it was deleted along with the engine.What
The arm is back.
duckdbruns this tree's harness, models, ladder and parquet cache under another checkout's interpreter; only the engine package comes from there. That is what keepsfleet,sectorand thexl/2xlrungs — all added since — covered rather than silently skipped.Two gaps had opened since it was deleted:
_engine()absorbsfarkasunder its old name.expression:surface replacedequations:, so today's models are a validation error to that engine._dialectrewrites the declaration on the way in, before the clock starts — a rewrite rather than a second copy of each model, because copies drift and the drift stays invisible until it changes an optimum.The gate is what makes that checkable: it agrees to
0.0e+00relative on all six cases, including the two the duckdb checkout has never seen.bench.report --arms SUBJECT AGAINSTrenders the comparison through the same tool as the published ladder rather than a second renderer that drifts.What the numbers say
Six cases,
xs..xl, both sinks, best of two. The duckdb arm ran at its own defaultmemory_limit='1GB'—fk.builddefaults to it, so this is the engine as it shipped against the engine as it ships.The write path is the column #189 never published, because its headline compares to a loaded solver where the build is ~9% of peak:
lxlxlThe peak gap widens with the model (1.11 / 1.25 / 1.63 / 2.69 / 3.15 up dispatch's ladder). On the solver path it collapses to 1.15–1.35×, and at
profiled/lpolars is lighter (0.95×) — kept in the table rather than smoothed.fleet/xlfailed on the duckdb arm,OutOfMemoryExceptionat 953 MiB. That is a 1 GB budget being too tight for 48M variables, not the engine breaking — and it is what a ceiling is for. Recorded rather than dropped.What this is not
Not a proposal to switch back.
duckdb-spike.mdprices the reversal honestly (~2,300 of 3,317 engine lines;primal()'s return type the one public break) and its conclusion is that this is a product question aboutlps.write, not an engine question — and that if the write path matters, a duckdb-backedlp_filesink is the cheaper move, since a sink is a module and not an engine.Reviewing
bench/duckdb-spike.mdis the argument; everything else is machinery.709 passed, 4 skipped, 1 xfailed; ruff clean.PolarsCompiler/PolarsExecutorviolate it in 44 places andtest_architecture.pydoes not enforce that clause. Not fixed here — flagged.🤖 Generated with Claude Code