feat!: absence propagates and drops the row, plus defined(v) by FBumann · Pull Request #239 · FBumann/lpspec

FBumann · 2026-07-28T11:11:11Z

Based on main. The language half of the v1 adoption, ported to the duckdb
engine so it lands independently of #189 — which then goes back to being what it
says on the tin, an engine swap the differential oracle can validate on its own.

Breaking. A term whose variable is masked out no longer contributes zero —
it makes the row absent.

x - rel_max * size <= 0     # size masked out on an unsized component

before:  c1: +x1 <= 0       # the flow is pinned to zero. feasible, plausible, no error
after:   (no row)

Goal 1 of linopy's v1 convention ("no silent wrong answers") and the whole of
PyPSA/linopy#712 — reachable here, not hypothetical.

What does not propagate

a reduction skips absent slots, so sum(x, over=d) stays defined when
only part of d exists — otherwise one masked component deletes a
system-wide accounting row
a parameter covering some coordinates is sparse encoding, not absence:
missing rows mean a zero coefficient (SPEC §8), which is what lets a
coefficient table hold live entries only

Absence is a property of variables. SPEC §6 now says so.

`defined(v)` — not optional

Dropping the row is the only reading available, so a model wanting the other one
must name those coordinates. A bare parameter name in a where already asks
"does this have a value here"; a bare variable name now asks "does this
exist here" — same grammar, and resolution already had the branch:

- expression: x - rel_max * size <= 0
  where: "size"
- expression: x <= 0
  where: "NOT size"

Otherwise the only spelling is a parameter mirroring the variable's own mask —
two sources for one fact, and when they drift the failure is a row quietly
pinned to zero rather than an error.

How it works in SQL

Once coeff × var are joined, the result cannot say which side removed a row —
so this is not an anti-join over the term stream. TermFragment.presence
carries a SELECT of the variable's own coordinates alongside, remapped by the
same shape operators, including shift, where the vacated edge is unioned back
in because SPEC §7 declares it contributes zero rather than being absent.

Two things worth reviewing closely:

populated only for a variable whose declaration has a where, read off
the plan before any data. A restriction is data, so it costs _label_frame
both of its arithmetic paths; an unmasked variable's presence is the whole
coordinate product and would remove nothing, so it never imposes that cost.
the semi-join goes in before the ROW_NUMBER. row is the solver's row
index, so deciding existence after labels exist would misalign it.

Verification

409 passed, 1 xfailed, on the duckdb engine against a v1 linopy oracle.
ruff and pyrefly clean.

The oracle is v1 and tests/oracle.py raises rather than skipping on a
linopy without options["semantics"]: legacy fills absent slots with zero, so
it agrees with the old reading, and a skip would go green having stopped
comparing the lanes on exactly the cases that changed. [tool.uv.sources] pins
PyPSA/linopy#717 by branch — a stale rev would measure us against a spec that
has moved.

Bug fixed on the way

A variable's own where naming itself was KeyError: 'p' out of linopy —
nothing in this package's voice. Now a load error: that mask is what decides
where the variable exists, so it cannot depend on it.

Known limits

defined(v) refuses a variable whose foreach exceeds the frame — masking a
[snapshot] constraint by a [snapshot, generator] variable would silently
widen the mask, and saying which reduction is meant needs ROADMAP Track 1 item 6
(all(x, over=d)). For the same reason it has no home in the dispatch-model
parity sweep, so COVERED_ELSEWHERE maps it to the test that does exercise it
rather than weakening the coverage guard.

Interaction with #189

#189 already contains this change (via #234, merged into polars-engine). Once
this lands on main, rebasing #189 drops the duplicate and leaves it a pure
engine swap — which is the point, and what makes the oracle able to check it.

The language half of the v1 adoption, on main's duckdb engine, so it lands independently of the polars swap (#189) and the swap goes back to being a swap. **What changes.** A term whose variable is masked out no longer contributes zero — it makes the row absent. `x + y >= 10` is *no constraint* where `y` is masked, not `x >= 10`. That old reading is how x - rel_max * size <= 0 silently became `x <= 0` on an unsized component: feasible model, plausible answer, no error. It is goal 1 of linopy's v1 convention ("no silent wrong answers") and the whole of PyPSA/linopy#712. **Two things deliberately do not propagate.** A *reduction* skips absent slots, so `sum(x, over=d)` stays defined when only part of `d` exists — otherwise one masked component deletes a system-wide accounting row. A *parameter* covering some coordinates is sparse encoding, not absence: its missing rows mean a zero coefficient (SPEC §8), which is what lets a coefficient table hold live entries only. Absence is a property of variables. **Telling them apart, in SQL.** Once `coeff x var` are joined the result cannot say which side removed a row, so this is not an anti-join over the term stream. `TermFragment.presence` carries a SELECT of the variable's own coordinates alongside, remapped by the same shape operators — including `shift`, where the vacated edge is unioned back in because SPEC §7 declares it contributes zero rather than being absent. It is populated only for a variable whose declaration carries a `where`, read off the plan before any data. That guard is load-bearing: a restriction is data, so it costs `_label_frame` both its arithmetic paths, and an unmasked variable's presence is the whole coordinate product and would remove nothing. The semi-join goes in before the `ROW_NUMBER`, because `row` *is* the solver's row index and deciding existence after labels exist would misalign it. **`defined(v)`, and it is not optional.** Dropping the row is now the only reading available, so a model wanting the other one needs to name those coordinates. A bare *parameter* name in a where already asks "does this have a value here"; a bare *variable* name now asks "does this exist here" — same grammar, and resolution already had the branch: - expression: x - rel_max * size <= 0 where: "size" - expression: x <= 0 where: "NOT size" Without it the only spelling is a parameter mirroring the variable's own mask: two sources for one fact, and when they drift the failure is a row quietly pinned to zero rather than an error. A variable's own where naming itself is now a load error. It used to be `KeyError: 'p'` out of linopy — nothing in this package's voice. 409 passed, 1 xfailed, on the duckdb engine against a v1 oracle. Closes #219. Refs #8

coderabbitai · 2026-07-28T11:11:26Z

Warning

Review limit reached

@FBumann, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 41 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4c20e457-447b-4fde-9a32-281182b03058

📥 Commits

Reviewing files that changed from the base of the PR and between ff61957 and 499efb0.

📒 Files selected for processing (14)

SPEC.md
pyproject.toml
src/farkas/dimensions.py
src/farkas/linopy/builder.py
src/farkas/linopy/semantics.py
src/farkas/lowering.py
src/farkas/relational/compiler.py
src/farkas/relational/executor.py
src/farkas/relational/plan.py
src/farkas/resolution.py
src/farkas/where_parser.py
tests/oracle.py
tests/test_relational.py
tests/test_resolution_parity.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/absence-drops-the-row

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Every number in this file predated #233, #234 and #238, and was measured against linopy 0.9.0 rather than the v1-semantics build this branch now pins. Re-taken in one run: five cases, two sinks, three arms, best of three, plus the marginal-cost pass. 408 timings and 72 loop records, no failures. **Ahead on both axes on every case through the hand-off**, which is the sink most callers use: wall 0.35x, 0.32x, 0.25x, 0.45x, 0.86x and peak 0.95x, 0.84x, 0.32x, 0.76x, 0.88x. `profiled` was the one exception until #238 stopped the duplicate-coordinate check grouping 12M rows to answer a yes/no question; it is a win now like the rest. The LP file is the weaker route and stays that way — 0.71x to 1.35x on wall, and `transport` 1.61x on peak — because most of an LP write is float-to-text, work neither lane avoids. Against duckdb: 2.2-5.2x faster on every case and both sinks, and duckdb 1.2-2.9x lighter. duckdb is slower than the eager lane everywhere here. The parity gate agrees across all three arms at 0.0e+00 on every shared case, which is also what proves this branch's v1 semantics (#234) and `main`'s (#239) build the same model rather than merely claiming to. Provenance now names a commit per arm rather than a version, since two of the three arms have no meaningful version string — the duckdb arm is a checkout and the polars arm is an editable install that reports its sync point. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…288) * docs(spec): three behaviours a consumer could only reach by probing A downstream port (#214) worked around all three, having established each by probing rather than by reading. This states them, and pins each with a differential test — an unrun claim in a language reference is the same shape of thing as an unrun example. #224 filed three items; one of them had already been overtaken by the engine. **Term absence** was item 1, and it is now half wrong: #239 landed absence-propagates hours after the issue, so a masked *variable* drops the row and the trap it describes is gone. The other half is live and unstated — the same expression with the *parameter* missing keeps the row and builds `x <= 0`, which is exactly the silently-wrong shape the v1 convention removed from the variable side. §6 already used `x - rel_max * size <= 0` as its example of what does *not* happen; a reader who stops there concludes they are safe. It now says which operand decides that, and that a coefficient table sparse because the *component* is absent wants a mask. **shift/roll over parameters** — §7's `array` covers it and every example took a variable, so the port shipped `dt_previous`, a hand-shifted copy of a table it already had, and deleted it once it knew. The vacated-position zero is the trap in RHS position: `x <= shift(dt, t=1)` pins the first coordinate rather than leaving it free, so `roll` is named as the contrast. **Pinning with equal bounds** is how a quantity that is data in one model and a decision in another stays one declaration, instead of a block per regime with coefficients whose names encode the regime. §2, next to bounds, with the two limits that bite: affinity is unchanged, and a pinned variable cannot bound another. Tests are `differential(..., lp=True)`, so each is asserted on both lanes and the LP re-solve. The parameter-absence one sits beside the benign case it is easy to mistake for — there the zero lands on a coefficient *and* the RHS, so the row constrains nothing; here the RHS is a literal 0. Refs #224 * docs(spec): name shift as the one place we hold both lanes off v1 (#289)

…tity of its position Two corrections to the same mistake, found by auditing for other places we answer on linopy's behalf. **The package never opted into v1.** `tests/oracle.py` sets `options['semantics'] = 'v1'` and argues for it well — measuring against legacy "would pin this package to the behaviour v1 classifies as a bug". `src/farkas/` did not, and linopy's default is legacy. So the tested configuration was not the shipped one, and the two lanes answered the same YAML differently: masked variable in a product, no test harness EAGER (as shipped) objective 25.0 x[b] = 0 legacy fills absent with 0 RELATIONAL objective 125.0 x[b] = 100 row dropped, SPEC §6 A wrong answer, not a wrong error, and live since #239 — this commit's shift work would merely have added a second instance. Set on import: the option is global, and linopy's own context manager cannot scope it (`__exit__` calls `reset()`, restoring *all* options to their defaults rather than their prior values, so it would discard a caller's `display_max_rows`). The two tests for it run in a **subprocess**. An in-process assertion would pass whether or not the package sets it, since importing the harness sets it — which is precisely how this shipped unnoticed. **`fill=` takes a number because the right fill is positional.** v1 refuses to fill on a caller's behalf for exactly this reason (`convention.rst` §7): 0 is the identity of a sum, 1 of a product. Restricting to the literal 0 got that wrong — `x * shift(eff, t=1, fill=1)` leaves the first coordinate governed by its own bound, where `fill=0` pins it, which is the pin this branch exists to remove wearing the coefficient's hat. Over data any number is accepted, since it is a data fill both lanes do natively; the relational lane now writes the rows for a nonzero one, because a const fragment reads a missing row as zero and `fill=1` exists only if something puts it there. Over an operand carrying a variable 0 remains the only representable fill — there a vacated slot contributes no term, and a nonzero one would be a constant standing where a term was. BREAKING CHANGE: importing `farkas.linopy` sets `linopy.options['semantics']` to 'v1' process-wide. Models built through this lane change answer wherever a variable is masked or shifted — that is the point, and it is the answer the native engine has always given.

…tity of its position (#293) * fix(linopy)!: the lane selects v1 on import, and fill= takes the identity of its position Two corrections to the same mistake, found by auditing for other places we answer on linopy's behalf. **The package never opted into v1.** `tests/oracle.py` sets `options['semantics'] = 'v1'` and argues for it well — measuring against legacy "would pin this package to the behaviour v1 classifies as a bug". `src/farkas/` did not, and linopy's default is legacy. So the tested configuration was not the shipped one, and the two lanes answered the same YAML differently: masked variable in a product, no test harness EAGER (as shipped) objective 25.0 x[b] = 0 legacy fills absent with 0 RELATIONAL objective 125.0 x[b] = 100 row dropped, SPEC §6 A wrong answer, not a wrong error, and live since #239 — this commit's shift work would merely have added a second instance. Set on import: the option is global, and linopy's own context manager cannot scope it (`__exit__` calls `reset()`, restoring *all* options to their defaults rather than their prior values, so it would discard a caller's `display_max_rows`). The two tests for it run in a **subprocess**. An in-process assertion would pass whether or not the package sets it, since importing the harness sets it — which is precisely how this shipped unnoticed. **`fill=` takes a number because the right fill is positional.** v1 refuses to fill on a caller's behalf for exactly this reason (`convention.rst` §7): 0 is the identity of a sum, 1 of a product. Restricting to the literal 0 got that wrong — `x * shift(eff, t=1, fill=1)` leaves the first coordinate governed by its own bound, where `fill=0` pins it, which is the pin this branch exists to remove wearing the coefficient's hat. Over data any number is accepted, since it is a data fill both lanes do natively; the relational lane now writes the rows for a nonzero one, because a const fragment reads a missing row as zero and `fill=1` exists only if something puts it there. Over an operand carrying a variable 0 remains the only representable fill — there a vacated slot contributes no term, and a nonzero one would be a constant standing where a term was. BREAKING CHANGE: importing `farkas.linopy` sets `linopy.options['semantics']` to 'v1' process-wide. Models built through this lane change answer wherever a variable is masked or shifted — that is the point, and it is the answer the native engine has always given. * docs(packaging): the linopy floor now guards the v1 opt-in, so raising it is a release step

…d it there (#314) Closes #311. `sum(x + y, over=f)` with `y` masked to `f=a` built different models on the two lanes: 400 eager, 240 native — a 40% gap, no error. Addition is fragment concatenation in the relational compiler, so `x + y` is two independent streams and each was reduced over its own coordinates. At row level that is right, since the executor intersects the presences when it assembles the row; a reduction consumes the expression before any row exists, so nothing intersected them. The eager lane was right and I first argued it was not. SPEC §6's carve-out — "a reduction skips what is absent" — has `sum(x, over=d)` over a bare variable as its example, and says a reduction must not propagate absence *outward*. It does not license distributing a compound summand, and distributing is what the relational lane was doing: rewriting `sum(x + y)` into `sum(x) + sum(y)` reads the absent `y[b]` as a zero, which is the reading v1 exists to remove and that #239 and #291 removed from masks and shifts. So reductions are not linear over operands of differing presence. That is the honest consequence of absence being a state rather than a value, and the two spellings are different questions: the total of the net where the net is defined, against the total in minus the total out. SPEC now puts them in a table side by side, because the composition of the two rules was nowhere written down — which is how I came to misread it. `tests/test_arithmetic_laws.py` is the other half. An arithmetic convention is a set of equalities between spellings and nothing else in the suite was checking those: a model can build, solve and agree across both lanes while `a + b` and `b + a` mean different things. Eight laws (commutativity, associativity, subtraction as negated addition, distribution over a variable-free factor and over a divisor, reduction linearity while every operand is total, roll invertibility, and commutativity *under* absence) plus three deliberate non-laws, each asserted through `differential` so a law holding is six numbers agreeing rather than two. The non-laws are the point. This defect is invisible to law-shaped tests — both spellings were individually consistent — and only a test saying "these two are supposed to disagree, and by exactly this much" would have caught it. Writing them also caught one of my own: a "law" that summing over a dim the operand lacks scales by that dim's cardinality. The language refuses that outright at load time. The scaling in `_sum_fragment` applies to a fragment lacking a dim the expression carries, which is not user-visible.

FBumann merged commit 5eb5943 into main Jul 28, 2026
3 checks passed

fluxopt-release Bot mentioned this pull request Jul 28, 2026

chore(main): release 0.0.0-alpha.33 #241

Merged

This was referenced Jul 28, 2026

feat!: rebuild the relational engine on polars #189

Merged

docs(bench): re-run all three engines on the current base #243

Merged

This was referenced Jul 29, 2026

feat!: shift() creates absence — and the lane actually speaks v1 #291

Merged

fix(linopy)!: the lane selects v1 on import, and fill= takes the identity of its position #293

Merged

This was referenced Jul 29, 2026

fix(relational): absence propagates into a reduction, and laws to hold it there #314

Merged

sum(a + b) and sum(a) + sum(b) disagree when b has absent slots — and only on the eager lane #311

Closed

FBumann mentioned this pull request Jul 29, 2026

SPEC: three behaviours only reachable by probing — term absence, shift() over parameters, pinned variables #224

Closed

FBumann deleted the feat/absence-drops-the-row branch July 31, 2026 10:51

Navigation Menu

feat!: absence propagates and drops the row, plus defined(v) - #239

feat!: absence propagates and drops the row, plus defined(v)#239
FBumann merged 1 commit into
mainfrom
feat/absence-drops-the-row

FBumann commented Jul 28, 2026

Uh oh!

coderabbitai Bot commented Jul 28, 2026

Review limit reached

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FBumann commented Jul 28, 2026

What does not propagate

defined(v) — not optional

How it works in SQL

Verification

Bug fixed on the way

Known limits

Interaction with #189

Uh oh!

coderabbitai Bot commented Jul 28, 2026

Review limit reached

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

`defined(v)` — not optional