test(ports): real models, checked against optima that did not come from us - #252
Conversation
|
Warning Review limit reached
Next review available in: 40 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (9)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
023604b to
186840b
Compare
…om us Every other test compares farkas against farkas. Even the differential harness compares two lanes consuming the same resolved AST (hard rule 1), so a shared misreading — both lanes agreeing on a meaning the modeller did not intend — passes the whole suite green. This is the net for that class. transport_dantzig 153.675, published with GAMS model library #1 pypsa_transport 22000.0, from PyPSA 1.2.4 and its own linopy 0.9.0 Both agree exactly, and pypsa_transport matches PyPSA's dispatch generator for generator. Reference scripts never run in CI — pinning PyPSA into this project would hand their release cadence a veto over the suite. They carry their deps inline (PEP 723) pinned to what produced the recorded number, and read the same instance the port binds, since a reference optimum against a different instance means nothing. What stays independent is the formulation. The corpus therefore needs no oracle and no extra dependency, and runs on the bare-install job. Two findings already: - transport_dantzig reaches 153.675 at a different vertex than the source prints. Alternative optima, on port zero — which is why the corpus asserts objectives and never primals. - PyPSA's `p_min_pu = -1` is a bound of minus the rating, and bounds take a name or a number, never arithmetic. The port ships neg_rating as data; the ledger in docs/ports.md records it as a second model asking for #31. docs/ports.md is a ladder rather than a model per network: rung 1 is the transport model, with ramp limits, storage, cyclic SOC and KVL each a later rung, so a rung that fails to match implicates one feature instead of five. A test asserts each YAML fence on the page equals its model file byte for byte — the trade linopy/semantics.py already makes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
186840b to
e0e604d
Compare
1 participant
Two ported models, each checked against an optimum produced by somebody else's code. 9 files, +511.
Why this is not another differential test
Every other test compares farkas against farkas. Even the differential harness compares two lanes consuming the same resolved AST — hard rule 1, and what makes them an oracle for each other. It is also what they cannot see: a shared misreading, both lanes agreeing on a meaning the modeller did not intend, passes the whole suite green. #197 was the near-miss.
transport_dantzigpypsa_transportpypsa_transportmatches PyPSA generator for generator on the dispatch too, and both links run saturated in every snapshot — so the flow variables are load-bearing rather than decorative.Two findings, both on the first day
Alternative optima, on port zero.
transport_dantzigreaches 153.675 at a different vertex than the source prints. That is why the corpus asserts objectives and never primals — an optimal objective is unique, an optimal solution need not be, and a corpus pinned to a vertex fails on a solver upgrade that broke nothing.One ledger row. PyPSA's
p_min_pu = -1is a bound of minus the rating, and bounds take a name or a number, never arithmetic (SPEC §2). The port shipsneg_ratingas data — logged as a second real model asking for bounds as expressions (#31).The shape
Reference scripts never run in CI. Pinning PyPSA into the dependency tree would hand their release cadence a veto over this suite. They carry their deps inline via PEP 723, pinned to the versions that produced the recorded number:
So the corpus needs no oracle and no new dependency — verified to leak no linopy/xarray/pandas/pyarrow, which means it runs on the bare-install job, where it is now the only evidence the engine solves models nobody wrote for it.
transport_dantzighas no script at all: its optimum is published, the strongest tier.The instance is data both sides read. A reference optimum against a different instance means nothing, so the reference builds its PyPSA network from the same JSON the port binds. What stays independent is the formulation — nothing in it imports farkas.
rtolis per port: a published optimum is rounded to the digits that fitted in a table, a solved one is good to solver precision.A ladder, not a model per network
Reproducing a full PyPSA objective means reproducing marginal and capital cost, ramp limits, storage cycling and KVL at once — a mismatch then implicates five features instead of one. So each network is a ladder: 1 transport model (this PR) · 2 ramp limits · 3 storage with SOC · 4 cyclic boundary condition · 5 KVL. A rung that matches is a row in the table; a rung that cannot be said is a row in the ledger. Both are evidence.
Also in here
docs/ports.mdequals its model file byte for byte — the tradelinopy/semantics.pyalready makes.ARCHITECTURE.mdpointing the ceiling section at the corpus.Checks
423 passed, 1 xfailed(pre-existing), ruff clean, pyrefly 0 errors. Rebased ontomainat ff07e1f.Open questions
examples/ports/*.yamlare not picked up bytest_language_boundary's non-recursiveexamples/*.yamlglob.test_ports.pyrunsfk.checkon each, so they are covered — but if that claim should live in one place, the glob is a one-word change.docs/ports.mdjoinTRACKEDintest_doc_examples.py? It would re-validate fences the byte-for-byte test already ties to checked files, so I left it out as duplicate work.🤖 Generated with Claude Code