Skip to content

refactor: remove demand-side mechanisms superseded by market response - #49

Open
koen-vg wants to merge 4 commits into
pmp-demand-responsefrom
remove-legacy-demand-mechanisms
Open

refactor: remove demand-side mechanisms superseded by market response#49
koen-vg wants to merge 4 commits into
pmp-demand-responsefrom
remove-legacy-demand-mechanisms

Conversation

@koen-vg

@koen-vg koen-vg commented Jul 31, 2026

Copy link
Copy Markdown
Member

Stacked on #48 (rebased onto its final state, including the capacity-lift refit and the legacy-chain restructure). Removes the legacy demand-side mechanisms that the market_response demand component supersedes, migrates the remaining consumers, and rewrites Tutorial 2 around demand response.

Removed

  • Consumer-values workflow (extract_consumer_values, the consumer_values.smk rules, validation, comparison plot, docs page): hand-rolled PMP phase 1 for the demand side, with the level-degeneracy problem the sequential calibration was built to fix.
  • food_utility_piecewise (+ calibrate_food_utility_blocks): piecewise utility from consumer values with ad-hoc decline/width knobs -- the demand curves carry the same content with exact calibration and literature elasticities.
  • food_incentives and the optimal_taxes workflow: disabled everywhere; the equivalent quantities are directly readable off the demand curves.

~1.9k lines of workflow code plus the config blocks, schema entries, solve-time prefixes, and manifest wiring. The deviation_penalty L1 machinery (and the legacy cost + stability steps) stay, as the anchor for coarser curve granularities and structurally divergent configs.

Migrations

Artefact refits

  • default and gbd-anchored: the legacy cost and stability artefacts are refit under the new regime (the four-step chain from Replace L1 stability anchoring with exactly calibrated PMP market-response curves (supply + demand) #48, including the capacity-lift market-response fit, is untouched). The calibrated L1 costs rise substantially (default: cropland 1.38 -> 6.54, grassland 0.23 -> 0.34, feed 0.06 -> 0.50 bn USD per unit deviation): without any demand-side valuation, the free-diet main solve reorganizes production much harder, and holding the 5% deviation target is genuinely more expensive. Legacy consumers run the same regime, so calibration and consumption match; expect stiffer L1 behaviour in legacy configs.
  • tutorial-02 (new): backs the demand-response tutorial at 200 regions, fit with the four-step chain only -- no legacy cost/stability artefacts ship with it. Config validation now requires the legacy stability yaml only when a solve would actually resolve the "calibrated" sentinel, which replaces the earlier seeding workaround.

Tutorial 2 rewritten

Now "Demand response: letting the diet adjust to policy": the reader calibrates a dedicated 200-region artefact set (tools/calibrate --base config/tutorial/02_demand_response.yaml, committed as tutorial-02), sees the exact-calibration property live (an unpriced solve reproducing the observed food system with nothing pinning it), then runs a carbon-price ladder plus two demand-side variants: the diet frozen at baseline (what flexibility is worth) and half elasticities (how much hinges on that assumption, changeable with no refit). The companion notebook compares consumption shifts, the supply_response/demand_response objective columns, and emissions across the three demand regimes.

Verification

  • 758 unit tests pass (including new coverage of the stability-yaml existence gate); DAG dryruns pass for the tutorial, gsa.yaml, gsa_fixed_diet.yaml, and doc_figures.yaml.
  • tools/calibrate --check is fully green for all three artefact sets; both legacy stability calibrations converged (default: 4 warm-started iterations, residual 0.016; gbd-anchored: residual 0.013).
  • Tutorial reference solve (unpriced, 200 regions, HiGHS, tutorial-02 set): baseline reproduction with nothing pinning it: consumption 0.003%, crops 0.107%, pasture 0.012%, animal feed 0.249% off observed 2020. Demand censoring minimal: 10 of 8403 groups, total pin slack 0.14 Mt.

koen-vg added 4 commits July 31, 2026 16:08
The consumer-values workflow, food_utility_piecewise, food_incentives
and optimal_taxes all predate the market_response demand component,
which carries the same economic content (a fitted marginal utility per
(food, country) consumption) with exact calibration and
literature-grounded elasticities instead of extraction-plus-tuning.

The GSA and doc-figure configs move onto the demand component, which
also removes their locally-solved baseline scenario (it existed only to
feed consumer-value extraction). The cost calibration's tight-band solve
keeps the diet pinned to baseline instead of valuing it with utility
blocks, and the legacy stability calibration runs under the base
config's own demand regime. Tutorial 2 is rewritten around demand
response with its own tutorial-02 artefact set.
…e it

Config validation demanded deviation_penalty.yaml exist whenever the
calibration block was enabled, even for configs that never enable the
L1 penalty. Artefact sets fit with the four-step default chain (like
the tutorial-02 set) ship without the legacy stability artefact, so
gate the existence check on the same condition under which the solve
actually opens the file (penalty enabled, l1 mode, calibrated
sentinel in use). This replaces the seeding workaround that copied the
default set's yaml into fresh sets.
The legacy cost and stability calibrations of the default and
gbd-anchored sets are refit under the pinned-diet cost regime and the
utility-free stability main solve. The L1 costs rise substantially
(default: cropland 1.38 -> 6.54, grassland 0.23 -> 0.34, feed
0.063 -> 0.50 bn USD per unit deviation): with no demand-side valuation
in the legacy regime, the main solve reorganizes production much more
aggressively, and holding the 5% deviation target is genuinely more
expensive. Consumers of these artefacts run the same regime, so the
calibration and consumption regimes match.

The new tutorial-02 set backs the demand-response tutorial at its
200-region resolution, fit with the four-step default chain only (no
legacy cost/stability artefacts). Demand censoring stays minimal
(10 of 8403 groups, total pin slack 0.14 Mt).
@koen-vg
koen-vg force-pushed the remove-legacy-demand-mechanisms branch from 324b7bb to 6644153 Compare July 31, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant