Skip to content

Reduce Rest test cost and move policy examples to YAML - #10010

Draft
hua7450 wants to merge 3 commits into
PolicyEngine:mainfrom
hua7450:trim-rest-test-cost
Draft

hua7450 wants to merge 3 commits into
PolicyEngine:mainfrom
hua7450:trim-rest-test-cost

Conversation

@hua7450

@hua7450 hua7450 commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

Rest repeats full-model setup and runs policy examples that the YAML suites can cover. Reuse the loaded baseline in seven read-only consumers, accelerate repeated YAML scans with compiled safe/base loaders (keeping the Python fallback), and move ordinary policy examples out of Python.

  • Delete core/test_ssi_disability_criteria_inputs.py: both checks are already covered by gov/ssa/ssi/meets_ssi_disability_criteria.yaml.
  • Move 12 Medicare cases and two farm-income cases into eight variable-named YAML files. Preserve the original inputs, monthly and lagged-year periods, household memberships, output values, and equal or tighter tolerances.
  • Remove the duplicate Medicare zero-premium case: the retained Medicare-cost case has the identical situation and identical first assertion.
  • Keep the Medicare calculation-order test and two country-Simulation input-remapping checks in Python, unchanged.
  • Remove the duplicate full-US empty-collection runner check and separate YAML syntax scan. The existing small-country Core CLI check retains the exit marker, exit code 5, and failed status; retained parameter checks parse the complete YAML tree.
  • Make baseline reuse and simulation independence explicit in AGENTS.md and CLAUDE.md.

The migration removes 17 Python cases: 14 moved to YAML and three duplicates deleted, including one entire Python file. Together with the earlier runner/syntax duplicates, this PR removes five duplicate checks. January-only Medicare eligibility inputs remain separate from existing annual-input YAML cases because their calculation paths differ.

The workflow still uses 26 runners, with sequential process isolation. Medicare cases use the existing Baseline contrib-hhs job; farm-income cases use Baseline irs; retained SSI coverage uses Baseline ssa-usda. No workflow, formula, parameter, or partner test changes, and no new YAML reform combinations.

Whole-suite measurements

The final CI run at 25ac672f8c passed all 26 jobs using 26 runner allocations. Rest remains the last job to finish.

Compare the #9744 baseline, including #10006 and #10007 and the earlier setup/scan revision:

Metric #9744 baseline Setup/scan revision 5f3ce45735 Final revision 25ac672f8c
Complete Rest job 57m03s 43m33s 53m49s
Python step 55m42s 42m30s 52m29s
Variables YAML step 0m59s 0m46s 1m00s
Highest process RSS in Rest 13.58 GiB 10.59 GiB 10.60 GiB
Baseline contrib-hhs job 22m16s 21m04s 20m42s
Baseline irs job 18m30s 14m08s 18m22s
Whole workflow elapsed 57m31s 44m09s 54m43s
Sum of all 26 runner job durations 9h11m43s 8h39m37s 9h30m31s

The final full Rest job was 3m14s shorter than the original baseline, with highest process RSS falling from 14,242,068 to 11,113,604 KiB. However, total runner job time was 18m48s higher than the original baseline. Compared with the earlier setup/scan revision, final Rest was 10m16s longer and total runner job time was 50m54s higher. This run therefore does not demonstrate additional savings from the deletion/YAML migration, or an overall reduction in CI runner time. Moving work between jobs alone is not evidence of lower total cost.

These are observational comparisons across distinct runner allocations. Baseline and final Rest used Python 3.14.7 and Ubuntu image 20260927.320.1; the earlier setup/scan run used Python 3.14.8 and image 20261004.327.1. Rest installed Core 3.32.21 in all three runs. The PR base also advanced from the #9744-era baseline through #9745; the setup/scan and final revisions share that base. Unchanged test groups varied materially, so neither improvements nor regressions here can be attributed solely to this patch.

Workflow elapsed is measured from the workflow start to the last job completion, including queue/dependency waits. Aggregate runner time sums each allocated job's start-to-completion duration, excluding queue waits; it is not a billing-minutes figure. RSS is the maximum reported by a single timed Rest process, not a measurement of the whole machine or concurrent jobs.

Final Rest artifacts contain 3,273 cases: 3,262 passed and 11 skipped, with zero failures/errors. They comprise 3,204 Python cases and 69 variables YAML cases. The 17-case reduction from the setup/scan run matches the 14 migrated and three deleted Python cases. Seven complete GNU time/memory reports, seven JUnit reports, full Rest logs, and every job/step's metadata have been saved for audit.

Validation

  • Initial setup/parser revision: 136 targeted checks passed locally, followed by 26/26 successful CI jobs.
  • Original parser validation: identical content for 6,249 parameter files and 4,892 policy YAML files, including identical reform-combination keys in the measured checkout.
  • Final migration: static comparison of all moved inputs/expected outputs and unchanged retained Python function bodies; all existing YAML cases preserved.
  • Duplicate cases mapped to retained coverage; removed names checked for references.
  • YAML parsed with duplicate-key detection; each destination is included exactly once in the existing full-suite batch plans, with no added reform combinations.
  • make format, git diff --check, and changelog fragment.
  • No local tests rerun for moves/deletions, per repository/workspace guidance.
  • Final revision 25ac672f8c: 26/26 CI jobs passed; full Rest timing/memory, affected YAML jobs, workflow elapsed, and aggregate runner time captured above.

axiom: n/a: test setup, deduplication, migration, and documentation only; no policy changes.

@hua7450 hua7450 changed the title Reduce redundant setup and scans in Rest tests Reduce Rest test cost and move policy examples to YAML Oct 8, 2026
@hua7450
hua7450 marked this pull request as draft October 8, 2026 13:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant