Skip to content

Publish a release only after its tests pass - #591

Open
MaxGhenis wants to merge 3 commits into
masterfrom
fix-publish-needs-test
Open

MaxGhenis wants to merge 3 commits into
masterfrom
fix-publish-needs-test

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

In .github/workflows/push.yaml, the Publish job (PyPI upload and git tag) had no needs:, so on each release commit it ran beside Test. A release could reach PyPI while its tests were still running, or after they failed. In the 2026-10-06 13:11Z release run (37468917800), Publish finished at 13:40:07 and Test at 13:44:04.

Publish now has needs: Test. Both jobs have the same if: (the canonical repository and the "Update PolicyEngine Core" commit), so Publish still runs on every release commit; it now waits for Test to pass.

Gating won't hold up current releases: all 69 of the last 69 "Update PolicyEngine Core" push runs passed.

Tests

test_release_tagging.py::test_publish_workflow_publishes_only_after_tests_pass checks four things:

  • every job that uses pypa/gh-action-pypi-publish needs Test;
  • the only such job is Publish;
  • Publish's if: is Test's;
  • Publish's if: contains no status function (always(), failure(), cancelled(), success()), which could let it run after a failed Test.
before (master workflow):  1 failed, 5 passed   (AssertionError: assert 'Test' in [])
after:                     6 passed

Impact

CI only; no package code changes. Each release now publishes about four minutes later, once Test finishes.

Noticed in passing, not changed here: the Test job's "Deploy documentation" step runs if: matrix.os == 'ubuntu-latest', but the job has no matrix, so the step has been skipped on every run. I'm raising that separately.

axiom: n/a: CI workflow change, no policy change

Other audit findings, handled elsewhere

The 2026-10-06 invariants audit of master 757147c7 also confirmed defects that this PR leaves alone, because other work owns them:

Branch staleness after set_input and raw-file restore are documented behaviour, not defects.

🤖 Generated with Claude Code

push.yaml ran Test and Publish side by side on the release commit, so a
release could reach PyPI while its tests were running or after they failed.
In the 2026-10-06 13:11Z release run, Publish finished at 13:40:07 and Test
at 13:44:04. Publish now has needs: Test. Both jobs share the same if
condition, so Publish still runs on every release commit and waits for Test
first.

Adds a check in test_release_tagging.py that every PyPI-publishing job
needs Test and runs under Test's condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 6, 2026 16:59
From the independent review of 9948a8d (approved, nit): adding always()
to both jobs' shared condition kept them equal and passed the check,
although it would let Publish run after a failed Test. The check now also
requires Publish's condition to contain no status function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the second independent review (approved, nit): Always() and ALWAYS()
passed the check. It now uses a case-insensitive regex; all five override
variants are rejected and the current workflow passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 8, 2026
* Deploy the documentation site from its own release job

The Test job lost its OS matrix in #282 (2024-09-26) but kept the
deploy step's `if: matrix.os == 'ubuntu-latest'` guard, so the step has
been skipped on every release since and gh-pages last moved on
2024-09-26.

Move the docs build and deploy into a Docs job that needs Test, has
`contents: write`, and uses the v4 action's documented inputs. Keeping
it out of Test means a failed deploy cannot hold back a PyPI release
once Publish needs Test (#591). Add sphinx.ext.githubpages so the build
writes the .nojekyll that a branch-sourced Pages site needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Deploy docs only after the release reaches PyPI

Review follow-ups on the Docs job:

- Need Publish as well as Test, so the site never documents a version
  that failed to reach PyPI. Publish still does not depend on Docs.
- Serialize deploys with a gh-pages-deploy concurrency group; a newer
  waiting deploy replaces an older one instead of racing it.
- Loosen the structural tests to the properties that matter (branch,
  folder, contents: write, an unconditional deploy step), parse the
  Makefile recipe by its tab-indented lines, and skip jobs whose matrix
  is built by an expression.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Tighten docs deploy checks from the delta review

- Say what the concurrency group guarantees: one deploy at a time, a
  newly queued deploy replacing one still waiting. GitHub does not
  promise release order.
- Require a constant concurrency group (a per-run group would not
  serialize anything) and accept the string shorthand.
- Match `jb build docs` as a whole recipe line, so a changed output
  path or a commented-out line no longer passes.
- Treat `strategy: null` like no strategy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant