Skip to content

ci: drive build versions from docs/packages/<pkg>.yaml - #1937

Merged
luhenry merged 4 commits into
mainfrom
yaml-driven-versions
Sep 16, 2026
Merged

luhenry merged 4 commits into
mainfrom
yaml-driven-versions

Conversation

@luhenry

@luhenry luhenry commented Sep 16, 2026

Copy link
Copy Markdown
Member

docs/packages/<pkg>.yaml becomes the source of truth for what a workflow builds. A - version: entry without tag:/files: is pending; _setup.yml turns the pending set into a matrix.version every job iterates over, and _publish-wheel.yml fills the entry in through the docs PR. Adding a package is two files (build-<pkg>.yml + docs/packages/<pkg>.yaml); upgrading one is a single - version: line.

Triggers

  • pull_request (to main) on the workflow and its YAML: builds every pending version, dry-runs the publish.
  • push to main: builds and publishes every pending version — replaces the manual post-merge dispatch. Merging the docs PR pushes an all-released YAML, so that run has an empty matrix and skips every job (green).
  • workflow_dispatch takes a version glob (2.5.*), released or not, for force-rebuilds.

Infra

  • _setup.yml: package (required) + version (glob) inputs; runs the new ci_scripts/pending_versions.py via uv run, outputs versions as JSON. The script also lints the YAML (tag/files pairing, duplicates).
  • update_doc.py: fills the declared entry in place (byte-for-byte otherwise); fails when the wheel's version is not declared (setuptools_scm dev versions no longer get documented) or the YAML is missing; still registers first-time packages in packages.txt.
  • generate_packages_doc.py: pending versions are skipped by the docs and the Simple API instead of raising.
  • check_versions.py: upgrade PRs append - version: lines and nothing else — no workflow bump, no Trigger: lines, no dispatch. pr-trigger.yml is deleted.
  • pr-checks.yml: lints changed package YAMLs and requires one for every changed build workflow.

Workflows (707 build-*.yml, 3 test-*.yml)
Mechanically rewritten: ${{ inputs.version || '<lit>' }}${{ matrix.version }}, workflow-level <PKG>_VERSION env moved to job level, if: needs.setup.outputs.versions != '[]' + version: matrix vector on every job (GHA rejects an empty matrix). Hand-finished:

  • sdist→bdist workflows (45): job outputs are not per matrix leg, so the sdist artifact is named <pkg>-<version>-sdist and consumers resolve it with a sdist_path step.
  • tag ≠ wheel version: 13 v-prefixed defaults move the v into ref: (their patch dirs renamed to the bare version); rtoml/dprint-py/point-cloud-utils/fabio/torch derive the tag from the version in a step.
  • build-mlx.yml/build-polars-runtime.yml build two packages; they pass the primary one and the sibling YAML must be kept in lockstep (commented).
  • torch/s5cmd keep a genuinely version-dependent job output; commented as one-version-at-a-time.

Verified on this branch

  • Every pull_request run: setup only, build/publish skipped, green.
  • gh workflow run build-ada-url.yml -f version='4.0.*' and build-cffi.yml -f version=2.1.0: matrix expands, wheels build, publish dry-run shows update_doc.py replacing the tag of the already-released entry.
  • pending_versions.py, update_doc.py (pending / released / undeclared / mid-file entry) and generate_packages_doc.py exercised locally; check_versions.py --self-test passes; actionlint reports nothing new versus main.

Known limitation: a second push touching a package's YAML before its docs PR lands rebuilds the still-pending version and creates a second release; update_doc.py keeps the latest tag.

A version entry without tag/files is pending; _setup.yml turns the pending
set (or a workflow_dispatch version glob) into a matrix that every job of a
build workflow iterates over, and _publish-wheel.yml fills in tag/files
through the docs PR. A push to main builds and publishes whatever is pending,
so no manual dispatch is needed after a merge.

Migrate build-ada-url.yml, build-cffi.yml and build-mlx.yml as pilots.

Signed-off-by: Ludovic Henry <git@ludovic.dev>
Every build-<pkg>.yml and test-<pkg>.yml now calls _setup.yml with its
package name, iterates every job over matrix.version, keys artifacts and
the publish pattern on it, and triggers on pull_request, push to main and
a workflow_dispatch version glob instead of a version literal.

Jobs cannot pass per-leg values through outputs, so the sdist->bdist
workflows name the sdist artifact after the version instead. Workflows
whose git tag differs from the wheel version (v-prefixed tags, rtoml,
fabio, dprint-py, point-cloud-utils, torch's +cpu) derive the tag from the
version; their patch directories follow the wheel version.

Signed-off-by: Ludovic Henry <git@ludovic.dev>
check_versions.py appends a bare '- version:' entry to
docs/packages/<pkg>.yaml and opens the PR; the PR's own pull_request run
builds it, so pr-trigger.yml and the workflow_dispatch calls go away.
pr-checks.yml lints changed package YAMLs and requires one for every
changed build workflow.

Signed-off-by: Ludovic Henry <git@ludovic.dev>
Signed-off-by: Ludovic Henry <git@ludovic.dev>
@luhenry
luhenry marked this pull request as ready for review September 16, 2026 09:59
@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-16 10:10 UTC

@luhenry
luhenry merged commit e7b2b20 into main Sep 16, 2026
2265 of 2267 checks passed
@luhenry
luhenry deleted the yaml-driven-versions branch September 16, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant