Skip to content

CI: build the variants from common layers, cache the layers of v5 - #428

Merged
mistraloz merged 6 commits into
thecodingmachine:v5from
mistraloz:perf/ci-layers-cache
Nov 6, 2026
Merged

mistraloz merged 6 commits into
thecodingmachine:v5from
mistraloz:perf/ci-layers-cache

Conversation

@mistraloz

@mistraloz mistraloz commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Note

The builder uses the release candidate moby/buildkit:v0.35.0-rc1 (see below): bump it in workflow.yml when
BuildKit 0.35.0 is released.

This PR divides the runner minutes of the CI by 2 to 3.5, and makes every run faster (see the measures below):

  • Bug
  • Feature
  • Breaking changes
  • The 3 variants of a PHP version are built in one job, from common layers: the base and the extensions are built once
    instead of once per variant. The extensions are installed once, then the slim part of the variant is applied on them for
    the fat image (see Contributing). The fat images no longer contain the layers of the slim
    image of their variant, their content is unchanged.
  • The tests of a job run 2 test files at a time (tests-suite/run_tests.sh), the NodeJS images are now tested.
  • The images are pushed 4 at a time, with retries (ghcr.io sometimes answers "unknown blob").
  • The layers are cached in ghcr.io/thecodingmachine/php-cache:
    • the runs of v5 read and write it, so a push or a manual run only rebuilds what changed;
    • a separate workflow (purge-cache.yml) deletes the cache each Sunday, 30 minutes before the weekly run, so that
      the images take the updates of the packages each week. It can also be run manually;
    • the pull requests only read it: no pull request can write in the cache of the published images.

BuildKit 0.34 does not find in the cache the layers of a target built from another target, with a new builder as on
GitHub. BuildKit 0.35.0-rc1 fixes it.

It also fixes mod_cgid (APACHE_EXTENSION_CGID=1), which did not start in the slim-apache image and with the
built-in Apache of the fpm variant: its socket directory was missing.

Duration of the whole workflow, from its start to its end, and runner minutes:

Run Before After
Weekly run of v5 (from an empty cache) 18 to 19 min, 211 15 min, 96 (fork)
Push or manual run during the week 18 min, 211 10.8 min, 69 (fork)
Pull request, without cache 13.2 min, 168 10.3 min, 75
Pull request, with the cache of v5 7.9 min, 58 (fork)

"Before" is measured on the last runs of this repository (weekly runs of October and November, CI of #429): their
24 build jobs waited up to 7 min in the queue. The new workflow runs 8 jobs at a time, they start within 5 s. The
cache does not exist in this repository yet, so the runs marked "fork" were measured on a fork (with the same queue
time). The purge of the cache (about 1 min) runs 30 min before the weekly run, it is not counted.

Build of PHP 8.4 measured locally with a new builder: 753 s of CPU without cache, 297 s with the cache.

Test plan (required)

  • The images of PHP 8.4 built before and after were compared (configuration and files): only the socket
    directory of mod_cgid differs.
  • New test test_cgid (variant-apache.sh), it fails on the current images.
  • The whole test suite passes on the 4 PHP versions, locally and in the runs of the fork.
  • The cache was checked on the fork: the purge workflow empties it, the next run fills it, a later run and a pull request
    reuse it. Locally, a new builder finds the 195 build steps of PHP 8.4 in the cache.
  • test_dockerStop waited 3 seconds before stopping the container: it now waits until its processes are started (it
    failed once on a loaded runner).

Checklist

  • I followed the guidelines in CONTRIBUTING guide
  • I have squashed any insignificant commits
  • This change has comments for package types, values, functions, and non-obvious lines of code

@mistraloz
mistraloz force-pushed the perf/ci-layers-cache branch from 386329f to a6e5052 Compare November 5, 2026 21:12
Apache is started without apache2ctl, which creates it: mod_cgid failed to start in the slim-apache image and with the
built-in Apache of the fpm variant. The fat apache image only had it because an extension package started apache2ctl
during its installation.
The slim Dockerfiles start from Dockerfile.base (the layers common to all the images). The extensions are installed
once on Dockerfile.base (Dockerfile.extensions), then the slim Dockerfile of each variant is applied on them to build
the fat images: the extensions were installed 3 times per PHP version and architecture. The images keep the same
content (configuration, files, packages and PHP extensions of each SAPI compared on PHP 8.4).

The CI builds and tests a PHP version and an architecture per job (8 jobs instead of 24) and runs the test files in
parallel (tests-suite/run_tests.sh). The NodeJS images are now tested: variant-node.sh was run without NODE.
A job publishes the 15 images of a PHP version and an architecture: pushing them one by one took about 3 minutes.
… week

The runs of v5 read and write the layers in a registry (ghcr.io/<owner>/php-cache): a push or a manual run only
rebuilds what changed. A separate workflow deletes the versions of the cache each week, 30 minutes before the
scheduled run of v5 (or when run manually), so that the images take the updates of the packages each week. The pull
requests only read the cache of v5, which no pull request can write. The GitHub Actions cache is not used: limited to
10 GB, and slow to write by 8 jobs at the same time.

BuildKit 0.34 does not find in the cache the layers of a target built from another target (contexts = target:...)
with a new builder, as on GitHub: the builder uses BuildKit 0.35.
The test waited 3 seconds: on a loaded runner, the container was still starting, and the entrypoint only handles the
stop signal at the end of its current step.
@mistraloz
mistraloz force-pushed the perf/ci-layers-cache branch from a6e5052 to bdff28f Compare November 6, 2026 08:39
@mistraloz
mistraloz merged commit 9cdb470 into thecodingmachine:v5 Nov 6, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant