Drop the automatic GitHub Actions cache, add a parallel build keeping the cache of services sharing a stage

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## Unreleased
+
+### Added
+
+* [`docker:build --parallel`](tasks.md#castor-dockerbuild) and the `build_parallel` context variable: each service builds on its own, in parallel, so services sharing a stage keep their build cache ([moby/buildkit#6418](https://github.com/moby/buildkit/issues/6418)). `docker:push` follows them too.
+
+### Removed
+
+* The GitHub Actions cache 0.9.0 added to every build on GitHub Actions. The registry cache misses it made up for come from a BuildKit race between caches describing the same stages, which a second cache only makes worse, and the `docker` driver of the runners failed the build exporting it. A service still takes any cache given to its `cacheFrom()`.
+
 ## 0.9.0 - 2026-10-06
 
 ### Added

doc/configuration.md

@@ -33,6 +33,7 @@ function default_context(): Context
 | `resolve_domains_via_host` | Whether the containers resolve the project's own public domains, `true` by default — see [below](#resolving-your-own-domains-from-a-container) |
 | `docker_profiles` | The compose profiles every task activates by default, `['default']` otherwise |
 | `router_autostart` | Whether `docker:up` starts the global router and `docker:stop` stops it, `true` by default — see [below](#starting-and-stopping-the-router-with-your-projects) |
+| `build_parallel` | Whether the services are built on their own, in parallel, so services sharing a stage keep their build cache, `false` by default — see [`docker:build`](tasks.md#castor-dockerbuild) |
 | `worktree` | The git worktree this checkout is, detected on boot — see [git worktrees](going-further/worktrees.md) |
 | `worktree_isolation` | Whether a git worktree runs a stack of its own, `true` by default |
 | `worktree_directory` | Where `worktree:create` checks a worktree out |

doc/tasks.md

@@ -58,6 +58,19 @@ castor docker:build app                 # a single service
 castor docker:build --profiles builder  # restrict to a profile
 ```
 
+`--parallel` builds each service on its own, all of them in parallel, and
+prints the log of each one once it is done. Built together, services sharing a
+stage — an application and its builder sharing `php-base` — race their build
+caches, and BuildKit randomly rebuilds the steps of all but one of them
+([moby/buildkit#6418](https://github.com/moby/buildkit/issues/6418)). Turn it on
+for a whole project with the `build_parallel` context variable: `castor docker:up
+--build` and `castor docker:push` follow it too, `--no-parallel` turns it off
+for a single command.
+
+```bash
+castor docker:build --parallel
+```
+
 Alias: `castor build`.
 
 ### `castor docker:up`
@@ -260,37 +273,6 @@ castor docker:push --tag "$(git rev-parse --short HEAD)"
 castor docker:push --dry-run
 ```
 
-#### On GitHub Actions
-
-When `GITHUB_ACTIONS` is set, every registry cache gets a
-[GitHub Actions cache](https://docs.docker.com/build/cache/backends/gha/) next
-to it, scoped by the cache name: every build reads both and writes the GitHub
-one, `castor docker:push` writes both. BuildKit sometimes misses layers of a
-registry cache without a word, the second one makes up for it.
-
-What a pull request writes stays with it: its next runs reuse it, the default
-branch never sees it. The layers it shares with the default branch are not
-stored twice.
-
-Buildx reaches that cache with variables Actions only hands to JavaScript
-actions, so every job that builds has to expose them first, reading the cache
-needing them as much as writing it:
-
-```yaml
-- uses: actions/github-script@v9
-  with:
-    script: |
-      core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN)
-      core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL)
-      core.exportVariable('ACTIONS_CACHE_SERVICE_V2', process.env.ACTIONS_CACHE_SERVICE_V2 || 'true')
-
-- run: castor docker:build
-```
-
-Without them buildx skips the cache silently, and the build only uses the
-registry. A matrix building several variants of the same service has to give
-each one its own cache name, or the jobs overwrite each other's cache.
-
 #### Publishing to ghcr.io
 
 GitHub attaches a package to a repository in two cases only: a push from a

phpstan.neon

@@ -13,6 +13,7 @@ parameters:
                 docker_profiles?: string[],
                 resolve_domains_via_host?: bool,
                 router_autostart?: bool,
+                build_parallel?: bool,
                 worktree?: ?string,
                 worktree_isolation?: bool,
                 worktree_directory?: string,

phpunit.xml.dist

@@ -6,11 +6,6 @@
          cacheDirectory=".phpunit.cache"
          failOnRisky="true"
 >
-    <php>
-        <!-- Adds a GitHub Actions cache to the generated compose files. -->
-        <env name="GITHUB_ACTIONS" value="" force="true"/>
-    </php>
-
     <testsuites>
         <testsuite name="unit">
             <directory>tests/Unit</directory>

src/Service/Builder/BuildBuilder.php

@@ -26,8 +26,6 @@ final class BuildBuilder
     private array $additionalContexts = [];
     /** @var array<string> */
     private array $cacheFrom = [];
-    /** @var array<string> */
-    private array $cacheTo = [];
 
     public function __construct(private readonly ServiceBuilder $serviceBuilder) {}
 
@@ -40,7 +38,6 @@ public function clone(ServiceBuilder $serviceBuilder): self
         $new->args = $this->args;
         $new->additionalContexts = $this->additionalContexts;
         $new->cacheFrom = $this->cacheFrom;
-        $new->cacheTo = $this->cacheTo;
 
         return $new;
     }
@@ -101,24 +98,13 @@ public function cacheFrom(string $image): self
     public function noCacheFrom(): self
     {
         $this->cacheFrom = [];
-        $this->cacheTo = [];
 
         return $this;
     }
 
     public function withRegistryCache(string $image): self
     {
         $this->cacheFrom = ['type=registry,ref=${REGISTRY:-}/' . $image . ':cache'];
-        $this->cacheTo = [];
-
-        // BuildKit silently misses layers of a registry cache on CI
-        // (jolicode/docker-starter#430), the GitHub Actions one backs it up.
-        // Every build writes it: GitHub keeps what a pull request writes to
-        // that pull request, whose next runs reuse it.
-        if ('true' === getenv('GITHUB_ACTIONS')) {
-            $this->cacheFrom[] = 'type=gha,scope=' . $image;
-            $this->cacheTo = ['type=gha,scope=' . $image . ',mode=max,ignore-error=true'];
-        }
 
         return $this;
     }
@@ -159,10 +145,6 @@ public function toArray(): array
             $build['cache_from'] = $this->cacheFrom;
         }
 
-        if (!empty($this->cacheTo)) {
-            $build['cache_to'] = $this->cacheTo;
-        }
-
         return $build;
     }
 }

src/tasks.php

@@ -8,6 +8,7 @@
 use Castor\Attribute\AsOption;
 use Castor\Attribute\AsRawTokens;
 use Castor\Attribute\AsTask;
+use Castor\Context;
 use Castor\Console\Output\VerbosityLevel;
 use Castor\Docker\Doctor\Doctor;
 use Castor\Docker\Doctor\HostSystemProbe;
@@ -19,6 +20,7 @@
 use Symfony\Component\Console\Formatter\OutputFormatter;
 use Symfony\Component\Console\Helper\TableSeparator;
 use Symfony\Component\Console\Input\InputOption;
+use Symfony\Component\Console\Output\OutputInterface;
 use Symfony\Component\Console\Terminal;
 use Symfony\Component\Process\Exception\ExceptionInterface;
 use Symfony\Component\Process\Process;
@@ -27,6 +29,7 @@
 use function Castor\capture;
 use function Castor\context;
 use function Castor\io;
+use function Castor\parallel;
 use function Castor\variable;
 use function Castor\run;
 
@@ -39,6 +42,8 @@ function build(
     ?string $service = null,
     #[AsOption(mode: InputOption::VALUE_IS_ARRAY | InputOption::VALUE_REQUIRED)]
     array $profiles = [],
+    #[AsOption(description: 'Build each service on its own, in parallel, so services sharing a stage keep their build cache', mode: InputOption::VALUE_NONE | InputOption::VALUE_NEGATABLE)]
+    ?bool $parallel = null,
 ): void {
     io()->title('Building infrastructure');
 
@@ -56,16 +61,96 @@ function build(
         $command[] = "{$key}={$value}";
     }
 
-    if ($service) {
-        $command[] = $service;
-    }
-
     if (!$profiles) {
         $profiles = get_default_profiles();
         $profiles[] = 'builder';
     }
 
-    docker_compose($command, profiles: $profiles);
+    $services = $service || !is_build_parallel($parallel) ? [] : get_buildable_services($profiles);
+
+    if (\count($services) < 2) {
+        docker_compose($service ? [...$command, $service] : $command, profiles: $profiles);
+
+        return;
+    }
+
+    build_separately($command, $services, $profiles);
+}
+
+/**
+ * Off by default: the logs of the services only show once each one is built.
+ */
+function is_build_parallel(?bool $option = null, ?Context $c = null): bool
+{
+    return $option ?? (($c ?? context())->data['build_parallel'] ?? false);
+}
+
+/**
+ * One build per service, run in parallel: in a single session BuildKit merges
+ * the stages services share, then races the caches of those services over them
+ * and randomly loses the steps of all but one (moby/buildkit#6418).
+ *
+ * @param list<string> $command
+ * @param list<string> $services
+ * @param list<string> $profiles
+ */
+function build_separately(array $command, array $services, array $profiles): void
+{
+    $c = context()->withQuiet()->withAllowFailure()->withTty(false)->withPty(false);
+
+    $processes = parallel(...array_map(
+        static fn(string $service): \Closure => static fn(): Process => docker_compose([...$command, $service], $c, $profiles, 'plain'),
+        $services,
+    ));
+
+    report_separate_builds($services, $processes);
+}
+
+/**
+ * @param list<string>  $services
+ * @param array<mixed>  $processes
+ */
+function report_separate_builds(array $services, array $processes): void
+{
+    $failed = [];
+
+    foreach ($services as $i => $service) {
+        $process = $processes[$i];
+
+        if (!$process instanceof Process) {
+            $failed[] = $service;
+
+            continue;
+        }
+
+        io()->section($service);
+        io()->write($process->getOutput() . $process->getErrorOutput(), false, OutputInterface::OUTPUT_RAW);
+
+        if (!$process->isSuccessful()) {
+            $failed[] = $service;
+        }
+    }
+
+    if ($failed) {
+        throw new \RuntimeException(\sprintf('The build of %s failed.', implode(', ', $failed)));
+    }
+}
+
+/**
+ * @param list<string> $profiles
+ *
+ * @return list<string>
+ */
+function get_buildable_services(array $profiles): array
+{
+    $config = json_decode(
+        docker_compose(['config', '--format', 'json'], context()->withQuiet(), profiles: $profiles)->getOutput(),
+        true,
+    );
+
+    $services = \is_array($config['services'] ?? null) ? $config['services'] : [];
+
+    return array_map('strval', array_keys(array_filter($services, static fn(mixed $config): bool => \is_array($config) && isset($config['build']))));
 }
 
 /**
@@ -689,6 +774,8 @@ function push(
     bool $dryRun = false,
     #[AsOption(description: 'The tag the images are published under')]
     string $tag = 'latest',
+    #[AsOption(description: 'Build each target on its own, in parallel, so targets sharing a stage keep their build cache', mode: InputOption::VALUE_NONE | InputOption::VALUE_NEGATABLE)]
+    ?bool $parallel = null,
 ): void {
     $registry = variable('registry');
 
@@ -730,16 +817,17 @@ function push(
     // apply: bake sees every service that has a "build".
     $command = ['docker', 'buildx', 'bake', '-f', $c->workingDirectory . '/compose.yaml'];
 
+    $options = [];
+
     foreach ($targets as $service => $cacheEntries) {
         $images = [];
+        $options[$service] = [];
 
         foreach ($cacheEntries as $cacheTo) {
             $cacheRef = get_cache_reference($cacheTo);
 
-            $command[] = '--set';
-            // A GitHub Actions cache only backs the registry one up: its
-            // outage must not keep the images from being pushed.
-            $command[] = \sprintf('%s.cache-to=%s,mode=max%s', $service, $cacheTo, str_contains($cacheTo, 'type=gha') ? ',ignore-error=true' : '');
+            $options[$service][] = '--set';
+            $options[$service][] = \sprintf('%s.cache-to=%s,mode=max', $service, $cacheTo);
 
             // "type=gha" or "type=local" names no repository to publish an
             // image to.
@@ -763,32 +851,26 @@ function push(
         }
 
         foreach ($images as $image) {
-            $command[] = '--set';
-            $command[] = \sprintf('%s.tags=%s', $service, $image);
+            $options[$service][] = '--set';
+            $options[$service][] = \sprintf('%s.tags=%s', $service, $image);
         }
 
         // Per target, rather than a global "--push": the services whose cache
         // is not a registry one must not be pushed anywhere.
-        $command[] = '--set';
-        $command[] = \sprintf('%s.output=type=registry', $service);
+        $options[$service][] = '--set';
+        $options[$service][] = \sprintf('%s.output=type=registry', $service);
         if (null !== $source) {
-            $command[] = '--set';
-            $command[] = \sprintf('%s.labels.org.opencontainers.image.source=%s', $service, $source);
+            $options[$service][] = '--set';
+            $options[$service][] = \sprintf('%s.labels.org.opencontainers.image.source=%s', $service, $source);
         }
 
         if (null !== $revision) {
-            $command[] = '--set';
-            $command[] = \sprintf('%s.labels.org.opencontainers.image.revision=%s', $service, $revision);
+            $options[$service][] = '--set';
+            $options[$service][] = \sprintf('%s.labels.org.opencontainers.image.revision=%s', $service, $revision);
         }
     }
 
-    if ($dryRun) {
-        $command[] = '--print';
-    }
-
-    // Naming the targets keeps out the services that build without a cache:
-    // bake's default group is every buildable service of the project.
-    run([...$command, ...array_keys($targets)], context: $c->withEnvironment([
+    $c = $c->withEnvironment([
         // bake does not go through docker_compose(), so the variables the
         // generated compose file interpolates have to be given to it here.
         'COMPOSE_PROJECT_NAME' => get_project_name($c),
@@ -797,7 +879,26 @@ function push(
         // The build contexts live in the plugin, outside of the project
         // directory, which bake asks to confirm on every run otherwise.
         'BUILDX_BAKE_ENTITLEMENTS_FS' => '0',
-    ]));
+    ]);
+
+    // Naming the targets keeps out the services that build without a cache:
+    // bake's default group is every buildable service of the project.
+    if ($dryRun || !is_build_parallel($parallel)) {
+        run([...$command, ...array_merge(...array_values($options)), ...($dryRun ? ['--print'] : []), ...array_keys($targets)], context: $c);
+
+        return;
+    }
+
+    // One bake per target, for the reason build_separately() gives.
+    $quiet = $c->withQuiet()->withAllowFailure()->withTty(false)->withPty(false);
+    $services = array_keys($targets);
+
+    $processes = parallel(...array_map(
+        static fn(string $service): \Closure => static fn(): Process => run([...$command, ...$options[$service], '--progress', 'plain', $service], context: $quiet),
+        $services,
+    ));
+
+    report_separate_builds($services, $processes);
 }
 
 /**