Reduce build times with sccache by josecelano · Pull Request #1905 · torrust/torrust-tracker

josecelano · 2026-06-11T16:36:35Z

Evidence-driven research for issue #1726.

Summary

This PR investigates whether sccache can reduce CI build times for the Torrust Tracker workspace. After controlled benchmarks in three contexts — local development, GHA bare builds, and GHA Docker builds — the final decision is documented in ADR docs/adrs/20260612000000_adopt_sccache_for_ci_bare_builds.md.

Final decisions:

Context	Decision	Reasoning
Local development	❌ Reject	Cold build 22 % slower; warm builds no better due to sccache overhead
Docker CI builds	❌ Reject	No measurable benefit on GHA runners (29 vs 30 min, within noise)
Bare CI builds	✅ Adopt	93.38 % hit rate; cold build ~60 % faster (479 s → 192 s)

Changes

Production CI (5 workflow files)

.github/workflows/testing.yaml — add sccache to unit, test-doc, clippy, fmt jobs
.github/workflows/os-compatibility.yaml — add sccache to build job
.github/workflows/db-compatibility.yaml — add sccache to build jobs
.github/workflows/coverage.yaml — add sccache, remove Swatinem/rust-cache
.github/workflows/db-benchmarking.yaml — add sccache, remove Swatinem/rust-cache

Experiment artifacts (archived under `contrib/dev-tools/experiments/sccache-docker/`)

3 local Docker build strategies (basic, multi-stage, GHA-backend) with full reports
2 experiment workflows: experiment-sccache-bare-build.yaml, experiment-sccache-docker.yaml
Containerfile.sccache-experiment for sccache-in-Docker experiments
Raw results: GHA cold/warm timing, Docker cold/warm timing, A/B benchmark report, compile hotspot analysis, Q&A

Documentation

ADR: docs/adrs/20260612000000_adopt_sccache_for_ci_bare_builds.md
Issue spec: docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md — all tasks checked complete
Reports: sccache A/B report, GHA experiment results, Docker GHA experiment results, compile hotspot analysis, Q&A
Skills: semantic-skill-link-convention.md, git hook skills updated (TORRUST_GIT_HOOKS_LOG_DIR=.tmp)

Drive-by fixes

Fix clippy::result_large_err by boxing large HandlerError tuple in packages/udp-server/src/handlers/
Suppress clippy::extra_unused_lifetimes on #[automock] traits in packages/tracker-core/src/databases/traits/
Fix clippy dead match arm and box large error variant in console/tracker-client/src/console/clients/

Checklist

Task 0: Local branch
Task 1: Local A/B benchmarks — sccache rejected for local development
Task 2: Local configuration decision — do not enable
Task 3a: GHA bare build experiment — 93.38 % hit rate, ~60 % reduction ✅
Task 3b: Docker build experiment — no measurable benefit, rejected ❌
Task 3c: Full E2E with sccache-warmed Docker — merged into Task 3b
Task 3d: Final decision documented — adopt sccache for bare CI, reject for Docker/local

See full spec: docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md

Update related-pr to point to PR 1905 for experiment tracking.

codecov · 2026-06-11T16:54:30Z

Codecov Report

❌ Patch coverage is 50.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.10%. Comparing base (126e746) to head (3538730).

Files with missing lines	Patch %	Lines
.../tracker-client/src/console/clients/udp/checker.rs	25.00%	3 Missing ⚠️
...-client/src/console/clients/checker/monitor/udp.rs	0.00%	2 Missing ⚠️
packages/udp-server/src/handlers/scrape.rs	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1905      +/-   ##
===========================================
- Coverage    79.17%   79.10%   -0.07%     
===========================================
  Files          326      326              
  Lines        22801    22804       +3     
  Branches     22801    22804       +3     
===========================================
- Hits         18052    18039      -13     
- Misses        4494     4501       +7     
- Partials       255      264       +9

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Update related-pr to point to PR 1905 for experiment tracking.

Cold build: 479.44 s (5.52 % cache hits, 133 write errors) Warm-after-change: 153.86 s (6.96 % cache hits) Key finding: the warm-after-change speedup comes from Cargo's own dependency tracking, not from sccache. Cross-run cache test (Run 2) is still pending.

Cross-run cache test confirms sccache GHA backend works well: - Cold with cache restore: 192.21 s (93.38 % hits, vs 479.44 s without) - Warm-after-change: 137.35 s (93.48 % hits) - Task 3a complete Proceeding to Task 3b: sccache inside Docker build.

Experiment 1: sccache with BuildKit cache mount works within single build Experiment 2: BuildKit cache mounts are stage-scoped, not shared across stages Experiment 3: SCCACHE_GHA_ENABLED=true fails hard without GHA creds; local disk fallback works Key finding: GHA backend must be passed into Docker via --secret-env. The Containerfile must NOT hardcode SCCACHE_GHA_ENABLED=true as it breaks local builds.

Three local Docker experiments conducted to determine best sccache integration strategy: - Exp 1: BuildKit cache mounts work within single build - Exp 2: Cache mounts are stage-scoped, not shared across stages - Exp 3: SCCACHE_GHA_ENABLED=true fails hard without GHA creds Decision: Use B2 (GHA backend passed via --secret-env into Docker build) instead of B1 (mount host cache), which was found infeasible.

- Created Containerfile.sccache-experiment with sccache in chef stage - Built chef stage locally: sccache 0.15.0 + cargo-chef verified - Remaining Task 3b items (GHA workflow, cross-run test) pending as they require GitHub Actions runner access - Updated ISSUE.md with current progress checkboxes

Reviewed Docker cache optimization docs, BuildKit docs, GHA cache backend docs, and sccache docs. Key insight: container.yaml already uses cache-from/ cache-to: type=gha, mode=max for BuildKit layer caching. sccache would be a second caching layer inside Docker layers — only helpful when Cargo.lock changes but individual crate sources don't. Decision remains B2 (GHA backend via --secret-env) but added github-token to mitigate cache API throttling risk. Strategy question: whether sccache inside Docker adds value beyond the already- working BuildKit layer cache will be answered by the experiment data.

- Containerfile.sccache-experiment: ARG for GHA credentials - New experiment-sccache-docker.yaml (Docker build + E2E) - Local Docker stage timings: third-party deps 52.75 s - Need GHA runner for cold/warm comparison Note: build-args for GHA tokens trigger Docker security warnings. Acceptable for experiment branches. Production would use --secret-env.

Applied sccache to all workflow jobs that compile Rust directly on GHA runners: - testing.yaml: unit + docker-e2e jobs - os-compatibility.yaml: build job (6 matrix combinations) - db-compatibility.yaml: mysql + postgres jobs - coverage.yaml: report job - db-benchmarking.yaml: sqlite3 + mysql + postgres jobs ADR docs/adrs/20260612000000_adopt_sccache_for_ci_bare_builds.md documents the decision: - Adopt sccache for bare CI builds (93.38 % hit rate, -60 % build time) - Reject sccache for local development (cold: +22 % slower) - Reject sccache for Docker builds (no benefit on GHA runners) Workflow files reference the ADR via '# adr:' comments.

Added guidance to semantic-skill-link-convention.md for: - Cross-referencing ADRs using frontmatter related-artifacts with issue #N - Linking workflows to ADRs via '# adr:' comments - Stability warning about issue spec file paths vs issue numbers

Added explanation that YAML frontmatter (--- delimiter) would create a second YAML document, and that the workflow schema rejects unknown top-level keys.

Updated run-pre-commit-checks, run-pre-push-checks, and commit-changes skills to instruct AI agents to set TORRUST_GIT_HOOKS_LOG_DIR=.tmp so per-step log files stay inside the workspace (git-ignored .tmp/) instead of /tmp.

Repo cache at 13.03 GB / 10 GB (over limit). sccache proved superior (93.38 % hit rate vs ~0 % for Swatinem). Removing Swatinem/rust-cache to free cache space and avoid eviction of sccache entries. Removed from: testing.yaml (unit + docker-e2e), db-compatibility.yaml (mysql + postgres), coverage.yaml, db-benchmarking.yaml (sqlite3 + mysql + postgres)

Copilot

Pull request overview

This PR documents and operationalizes the outcome of issue #1726’s sccache investigation: rejecting sccache for local development and Docker-based CI builds, while adopting it for “bare” (non-Docker) GitHub Actions jobs to reduce repeat-run compile time via the GHA backend.

Changes:

Adds benchmark reports + analysis docs (local A/B, GHA bare build results, Docker build experiment results) and an ADR capturing the final decision.
Updates multiple CI workflows to replace Swatinem/rust-cache with mozilla-actions/sccache-action + RUSTC_WRAPPER=sccache for bare Rust compilation steps.
Archives experiment workflows / Containerfile variants and supporting Docker experiments under contrib/dev-tools/experiments/.

Reviewed changes

Copilot reviewed 37 out of 41 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
project-words.txt	Adds new spellcheck dictionary words used by the new docs/CI experiment writeups.
docs/skills/semantic-skill-link-convention.md	Documents how to semantically link long-lived artifacts (ADRs/workflows/issues), including ADR references from workflows.
docs/issues/open/1726-1840-workflow-performance-sccache/sccache-a-b-report.md	Adds the local A/B benchmark report with commands, timings, and sccache stats.
docs/issues/open/1726-1840-workflow-performance-sccache/Q-and-A.md	Adds a Q&A log for reviewer questions and evidence-driven answers.
docs/issues/open/1726-1840-workflow-performance-sccache/ISSUE.md	Updates the research spec with completed tasks, results, and the final recommendation/decision trail.
docs/issues/open/1726-1840-workflow-performance-sccache/experiment-results-gha.md	Captures GHA bare-build experiment timings/stats for cross-run sccache caching.
docs/issues/open/1726-1840-workflow-performance-sccache/experiment-docker-gha-results.md	Captures GHA Docker-build experiment results and conclusions.
docs/issues/open/1726-1840-workflow-performance-sccache/compile-hotspot-analysis.md	Updates baseline timings and revises recommendations based on measured A/B data.
docs/adrs/20260612000000_adopt_sccache_for_ci_bare_builds.md	New ADR formalizing “adopt for bare CI only; reject locally and for Docker builds”.
contrib/dev-tools/experiments/sccache-docker/test-crate/src/main.rs	Test crate used by Docker/sccache experiments.
contrib/dev-tools/experiments/sccache-docker/test-crate/Cargo.toml	Test crate manifest for Docker/sccache experiments.
contrib/dev-tools/experiments/sccache-docker/test-crate/Cargo.lock	Locked dependency set for reproducible Docker/sccache experiments.
contrib/dev-tools/experiments/sccache-docker/README.md	Explains experiment structure, how to run, and how results are tracked.
contrib/dev-tools/experiments/sccache-docker/Dockerfile.test	Minimal BuildKit-cache-mount Dockerfile to validate sccache behavior in-container.
contrib/dev-tools/experiments/sccache-docker/04-gha-workflow-experiments/REPORT.md	Summarizes the GHA workflow experiments and the key fixes discovered.
contrib/dev-tools/experiments/sccache-docker/04-gha-workflow-experiments/experiment-sccache-docker.yaml	Archived workflow used to benchmark sccache inside Docker builds on GHA.
contrib/dev-tools/experiments/sccache-docker/04-gha-workflow-experiments/experiment-sccache-bare-build.yaml	Archived workflow used to benchmark bare `cargo build --release` with sccache on GHA.
contrib/dev-tools/experiments/sccache-docker/04-gha-workflow-experiments/Containerfile.sccache-experiment	Archived Containerfile variant used to test sccache inside Docker build stages.
contrib/dev-tools/experiments/sccache-docker/03-gha-backend/src/main.rs	Test program for validating sccache behavior under “GHA backend enabled” scenarios.
contrib/dev-tools/experiments/sccache-docker/03-gha-backend/REPORT.md	Documents the “fails hard without creds” behavior and the corrected approach.
contrib/dev-tools/experiments/sccache-docker/03-gha-backend/Dockerfile	Docker experiment to validate secret-based credential passing strategy.
contrib/dev-tools/experiments/sccache-docker/03-gha-backend/Cargo.toml	Manifest for the GHA-backend experiment crate.
contrib/dev-tools/experiments/sccache-docker/03-gha-backend/Cargo.lock	Locked deps for the GHA-backend experiment crate.
contrib/dev-tools/experiments/sccache-docker/02-multi-stage/src/main.rs	Test program for multi-stage build experiments.
contrib/dev-tools/experiments/sccache-docker/02-multi-stage/REPORT.md	Documents findings about cache-mount scoping across Docker stages.
contrib/dev-tools/experiments/sccache-docker/02-multi-stage/Dockerfile	Multi-stage Docker experiment mirroring the production Containerfile structure.
contrib/dev-tools/experiments/sccache-docker/02-multi-stage/Cargo.toml	Manifest for multi-stage experiment crate.
contrib/dev-tools/experiments/sccache-docker/02-multi-stage/Cargo.lock	Locked deps for multi-stage experiment crate.
contrib/dev-tools/experiments/sccache-docker/01-basic-build/src/main.rs	Test program for the basic BuildKit cache-mount experiment.
contrib/dev-tools/experiments/sccache-docker/01-basic-build/REPORT.md	Records timings/stats and conclusions for the basic Docker experiment.
contrib/dev-tools/experiments/sccache-docker/01-basic-build/Dockerfile	Basic Docker experiment verifying sccache + cache mounts.
contrib/dev-tools/experiments/sccache-docker/01-basic-build/Cargo.toml	Manifest for basic-build experiment crate.
contrib/dev-tools/experiments/sccache-docker/01-basic-build/Cargo.lock	Locked deps for basic-build experiment crate.
.github/workflows/testing.yaml	Enables sccache GHA backend for bare compilation steps; removes rust-cache in affected jobs.
.github/workflows/os-compatibility.yaml	Enables sccache for cross-run caching on OS compatibility builds.
.github/workflows/db-compatibility.yaml	Enables sccache for DB compatibility jobs to reduce repeat-run compile time.
.github/workflows/db-benchmarking.yaml	Enables sccache for DB benchmarking jobs; removes rust-cache steps.
.github/workflows/coverage.yaml	Enables sccache in coverage job (keeping `CARGO_INCREMENTAL=0` from job env).
.github/skills/dev/git-workflow/run-pre-push-checks/SKILL.md	Updates instructions to set `TORRUST_GIT_HOOKS_LOG_DIR=.tmp` for agents.
.github/skills/dev/git-workflow/run-pre-commit-checks/SKILL.md	Same `.tmp` logging guidance for pre-commit checks.
.github/skills/dev/git-workflow/commit-changes/SKILL.md	Same `.tmp` logging guidance for commit workflow.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Fix project-words.txt alphabetical order (endgroup/endianness) - Fix Q&A: clarify token scope ≠ cross-run block (Task 3a proved 93% hits work) - Fix experiment-results-gha: 'incremental compilation' → 'dependency fingerprinting' - Fix experiment-results-gha: remove TBD placeholder table at end - Fix ISSUE.md: replace [- ] with [ ] ~strikethrough~ for GitHub task list compatibility - Fix sccache-a-b-report: mark cargo install sccache section as skipped - Fix testing.yaml ADR comment to mention Swatinem removal

…and tracker-client - packages/udp-server/: box large HandlerError tuple to fix clippy::result_large_err - packages/tracker-core/: suppress clippy::extra_unused_lifetimes on #[automock] traits - console/tracker-client/: box large Error variant and fix dead match arm - docs/issues/open/1726-1840-workflow-performance-sccache/: fix typo in checkbox PR: torrust#1905

josecelano · 2026-06-14T06:08:25Z

ACK 3538730

josecelano temporarily deployed to coverage June 11, 2026 16:36 — with GitHub Actions Inactive

josecelano added a commit to josecelano/torrust-tracker that referenced this pull request Jun 11, 2026

docs(build): link draft PR torrust#1905 in issue spec

ad135df

Update related-pr to point to PR 1905 for experiment tracking.

josecelano temporarily deployed to coverage June 11, 2026 16:37 — with GitHub Actions Inactive

josecelano self-assigned this Jun 11, 2026

josecelano temporarily deployed to coverage June 11, 2026 16:58 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 11, 2026 17:13 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 11, 2026 18:00 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 11, 2026 18:11 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 11, 2026 19:25 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 12, 2026 06:12 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 12, 2026 07:23 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 12, 2026 07:32 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 12, 2026 08:31 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 12, 2026 09:15 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 12, 2026 17:43 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 12, 2026 17:45 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 12, 2026 17:47 — with GitHub Actions Inactive

josecelano added 12 commits June 12, 2026 18:50

docs(build): record local sccache A/B benchmark results

6a0e211

docs(build): document local sccache decision (reject)

1eef743

docs(build): refine task 3 strategy with B1/B2 rationale

0132701

ci(experiment): add sccache bare build workflow (task 3a)

b281768

docs(build): link draft PR torrust#1905 in issue spec

ab849b1

Update related-pr to point to PR 1905 for experiment tracking.

josecelano added 3 commits June 12, 2026 18:50

docs(project): clarify why workflows cannot use frontmatter

b093328

Added explanation that YAML frontmatter (--- delimiter) would create a second YAML document, and that the workflow schema rejects unknown top-level keys.

josecelano force-pushed the 1726-reduce-build-times-sccache branch from f7bec0b to b093328 Compare June 12, 2026 17:51

josecelano temporarily deployed to coverage June 12, 2026 17:51 — with GitHub Actions Inactive

docs(skills): add TORRUST_GIT_HOOKS_LOG_DIR=.tmp for AI agents

691968b

Updated run-pre-commit-checks, run-pre-push-checks, and commit-changes skills to instruct AI agents to set TORRUST_GIT_HOOKS_LOG_DIR=.tmp so per-step log files stay inside the workspace (git-ignored .tmp/) instead of /tmp.

josecelano temporarily deployed to coverage June 12, 2026 18:05 — with GitHub Actions Inactive

docs(project): fix spelling in skill-link-convention

48d0e34

josecelano temporarily deployed to coverage June 12, 2026 18:08 — with GitHub Actions Inactive

docs(build): record Q&A about cache limits and Swatinem removal

3871822

josecelano temporarily deployed to coverage June 12, 2026 18:21 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 12, 2026 18:31 — with GitHub Actions Inactive

josecelano marked this pull request as ready for review June 12, 2026 18:33

josecelano requested a review from a team as a code owner June 12, 2026 18:33

Copilot AI review requested due to automatic review settings June 12, 2026 18:33

Copilot started reviewing on behalf of josecelano June 12, 2026 18:34 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

josecelano temporarily deployed to coverage June 12, 2026 19:02 — with GitHub Actions Inactive

ci: trigger workflows after allowlist fix

909231a

josecelano temporarily deployed to coverage June 13, 2026 08:07 — with GitHub Actions Inactive

ci: retry with sccache-action wildcard allowed

97fb88d

josecelano temporarily deployed to coverage June 13, 2026 08:36 — with GitHub Actions Inactive

josecelano temporarily deployed to coverage June 13, 2026 22:02 — with GitHub Actions Inactive

josecelano changed the title ~~WIP: Reduce build times with sccache (experiment)~~ Reduce build times with sccache Jun 14, 2026

josecelano merged commit 567e216 into torrust:develop Jun 14, 2026
25 of 26 checks passed

josecelano mentioned this pull request Jun 14, 2026

Research: Reduce Build Times with sccache #1726

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce build times with sccache#1905

Reduce build times with sccache#1905
josecelano merged 29 commits into
torrust:developfrom
josecelano:1726-reduce-build-times-sccache

josecelano commented Jun 11, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jun 11, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

josecelano commented Jun 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

josecelano commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Production CI (5 workflow files)

Experiment artifacts (archived under contrib/dev-tools/experiments/sccache-docker/)

Documentation

Drive-by fixes

Checklist

Uh oh!

codecov Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

josecelano commented Jun 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

josecelano commented Jun 11, 2026 •

edited

Loading

Experiment artifacts (archived under `contrib/dev-tools/experiments/sccache-docker/`)

codecov Bot commented Jun 11, 2026 •

edited

Loading