We set rust_cache_root=/cache on a workstation so the Rust build caches live on
its dedicated cache volume, and it only moved part of them. sccache kept its
default ~/.cache/sccache and sat on the root filesystem holding 22G. That was a
decent chunk of why root got to 84%.
Where rust_cache_root actually lands:
- the pooled cargo build-dir, via
hyperi-rust-setup --cache-root (rust.yml:642) -- becomes <root>/hyperi-rust-build
- the prune and the prune guard, via
--cache-root in their unit and plist templates
Where it does not:
- sccache --
hyperi-sccache.service.j2 sets SCCACHE_CACHE_SIZE and SCCACHE_BASEDIRS but never SCCACHE_DIR, so the server falls back to ~/.cache/sccache
- ccache -- the role writes
CCACHE_MAXSIZE into the profile.d snippet but never CCACHE_DIR
In hyperi-rust-setup, --cache-root only feeds build_dir_path(). Nothing else
reads it.
The sccache half is the one that bites, and it has a trap in it. A systemd USER
service reads neither /etc/environment nor /etc/profile.d. So setting
SCCACHE_DIR in either place looks like a fix and does nothing -- the unit is the
only place that reaches the server. Worth a comment next to the fix, because the
next person will try /etc/environment first. We did.
It is also not a one-off leak. sccache evicts against rust_cache_sccache_max, so
it holds that much of / indefinitely. Every host that took rust_cache_root has
the same problem sitting there quietly.
We already get this right on the CI runner images, which set both SCCACHE_DIR and
CCACHE_DIR to the cache volume. The workstation role just missed it.
Done when a host provisioned with rust_cache_root=/cache has sccache and ccache
writing under /cache with nothing of theirs left in ~/.cache.
Two things for whoever picks it up:
- one workstation is patched BY HAND right now --
Environment=SCCACHE_DIR=/cache/sccache added to ~/.config/systemd/user/hyperi-sccache.service, plus a matching SCCACHE_DIR line in /etc/environment, and the 21G cache moved across. The next ansible run reverts the unit, so the role fix wants to land before we re-run there.
- the cap disagrees with itself on that box: the rendered unit says
SCCACHE_CACHE_SIZE=50G, the role default and the profile.d snippet say 20G. The unit is what the server reads, so 50G wins. Might be worth the role owning that value outright rather than leaving three copies.
We set
rust_cache_root=/cacheon a workstation so the Rust build caches live onits dedicated cache volume, and it only moved part of them. sccache kept its
default
~/.cache/sccacheand sat on the root filesystem holding 22G. That was adecent chunk of why root got to 84%.
Where
rust_cache_rootactually lands:hyperi-rust-setup --cache-root(rust.yml:642) -- becomes<root>/hyperi-rust-build--cache-rootin their unit and plist templatesWhere it does not:
hyperi-sccache.service.j2setsSCCACHE_CACHE_SIZEandSCCACHE_BASEDIRSbut neverSCCACHE_DIR, so the server falls back to~/.cache/sccacheCCACHE_MAXSIZEinto the profile.d snippet but neverCCACHE_DIRIn
hyperi-rust-setup,--cache-rootonly feedsbuild_dir_path(). Nothing elsereads it.
The sccache half is the one that bites, and it has a trap in it. A systemd USER
service reads neither
/etc/environmentnor/etc/profile.d. So settingSCCACHE_DIRin either place looks like a fix and does nothing -- the unit is theonly place that reaches the server. Worth a comment next to the fix, because the
next person will try
/etc/environmentfirst. We did.It is also not a one-off leak. sccache evicts against
rust_cache_sccache_max, soit holds that much of
/indefinitely. Every host that tookrust_cache_roothasthe same problem sitting there quietly.
We already get this right on the CI runner images, which set both
SCCACHE_DIRandCCACHE_DIRto the cache volume. The workstation role just missed it.Done when a host provisioned with
rust_cache_root=/cachehas sccache and ccachewriting under
/cachewith nothing of theirs left in~/.cache.Two things for whoever picks it up:
Environment=SCCACHE_DIR=/cache/sccacheadded to~/.config/systemd/user/hyperi-sccache.service, plus a matchingSCCACHE_DIRline in/etc/environment, and the 21G cache moved across. The next ansible run reverts the unit, so the role fix wants to land before we re-run there.SCCACHE_CACHE_SIZE=50G, the role default and the profile.d snippet say 20G. The unit is what the server reads, so 50G wins. Might be worth the role owning that value outright rather than leaving three copies.