Skip to content

fix: developer-rust never points sccache or ccache at rust_cache_root, so they stay on / #94

Description

@catinspace-au

We set rust_cache_root=/cache on a workstation so the Rust build caches live on
its dedicated cache volume, and it only moved part of them. sccache kept its
default ~/.cache/sccache and sat on the root filesystem holding 22G. That was a
decent chunk of why root got to 84%.

Where rust_cache_root actually lands:

  • the pooled cargo build-dir, via hyperi-rust-setup --cache-root (rust.yml:642) -- becomes <root>/hyperi-rust-build
  • the prune and the prune guard, via --cache-root in their unit and plist templates

Where it does not:

  • sccache -- hyperi-sccache.service.j2 sets SCCACHE_CACHE_SIZE and SCCACHE_BASEDIRS but never SCCACHE_DIR, so the server falls back to ~/.cache/sccache
  • ccache -- the role writes CCACHE_MAXSIZE into the profile.d snippet but never CCACHE_DIR

In hyperi-rust-setup, --cache-root only feeds build_dir_path(). Nothing else
reads it.

The sccache half is the one that bites, and it has a trap in it. A systemd USER
service reads neither /etc/environment nor /etc/profile.d. So setting
SCCACHE_DIR in either place looks like a fix and does nothing -- the unit is the
only place that reaches the server. Worth a comment next to the fix, because the
next person will try /etc/environment first. We did.

It is also not a one-off leak. sccache evicts against rust_cache_sccache_max, so
it holds that much of / indefinitely. Every host that took rust_cache_root has
the same problem sitting there quietly.

We already get this right on the CI runner images, which set both SCCACHE_DIR and
CCACHE_DIR to the cache volume. The workstation role just missed it.

Done when a host provisioned with rust_cache_root=/cache has sccache and ccache
writing under /cache with nothing of theirs left in ~/.cache.

Two things for whoever picks it up:

  • one workstation is patched BY HAND right now -- Environment=SCCACHE_DIR=/cache/sccache added to ~/.config/systemd/user/hyperi-sccache.service, plus a matching SCCACHE_DIR line in /etc/environment, and the 21G cache moved across. The next ansible run reverts the unit, so the role fix wants to land before we re-run there.
  • the cap disagrees with itself on that box: the rendered unit says SCCACHE_CACHE_SIZE=50G, the role default and the profile.d snippet say 20G. The unit is what the server reads, so 50G wins. Might be worth the role owning that value outright rather than leaving three copies.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions