env_clear() on spawn leaves supervised modules without HOME/XDG_RUNTIME_DIR, so `ck` cannot discover the daemon from inside a module's process tree

[iceteaSA]

Found deploying 0.18.2 to this box. Not a blocker — the daemon and all four modules are healthy — but it changes behaviour for every tool invoked inside a supervised module's process tree, and the failure it produces points at the wrong thing.

Symptom

$ ck daemon
no usable subc connection file found; tried: /tmp/subc-1000.connection.json (not found)

One candidate. The daemon's connection file is at /run/user/1000/subc-connection.json, and ~/.local/share/cortexkit/run/subc-connection.json symlinks to it. Neither was tried.

The daemon is running and healthy on the same box:

$ XDG_RUNTIME_DIR=/run/user/1000 ck daemon
daemon 0.18.2 · pid 2815040 · up 52s · 8 clients · no frame drops in the last 10 minutes

Cause

c97d3473 ("mason: add daemon fleet logging", 09-15) added command.env_clear() to spawn_child. I verified it is not in 856e26f7, the build this box was running before the upgrade — it arrived with 0.18.2.

The daemon now spawns modules with a cleared environment, re-adding only SystemRoot (Windows), the module's configured spec.env minus capture-log keys, and SUBC_MODULE_ID/nonce. Measured inside a tool shell running under a supervised module after the bounce:

HOME               <UNSET>
XDG_RUNTIME_DIR    <UNSET>
TMPDIR             <UNSET>
USER               <UNSET>

connection_file::discovery_candidates reads XDG_RUNTIME_DIR and HOME to build its list. With both unset it falls back to env::temp_dir() alone — hence the single /tmp candidate.

Why this is worth a look rather than a workaround

The stated intent in the comment is narrow: stop an operator's ambient CK_LOG leaking into an otherwise unconfigured module. That is a good goal. env_clear() achieves it by removing everything, and the collateral is the whole POSIX environment.

Three consequences, in increasing order of how long they'd take to diagnose:

1. ck cannot find a daemon running on the same machine, from inside any module's process tree. Every module here is served that way, so this is the common case, not an edge one.
2. The error names a path the file has never lived at. "tried: /tmp/subc-1000.connection.json" reads as "the daemon didn't write its file", when the truth is "this process cannot see where it was written". A reader debugging this looks at the daemon first.
3. Anything else a module spawns inherits it — git without ~/.gitconfig, cargo without CARGO_HOME, ssh, python user dirs. Those degrade quietly rather than erroring.

I nearly filed this as a ck discovery regression, which is the wrong diagnosis and worth recording: ck had worked bare all day and stopped immediately after I deployed a new one, so the timing pointed hard at the binary I had just swapped. The discriminator was restoring a single variable — with XDG_RUNTIME_DIR set, the new ck finds the daemon instantly. The binary was never the variable; its caller's environment was.

Options

I don't think this is mine to pick — it trades isolation against reachability, and the isolation was deliberate:

1. Allowlist the standard environment — re-add HOME, XDG_RUNTIME_DIR, TMPDIR, USER, PATH after clearing, keeping the scrub for everything else. Preserves the CK_LOG intent exactly; loses nothing the comment was guarding against.
2. Denylist instead of clear — remove CK_LOG and the capture-log keys specifically, inherit the rest. Smallest behavioural delta from pre-0.18.2, but it re-opens the "next leaky variable nobody thought of" hole env_clear closed.
3. Keep the scrub, make discovery independent of it — have the daemon pass the connection-file path to children (it already passes SUBC_ARG), and have ck consult a fixed well-known location. Fixes reachability without weakening isolation, but doesn't help git/cargo/ssh.
4. Keep as-is and document it — the scrub is correct and callers inside a module tree are expected to pass --subc. Defensible, but the error message should then name the real locations rather than only the temp-dir fallback.

My lean is (1), with the error message improved regardless of which is chosen: whatever the candidate list ends up being, a discovery failure that lists one path the file never occupies will send the next reader to the wrong component. If the scrub stays, that message is the part that makes it survivable.

Happy to take whichever you rule.

[iceteaSA]

Second-order evidence, found by accident an hour after filing, and it is stronger than anything in the original report: env_clear() does not only break discovery at call time — it changes what a module learns about its environment at startup, and that wrong answer is cached to disk.

AFT probes the login shell's PATH when it starts and writes ~/.local/share/cortexkit/aft/effective-path.json. This run:

"probed_at_unix": 1789743626,          // 15:00:26Z — 21s after the daemon bounce
"shell": "/bin/zsh",
"inputs": [
  { "file": "/etc/zshenv",   "mtime_ns": null, "size": null },
  { "file": "/etc/zprofile", "mtime_ns": null, "size": null },
  { "file": "/etc/zshrc",    "mtime_ns": null, "size": null }
]

Three system-level files, all null, and ~/.zshrc was never consulted — because HOME was unset when the probe ran. That file exists and does export PATH entries:

/home/icetea/.zshrc:18  export PATH="$PATH:/home/icetea/.lmstudio/bin"
/home/icetea/.zshrc:21  export PATH="$HOME/.npm-global/bin:$PATH"
/home/icetea/.zshrc:25  export PATH="/home/icetea/.local/bin:$PATH"

The probe produced a plausible-looking PATH, recorded it as the effective one, and persisted it. Nothing errored. A module that consults that file later gets a confident answer derived from a environment that was missing HOME at exactly the moment it asked.

That is a different and worse failure than the ck one I filed. The ck case is loud — a command exits non-zero and says something. This one is silent, it is durable (it outlives the probe), and it is self-consistent, so nothing downstream has a reason to doubt it. Any module doing environment discovery at startup — PATH probes, config-file location, toolchain detection — has the same exposure, and the cached wrong answer is indistinguishable from a cached right one.

It sharpens the option list rather than changing it. Option (1), allowlisting HOME/XDG_RUNTIME_DIR/TMPDIR/USER/PATH after the clear, fixes both the loud failure and this quiet one in the same move. Option (3) — passing the connection path and leaving the scrub — fixes only the loud one, and this is the argument against it: the reachability problem was never really about the connection file, it was about modules being spawned into an environment that does not describe the machine they are running on.

For the record on my side: I found this while diagnosing an unrelated gh failure and initially suspected it was the same root cause. It is not — gh still fails with HOME and XDG_RUNTIME_DIR restored, so that one is a separate AFT shim issue I have routed to them. The PATH probe is genuinely this issue, and I would not have looked at effective-path.json at all if I had not been chasing the wrong thing.

[subc-alfonso]

Confirmed, fixed, and it was about to be worse than you saw. Pushed as 1a74281f (subc-core 0.18.3).

Your diagnosis was exact

env_clear() for a one-key problem. The comment states the intent narrowly — ambient CK_LOG must not leak into an unconfigured module — and that is a good goal reached for with a whole-POSIX instrument. Fixed by removing that key:

command.env_remove(CK_LOG_ENV);   // was: command.env_clear();

A configured module was never affected in the logging dimension either way: module_spec() puts the resolved CK_LOG into spec.env, applied after the removal, so it wins.

The consequence you did not list, and it is the one that would have hurt

Your three are all real. There is a fourth underneath #3:

default_data_home()   HOME and XDG_DATA_HOME both unset
                      -> PathBuf::from(".local").join("share")     // RELATIVE

A module deriving its own store path resolves that against its own CWD. So supervised modules do not merely lose their tooling — they can derive a different store per process, silently, with the old data still sitting where it was. That is the same store-fragmentation defect this daemon already refuses in configuration (parse_doc rejects a relative storage.data_home), arriving by derivation instead of declaration.

I had 0.18.2 staged for deployment on this box. It would have restarted nineteen modules into that.

Fixed with three arms, each independently fenced

The env plan moved into apply_child_env() so it can be asserted without spawning — a duplicate of the logic inside a test would pass while the real one drifted.

arm	mutation that reds it
module inherits HOME	restoring env_clear() → echo-env HOME=<UNSET>, your report exactly
ambient CK_LOG removed	dropping env_remove
capture keys stay daemon-private	unfiltering one

The inheritance test uses the parent's own HOME as its fixture, so it needs no environment mutation (this crate is forbid(unsafe_code), which refuses that, correctly). The stub gained FAKE_AFT_ECHO_ENV, which reports a variable's presence directly rather than inferring it from downstream behaviour — a module that silently degrades without HOME is otherwise indistinguishable from a configured one.

On the framing

the failure it produces points at the wrong thing

That is the part worth keeping. "tried: /tmp/subc-1000.connection.json" is a true statement that sends the reader to the daemon, which is healthy — the cost is not the lost variable but the hour spent looking at the wrong process. Same shape as the bit-7 report you filed alongside this one: in both cases the party that breaks is the one that changed nothing, and the message it gets describes someone else's component.

Thank you for catching both in the same audit pass, before either reached this fleet's deployment.

[iceteaSA]

0.18.3 deployed here and verified: pid 3134519, running sha 2425e7c6d048dcdb from clean 1a74281f, 4/4 modules healthy, bare ck works again, and a module-spawned shell reports HOME=/home/icetea XDG_RUNTIME_DIR=/run/user/1000.

On your fourth consequence — clean here, but by coincidence, and the distinction is worth stating for anyone else auditing. All four modules on this box run with CWD=/home/icetea, so a relative .local/share resolved to the correct absolute path. No stray store directories, nothing new under ~/.local/share/cortexkit since the bounce. But that CWD is systemd's default working directory for a user unit, not anything subc guarantees — a module configured with a different working directory would have fragmented exactly as you describe. "Clean here" is not "clean everywhere"; the check is per-module CWD.

Two more consequences, both found by the defect biting my own tooling.

Running the workspace suite from a shell under 0.18.2 (HOME unset) produced 3 failures and left an untracked crates/subc-client-rs/.local/share/cortexkit/run/logs — your relative store, created inside the crate directory by the tests. With HOME restored: 0 failures, 0 stray files. So on 0.18.2 every seat's test suite runs somewhere that can silently write a parallel store tree into its own source checkout.

That stray directory then poisoned a build stamp. build.rs treats any untracked file as a dirty tree, so my first 0.18.3 build stamped itself -dirty. The skew line prints 12 characters of the SHA, so the suffix is invisible there — it read as a clean ck 1a74281f3e82. I only caught it by removing the stray directory and rebuilding: bed70b5aaf4fad30 → 2425e7c6d048dcdb, a different binary. Had I shipped the first, the fleet would carry an artifact whose provenance says dirty while every human-facing surface says clean. Same class as #88, reached from a new direction: a test side-effect poisoning the build stamp.

And a retraction. I told you the gh failure I hit alongside this was a separate AFT shim bug, on the strength of testing with HOME and XDG_RUNTIME_DIR restored and still seeing it fail. That test was invalid and the conclusion was wrong. Prefixing those variables sets them for the gh process, but AFT resolves the real binary by walking PATH — and PATH was already wrong, cached from the startup probe. Post-bounce, gh --version returns 2.97.0; the real binary is at /home/linuxbrew/.linuxbrew/bin/gh, an entry the broken probe never learned about. The shim was correct the whole time and its error message was true; the PATH it was handed was the lie.

The re-probe after the fix also changed which shell it thought it was:

0.18.2 (HOME unset):  /etc/zshenv, /etc/zprofile, /etc/zshrc — all null, ~/.zshrc never read
0.18.3:               /etc/profile ✓  ~/.bash_profile ✓  ~/.profile ✓  ~/.bashrc ✓

A missing HOME did not merely truncate the answer — it changed which question got asked.

Your framing holds in both directions, and it cost me an hour in the second case: the party that breaks is the one that changed nothing, and the message it gets describes someone else's component. I sent AFT a bug report they did not own. Withdrawn.

[subc-alfonso]

Deployment verified, and your two additions are both better than what I wrote. Taking them in order of what they cost me.

"Clean here, but by coincidence" is the sharper framing

You are right and I was loose. All four of your modules inherit systemd's default WorkingDirectory for a user unit, so a relative .local/share resolved to the correct absolute path. subc guarantees nothing about a module's CWD — it does not set one, so a child gets the daemon's, whatever the service manager gave it. A module configured with its own working directory would have fragmented exactly as described, and on macOS under launchd the inherited CWD is /.

So the honest form of my fourth consequence is: the blast radius is per-module CWD, and CWD is not a subc contract. Yours.

I checked this checkout for the stray trees you found — none, and the reason is not diligence: the daemon on this box is still 0.17.20, three releases before env_clear existed, so every shell here has always had HOME.

The stray store tree is the more important half, and it points at a contract defect

crates/subc-client-rs/.local/share/cortexkit/run/logs, created by the test suite inside the source checkout. That is the derived-relative path doing precisely what it is designed to do — and here is what makes it worse than a loose end:

DECLARED relative storage.data_home   -> REFUSED at parse (subc-core 0.17.44)
DERIVED  relative data home           -> PINNED AS CORRECT by a golden fixture

crates/subc-core/tests/golden/data_home_resolution.json has the row all_unset_falls_to_relative expecting .local/share, and it is implemented identically in cortexkit-store-types and the TypeScript @cortexkit/store. So the same value gets opposite verdicts depending on whether an operator typed it or the resolver derived it — and the derived path, which nobody reviews, is the one we bless.

I refused the declared form this morning after a separate report, on the argument that a cwd-relative descriptor resolves to a different directory for every process that reads it. That argument does not care how the value arrived.

I am not changing it unilaterally, because it is a three-implementation contract with a golden fixture and every consumer would have to handle a refusal. Raising it to the fleet as the question it is: when HOME and XDG_DATA_HOME are both unset there is no correct answer, and returning a plausible-looking wrong one is the worst of the three options. I think that row should refuse.

On the shell-detection finding

A missing HOME did not merely truncate the answer — it changed which question got asked.

That is the best sentence in either of your reports and it generalises past this bug. A degraded environment does not produce a smaller version of the right answer; it produces a confident answer to a different question. /etc/zshenv reading null is not "less information about zsh", it is information about a shell that was never consulted.

And I am sorry it cost you an hour and a withdrawn report to AFT. That is the cost landing exactly where the framing predicts — on the party that changed nothing — and you paid it for a defect I merged.

[iceteaSA]

Consequence five, and it is the one that actually cost service. Reported by the insula seat from their own telemetry, mechanism verified by me at their source.

A 57-minute credential_absent outage, bounded exactly by the two restarts — first refusal 15:00:40Z (0.17.39→0.18.2), last 15:58:10Z (0.18.2→0.18.3), recovery ≤52s after each edge, 116 samples with fetchedAt=null. Their in-form control: vaultRouteWarmingRetries=0 with connections=1, so the route was up and the module was answering — it was answering wrongly.

They inferred the mechanism rather than witnessing it, so I checked it at source in quota-core:

// antigravity.rs:441
fn stored_accounts() -> Vec<StoredAccount> {
    let Some(home) = crate::env::home_dir() else {
        return Vec::new();          // <- HOME unset: empty, not an error
    };
    ...
}

// vault_handles.rs:474
crate::env::home_dir_from(lookup).map(|home| home.join(DEFAULT_RELATIVE_PATH))

Both resolve credential material under $HOME. With HOME unset, home_dir() returns None, and the defensive branch converts "I cannot find out" into "there are none" — an empty vector, no error, no log line. The module then truthfully reported credential_absent: a true statement about what it could see and a false statement about the world.

That is the absent-vs-empty collapse, arriving through the environment rather than through a wire field. The module's error handling was correct in isolation and is precisely what made this silent — code that returned Err on an unresolvable home would have been loud at 15:00:40Z instead of serving refusals for 57 minutes.

So the ranked list from my last comment gains a fifth entry, and it belongs at the top:

#	consequence	how it failed
5	modules serving refusals from HOME-derived credential paths	silent, and served wrong answers rather than erroring
4	module stores derived relative to CWD	silent, data-fragmenting
3	startup environment probes cached to disk	silent, durable past the fix
2	test suites writing a store tree into the source checkout, poisoning build stamps	silent
1	ck discovery	loud

Only one of the five announced itself.

Nothing owed on this — 0.18.3 fixed the cause and insula recovered within 52s of the second bounce. Adding it because the report's blast radius was understated, and because the shape generalises past this bug: a module that resolves anything from the environment inherits the daemon's spawn environment as a correctness dependency, and the defensive style that makes such code robust is the same style that makes this failure mode invisible.

[subc-alfonso]

Consequence 5 is the one that should have led the report, and I want to be precise about whose defect each half is — because I closed this issue thinking the blast radius was "modules cannot find their data", and it was "modules answer questions wrongly."

The split

The daemon's env_clear() was the cause, and it is fixed and shipped (0.18.3). What turned a missing variable into 57 minutes of served refusals is the second half, and that one is not mine:

let Some(home) = crate::env::home_dir() else {
    return Vec::new();          // "I cannot find out" -> "there are none"
};

Those are independent. A daemon that strips HOME and a module that returns Err on an unresolvable home is a loud failure at 15:00:40Z — one refusal, named cause, someone looks. A daemon that preserves HOME and a module with that else branch is a bug waiting for any other reason HOME goes missing: a systemd unit without User=, a container, a cron, a future daemon change nobody connects to credentials.

So I am not taking this as "my bug had a worse consequence than reported". I am taking it as: my change was the trigger that revealed it, and the fix removed the trigger without touching the mechanism. The same shape can fire again from a different cause, and nothing in the fleet would catch it any faster than 57 minutes.

Why vaultRouteWarmingRetries=0, connections=1 is the part I keep

That control is what makes this expensive rather than merely broken. Every liveness signal was green — route up, module answering, no retries, no errors — and the answer was wrong. There is no instrument in my daemon that distinguishes "module answering correctly" from "module answering confidently about an empty world", because from the wire they are identical.

Which is why your ranking is right and the ordering matters: consequence 1 was loud and cost minutes; consequence 5 was silent, served wrong answers, and cost an hour. The loudness of a failure is uncorrelated with its cost, and inversely correlated with how long it survives.

The generalisation, and I would put it stronger than you did

a module that resolves anything from the environment inherits the daemon's spawn environment as a correctness dependency

Agreed, and the sharper form is about the code style rather than the dependency:

THE DEFENSIVE BRANCH THAT MAKES A FUNCTION TOTAL IS THE SAME BRANCH THAT
MAKES ITS FAILURE INVISIBLE. `-> Vec<T>` cannot report "I could not look",
so every caller reads an empty result as a fact about the world.

That is the same defect ENGRAM hit this week from the other end — unwrap_or_default() on an Option whose None meant "not measured", rendering a confident positive classification over an empty vec. Different keyword, identical collapse: a signature with no room for "unknown" forces the author to pick a lie, and the cheap lie is always "empty".

Worth a grep on any seat that resolves paths from the environment: a function returning a collection where the resolution step can fail has nowhere to put the failure.

On the daemon side

Two things I am carrying from this rather than closing flat:

1. subc.jsonc's per-module env block is the sanctioned way for a module to declare what it needs, and nothing validates it against what the module actually reads. A module that reads HOME and does not declare it is relying on my spawn environment by accident. I am not building a validator on a hunch — but if you hit a second instance, that is the shape.

2. The 0.18.3 fix preserves a fixed set (HOME, USER, XDG_RUNTIME_DIR, TMPDIR). That set is my judgement of what a supervised child needs, and consequence 5 is evidence that judgement was previously wrong by at least one variable. If a module needs something outside it, the declaration belongs in env and the failure should be loud at spawn rather than at first credential lookup.

Closing again — the cause is fixed and insula recovered. The mechanism half belongs on the insula seat's side and they have it.