Skip to content

Commit 585edf7

Browse files
docs(scripts): scope the ablation-on-dist hazard by resolution, not by suite (#8951)
The header framed the false green as a property of `packages/qa/dogfood`. It is not. The true condition is any test whose subject resolves through the dependency's `exports` (which point at that package's built `dist/`) with no vitest alias redirecting the specifier back to source. That set is already enumerated as `KNOWN_UNALIASED_TEST_IMPORTS` in `scripts/check-test-source-alias.mjs` -- 61 packages, 305 pairs -- and every entry carries the hazard. Measured twice outside dogfood: plugin-email -> platform-objects and plugin-auth -> core. Also recorded, all in the same header: - why the false green is worse than uninformative: an un-rebuilt ablation whose purpose was to prove a new gate can fail returns "the gate did not fire", which reads as evidence the gate is broken -- so a dev hunts a fix that is not needed, or weakens a working gate until it "fires"; - both halves of the mechanical rule -- rebuild after mutate AND rebuild after restore, proving each reached dist/ -- since the restore half is the one that gets skipped; - the measured limitation of the whole-package scan: a marker the package also writes elsewhere breaks both modes in opposite directions, so a per-field ablation needs a property read rather than a substring scan. The worked example is kept close to verbatim from the filing measurement. Comment-only: every changed line is a `//` comment and the executable code is byte-identical. Claude-Session: https://claude.ai/code/session_011RB4waLuNbdruCo6X9oobm Co-authored-by: Claude <noreply@anthropic.com>
1 parent d6e80b2 commit 585edf7

1 file changed

Lines changed: 85 additions & 11 deletions

File tree

‎scripts/ablation-dist-preflight.mjs‎

Lines changed: 85 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,29 @@
1010
//
1111
// ## The failure this exists to stop
1212
//
13-
// `packages/qa/dogfood` resolves the code under test from each package's built
14-
// `dist/`, not `src/` -- deliberately, because that is what covers packaging and
15-
// export-surface defects. Editing `src` therefore has no effect on the suite
16-
// until that package is rebuilt, and the two directions of forgetting are NOT
17-
// equally dangerous:
13+
// The hazard is a property of RESOLUTION, not of any one suite. Its true
14+
// condition is: **any test whose subject resolves through the dependency's
15+
// `exports`** -- which point at that package's built `dist/`, not `src/` --
16+
// with no vitest alias redirecting the specifier back to source. Such a test's
17+
// verdict is a function of BUILD state, so editing `src` has no effect on it
18+
// until that package is rebuilt.
19+
//
20+
// That set is enumerable, and already enumerated: `KNOWN_UNALIASED_TEST_IMPORTS`
21+
// in `scripts/check-test-source-alias.mjs` is the measured, shrink-only ledger
22+
// of exactly those package-dependency pairs -- 61 packages, 305 pairs when this
23+
// paragraph was written. Every one of them carries the failure below.
24+
//
25+
// `packages/qa/dogfood` is the most familiar instance -- it consumes `dist/`
26+
// deliberately, because that is what covers packaging and export-surface
27+
// defects -- but it is an INSTANCE, not the definition. Reading the hazard as
28+
// dogfood-only is how an ablation gets trusted in a plain unit suite where it
29+
// proves nothing: measured in `plugin-email` -> `platform-objects` (375 passes
30+
// on an ablated field; 4 of them went red once `platform-objects` was rebuilt),
31+
// and again in `plugin-auth` -> `core`. Neither package aliases the dep it
32+
// mutated; both are ordinary ledger entries, and `plugin-email` has no
33+
// `vitest.config.*` at all.
34+
//
35+
// The two directions of forgetting are NOT equally dangerous:
1836
//
1937
// forgot to rebuild a FIX -> false RED. Costs a lap, and gets noticed.
2038
// forgot to rebuild an ABLATION -> false GREEN. Silently certifies a vacuous
@@ -30,6 +48,17 @@
3048
// by hand-grepping `dist/` for the mutation marker. This script is that grep,
3149
// mechanized, with the traps the manual version cannot see.
3250
//
51+
// ### Worse than uninformative -- it points confidently the WRONG WAY
52+
//
53+
// When the ablation's purpose was to prove a NEW GATE is capable of failing,
54+
// the false green is not a null result. It is the observation "the gate did not
55+
// fire", and in that context that reads as evidence the GATE is broken rather
56+
// than evidence the harness is. A dev acting on it goes hunting for a fix that
57+
// is not needed -- or, the expensive outcome, WEAKENS a working gate until it
58+
// "fires", destroying the thing the ablation was written to certify. That is
59+
// the `plugin-auth` -> `core` shape above: ablation legs whose entire purpose
60+
// was to demonstrate a new gate can fail, every one of them dist-mediated.
61+
//
3362
// ## The two ablation shapes, hence the two modes
3463
//
3564
// PLANT (default) the mutation ADDS something identifiable -- a changed
@@ -39,10 +68,20 @@
3968
// so the assertion inverts: a literal unique to the deleted
4069
// code must be GONE from dist. Same check, mirrored.
4170
//
42-
// `--absent` is also the restore leg: after putting the fix back and rebuilding,
43-
// it proves the marker really left the artifact. That leg matters more than it
44-
// looks -- a marker left behind in `dist/` keeps mutated code live for every
45-
// later suite run in that worktree, long after the ablation is "finished".
71+
// The rule they mechanize has TWO halves, and only one of them is intuitive:
72+
//
73+
// mutate -> rebuild -> prove the marker is IN dist/ (default mode)
74+
// restore -> rebuild -> prove the marker is GONE (`--absent`)
75+
//
76+
// So `--absent` is two things at once: the mode for a DELETE ablation, and the
77+
// restore leg of a PLANT one. The restore half is the one that gets skipped --
78+
// rebuilding after mutating is obvious, while remembering that RESTORING also
79+
// needs a rebuild before the NEXT measurement is trustworthy is not. It matters
80+
// more than it looks: a marker left behind in `dist/` keeps mutated code live
81+
// for every later suite run in that worktree, long after the ablation is
82+
// "finished", so the runs that follow are measuring the wrong tree. Done by
83+
// hand the leg reads `grep -c <marker> packages/core/dist/index.js` -> 0 after
84+
// the `git checkout`; this script is that, plus the traps below.
4685
//
4786
// ## Sourcemap-only matches are RED, not green
4887
//
@@ -51,14 +90,49 @@
5190
// false green this script exists to prevent, so `.map` hits are reported and
5291
// excluded from the verdict.
5392
//
93+
// ## The scan is WHOLE-PACKAGE, so the marker must be unique to the mutation
94+
//
95+
// Both modes grep the package's entire `dist/` tree, deliberately -- a mutation
96+
// can land in any emitted chunk, and guessing which one is how the manual
97+
// version missed things. The price is that the scan cannot tell YOUR literal
98+
// from the same literal written elsewhere in the same package, and both modes
99+
// assume it is unique. When it is not, both go wrong, in opposite directions:
100+
// `--absent` reports surviving hits that were never yours (a false RED), and
101+
// the default mode passes on a sibling's hit alone (a false GREEN -- the one
102+
// this script exists to stop, reintroduced through the marker).
103+
//
104+
// Measured: ablating `internal: true` on ONE field of `sys_email`, then running
105+
//
106+
// node scripts/ablation-dist-preflight.mjs @objectstack/platform-objects 'internal: true' --absent
107+
//
108+
// reported 6 surviving hits, every one of them a legitimate `internal: true` on
109+
// an identity object in `dist/identity/` and none of them the ablated field.
110+
//
111+
// There is no per-symbol mode. When the marker cannot be made unique -- a
112+
// per-FIELD ablation of a flag the package also uses elsewhere is the standard
113+
// case -- do NOT weaken the marker to make this script agree with you. Verify
114+
// by PROPERTY READ against the same artifact instead, which is exact where a
115+
// substring scan cannot be:
116+
//
117+
// node -e "const {SysEmail}=require('./packages/platform-objects/dist/audit/index.js');
118+
// console.log(SysEmail.fields.headers_json.internal)"
119+
// # before rebuild: true (ablation NOT in the artifact -> the green was vacuous)
120+
// # after rebuild: undefined
121+
//
122+
// Same question, same moment in the procedure, same two halves (mutate and
123+
// restore); only the instrument changes. What is not negotiable is that SOME
124+
// instrument reads the built artifact before the ablation's colour is believed.
125+
//
54126
// ## Why this is not a `check:*` gate
55127
//
56128
// It judges a deliberately mutated working tree, so it can only be run by the
57129
// agent performing the ablation, at one specific moment between "mutate" and
58130
// "run the suite". CI has no ablation in flight and nothing to assert. It is
59131
// dev-side agent tooling, invoked from the ablation procedure in
60-
// `.claude/agents/os-dev.md` and `.claude/skills/dogfood-verification/SKILL.md`
61-
// -- keep those two and this file's usage line in step.
132+
// `.claude/agents/os-dev.md`, `.claude/skills/dogfood-verification/SKILL.md`
133+
// and `packages/qa/dogfood/README.md` step 4 -- keep those three and this
134+
// file's usage line in step. (Those three state the procedure scoped to the
135+
// dogfood suite; the true condition is the resolution one stated at the top.)
62136
//
63137
// Anything this script cannot see is RED, never a skip: a missing `dist/`, a
64138
// `dist/` with nothing readable in it, or a package name that resolves to

0 commit comments

Comments
 (0)