|
10 | 10 | // |
11 | 11 | // ## The failure this exists to stop |
12 | 12 | // |
13 | | -// `packages/qa/dogfood` resolves the code under test from each package's built |
14 | | -// `dist/`, not `src/` -- deliberately, because that is what covers packaging and |
15 | | -// export-surface defects. Editing `src` therefore has no effect on the suite |
16 | | -// until that package is rebuilt, and the two directions of forgetting are NOT |
17 | | -// equally dangerous: |
| 13 | +// The hazard is a property of RESOLUTION, not of any one suite. Its true |
| 14 | +// condition is: **any test whose subject resolves through the dependency's |
| 15 | +// `exports`** -- which point at that package's built `dist/`, not `src/` -- |
| 16 | +// with no vitest alias redirecting the specifier back to source. Such a test's |
| 17 | +// verdict is a function of BUILD state, so editing `src` has no effect on it |
| 18 | +// until that package is rebuilt. |
| 19 | +// |
| 20 | +// That set is enumerable, and already enumerated: `KNOWN_UNALIASED_TEST_IMPORTS` |
| 21 | +// in `scripts/check-test-source-alias.mjs` is the measured, shrink-only ledger |
| 22 | +// of exactly those package-dependency pairs -- 61 packages, 305 pairs when this |
| 23 | +// paragraph was written. Every one of them carries the failure below. |
| 24 | +// |
| 25 | +// `packages/qa/dogfood` is the most familiar instance -- it consumes `dist/` |
| 26 | +// deliberately, because that is what covers packaging and export-surface |
| 27 | +// defects -- but it is an INSTANCE, not the definition. Reading the hazard as |
| 28 | +// dogfood-only is how an ablation gets trusted in a plain unit suite where it |
| 29 | +// proves nothing: measured in `plugin-email` -> `platform-objects` (375 passes |
| 30 | +// on an ablated field; 4 of them went red once `platform-objects` was rebuilt), |
| 31 | +// and again in `plugin-auth` -> `core`. Neither package aliases the dep it |
| 32 | +// mutated; both are ordinary ledger entries, and `plugin-email` has no |
| 33 | +// `vitest.config.*` at all. |
| 34 | +// |
| 35 | +// The two directions of forgetting are NOT equally dangerous: |
18 | 36 | // |
19 | 37 | // forgot to rebuild a FIX -> false RED. Costs a lap, and gets noticed. |
20 | 38 | // forgot to rebuild an ABLATION -> false GREEN. Silently certifies a vacuous |
|
30 | 48 | // by hand-grepping `dist/` for the mutation marker. This script is that grep, |
31 | 49 | // mechanized, with the traps the manual version cannot see. |
32 | 50 | // |
| 51 | +// ### Worse than uninformative -- it points confidently the WRONG WAY |
| 52 | +// |
| 53 | +// When the ablation's purpose was to prove a NEW GATE is capable of failing, |
| 54 | +// the false green is not a null result. It is the observation "the gate did not |
| 55 | +// fire", and in that context that reads as evidence the GATE is broken rather |
| 56 | +// than evidence the harness is. A dev acting on it goes hunting for a fix that |
| 57 | +// is not needed -- or, the expensive outcome, WEAKENS a working gate until it |
| 58 | +// "fires", destroying the thing the ablation was written to certify. That is |
| 59 | +// the `plugin-auth` -> `core` shape above: ablation legs whose entire purpose |
| 60 | +// was to demonstrate a new gate can fail, every one of them dist-mediated. |
| 61 | +// |
33 | 62 | // ## The two ablation shapes, hence the two modes |
34 | 63 | // |
35 | 64 | // PLANT (default) the mutation ADDS something identifiable -- a changed |
|
39 | 68 | // so the assertion inverts: a literal unique to the deleted |
40 | 69 | // code must be GONE from dist. Same check, mirrored. |
41 | 70 | // |
42 | | -// `--absent` is also the restore leg: after putting the fix back and rebuilding, |
43 | | -// it proves the marker really left the artifact. That leg matters more than it |
44 | | -// looks -- a marker left behind in `dist/` keeps mutated code live for every |
45 | | -// later suite run in that worktree, long after the ablation is "finished". |
| 71 | +// The rule they mechanize has TWO halves, and only one of them is intuitive: |
| 72 | +// |
| 73 | +// mutate -> rebuild -> prove the marker is IN dist/ (default mode) |
| 74 | +// restore -> rebuild -> prove the marker is GONE (`--absent`) |
| 75 | +// |
| 76 | +// So `--absent` is two things at once: the mode for a DELETE ablation, and the |
| 77 | +// restore leg of a PLANT one. The restore half is the one that gets skipped -- |
| 78 | +// rebuilding after mutating is obvious, while remembering that RESTORING also |
| 79 | +// needs a rebuild before the NEXT measurement is trustworthy is not. It matters |
| 80 | +// more than it looks: a marker left behind in `dist/` keeps mutated code live |
| 81 | +// for every later suite run in that worktree, long after the ablation is |
| 82 | +// "finished", so the runs that follow are measuring the wrong tree. Done by |
| 83 | +// hand the leg reads `grep -c <marker> packages/core/dist/index.js` -> 0 after |
| 84 | +// the `git checkout`; this script is that, plus the traps below. |
46 | 85 | // |
47 | 86 | // ## Sourcemap-only matches are RED, not green |
48 | 87 | // |
|
51 | 90 | // false green this script exists to prevent, so `.map` hits are reported and |
52 | 91 | // excluded from the verdict. |
53 | 92 | // |
| 93 | +// ## The scan is WHOLE-PACKAGE, so the marker must be unique to the mutation |
| 94 | +// |
| 95 | +// Both modes grep the package's entire `dist/` tree, deliberately -- a mutation |
| 96 | +// can land in any emitted chunk, and guessing which one is how the manual |
| 97 | +// version missed things. The price is that the scan cannot tell YOUR literal |
| 98 | +// from the same literal written elsewhere in the same package, and both modes |
| 99 | +// assume it is unique. When it is not, both go wrong, in opposite directions: |
| 100 | +// `--absent` reports surviving hits that were never yours (a false RED), and |
| 101 | +// the default mode passes on a sibling's hit alone (a false GREEN -- the one |
| 102 | +// this script exists to stop, reintroduced through the marker). |
| 103 | +// |
| 104 | +// Measured: ablating `internal: true` on ONE field of `sys_email`, then running |
| 105 | +// |
| 106 | +// node scripts/ablation-dist-preflight.mjs @objectstack/platform-objects 'internal: true' --absent |
| 107 | +// |
| 108 | +// reported 6 surviving hits, every one of them a legitimate `internal: true` on |
| 109 | +// an identity object in `dist/identity/` and none of them the ablated field. |
| 110 | +// |
| 111 | +// There is no per-symbol mode. When the marker cannot be made unique -- a |
| 112 | +// per-FIELD ablation of a flag the package also uses elsewhere is the standard |
| 113 | +// case -- do NOT weaken the marker to make this script agree with you. Verify |
| 114 | +// by PROPERTY READ against the same artifact instead, which is exact where a |
| 115 | +// substring scan cannot be: |
| 116 | +// |
| 117 | +// node -e "const {SysEmail}=require('./packages/platform-objects/dist/audit/index.js'); |
| 118 | +// console.log(SysEmail.fields.headers_json.internal)" |
| 119 | +// # before rebuild: true (ablation NOT in the artifact -> the green was vacuous) |
| 120 | +// # after rebuild: undefined |
| 121 | +// |
| 122 | +// Same question, same moment in the procedure, same two halves (mutate and |
| 123 | +// restore); only the instrument changes. What is not negotiable is that SOME |
| 124 | +// instrument reads the built artifact before the ablation's colour is believed. |
| 125 | +// |
54 | 126 | // ## Why this is not a `check:*` gate |
55 | 127 | // |
56 | 128 | // It judges a deliberately mutated working tree, so it can only be run by the |
57 | 129 | // agent performing the ablation, at one specific moment between "mutate" and |
58 | 130 | // "run the suite". CI has no ablation in flight and nothing to assert. It is |
59 | 131 | // dev-side agent tooling, invoked from the ablation procedure in |
60 | | -// `.claude/agents/os-dev.md` and `.claude/skills/dogfood-verification/SKILL.md` |
61 | | -// -- keep those two and this file's usage line in step. |
| 132 | +// `.claude/agents/os-dev.md`, `.claude/skills/dogfood-verification/SKILL.md` |
| 133 | +// and `packages/qa/dogfood/README.md` step 4 -- keep those three and this |
| 134 | +// file's usage line in step. (Those three state the procedure scoped to the |
| 135 | +// dogfood suite; the true condition is the resolution one stated at the top.) |
62 | 136 | // |
63 | 137 | // Anything this script cannot see is RED, never a skip: a missing `dist/`, a |
64 | 138 | // `dist/` with nothing readable in it, or a package name that resolves to |
|
0 commit comments