ci: pin the locale composition of llms.txt and llms-full.txt

[hotlong]

Fixes #184

.github/scripts/check-locale-surface.mjs gained a locale oracle for the sitemap in #193. This adds the two llms endpoints to the same harness. Nothing in the repo asserted which locales they serve, and the same missing-language-argument defect has now been found and fixed in all three of them (#176 / #179 / #183).

One file changed: .github/scripts/check-locale-surface.mjs. tools/ci-scripts/run-self-tests.mjs needed no edit — the script was already registered there, and ci.yml already runs the gate after build. No apps/docs/** source file is touched.

The assertion

The llms bodies never emit a page's own URL — llms-full.txt is page texts concatenated, each opening with the # heading line getLLMText prepends — so the sitemap's URL-set comparison cannot be reused verbatim. They do carry page titles, so the comparison runs over the titles that belong to exactly one locale:

• every en-exclusive title must be in the body — one going missing means English pages stopped being served;
• no other locale's exclusive title may be — one appearing means the language argument was dropped again.

Both directions, as #175 argued for the sitemap. Measured on this tree: 60 / 52 / 31 / 25 / 32 / 28 / 31 exclusive titles for en / zh-Hans / ja / de / es / fr / ko, with both bodies reading 60 / 0 / 0 / 0 / 0 / 0 / 0. The card recorded 63/53/31/25/32/28/31 → 63/0×6 one tree earlier; the numbers move with the content and nothing here pins them.

Restricting to exclusive titles is what makes it stable rather than merely strict. A plain "every title in the body equals every English title" comparison would go red on a correct build twice over: three pairs of English pages share a title today (Approvals, Dashboards, Notifications), and any page growing an # heading in its own body would read as an extra page.

Composition, never sequence. llms-full.txt's page order is non-deterministic across builds — observed directly here, twice: two builds of an identical tree produced llms-full.txt.body hashes a6b849f1… and 28cccb8b… while the gate stayed green on both. #196 is changing that order on purpose. Any assertion on sequence, a golden file, or a diff against a recorded body would flake today and break when #196 lands. llms.txt walks the page tree, whose order comes from meta.json, so it is not exposed to the same non-determinism; it is written the same way anyway.

Rules

Two new rules, unexpected-locale-title and missing-locale-title, rather than reusing unexpected-url / missing-url. A shared message would be a wrong message: telling a reader that データモデル "is advertised but the content tree has no source file for it" is false twice — it is not a URL and it has a source file. artifact-missing and artifact-empty are reused as-is, with the existing fixtures extended to cover the new artifacts.

A third rule, nothing-expected, closes a hole this design opens: if no English title were exclusive to English, the comparison would have nothing to compare and both artifacts would pass without measuring anything. That is the same failure artifact-empty already treats as a failure rather than a skip.

Verification

Every claim below is a captured exit code read before any pipe, not a bare $? after one.

The real defect, reconstructed in llms-full.txt — source.getPages(LANG) → source.getPages(), proven on disk (anchor count 1 → 0; blob 722f1479… → 3d6440d4…), rebuilt with --force, then restored and proven restored (blob back to 722f1479…, git diff HEAD empty):

| `llms-full.txt` | locale-exclusive title | 553 | 481 | 259 | 60 | 199 | 0 | n/a |

✗ 199 locale-surface finding(s)   [gate exit 1]

  - llms-full.txt: "AI サービス" is in the body, and that title exists only in ja
    (content/docs/configure/ai.ja.mdx) — a non-en page is being served here, which is
    the language argument having been dropped from this route's page lookup

read goes 79 → 553, matching the 553 that lib/seo.ts documents for a bare getPages(). The other two artifacts stay green: the failure localizes to the one that broke.

The real defect, reconstructed in llms.txt. Worth recording precisely, because the obvious mutation is a no-op. Dropping the argument from source.getPageTree(LANG) produces a byte-identical llms.txt.body (883ebd38… both ways) — getPageTree() resolves to the default language in fumadocs-core 16.8.12, so the gate staying green there is correct, not a miss. The actual historical defect is a bare source.getPages() flat list, which is what 9c29df6^ contains. Restoring that exact blob (6639c252…, verified equal to the historical blob on disk) and rebuilding:

| `llms.txt` | locale-exclusive title | 553 | 481 | 259 | 60 | 199 | 0 | n/a |

✗ 199 locale-surface finding(s)   [gate exit 1]

llms.txt.body moved to 21c61abb…, so the mutation reached the build output and not just the source. Restored, blob back to 790d942a…, git diff HEAD empty.

The other direction, on the real tree. getPages(LANG) → getPages('ja'):

| `llms-full.txt` | locale-exclusive title | 79 | 71 | 68 | 60 | 31 | 23 | n/a |

**unexpected-locale-title** — 31
**missing-locale-title** — 23

  - llms-full.txt: "AI Service" is a page title that exists only in en
    (content/docs/configure/ai.mdx), and it is NOT in the body — en pages have
    stopped being served here

23 rather than 60 missing, because the 37 English pages with no Japanese translation still fall back to English — the behaviour lib/seo.ts documents.

The sitemap is unchanged. The pre-change script, run from the same directory against the same build, reports the same row it reports now — 346 | 346 | 346 | 0 | 0 | 0 — and its --self-test fires the identical rule set per case. BY_URL encodes exactly the previous behaviour, including duplicate-url, which the title vocabulary deliberately does not use (a title is a label, not an identity).

Self-test, on f4a7902, exit 0 — 16 cases over 9 rules and 3 artifacts:

✓ llms-full.txt carries another locale (the #177 shape) fired [unexpected-locale-title]
✓ llms.txt carries another locale (the #170 shape)     fired [unexpected-locale-title]
✓ llms-full.txt has dropped an English page            fired [missing-locale-title]
✓ llms.txt has dropped an English page                 fired [missing-locale-title]
✓ llms-full.txt present but unreadable                 fired [artifact-empty]
✓ llms.txt present but unreadable                      fired [artifact-empty]
✓ a `# ` line inside a code fence is not a page title  fired [—]
✓ a title named in prose is not an advertisement       fired [—]
✓ no title is exclusive to any locale                  fired [nothing-expected]
✓ locale-exclusive titles {"en":["Deep","Guide","Home"],"zh-Hans":[],"ja":["ガイド"]}

The last three green cases are assertions, not filler. Both readers are structured rather than substring searches, because a substring search over these bodies produces four false hits on today's tree — Glossar inside Glossary, and Datasources, Roles, Licence sitting in English prose — each reading as a foreign page being served. The llms-full.txt reader is fence-aware for the same reason: 14 lines in the corpus open with # inside a shell fence.

The self-test's own new check was proven able to fail. RULES coverage alone stopped being sufficient once one rule could fire for any of three artifacts, so the self-test now also asserts every ARTIFACTS entry was driven red on its own content. Injecting a fourth, always-green artifact into a copy outside the repo made exactly that check fail (exit 1, all 21 other assertions passing) — and closing a loophole found while testing it: artifact-missing does not count toward the tally, since the no built artifact case omits every file at once and would hand any new entry free coverage.

Gates run on f4a7902: pnpm turbo run type-check build (2 tasks successful), pnpm turbo run test (✓ 3 self-test(s) passed), and the gate itself (exit 0). All heavy runs went through the container's shared verify lock.

Notes for review

• #196 and #194 are in flight and neither touches this file. #196 is not addressed here — it remains open, and this gate is deliberately blind to the ordering it changes.
• The summary table gained a compared as column and split advertised into read / in scope, so a vocabulary that restricts the comparison cannot do so invisibly. The sitemap's numbers are identical in both columns.

Generated by Claude Code

---

Generated by Claude Code