fix(docs): match language tags case-insensitively, key folded and value never

[hotlong]

Fixes #220

Notation: this repo's body sanitizer strips angle brackets and decodes HTML numeric character references, including inside code fences. Nothing below needs either — RFC section numbers are written as the word "section", and redirect steps are written as the word "to".

All readings below are from 90e6c3f, the head of this branch.

What was wrong

apps/docs/middleware.ts looked a browser tag up with plain string equality. BCP 47 tags are case-insensitive — RFC 5646 section 2.1.1: "the tags and their subtags ... are not case sensitive" — and the casing the spec recommends (lowercase language, Titlecase script, UPPERCASE region) is a writing convention, not a constraint on the wire. Both lookup tables are written in that convention, so a client spelling its tags differently missed rows that name it exactly.

Measured on a rebuilt pre-fix tree, one tag, no q-list, asserting Location on /docs/quickstart: zh-tw, zh-hk, zh-mo, zh-hant and zh-hant-tw all negotiated to Simplified, because the Chinese keys are the only ones carrying a script or region subtag. Those are the five rows the card was filed for, and they reproduce exactly.

The same run found the defect is wider than the card's picture of it. The card says the other six locales are unaffected because their supported tags are already all-lowercase. That holds for the lowercase direction only. Any other casing missed every locale: DE-AT, JA-jp and KO-kr fell through to English, and ZH-TW and ZH-HANT-TW did too — a bare de is lowercase by convention as well, not by rule. So the fix is written as a fold over the whole key space, not as five more table rows, and it closes the uppercase direction in the same stroke. Those rows are in table A2 below.

The change

One file, apps/docs/middleware.ts, as the card's surface allowed.

One folded index, consulted by #217's walk

PUBLISHED_BY_FOLDED_LOCALE maps each folded locale spelling to the canonical string i18n.languages itself holds. CANONICAL_BY_FOLDED_TAG is that map plus the LANGUAGE_MAPPING rulings, and resolveSupportedLanguage — the RFC 4647 truncation walk that landed with #217 — now folds each candidate and takes a single look at it. The walk is otherwise untouched: still long to short, still consulting the rulings at every step.

Merging the two tables preserves #217's second ordering decision structurally rather than by repetition: the ruled rows are written last, so where a tag is both published and ruled, the ruling wins. No such tag exists today, and the order is fixed in code so that adding one cannot silently reverse the rule.

The key is folded. The value never is.

The payload is always the canonical string taken from i18n.languages, never the folded key and never the caller's spelling. A folded value would build a redirect to /zh-hant/..., which the site does not publish: it 404s, that 404 re-enters negotiation, and a wrong-script page becomes a loop.

The membership check is still the only exit

#217 made a single check against i18n.languages the one door out of the walk. That check is not removed here — it is moved to where the index is built, and paid once. A supported locale enters the index as itself; a LANGUAGE_MAPPING row enters only if its target resolves through the published map. Every value in the index is therefore a member of i18n.languages by construction, so the walk still has exactly one exit and still cannot hand back a locale the site does not publish. Same invariant, established at module load instead of re-checked per lookup. No second door.

The FD_LOCALE cookie is deliberately not folded, and the file says so: it is written by this file and nowhere else, always in canonical form, so folding it would widen only what a hand-set cookie can say. The fold belongs where strings arrive from outside — the header, and the path.

The decision: pathnameHasLocale now accepts a case variant and 308s to canonical

The card and the dispatch both asked for this to be settled either way rather than left to fall out of the matcher. It is settled in the code, with the reasoning written there.

Both answers are defensible on their face. URL path segments are case-sensitive (RFC 3986 section 6.2.2.1), so reading /zh-hant/... as a different resource is legitimate, and an extra redirect hop is a real cost.

What decided it is what the file actually did, measured before choosing:

request, on the pre-fix build	result
GET /zh-hant/docs/quickstart	404
GET /zh-hant/docs/quickstart with Accept-Language: zh-Hant	307 to /zh-Hant/zh-hant/docs/quickstart, which 404s

The second row settles it. A lowercase locale path did not fail as an unknown URL. It fell through to negotiation, which prepended a locale to a path that already named one and produced a URL nobody could have meant. That is the header-side defect's own shape — a wrong answer rather than an absent one, arrived at by accident — and it is reachable the same way, by a proxy that lowercases what it forwards.

So the first path segment is matched against the published locales case-insensitively, and a non-canonical spelling gets a 308 to the spelling the site publishes. 308 for the same reason as the legacy /cn/ rows: permanent, and it collapses the aliases onto one indexable URL. /zh-Hant/... still costs zero hops; only a URL that is broken today pays anything.

Three things it deliberately does not do, each stated in the file:

• Published locales only, never LANGUAGE_MAPPING. That table rules on what a browser tag means; it is not a list of URL aliases. /zh-TW/docs stays a 404 rather than becoming a second name for /zh-Hant/docs — serving one page under two URLs is an SEO problem this file has no mandate to create.
• No cookie is written. A locale-bearing path does not set one today (the branch ends in next()); this hop is URL normalisation, not negotiation, and /zh-hant/x should not carry a side effect that /zh-Hant/x lacks.
• It cannot loop. The target segment comes from i18n.languages, so the redirected request matches canonically and falls straight through. Confirmed by following the chains, below.

/EN/docs therefore takes two hops — 308 to /en/docs, then the default-locale strip 307s to /docs. Same shape as the pre-existing two-hop /cn/ case; each rule stays responsible for one thing.

Verification

Against a running next start, Accept-Language set to exactly one tag with no q-list, asserting the Location header. 89 rows, run twice — 71 unchanged, 18 changed, and every one of the 18 is an intended row of this fix.

How the before column was produced

The before column is a real second run on a rebuilt pre-fix tree, not a reading of the source.

Method, and a deliberate deviation from the mutate-measure-restore shape #223 used: the pre-fix column was measured first, on the pristine cae6bf5 tree, before this branch had any edit on it. That is strictly stronger than mutating and restoring — there was never a fixed build in the worktree to leak into the before column, and no restore to get wrong. The equivalent proofs were taken at the moment of the run:

PREFIX blob on disk: 9ec9fb44208b9e8d6e434a9cea3c43772b40fdb7
HEAD blob:           9ec9fb44208b9e8d6e434a9cea3c43772b40fdb7
git diff HEAD:       (empty)
BUILD_BEFORE_EXIT=0  turbo run build --force printed "Compiled successfully in 63s"

Both builds used --force: the turbo cache is shared across worktrees in this container, and a replayed sibling green would make the comparison meaningless. Both builds reported cache bypass, force executing.

The edit itself was confirmed on disk by content, not by an editor's exit code — the patch script fails loudly on a zero-match anchor, and the landing was checked by blob hash plus grep counts of the inserted and the removed text:

HEAD blob:    9ec9fb44208b9e8d6e434a9cea3c43772b40fdb7
on-disk blob: 3d9044c4fba3a18f2490e6c065baea23b2cda5c4   (changed)
inserted: CANONICAL_BY_FOLDED_TAG 3   toLowerCase() 5
removed:  pathnameHasLocale 0   SUPPORTED_LANGUAGES.includes(candidate) 0

The committed tree is byte-identical to the tree the after column was measured on: git rev-parse HEAD:apps/docs/middleware.ts is 63661d4991cf29df567b27359f0eb97cf77107b2, the same blob the after build was made from, and git status is clean.

A. #223's 33 single-tag rows, GET /docs/quickstart

Re-run in full. Two rows move; they are the two #220 was filed for.

Accept-Language	before	after	moved
de-AT	307 /de/docs/quickstart	307 /de/docs/quickstart	—
fr-CA	307 /fr/docs/quickstart	307 /fr/docs/quickstart	—
es-MX	307 /es/docs/quickstart	307 /es/docs/quickstart	—
ja-JP	307 /ja/docs/quickstart	307 /ja/docs/quickstart	—
zh-Hant-TW	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-TW	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-HK	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-MO	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-Hant	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-CN	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh-Hans	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh-SG	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh-Hans-CN	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
ko	307 /ko/docs/quickstart	307 /ko/docs/quickstart	—
ko-KR	307 /ko/docs/quickstart	307 /ko/docs/quickstart	—
de	307 /de/docs/quickstart	307 /de/docs/quickstart	—
es	307 /es/docs/quickstart	307 /es/docs/quickstart	—
fr	307 /fr/docs/quickstart	307 /fr/docs/quickstart	—
ja	307 /ja/docs/quickstart	307 /ja/docs/quickstart	—
en	200, English, no redirect	200, English, no redirect	—
en-GB	200, English, no redirect	200, English, no redirect	—
xx-YY	200, English, no redirect	200, English, no redirect	—
tlh	200, English, no redirect	200, English, no redirect	—
nan-Hant-TW	200, English, no redirect	200, English, no redirect	—
yue-HK	200, English, no redirect	200, English, no redirect	—
*	200, English, no redirect	200, English, no redirect	—
zh-TW-x-private	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-Hant-x-private	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-hant-tw	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
zh-tw	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
de-at	307 /de/docs/quickstart	307 /de/docs/quickstart	—
ko-kr	307 /ko/docs/quickstart	307 /ko/docs/quickstart	—

A2. The rest of #220's rows, and the case space around them

zh-hk, zh-mo and zh-hant complete the card's five. The uppercase rows were not in the card and are the reason the fix is written as a fold rather than as five extra table entries.

Accept-Language	before	after	moved
zh-hk	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
zh-mo	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
zh-hant	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
ZH-HANT-TW	200, English, no redirect	307 /zh-Hant/docs/quickstart	yes
zh-HANT	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
ZH-TW	200, English, no redirect	307 /zh-Hant/docs/quickstart	yes
DE-AT	200, English, no redirect	307 /de/docs/quickstart	yes
EN-GB	200, English, no redirect	200, English, no redirect	—
KO-kr	200, English, no redirect	307 /ko/docs/quickstart	yes
JA-jp	200, English, no redirect	307 /ja/docs/quickstart	yes
zh-hans	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh-hans-cn	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh-tw-x-private	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
NAN-HANT-TW	200, English, no redirect	200, English, no redirect	—
YUE-hk	200, English, no redirect	200, English, no redirect	—

EN-GB does not move, but it is right for a different reason afterwards: before, it missed both en-GB and en and fell through to the default, which happens to be English. Now it matches en and negotiates to it.

B. q-lists — unchanged

Accept-Language	before	after	moved
ja-JP,ja;q=0.9	307 /ja/docs/quickstart	307 /ja/docs/quickstart	—
zh-TW,zh;q=0.9	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
de-AT,de;q=0.9	307 /de/docs/quickstart	307 /de/docs/quickstart	—
xx-YY,de;q=0.9	307 /de/docs/quickstart	307 /de/docs/quickstart	—
en-US,en;q=0.9	200, English, no redirect	200, English, no redirect	—
zh-Hant-TW,zh-Hant;q=0.9,zh;q=0.8	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—

C. Root path GET / — unchanged

Accept-Language	before	after	moved
de-AT	307 /de	307 /de	—
zh-Hant-TW	307 /zh-Hant	307 /zh-Hant	—
ja-JP	307 /ja	307 /ja	—
en	307 /docs	307 /docs	—
xx-YY	307 /docs	307 /docs	—

D. #204's six-case host table — unchanged

request	before	after
Host: www.objectos.app /docs/architecture	308 https://docs.objectos.ai/docs/architecture	308 https://docs.objectos.ai/docs/architecture
Host: www.objectos.app /	308 https://docs.objectos.ai/	308 https://docs.objectos.ai/
Host: www.objectos.app /docs/architecture?q=1	308 https://docs.objectos.ai/docs/architecture?q=1	308 https://docs.objectos.ai/docs/architecture?q=1
Host: www.objectos.app /zh-Hans/docs/architecture	308 https://docs.objectos.ai/zh-Hans/docs/architecture	308 https://docs.objectos.ai/zh-Hans/docs/architecture
Host: docs.objectos.ai /docs/architecture	200, no redirect	200, no redirect
Host: docs.objectos.ai /cn/docs/architecture	308 /zh-Hans/docs/architecture	308 /zh-Hans/docs/architecture

E. Legacy /cn/ redirects — unchanged

request	before	after
/cn	308 /zh-Hans	308 /zh-Hans
/cn/	308 /cn	308 /cn
/cn/docs	308 /zh-Hans/docs	308 /zh-Hans/docs
/cn/docs/quickstart	308 /zh-Hans/docs/quickstart	308 /zh-Hans/docs/quickstart
/cnx/docs	404	404
/docs/cn	404	404

F. Dotted machine-facing routes, all sent with Accept-Language: zh-Hant-TW — unchanged

The worst-case header, to confirm the matcher still excludes these routes from negotiation entirely.

request	before	after	content-type
/llms.txt	200, no redirect	200, no redirect	text/plain;charset=UTF-8
/llms-full.txt	200, no redirect	200, no redirect	text/plain;charset=UTF-8
/sitemap.xml	200, no redirect	200, no redirect	application/xml
/robots.txt	200, no redirect	200, no redirect	text/plain
/llms.mdx/docs/architecture	200, no redirect	200, no redirect	text/markdown

G. The pathnameHasLocale decision, before and after

Requested explicitly: the before and after for /zh-hant/docs/quickstart and /zh-Hant/docs/quickstart, plus the neighbours that show the rule's edges. No Accept-Language header is sent on these rows. A 200 here is the localized page itself, not an English fallback.

request	before	after
/zh-hant/docs/quickstart	404	308 /zh-Hant/docs/quickstart
/zh-Hant/docs/quickstart	200, no redirect	200, no redirect
/zh-hans/docs/quickstart	404	308 /zh-Hans/docs/quickstart
/zh-Hans/docs/quickstart	200, no redirect	200, no redirect
/EN/docs/quickstart	404	308 /en/docs/quickstart
/en/docs/quickstart	307 /docs/quickstart	307 /docs/quickstart
/DE/docs/quickstart	404	308 /de/docs/quickstart
/de/docs/quickstart	200, no redirect	200, no redirect
/zh-hant	404	308 /zh-Hant
/zh-Hant	307 /zh-Hant/docs	307 /zh-Hant/docs
/zh-Hantfoo/docs/quickstart	404	404

/zh-Hantfoo/docs/quickstart is the control: matching is on the whole first segment, so a locale that is merely a prefix of the segment is still not a locale.

G2. The row that decided it

A Traditional browser on the lowercase path. The before column is not a clean 404 — it is a redirect that prepends a locale to a path that already names one.

request	before	after
/zh-hant/docs/quickstart (Accept-Language: zh-Hant)	307 /zh-Hant/zh-hant/docs/quickstart	308 /zh-Hant/docs/quickstart
/zh-Hant/docs/quickstart (Accept-Language: zh-Hant)	200, no redirect	200, no redirect

Chains followed, to prove the new 308 resolves

/zh-hant/docs/quickstart     1 hop  to 200 /zh-Hant/docs/quickstart
/zh-Hant/docs/quickstart     0 hops to 200 /zh-Hant/docs/quickstart
/zh-hans/docs/quickstart     1 hop  to 200 /zh-Hans/docs/quickstart
/DE/docs/quickstart          1 hop  to 200 /de/docs/quickstart
/EN/docs/quickstart          2 hops to 200 /docs/quickstart
/zh-hant                     2 hops to 200 /zh-Hant/docs

And the page actually served is in the right script, not merely the right Location:

Accept-Language: zh-tw   served page carries lang="zh-Hant"
Accept-Language: zh-cn   served page carries lang="zh-Hans"

Machine-checked against #223's recorded rows

The after column was asserted row by row against the values PR #223 recorded, rather than eyeballed: 33 table-A rows, 6 q-list rows, 5 root-path rows, 6 host rows, 6 legacy /cn/ rows and 5 dotted routes with their content-types, plus the three remaining #220 rows. 64 assertions.

#223 table A rows declared: 33 (expected 33)
assertions run: 64
VERDICT: PASS - every asserted row matches

Four rows worth reading twice, all unchanged: nan-Hant-TW and yue-HK still reach English — folding the key does not leak Min Nan or Cantonese into zh-Hant through the script subtag, because the walk still has to match a whole subtag prefix and none of nan-hant-tw, nan-hant, nan is in the index. xx-YY and tlh still invent nothing — they exhaust the tag and negotiation moves on. That is the membership check still doing its job from inside the index build.

Gates

All at 90e6c3f, re-run as a union after the final commit, each read from the command's own verdict line with the exit code captured before any pipe.

gate	result
pnpm turbo run type-check --continue --force	exit 0 — Tasks: 1 successful, 1 total (cache bypass, force executing)
pnpm turbo run build --force	exit 0 — Compiled successfully in 58s
pnpm turbo run test --force	exit 0 — 4 self-test(s) passed
node .github/scripts/check-locale-surface.mjs	exit 0 — sitemap 409 read / 409 in scope / 409 expected / 0 unexpected / 0 missing / 0 duplicated
node apps/docs/scripts/gen-zh-hant.mjs --check	exit 0 — 73 generated file(s) match the zh-Hans sources byte for byte
node .github/scripts/check-node-floor.mjs	exit 0 — Every declared floor clears what the dependency tree requires

The test gate reads 4 self-test(s) passed where #223 recorded 3. That is not this change: #223 branched from d454ccb and so never carried #222's added self-test, which is on main underneath this branch.

check-translation-ownership.mjs was not run as a gate — invoked bare it exits 1 on its own usage message, because it takes --actor and --files from the workflow. That is an invocation error, not a measurement, so it is recorded here as not measured rather than as a result. CI runs it with its arguments.

Scope

One file. No changeset — this repo has no changeset flow, and no .changeset directory. No content, no locale files, no i18n.ts: the locale list is untouched, so the oracle the locale-surface gate derives is untouched too, which is why it still reads 409.

Considered and deliberately left alone: LEGACY_LOCALE_REDIRECTS is still matched case-sensitively, so /CN/docs remains a 404. Folding it would be a change to a legacy alias table rather than to language-tag matching, cn is not a BCP 47 tag, and no measured client produces the uppercase form. Naming it here so it is visible rather than discovered.

No new issues were filed: nothing was found outside this card's own defect.

Generated by Claude Code

---

Generated by Claude Code