fix(docs): resolve a browser tag by BCP 47 truncation, one subtag at a time

[hotlong]

Fixes #217

Notation: this repo's body sanitizer strips angle brackets and decodes HTML numeric character references, including inside code fences. Nothing below needs either — truncation steps are written with the word "to", and header captures are quoted as plain text.

All readings below are from 2dd7630, the head of this branch.

What was wrong

normalizeLanguage took exactly two shots at every Accept-Language tag: the exact tag, then the substring before the first hyphen. LANGUAGE_MAPPING held only Chinese and Korean rows, so for every other language both shots missed and the function handed back the tag unchanged — which is not in i18n.languages once it carries a region, so negotiation fell through to English.

Measured, not reasoned: four of the six locales we pay to translate were unreachable for a browser sending a single region tag, and zh-Hant-TW — whose middle subtag names the script — skipped straight past zh-Hant to zh and landed on Simplified. That last row is the one #218 made newly wrong in kind rather than degree: now that Traditional ships, it is a wrong answer sitting beside four right ones.

Most browsers send de-AT,de;q=0.9, and the bare de in the second list entry rescued them. That is why this survived unnoticed, and why every measurement below uses a single tag with no q-list.

The change

One file, apps/docs/middleware.ts, as the card's surface allowed.

normalizeLanguage becomes resolveSupportedLanguage(tag), which walks the tag the way RFC 4647 section 3.4 defines lookup: whole tag, drop the last subtag, repeat — zh-Hant-TW, then zh-Hant, then zh. It returns a supported locale or undefined; the caller takes the first tag that resolves to something.

The two questions the card asked to be settled explicitly are settled in the code, not by accident:

1. Most specific first. The walk runs long to short, so zh-Hant-TW reaches zh-Hant before it can reach zh. Short to long would answer Simplified for every Traditional tag that carries a region — the exact defect, with the sign flipped.

2. The table beats truncation, at every step of the walk. LANGUAGE_MAPPING is a ruling about what a tag MEANS; truncation is the mechanical default for tags nobody has ruled on. Where they disagree the ruling wins — that is what keeps zh-TW on Traditional instead of the Simplified that dropping TW would reach. Consulting the table at every step rather than only on the full tag is also what lets a longer tag built on a ruled one land correctly: zh-TW-x-private reaches Traditional (row in the matrix), where a table consulted only on the full tag would have dropped to zh and answered Simplified.

The membership check is the only exit. Both paths leave through the same SUPPORTED_LANGUAGES.includes(...), so neither mechanism can hand back a locale the site does not publish. Truncation manufactures strings out of whatever the client sent (xx, tlh), and a table row outliving the locale it names would do the same; either would otherwise produce a redirect to a path that 404s and re-enters negotiation.

The table is now an exceptions list, and six rows left it

Truncation subsumes every row whose answer is its own base language, so zh-CN, zh-SG, zh-Hans, zh-Hant, ko and ko-KR are removed. What remains is the one decision truncation cannot make, because the information is not in the tag:

row	why it cannot be derived
zh to zh-Hans	bare zh is not a supported tag at all; the house answer for unqualified Chinese is Simplified
zh-TW / zh-HK / zh-MO to zh-Hant	the region implies the Traditional script; truncation would drop it, reach zh, and answer Simplified

This is not cosmetic. A table containing zh-CN, ko-KR and zh-Hans teaches the next reader — human or model — that region tags belong in the table, which is precisely why de-AT was never added and never worked. The comment above the table now says the rule: add a row only when the mechanical answer is WRONG, never when it is merely absent.

Pruning is proven behaviour-neutral, not assumed. A differential harness ran three algorithms over a 167-tag corpus (every old table key, every supported tag, every card tag, the case variants, and a 12-region cross-product over 11 base languages): the old two-shot lookup, truncation with the FULL old table, and truncation with the pruned table that ships here.

corpus: 167 tags

A) pruning the subsumed rows (full table vs pruned table, both with truncation):
   IDENTICAL on all 167 tags — the pruned rows were subsumed by truncation.

B) behaviour changes from the fix (old two-shot vs shipped):  66 tag(s)

All 66 changed tags are improvements in the card's direction: 13 de-*, 12 es-*, 12 fr-*, 12 ja-* region tags moving from English to their language, and 17 zh-Hant-* / zh-*-x-* tags moving from Simplified to Traditional.

Verification

Against a running next start on the build of this branch, Accept-Language set to exactly one tag with no q-list, asserting the Location header. The before column is a real second run, not a reading of the source: the pre-fix middleware.ts was checked out over the tree, the mutation confirmed on disk by blob hash against d454ccb, rebuilt with turbo run build --force, measured, then restored — restore proven by blob hash identity with HEAD and an empty git diff HEAD, not by an exit code.

A. One tag, no q-list — GET /docs/quickstart

This is the shape the defect lives in.

Accept-Language	before	after	moved
de-AT	200, English, no redirect	307 /de/docs/quickstart	yes
fr-CA	200, English, no redirect	307 /fr/docs/quickstart	yes
es-MX	200, English, no redirect	307 /es/docs/quickstart	yes
ja-JP	200, English, no redirect	307 /ja/docs/quickstart	yes
zh-Hant-TW	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
zh-TW	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-HK	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-MO	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-Hant	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart	—
zh-CN	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh-Hans	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh-SG	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh-Hans-CN	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
ko	307 /ko/docs/quickstart	307 /ko/docs/quickstart	—
ko-KR	307 /ko/docs/quickstart	307 /ko/docs/quickstart	—
de	307 /de/docs/quickstart	307 /de/docs/quickstart	—
es	307 /es/docs/quickstart	307 /es/docs/quickstart	—
fr	307 /fr/docs/quickstart	307 /fr/docs/quickstart	—
ja	307 /ja/docs/quickstart	307 /ja/docs/quickstart	—
en	200, English, no redirect	200, English, no redirect	—
en-GB	200, English, no redirect	200, English, no redirect	—
xx-YY	200, English, no redirect	200, English, no redirect	—
tlh	200, English, no redirect	200, English, no redirect	—
nan-Hant-TW	200, English, no redirect	200, English, no redirect	—
yue-HK	200, English, no redirect	200, English, no redirect	—
*	200, English, no redirect	200, English, no redirect	—
zh-TW-x-private	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
zh-Hant-x-private	307 /zh-Hans/docs/quickstart	307 /zh-Hant/docs/quickstart	yes
zh-hant-tw	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
zh-tw	307 /zh-Hans/docs/quickstart	307 /zh-Hans/docs/quickstart	—
de-at	200, English, no redirect	307 /de/docs/quickstart	yes
ko-kr	307 /ko/docs/quickstart	307 /ko/docs/quickstart	—

Four rows are worth reading twice:

• xx-YY and tlh land on English, not on an invented locale. Truncation exhausts the tag and returns nothing; negotiation moves on. That is the membership check doing its job.
• nan-Hant-TW and yue-HK also land on English. Min Nan and Cantonese are not Mandarin, and truncation does not leak them into zh-Hant through the script subtag: the walk tries nan-Hant-TW, nan-Hant, nan, matches none, and stops. The tag would have to say zh for the Chinese rows to apply.
• zh-TW-x-private reaches Traditional. This is the table being consulted at every step and not only on the full tag.
• zh-hant-tw and zh-tw still reach Simplified. Matching is case-sensitive; this is pre-existing, unchanged in both columns, out of the card's scope, and now filed as Language-tag matching is case-sensitive, so a lowercase zh-tw or zh-hant-tw negotiates to Simplified #220 with the comment in the code pointing at it.

B. q-lists — the shape that already worked, confirmed not broken

Accept-Language	before	after
ja-JP,ja;q=0.9	307 /ja/docs/quickstart	307 /ja/docs/quickstart
zh-TW,zh;q=0.9	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart
de-AT,de;q=0.9	307 /de/docs/quickstart	307 /de/docs/quickstart
xx-YY,de;q=0.9	307 /de/docs/quickstart	307 /de/docs/quickstart
en-US,en;q=0.9	200, English, no redirect	200, English, no redirect
zh-Hant-TW,zh-Hant;q=0.9,zh;q=0.8	307 /zh-Hant/docs/quickstart	307 /zh-Hant/docs/quickstart

xx-YY,de;q=0.9 is the one to keep: an unresolvable first entry is skipped rather than ending the search, so German is still reached.

C. Root path — GET /

Accept-Language	before	after
de-AT	200 (English rewrite)	307 /de
zh-Hant-TW	307 /zh-Hans	307 /zh-Hant
ja-JP	200 (English rewrite)	307 /ja
en	307 /docs	307 /docs
xx-YY	307 /docs	307 /docs

D. #204's six-case redirect table, unchanged

request	status	Location
Host: www.objectos.app /docs/architecture	308	https://docs.objectos.ai/docs/architecture
Host: www.objectos.app /	308	https://docs.objectos.ai/
Host: www.objectos.app /docs/architecture?q=1	308	https://docs.objectos.ai/docs/architecture?q=1 (query kept)
Host: www.objectos.app /zh-Hans/docs/architecture	308	https://docs.objectos.ai/zh-Hans/docs/architecture
Host: docs.objectos.ai /docs/architecture	200	(control: canonical host not redirected away)
Host: docs.objectos.ai /cn/docs/architecture	308	/zh-Hans/docs/architecture

Identical to the table #204 recorded.

E. Legacy /cn/ redirects

request	status	Location
/cn	308	/zh-Hans
/cn/	308	/cn (Next strips the trailing slash first, then the row above applies)
/cn/docs	308	/zh-Hans/docs
/cn/docs/quickstart	308	/zh-Hans/docs/quickstart
/cnx/docs	404	(control: prefix match only, cnx is not cn)
/docs/cn	404	(control: first segment only)

The two-hop /cn/ case is pre-existing Next path normalisation, identical in both columns, and left alone.

F. Dotted machine-facing routes, with the worst-case header

Sent with Accept-Language: zh-Hant-TW — the tag that now negotiates hardest — to confirm the matcher still excludes them.

request	status	Location	content-type
/llms.txt	200	none	text/plain;charset=UTF-8
/llms-full.txt	200	none	text/plain;charset=UTF-8
/sitemap.xml	200	none	application/xml
/robots.txt	200	none	text/plain
/llms.mdx/docs/architecture	200	none	text/markdown

Gates

All at 2dd7630, each read from the command's own verdict line, with the exit code captured before any pipe.

gate	result
pnpm turbo run type-check --continue	exit 0 — Tasks: 1 successful, 1 total (cache miss, real execution)
pnpm turbo run build --force	exit 0 — Compiled successfully in 60s
pnpm turbo run test --force	exit 0 — 3 self-test(s) passed
node .github/scripts/check-locale-surface.mjs	exit 0 — sitemap 409 read / 409 expected / 0 unexpected / 0 missing / 0 duplicated
node apps/docs/scripts/gen-zh-hant.mjs --check	exit 0 — 73 generated file(s) match the zh-Hans sources byte for byte
node .github/scripts/check-node-floor.mjs	exit 0 — Every declared floor clears what the dependency tree requires

The locale surface reads 409 URLs, up from the 346 #204 recorded, which is #218's Traditional set now being advertised. --force on build and test because the turbo cache is shared across worktrees in this container and a replayed sibling green would make the before/after comparison meaningless.

Scope

One file. No changeset (this repo has no changeset flow). No content, no locale files, no i18n.ts — the locale list is untouched, so the oracle the locale gate derives is untouched too.

Filed while here, not touched here: #220, the case-sensitivity gap described above.

Generated by Claude Code

---

Generated by Claude Code