Speed up search by ranking from a title map - #3408
Open
enf0rc3 wants to merge 4 commits into
Open
Conversation
|
Pull request environment is available at https://stoctodocspr3408.z22.web.core.windows.net. You can view the ephemeral environment status in Octopus Deploy. This environment will be automatically deprovisioned when the pull request is closed, or after 7 days of inactivity. |
Ranking needed a url and a title, and Pagefind keeps both in the per-page fragment — so ordering thirty results meant fetching thirty files before the panel could draw, and a page ranked past thirty could not be reached at all. The build now writes a map of result id to url and title beside the index, which is the join Pagefind's own result stub already carries. So the whole result set is ranked before anything is fetched, and fragments are fetched only for the rows being drawn: ten per batch, against thirty to thirty-five for every settled query before. On Slow 4G with the map served uncompressed, first results arrive in 7.2s against 8.9s; the map is 28 KB gzipped, so most of that 122 KB is transfer a CDN removes. The shallow-page search this replaces is gone with it — the second Pagefind query, the landing filter, LANDING_DEPTH and the attribute it needed on every page's content div. A page the query names now wins from anywhere in the list rather than from a shortlist of 227. Relevance holds on both traffic-weighted sets: real-searches 57% w-S@1 and 84% w-S@5, top-pages 90% and 98%, unchanged either side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enf0rc3
force-pushed
the
willlaugesen/search-title-map
branch
from
August 26, 2026 03:36
176d4d7 to
399b5a1
Compare
enf0rc3
marked this pull request as ready for review
August 26, 2026 04:53
Two things found reviewing the branch before publishing it. The fallback for a missing or mismatched title map drew results in Pagefind's own order and never ranked them, while its comment claimed otherwise. That is the order the reorder exists to correct — a bare BM25 list puts a getting-started page above the section it belongs to. `draw` now orders each batch as it draws it when the list reached it unranked, which is what the engine did before the map existed, and `more()` carries the same flag so later batches match. Verified by blocking the map: three of four sample queries still land on the right page, and the fourth is one whose answer ranks past the batch. A map that matches the index only partly dropped the unmatched results quietly, because the fallback fires only when nothing joins at all. It now warns with the counts. Also removed a stale doc comment left on the Ranked type. Relevance unchanged: real-searches holds at 57% w-S@1 and 84% w-S@5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An unhashed name lets a cache serve one build's map against another build's index. Pagefind hashes its own chunks for that reason, and the map has to be versioned with them: read against the wrong index it joins against nothing. The build now reads the hash out of pagefind-entry.json and writes docs-titles.<hash>.json; the client reads the same file to build the same name. Two requests where there was one, both while the index is warming and neither on the path of a search. It also means the map can carry an immutable cache header, which an unhashed name could not. The build fails if the index has more than one language, because one map cannot carry two hashes and the client would look under a name that does not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enf0rc3
force-pushed
the
willlaugesen/search-title-map
branch
from
August 26, 2026 05:30
efad2a1 to
3cf4285
Compare
Front Door caches `.js` for a week as immutable and compressed, and gives anything it does not recognise `no-cache`. It also strips ETags, so revalidating is a full re-download — and the map loads on every page that opens the overlay. A `.json` name meant paying 29 KB per page view. The map is now `docs-titles.<hash>.js`, an `export default`, loaded with the same dynamic import the engine already uses for `pagefind.js`. It lands in the existing static-content rule with no infrastructure change, and the hash in the name is what makes a week of immutable caching safe. `pagefind-entry.json` stays on the default no-cache rule, which is correct: it changes every build and the client reads the hash out of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Ranking a result needs its URL and title. Pagefind keeps both inside that result's own file. So ordering 30 results meant downloading 30 files before drawing a single row.
The change
The build writes one map of result id → URL and title. Ranking reads the map. Result files are downloaded only for the rows on screen.
Effects
Removed
landingfilter attribute on every pageA page the query names now wins from anywhere in the results.
Costs
Safety