feat: Add dynamic markdown routing for llms.txt - #8132
Closed
artimath wants to merge 4 commits into
Closed
Conversation
Fixes #8117 Adds llms.txt and llms-full.txt generation following https://llmstxt.org/ spec to ensure AI models can access the latest React documentation. - llms.txt: 13KB hierarchical link index (167 pages) - llms-full.txt: 2.7MB full embedded documentation Generated from sidebar configs and markdown content at build time. Integrated into build pipeline via package.json scripts.
- Create src/utils/llms.js following src/utils/rss.js pattern - Refactor scripts/generate-llms-txt.js to thin wrapper - Remove llms-full.txt (simplify to single file approach) - Add dual copyright headers matching codebase style - Extract helper functions for cleaner organization Generates llms.txt (13KB, 167 links) covering 94% of sidebar entries. Missing 10 files are section index pages with path mismatches. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Upgrade 'remark' to version 12.0.1 and related packages for improved functionality. - Remove obsolete .yarnrc.yml file as it is no longer needed.
- Convert static llms.txt files to Next.js dynamic routes - Add /api/raw-md endpoint for serving markdown content - Create lib/llms utilities for documentation extraction - Update build script to support dynamic generation - Remove static public files in favor of API endpoints 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Size changesDetails📦 Next.js Bundle Analysis for react-devThis analysis was generated by the Next.js Bundle Analysis action. 🤖
|
| Page | Size (compressed) |
|---|---|
global |
115.14 KB (🟡 +4.6 KB) |
Details
The global bundle is the javascript bundle that loads alongside every page. It is in its own category because its impact is much higher - an increase to its size means that every page on your website loads slower, and a decrease means every page loads faster.
Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis
If you want further insight into what is behind the changes, give @next/bundle-analyzer a try!
Contributor
Author
|
closing to recreate with correct base branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #8130 (stacked PR) to add dynamic markdown routing, enabling AI agents to fetch individual doc pages as markdown via clean URLs.
Stack
Summary
/api/raw-md/[[...slug]]API route to serve markdown files dynamically/:path*.mdroutes to the API endpoint.mdformat in follow-up commitChanges Beyond Base PR (#8130)
Dynamic markdown API (
src/pages/api/raw-md/[[...slug]].ts):.md,.mdx,/index.md,/index.mdx)Refactored generation (
src/lib/llms-utils.ts):<Intro>,<Sandpack>, etc)Updated generation (
src/utils/llms.js):Routing config (
next.config.js):/:path*.md→/api/raw-md/:path*Examples
With this PR, AI agents can:
/learn.mdfor the learn index page/reference/react/hooks.mdfor specific API docs.mdformat llms.txtFiles Changed vs Base
next.config.js: rewrite rulessrc/lib/llms-utils.ts: shared markdown utilities (new)src/pages/api/raw-md/[[...slug]].ts: dynamic route handler (new)src/utils/llms.js: refactored to use shared utilsyarn.lock: updated dependenciesKnown Issues
.mdlinks instead of HTML URLsTest Plan
/learn.mdreturns markdown/reference/react/hooks.mdworks/reference/react-dom/client/createRoot.mdyarn buildcompletes successfully