Skip to content

[WBDOCS-828] Autogenerate Query Expression Language reference from weave-internal#2486

Merged
mdlinville merged 8 commits intomainfrom
WBDOCS-828
Apr 22, 2026
Merged

[WBDOCS-828] Autogenerate Query Expression Language reference from weave-internal#2486
mdlinville merged 8 commits intomainfrom
WBDOCS-828

Conversation

@mdlinville
Copy link
Copy Markdown
Contributor

@mdlinville mdlinville commented Apr 17, 2026

Description

Adds an automated pipeline that regenerates English Query Expression Language (QEL) datatype reference pages under models/ref/query-panel/ from the same TypeScript source as the product: wandb/weave-internal (weave-js/), using upstream src/core/generateDocs.ts (run with vite-node after a yarn install --ignore-scripts).

This is query panel documentation, not the W&B Weave LLM product (/weave/).

Fixes WBDOCS-828

What changed

  • New scripts under scripts/reference-generation/query-panel/:
    • generate_query_panel_reference.sh – optional shallow clone of weave-internal (PAT in QUERY_PANELS_READ_TOKEN), bootstraps docs/README.md, runs Yarn and upstream doc generation, then calls the converter.
    • convert_query_panel_md.py – turns docs_gen/*.md into Mintlify MDX: front matter, internal links to /models/ref/query-panel/.../, table header fix, HTML h3 → ### anchors, duplicate-id handling (including -list suffixes for List Ops), refresh of the datatype list in models/ref/query-panel.mdx, and sync of the Query Expression Language page lists in docs.json for en / ja / ko.
    • README.md – scope, prerequisites, local and CI usage.
    • New workflow .github/workflows/generate-query-panel-reference.yml – manual workflow_dispatch, uses repo secret QUERY_PANELS_READ_ONLY_PAT, Node 20, Yarn 1 (Corepack), runs the shell script + unittest, opens a draft PR when there are changes (persist-credentials: false on checkout, same pattern as code examples sync).
    • New tests scripts/reference-generation/query-panel/tests/test_query_panel_mdx_quality.py – no duplicate anchor ids, no legacy docs.wandb.ai/ref/weave links, overview markers present.
  • Regenerated content – models/ref/query-panel/.mdx and models/ref/query-panel.mdx (datatype list wrapped in MDX {/ ... */} markers so Mintlify parses the file).
    • Fixes duplicated in-page anchors (only the first instance of an anchor is navigable).
    • Handles indefinite articles when the noun starts with a vowel ("a ArtifactType" -> "an ArtifactType".
    • No bulk edits to ja/ or ko/ QEL pages.

Why?

WBDOCS-828: QEL is defined in TypeScript; docs were previously maintained by hand and could drift (for example duplicate anchors in Chainable vs List sections). This ties public docs to the same generator path the internal repo uses, with Mintlify-specific post-processing.

How to test (reviewers)

  1. Unit tests

    From repo root:

    python3 -m unittest discover -s scripts/reference-generation/query-panel/tests -p 'test_*.py' -v
  2. Local full pipeline (recommended if you have weave-internal)
    With an existing weave-js checkout:

    export WEAVE_JS_ROOT=/path/to/weave-internal/weave-js
    ./scripts/reference-generation/query-panel/generate_query_panel_reference.sh

    Or rely on clone + read token (same variable name as CI):

    export QUERY_PANELS_READ_TOKEN="<PAT with contents:read on wandb/weave-internal>"
    ./scripts/reference-generation/query-panel/generate_query_panel_reference.sh

    Then inspect git diff under models/ref/query-panel/, models/ref/query-panel.mdx, and docs.json.

  3. Mintlify

    • Optionally, run commands like mint broken-links, mint validate, mint dev locally. This PR's CI will also check these.
    • Spot-check Reference → Query Expression Language and a few datatype pages (for example run, boolean, string).
  4. GitHub Actions after this PR merges
    Run workflow Generate Query Panel (QEL) reference on this branch. Confirm the job clones weave-internal, completes generation and tests, and that a draft PR is created only when the working tree would change (or no PR if output matches main).

Testing

  • Use the tooling to generate the references locally
  • Local build succeeds without errors (mint dev)
  • Local link check succeeds without errors (mint broken-links)
  • PR tests succeed

@mintlify
Copy link
Copy Markdown
Contributor

mintlify Bot commented Apr 17, 2026

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
wandb 🟢 Ready View Preview Apr 17, 2026, 8:31 PM

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 17, 2026

📚 Mintlify Preview Links

🔗 View Full Preview

✨ Added (1 total)

⚙️ Other (1)
File
scripts/reference-generation/query-panel/README.md

📝 Changed (25 total)

📄 Pages (25)

File Preview
models/ref/query-panel.mdx Query Panel
models/ref/query-panel/artifact-type.mdx Artifact Type
models/ref/query-panel/artifact-version.mdx Artifact Version
models/ref/query-panel/artifact.mdx Artifact
models/ref/query-panel/audio-file.mdx Audio File
models/ref/query-panel/bokeh-file.mdx Bokeh File
models/ref/query-panel/boolean.mdx Boolean
models/ref/query-panel/entity.mdx Entity
models/ref/query-panel/file.mdx File
models/ref/query-panel/float.mdx Float
... and 15 more files

🤖 Generated automatically when Mintlify deployment succeeds
📍 Deployment: 9e0a5fd at 2026-04-22 20:42:50 UTC

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 17, 2026

🔗 Link Checker Results

All links are valid!

No broken links were detected.

Checked against: https://wb-21fd5541-wbdocs-828.mintlify.app

- Add script to generate the query panel
  references from https://github.com/wandb/weave-internal/tree/master/weave-js
- Add Github action to run the script on demand

  The action accesses the private repo using a fine-grained
  access token stored as a secret in the docs repo
@mdlinville mdlinville marked this pull request as ready for review April 17, 2026 20:52
@mdlinville mdlinville requested a review from a team as a code owner April 17, 2026 20:52
@domphan-wandb domphan-wandb self-requested a review April 17, 2026 21:13
Copy link
Copy Markdown

@domphan-wandb domphan-wandb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving with minor comment.

the grammar is off due to the logic that uses an or a based on the leading vowel.

for example, "an user". maybe you could add a list of nouns that could override the behavior?

@mdlinville mdlinville disabled auto-merge April 22, 2026 20:36
@mdlinville mdlinville merged commit 1165dcc into main Apr 22, 2026
10 checks passed
@mdlinville mdlinville deleted the WBDOCS-828 branch April 22, 2026 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants