feat: data-aware MCP tutorial (scout, real-schema quests, demo seeding)#600
Draft
joethreepwood wants to merge 1 commit into
Draft
feat: data-aware MCP tutorial (scout, real-schema quests, demo seeding)#600joethreepwood wants to merge 1 commit into
joethreepwood wants to merge 1 commit into
Conversation
Make the MCP tutorial look at the project before it talks. A fresh install with no events used to hit "show me my top events", get nothing back, and the agent would improvise — a data-less dead end. This adds a best-effort scout after auth and routes every branch on what it finds. - Scout (src/lib/mcp-project-profile.ts): one HogQL query for event volume + the project's real event names, plus parallel limit=1 REST existence checks for replay/surveys/flags/experiments/warehouse. Time-boxed and best-effort — a slow/failed probe returns a `degraded` profile and the tutorial falls back to the legacy static kit. Never throws, never blocks. - Real-schema quests: the picker is generated from the project's own top custom events (funnel/trend/breakdown) instead of a hardcoded list. The agent orders funnel steps, so volume-sorted input is fine. - Activation cross-sells: a product the scout found NO data for becomes a "here's how to turn on Session Replay / Error Tracking" prompt (routed through docs-search, which never dead-ends) ordered by role affinity — turning a dead end into a product-discovery beat. - Seeding (src/lib/mcp-seed-events.ts): empty projects are offered a backdated SaaS funnel via posthog-node, tagged `wizard_seed:true` for one-click cleanup. $exception is left out so error tracking stays an activation cross-sell. getTutorialPicker() is the single composition point (empty → write-only + activation, rich → generated quests, sparse → safe read + quest, degraded → legacy). Product knowledge stays in the mcp-* modules and copy JSON; the screen + services stay orchestration. New phases: Scouting, SeedOffer, Seeding. Playground gains a `T` tier toggle (rich/sparse/empty) to preview every branch without the network. +36 tests (profile classification, seed dataset, generated quests, activation selection, picker composition). Builds clean, 848/848 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| return 'http://localhost:8010'; | ||
| } | ||
|
|
||
| if (host.includes('eu.posthog.com') || host.includes('eu.i.posthog.com')) { |
| return 'http://localhost:8010'; | ||
| } | ||
|
|
||
| if (host.includes('eu.posthog.com') || host.includes('eu.i.posthog.com')) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The MCP tutorial offers the same canned prompts to everyone, and nothing in it is data-aware. A freshly-installed project usually has near-zero events, so prompts like "show me my top events", "build a funnel", or "top errors this week" come back empty — and the agent, told it's a live demo, improvises and narrates emptiness. Users can wander into a data-less dead end where no path pays off.
Builds on the role-aware tutorial (#508) and the read-only/write scope split in
src/lib/oauth/program-scopes.ts— reads are exploration, the writable surfaces (dashboards/insights/notebooks/annotations) are the guaranteed wins.Changes
A best-effort scout runs once after auth and produces a
ProjectDataProfilethat routes every downstream branch, so the tutorial only ever offers playable moves.src/lib/mcp-project-profile.ts: one HogQL query (POST /api/projects/:id/query/) for event volume + the project's real event names, plus parallellimit=1REST existence checks for replay/surveys/flags/experiments/warehouse. Time-boxed (5s) and best-effort: a slow/failed probe returns adegradedprofile and the tutorial falls back to the legacy static kit. Never throws, never blocks.topCustomEvents(a funnel + trend + breakdown) instead of a hardcoded list. The agent orders funnel steps, so volume-sorted input is fine.docs-search, which never dead-ends), ordered by role affinity. A dead end becomes a product-discovery beat.src/lib/mcp-seed-events.ts: empty projects are offered a small backdated SaaS funnel ($pageview → signed_up → activated → feature_used → upgraded_to_paid) sent viaposthog-node, taggedwizard_seed:truefor one-click cleanup. Backdated across 7 days so time-windowed charts light up immediately.$exceptionis left out on purpose so error tracking stays an activation cross-sell.getTutorialPicker()is the single composition point (empty → write-only + activation, rich → generated quests, sparse → safe read + quest, degraded → legacy). New screen phases: Scouting / SeedOffer / Seeding. Product knowledge stays in themcp-*modules + copy JSON; the screen and services stay dumb orchestration; the agent SDK runner is untouched.Files
src/lib/mcp-project-profile.ts— probe + pureassembleProfilesrc/lib/mcp-seed-events.ts— seeder + deterministic dataset buildersrc/lib/mcp-role-prompts.ts/.copy.json— generated quests, activation cross-sells, write-only quests, seed-offer copysrc/ui/tui/screens/McpSuggestedPromptsScreen.tsx— Scouting/SeedOffer/Seeding phases, profile-driven pickersrc/ui/tui/services/mcp-suggested-prompts-services.ts—probeProjectData/seedDemoEventssrc/utils/urls.ts—getIngestionHostFromHostTtier toggle (rich/sparse/empty)Test plan
pnpm buildclean,tsc --noEmitcleanpnpm test— 848/848 (+36: profile classification, seed dataset, generated quests, activation selection, picker composition)pnpm lint— 0 errors, 0 new warnings in touched filespnpm try --playground→ "MCP tutorial"):Tcyclesrich → sparse → empty; confirm rich leads with generated quests, empty walks SeedOffer → Seeding,Rreorders activation cross-sells.pnpm try mcp tutorial --debug): walk an empty project (seed fork) and a project with events (generated quests);tail -f /tmp/posthog-wizard.log | grep -E "probeProjectData|seedDemoEvents".Note
The HogQL query/REST response shapes weren't validated against a live instance — the code degrades safely (legacy kit, never crashes) if a shape differs, but the probe wants one real run in wizard-workbench to confirm tier classification. Seeding writes ~89 demo events into whatever project you log into (tagged
wizard_seed:true) — use a throwaway.LLM context
Co-authored with Claude Code (Opus 4.8). Scope was deliberately the data-aware spine (ideas: scout, activation cross-sell, real-schema quests, seeding); a quest-log HUD and a role capstone were left as follow-ups.
🤖 Generated with Claude Code