feat(cli/telemetry): surface unrecognized agents in the agent_runtime=null bucket

[jrusso1020]

Why

agent_runtime (shipped v0.6.37) is a closed allowlist — an agent we have no VENDOR_RULE for collapses to agent_runtime=null with no trace of what it was. Today that null bucket is ~87k users / ~18% of CLI users over the last 30 days, and it's on current 0.7.x versions (not stale clients). ~50k of those are non-tty / non-CI / non-container — i.e. something is driving the CLI programmatically and we can't name it. New agents stay invisible until someone reverse-engineers their marker by hand.

This adds a self-populating discovery signal so an unrecognized agent surfaces on its own instead of vanishing into null.

What

New detectAgentHints() in agent_runtime.ts, computed only for the null bucket (gated off the ~82% of classified events in system.ts):

• agent_hint — value of AGENT / AI_AGENT, the emerging self-identification convention (Crush and Goose already set AGENT=<name>). Directly names agents the allowlist misses.
• term_program — raw TERM_PROGRAM (editor name). Catches the IDE-terminal class (Zed, new VS Code forks) the same way the existing cursor/windsurf rules do.
• agent_env_hints — sorted, comma-joined list of "agent-ish" env-var key names present but matched by no vendor rule (AGENT|ASSISTANT|COPILOT|CODEX|CLAUDE|CODING|LLM|_THREAD_ID|_SESSION_ID, minus SSH/GPG false-friends). A compact fingerprint that clusters by agent.

Wired through getSystemMeta() → trackEvent() like the existing agent_runtime super-property.

Privacy

Consistent with the file's existing "never read secret-shaped values" stance:

• agent_env_hints emits key names only, never values.
• The three value-reads (AGENT/AI_AGENT/TERM_PROGRAM) are vars whose sole purpose is non-secret identification, each passed through a strict ^[a-z0-9_.-]{1,32}$ slug filter — anything long, spaced, or secret-shaped is dropped to null. Test asserts an sk-ant-…-shaped value returns null.

No PII, paths, or content — the existing telemetry notice still holds.

Using it

Once released and users upgrade, breakdown agent_hint / agent_env_hints filtered to agent_runtime IS NULL AND is_tty = false → a ranked leaderboard of new agents. When one climbs, promote it to a VENDOR_RULE and it moves into the named buckets on the existing DAU-by-agent_runtime insight.

Test plan

• bunx vitest run packages/cli/src/telemetry/agent_runtime.test.ts — 6 new tests (AGENT/AI_AGENT capture, secret-shaped drop, TERM_PROGRAM, unknown key surfaced, SSH false-friend excluded)
• oxlint / oxfmt / tsc clean; pre-commit gates pass

🤖 Generated with Claude Code

[miguel-heygen]

Reviewed b9bccf87; no prior reviews/comments were present. CI is green.

Strengths:

• packages/cli/src/telemetry/system.ts:77 correctly gates discovery hint collection to agent_runtime === null, so known-agent events do not get extra telemetry.
• packages/cli/src/telemetry/agent_runtime.ts:323 keeps agent_env_hints to env-var key names only, sorted/capped, which matches the intended low-content discovery signal.

Blocker:

• packages/cli/src/telemetry/agent_runtime.ts:293 / packages/cli/src/telemetry/agent_runtime.ts:320 / packages/cli/src/telemetry/client.ts:95: sanitizeHint() only lowercases and applies ^[a-z0-9_.-]{1,32}$, then trackEvent() sends the result as agent_hint/term_program. That means short credential-shaped values still pass the “safe slug” filter and get sent to PostHog. I verified the exact regex accepts examples like AGENT=sk-ant-api03, AGENT=AKIAIOSFODNN7EXAMPLE, and AGENT=github_pat_abc after lowercasing. The PR privacy contract says secret-shaped values are dropped, but the test at packages/cli/src/telemetry/agent_runtime.test.ts:358 only proves an overlong secret-looking value is dropped. Please add an explicit secret-shape/credential-prefix/entropy guard, or otherwise avoid value capture for unknown values, plus tests for short credential-shaped strings.

Verdict: REQUEST CHANGES
Reasoning: The null-bucket discovery shape is sound, but the current sanitizer can still emit short credential-shaped env values as telemetry, which violates the privacy boundary this PR relies on.

— Magi

[miguel-heygen]

Re-reviewed 99a2ccc58 against the privacy blocker from my prior review.

The blocker is resolved:

• packages/cli/src/telemetry/agent_runtime.ts:302 adds explicit credential/token prefixes, including the short examples I flagged.
• packages/cli/src/telemetry/agent_runtime.ts:323 adds the long unbroken alphanumeric-run guard, so short prefixless token bodies are also dropped.
• packages/cli/src/telemetry/agent_runtime.ts:330 now requires both the safe slug shape and the credential guard before returning a hint.
• packages/cli/src/telemetry/agent_runtime.test.ts:366 pins the exact short secret-shaped cases plus the positive multi-segment agent-name case.

I also sanity-checked the guard locally against the reviewed examples: sk-ant-api03, AKIAIOSFODNN7EXAMPLE, github_pat_abc, ghp_0123456789abcdef, ya29.a0veryrealtoken, and deadbeefdeadbeef01 all drop to null, while gemini_managed_agent, zed, and crush still pass.

Note: at the time of this review, the core CLI/code checks were green (Test, Typecheck, Lint, Build, CLI smoke, CodeQL, Windows CLI shim), while some long render/regression jobs were still pending in GitHub. No failing checks were present.

Verdict: APPROVE
Reasoning: The privacy boundary I requested is now enforced in code and covered by focused regression tests; remaining merge gating is CI completion, not code review.

— Magi