Add AI-assisted TechDocs Assessment spec

[nate-double-u]

Adds specs/ai-assisted-techdocs-assessment-spec.md: an AI-assisted, human-supervised TechDocs assessment workflow.

Goal: compress the 6–8 week assessment toward 2 weeks, so the budget that funds assessments serves more projects. A Copilot agent drafts, humans review and decide, in the open. Automates the existing methodology; does not change it.

Shape: Part I, requirements; Part II, the binding to the Copilot cloud agent, GitHub-native lifecycle, provenance block, and build plan. Part I wins on conflict.

Key points:

• The agent cannot write outside cncf/techdocs: platform-enforced, not promised.
• Expert review and human-only approval at every phase gate; an adversarial verifier agent fact-checks drafts by default, and a reviewer who skips a pass records why on the PR.
• Every deliverable discloses how it was produced, AI-drafted or hand-written, and carries its evidence trail (provenance block).

Status: proposal for discussion. Open items are collected in section 17.

---

"Deploy preview": https://github.com/nate-double-u/cncf-techdocs/blob/ai-assisted-assessment-spec/specs/ai-assisted-techdocs-assessment-spec.md

[netlify]

✅ Deploy Preview for cncf-techdocs ready!

Name	Link
🔨 Latest commit	2f872d5
🔍 Latest deploy log	https://app.netlify.com/projects/cncf-techdocs/deploys/6a7e23ca259aa4000754e4c6
😎 Deploy Preview	https://deploy-preview-363--cncf-techdocs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

Adds a proposal for an AI-assisted, human-supervised TechDocs assessment workflow.

Changes:

• Defines goals, constraints, roles, and phase-gated lifecycle.
• Specifies deliverables, security controls, and acceptance criteria.
• Documents open questions and pilot measurements.

Suppressed comments (3)

docs/analysis/ai-assisted-techdocs-assessment-spec.md:204

• issues/*.md contradicts the current methodology: docs/analysis/howto.md:173-187 defines one _PROJECT_-issues.md backlog, and docs/analysis/templates/issue.md:6-9 says to combine all proposed issues when they remain in cncf/techdocs. Use one backlog document here so this workflow actually automates rather than changes the canonical method.

- Phase C: Issue backlog (`issues/*.md`, one file per issue, each scoped to

docs/analysis/ai-assisted-techdocs-assessment-spec.md:291

• This restatement again permits the reviewer—who refined and owns the draft—to provide final sign-off merely because the agent drafted it. That is grading one's own work under the role model in section 4; require the approver to have served as neither drafter nor reviewer.

  approver (section 4), a writer who did not draft that phase, to avoid grading
  one's own work. The bar is parity with the human baselines (Flatcar, Knative,

docs/analysis/ai-assisted-techdocs-assessment-spec.md:325

• The staffing constraint repeats the insufficient "did not draft" test. Since the reviewer edits and owns the draft, approver separation also has to exclude anyone who reviewed that phase; otherwise the staffing plan can assign both roles to the same writer and defeat independent sign-off.

- Small-team staffing. Sustaining the approver separation (a phase's approver
  must be a writer who did not draft it; sections 4, 10) when the same few
  writers wear multiple hats.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (5)

docs/ai-assisted-techdocs-assessment-spec.md:342

• Part II promises components/layout, agent definitions, provenance, and a build plan, but the document jumps from section 12 to section 17. This also leaves live references to sections 13, 15, and 16 unresolved, so the claimed operational specification cannot yet be implemented or reviewed. Add sections 13–16 before merging, or remove/defer these promises and dangling references explicitly.

assumed. Part II covers, in order: the platform binding, lifecycle bindings,
components and repository layout, agent definitions, the provenance block, and
the build plan.

docs/ai-assisted-techdocs-assessment-spec.md:295

• The cited methodology supports independent, time-bounded issues, but it does not specify a four-hour ceiling. Remove that unsupported precision so this grounding statement remains accurate.

- The 4-hour, independent, time-bounded issue scoping (Phase C) follows the
  howto and assistance-program guidance on backlog creation.

docs/ai-assisted-techdocs-assessment-spec.md:1

• The PR description says this specification is added at docs/analysis/ai-assisted-techdocs-assessment-spec.md, but the file is actually being added at the documentation root. Please either move it under docs/analysis/ as described or update the PR description to make the intended placement clear.

---

docs/ai-assisted-techdocs-assessment-spec.md:61

• This makes the implementation plan mandatory while claiming to match the existing methodology. The canonical how-to explicitly allows small projects with independent, time-bound recommendations to skip the implementation plan (docs/analysis/howto.md:161-165), and Knative—named here as a baseline—has no implementation document. Either preserve that optional path throughout the lifecycle and completeness check, or identify mandatory Phase B as another proposed methodology change.

This issue also appears on line 294 of the same file.

- G-7: Produce the three standard deliverables. Assessment, implementation plan,
  and an issue backlog, matching the existing methodology.

docs/ai-assisted-techdocs-assessment-spec.md:214

• The four-hour limit is a second, undisclosed methodology change. The canonical how-to allows “a few hours or a couple of days at most” (docs/analysis/howto.md:180-187), and the assistance guidance similarly allows hours or days (docs/assistance.md:82-91). Retain that range here, or explicitly list the tighter limit as a proposal alongside the file-layout change.

This issue also appears on line 340 of the same file.

- Phase C: Issue backlog. Each proposed issue is a separate file scoped to
  roughly 4 hours for someone experienced with the project and comfortable
  writing, plus an umbrella/index file that lists them. Effort estimates are the

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (6)

docs/ai-assisted-techdocs-assessment-spec.md:295

• Neither cited source specifies a four-hour scope: the how-to permits a few hours through a couple of days, and the assistance document only requires time-bounded issues. Claiming that this exact limit comes from the methodology makes the provenance section inaccurate.

- The 4-hour, independent, time-bounded issue scoping (Phase C) follows the
  howto and assistance-program guidance on backlog creation.

docs/ai-assisted-techdocs-assessment-spec.md:61

• The canonical how-to makes the implementation plan optional for small projects whose recommendations are already independent and time-bound (docs/analysis/howto.md:163-165). Requiring all three deliverables here—and again for acceptance—therefore changes the existing methodology despite NG-4. Either preserve that skip path throughout the lifecycle or explicitly identify and justify this additional methodology change.

- G-7: Produce the three standard deliverables. Assessment, implementation plan,
  and an issue backlog, matching the existing methodology.

docs/ai-assisted-techdocs-assessment-spec.md:216

• The four-hour limit is not the canonical methodology: the how-to allows “a few hours or a couple of days at most” (docs/analysis/howto.md:186-187), while the assistance guidance only says time-bounded. This silently narrows issue scope even though NG-4 names the file layout as the sole methodology change. Keep the existing range unless this is documented as another deliberate change.

This issue also appears on line 294 of the same file.

- Phase C: Issue backlog. Each proposed issue is a separate file scoped to
  roughly 4 hours for someone experienced with the project and comfortable
  writing, plus an umbrella/index file that lists them. Effort estimates are the
  agent's first pass and are sanity-checked by the reviewer; agent estimates are
  not reliable on their own.

docs/ai-assisted-techdocs-assessment-spec.md:308

• HC-7 says this section defines the verification method, but “a set number” supplies neither a minimum nor a complete sampling rule. The alternative to checking every finding is therefore not measurable, so reviewers could satisfy the same acceptance gate with materially different coverage. Specify the threshold and selection rule, or mark this as an unresolved requirement that must be settled before the pilot.

  definition is an open item; section 17). Verification is not a token sample:
  the reviewer checks every rating-bearing finding, or at minimum a set number
  per criterion, biased toward the highest-risk claims, and records in the
  deliverable which findings were verified (HC-7). Final sign-off is given by

docs/ai-assisted-techdocs-assessment-spec.md:438

• The GitHub timeline records elapsed time between transitions, but it cannot distinguish active writer work from waiting within those intervals. It therefore cannot by itself produce the writer-working-time-versus-waiting decomposition required by section 10; that measurement still needs separate instrumentation, as section 17 already acknowledges.

- The timeline is the measurement. Gate transitions (opened, accepted, draft PR,
  ready, approved, merged) are timestamped in issue and PR history, so the
  pilot's cycle-time decomposition (section 10) is harvested from the GitHub
  timeline rather than a separate log.

docs/ai-assisted-techdocs-assessment-spec.md:480

• Sections 14–16 are absent even though Part II promises agent definitions, the provenance block, and the build plan (lines 340–342), and multiple earlier passages link to those section numbers. As published, those references are dead and the operational specification omits three promised components. Add the missing sections before section 17, or remove and renumber every dependent reference if they are intentionally deferred.

## 17. Open questions and future work

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (6)

docs/ai-assisted-techdocs-assessment-spec.md:61

• Requiring all three deliverables changes the canonical workflow: docs/analysis/howto.md:163-165 explicitly allows small projects to skip the implementation plan and proceed directly to the backlog. Because the PR states that the existing methodology is unchanged, the lifecycle and completeness criteria need an optional Phase A→C transition, or this must be identified as another proposed methodology change.

- G-7: Produce the three standard deliverables. Assessment, implementation plan,
  and an issue backlog, matching the existing methodology.

docs/ai-assisted-techdocs-assessment-spec.md:309

• The claimed 4-hour rule is not in the canonical how-to: docs/analysis/howto.md:180-187 permits anything from a few hours to a couple of days. This spec imposes roughly four hours in sections 6 and 14 while NG-4 identifies only the file layout as an intentional methodology change; retain the canonical range or flag this as an additional proposal.

- The 4-hour, independent, time-bounded issue scoping (Phase C) follows the
  howto and assistance-program guidance on backlog creation.

docs/ai-assisted-techdocs-assessment-spec.md:372

• Current GitHub documentation says the cloud agent can leave changes on a branch without opening a pull request, so “through a single pull request per task” is not a platform guarantee. Since HC-4 requires a reviewed PR, make that an explicit lifecycle requirement rather than part of the containment claim.

- The write boundary (HC-1). The cloud agent can only make changes in the
  repository where its task was started, on a single branch, through a single
  pull request per task, using a token the platform issues for that run. The

docs/ai-assisted-techdocs-assessment-spec.md:454

• GitHub timeline events provide wall-clock transition times, but they cannot distinguish active writer work from idle or stakeholder-waiting time as section 10 requires. Add an explicit work-time logging mechanism, or limit the acceptance criterion to elapsed gate intervals; the current claim also conflicts with the unresolved measurement item in section 17.

- The timeline is the measurement. Gate transitions (opened, accepted, draft PR,
  ready, approved, merged) are timestamped in issue and PR history, so the
  pilot's cycle-time decomposition (section 10) is harvested from the GitHub
  timeline rather than a separate log.

docs/ai-assisted-techdocs-assessment-spec.md:546

• The document jumps from section 14 to section 17, but sections 15 and 16 are referenced repeatedly for the provenance block, data collection, checks, and build plan. Without those sections, core operational requirements and the promised implementation plan are undefined; add them or renumber/remove all dangling references.

## 17. Open questions and future work

docs/ai-assisted-techdocs-assessment-spec.md:550

• A tool that creates issues in a project's repository necessarily writes outside cncf/techdocs, so it cannot literally “preserve HC-1.” Clarify that this future tool is a separate opt-in trust boundary with its own credential and safeguards rather than claiming compatibility with the current hard constraint.

- Filing issues into project repos. A separate, opt-in tool to create the
  backlog issues in a project's own repository (NG-2). Out of scope for phase
  one, and it must preserve HC-1.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (3)

specs/ai-assisted-techdocs-assessment-spec.md:415

• Reviewing the workflow file does not constrain runtime destinations derived from an intake or assessed repository. Because setup runs before the firewall with normal runner egress, an attacker-controlled URL or redirect can bypass the explicit, audited allowlist described above. The setup path needs an equivalent egress boundary, not only code review.

  The firewall's scope is the agent's own session: the environment-setup
  workflow (see Execution environment below) runs outside it, so what setup
  fetches is governed by review of that versioned workflow file, not by the
  firewall.

specs/ai-assisted-techdocs-assessment-spec.md:763

• The plan permits choosing the pilot while the quality rubric remains only an open item. Without defining the rubric and scoring the human baselines first, the pilot's required quality parity cannot be evaluated objectively and the rubric could be fitted after seeing the result. Make that work an explicit pre-pilot dependency.

The build is complete when steps 1 through 8 are merged, the section 17
build-time checks have recorded answers, and one end-to-end walkthrough of the
section 5 lifecycle on a fixture project has run clean. Then the pilot is
chosen, deliberately, per the caveat in section 10.

specs/ai-assisted-techdocs-assessment-spec.md:533

• Re-running proves that outputs were unmodified only for immutable inputs. Live sites can change after collection, and section 15 records only their retrieval date, so a later mismatch cannot distinguish legitimate source drift from an agent edit. Preserve a trusted pre-model snapshot or publish a checksum/attestation outside the agent-writable workspace, then validate the committed output against that evidence.

  unmodified with the draft. Unmodified is checkable, not assumed: re-running
  the committed command reproduces the committed outputs or exposes the edit.

[iRaindrop]

Next phase eligible, or perhaps a repeat of steps 3 through 6 as warranted. (?)

[iRaindrop]

This is an amazing forward thinking document and helps form a wide macro view of the project and its capabilities. Noteworthy is that employs multiple AI agents that can service different roles.
Hence we might need an MCP spec for our AI agents. For the agent using predefined prompts (at the low level) would need an MCP Sever (at the high level) - or something like that?
This spec is also very helpful for project planning - making things easier in that regard.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

specs/ai-assisted-techdocs-assessment-spec.md:548

• Checking only for confirmed is bypassable because any collaborator with Triage or greater access can apply or dismiss PR labels. The merge gate must verify an authorized stakeholder's /confirm event (or a protected check derived from it), rather than trusting a manually mutable label.

- Merge (step 6): merge plus one workflow. The approver (section 4) merges,
  checking the `confirmed` label is present first: stakeholder confirmation is
  required to advance (step 5, HC-2), and because a ready PR awaiting
  stakeholders is a normal state rather than a failure, the check is the
  approver's duty, not CI's. A merge-triggered GitHub Actions workflow (standard

specs/ai-assisted-techdocs-assessment-spec.md:221

• The PR description presents the adversarial verifier as running before every human review, but this exception allows a draft to advance without that safeguard. Either make the verifier pass mandatory or update the PR description and acceptance criteria to state when skipping it is acceptable.

4. Review. The verifier's fact-check pass runs first; the reviewer may skip it
   deliberately, recording why (section 12). The reviewer then refines the draft

specs/ai-assisted-techdocs-assessment-spec.md:762

• A retrieval date does not pin a live site's content. After the site changes or disappears, reviewers cannot reconstruct what the drafter saw, so this does not satisfy P-4's requirement that every claim trace to a committed source. Commit a normalized snapshot/evidence extract (with URL, time, and digest), use a durable archive, or explicitly narrow the reproducibility requirement.

  the retrieval date for each live site. The pins record which state of the
  sources each claim describes: a pinned repository lets the reviewer check out
  exactly what the drafter saw (section 10), a dated site bounds when the claim
  held, and when the sources move on, drift reads as drift rather than error.

specs/ai-assisted-techdocs-assessment-spec.md:558

• A closed intake does not necessarily mean the assessment is done: /decline for phase B/C and the abort path also close it. Define distinct terminal states (for example completed, declined, and aborted labels) and use those states for portfolio queries instead of equating every closed issue with completion.

  intake's phase label. The final phase's merge closes the intake issue: an open
  intake is an assessment in flight, a closed one is done. Approver independence

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (6)

specs/ai-assisted-techdocs-assessment-spec.md:416

• Reviewing the versioned setup workflow does not constrain its per-run destinations. The proposed setup invokes data collection for documentation domains supplied through intake, but setup processes are outside the firewall; a malicious or mistaken URL could therefore make the runner fetch arbitrary network locations. Require the setup script to validate every destination against a trusted, acceptance-time allowlist (including redirects), or perform these fetches inside the firewall boundary.

  The firewall's scope is the agent's own session: the environment-setup
  workflow (see Execution environment below) runs outside it, so what setup
  fetches is governed by review of that versioned workflow file, not by the
  firewall.

specs/ai-assisted-techdocs-assessment-spec.md:223

• This makes the verifier optional even though the PR description promises that the adversarial verifier fact-checks drafts first and the role definition at lines 171–176 says it runs on every draft. Either require the pass here or revise those guarantees; otherwise the documented safety workflow has an unadvertised bypass.

4. Review. The verifier's fact-check pass runs first; the reviewer may skip it
   deliberately, recording why (section 12). The reviewer then refines the draft

specs/ai-assisted-techdocs-assessment-spec.md:76

• This is a material departure from the canonical methodology, not just automation of it: docs/analysis/howto.md:47-50 lists creating GitHub issues as a follow-up, and lines 209–219 require creating the individual and umbrella issues in the project's repository. Either retain that step as an explicitly human-run follow-up outside the agent boundary, or acknowledge that phase one implements only a subset of the existing methodology rather than claiming it is unchanged.

- NG-2: No auto-filing of issues. The backlog is delivered as files in
  cncf/techdocs. Filing them as real issues in project repos is future work (a
  separate, opt-in script; see section 17), explicitly out of scope now. The
  per-file backlog format proposed in section 6 is chosen partly to make that
  future script straightforward.

specs/ai-assisted-techdocs-assessment-spec.md:264

• This conflicts with the documented hand-written fallback in section 15, whose provenance block correctly says that no agent was involved. Requiring every deliverable to claim it was AI-drafted would make that fallback produce a false disclosure.

Every deliverable carries a header noting it was AI-drafted and human-reviewed
(HC-6).

specs/ai-assisted-techdocs-assessment-spec.md:597

• The claimed isolation fails for duplicate requests for the same project and year: both can be accepted independently and target the same directory, causing conflicting deliverable PRs and ambiguous phase tracking. /accept needs to atomically reject or link a request when an in-flight assessment already reserves that project/year namespace.

assessment keeps its own namespace, the intake issue and the deliverable
directory (`analyses/<year>/<project>/`, section 13), one assessment per project
per year, with the tracking-issue link authoritative and the path the
human-readable key. Admission control is `/accept` itself: eligible issues wait

specs/ai-assisted-techdocs-assessment-spec.md:634

• Re-running a live-site collection later cannot prove that committed output was unmodified: a difference may be ordinary source drift rather than an edit. Preserve and hash the raw response during the original collection (or run analysis against a committed immutable snapshot) so integrity can be checked without refetching mutable content.

- Data collection: `scripts/assessment/`. The deterministic inventory scripts
  behind HC-5 (section 15). The drafting session's setup steps (section 11) run
  them and leave the outputs in the workspace; the drafter commits them
  unmodified with the draft. Unmodified is checkable, not assumed: re-running
  the committed command reproduces the committed outputs or exposes the edit.