DOC: Add scenario doc pages for all 7 undocumented scenarios#1558
Conversation
|
wondering if we should add integration tests for this (obviously we should skip most for the day to day integration tests). If not, we should add this to the release notes that these should be validated |
|
we should also update the configuring scenarios docs bc they're specific to foundry so I would add another redteam scenario notebook and maybe make that one more generic or merge it into the 0_scenarios files |
ValbuenaVC
left a comment
ValbuenaVC
left a comment
There was a problem hiding this comment.
What is our plan for documenting changes to scenarios? For extensibility should we add something about it to a style guide or release notes
As long as each notebook is running a small subset of objectives/strategies, it might be okay to run regularly like our other notebooks? @varunj-msft probably has better grasp on how long these take though |
hannahwestra25
left a comment
hannahwestra25
left a comment
There was a problem hiding this comment.
few minor comments :)
5 participants
Description
PR 1 of 3 for User Story 7933 - Migrate Cookbooks to Scenarios.
Creates a doc page for each of the 7 scenarios.
New pages:
2_content_harms — ContentHarms (7 harm categories)
3_psychosocial — Psychosocial (crisis escalation, therapist impersonation). Incorporates key concepts from cookbook 5: rubric design, MHFA/ALGEE methodology, conversation-level scoring via create_conversation_scorer.
4_cyber — Cyber (malware/exploitation)
5_jailbreak — Jailbreak (template-based prompt injection)
6_leakage — Leakage (IP/sensitive data). Incorporates key concepts from cookbook 3: copyright testing with FirstLetterConverter, PlagiarismScorer with LCS/Levenshtein/Jaccard metrics.
7_scam — Scam (phishing/fraud)
8_garak_encoding — GarakEncoding (17 encoding-based probes)
Each page follows the pattern from 1_configuring_scenarios.py: overview, strategy table, CLI usage, programmatic usage, result interpretation.
Also updates doc/myst.yml TOC with the 7 new entries.
Subsequent PRs will migrate remaining cookbook content (PR 2) and delete the cookbooks directory with cross-reference fixes (PR 3).
Tests and Documentation
This is a documentation-only PR. No source code or test changes.
All 7 notebooks were executed locally committed with outputs. Execution times ranged from 23s (Jailbreak) to ~2m (ContentHarms, Psychosocial). Ruff check passes on all files. All .py/.ipynb pairs round-trip correctly.