Feat: Add end-to-end correctness suite for AIAC eval framework

aiac/docs/specs/PRD.md

@@ -600,6 +600,7 @@ Beyond the marker-gated pytest tests above, individual integration tests are spe
 | `policy-eval-scenarios` — `test_policy_pipeline_eval.py` + guardrail tests | Generalized evaluation suite extending `policy-pipeline`'s single-agent/single-tool proof to ten scenarios: baseline-scale (many entities, names decoupled from roles, one agent→agent delegation grant), missing-details (emergent unreachability/zero-access under deny-by-default, a broad-sounding clause narrowed by an explicit qualifier, wildcard-grant expansion), adversarial-authoring (misleading names/descriptions, an identity/boundary-confusion probe, empty descriptions), and ambiguous-and-contradictory / adversarial-injection-and-edge-cases (whole-document `xfail` checks against the PRB directly, no Keycloak or `opa`). The eight heavy scenarios (`@pytest.mark.eval_extended`, scenario modules under `eval/scenarios/` except `agent_delegation`) assert full per-cell `opa eval` truth tables; the two light scenarios (`@pytest.mark.integration`) assert PRB-level rejection. | [eval/policy-eval-scenarios.md](eval/policy-eval-scenarios.md) |
 | `policy-eval-robustness-consistency` — `test_policy_pipeline_consistency.py` + `test_policy_pipeline_robustness.py` | Companion to `policy-eval-scenarios`, reusing its 8-scenario corpus to check the PRB's raw grant decisions (no OPA/PCE/k8s) for **consistency** (`@pytest.mark.eval_consistency`: N repeated runs on the same input, exact grant-set equality) and **robustness** (`@pytest.mark.eval_robustness`: mechanical text/order perturbation + a hand-reworded semantic-sibling corpus under `eval/scenarios_perturbed/`, both checked against the truth-table oracle). No Keycloak/`opa` needed — only `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`. | [eval/policy-eval-robustness-consistency.md](eval/policy-eval-robustness-consistency.md) |
 | `policy-eval-correctness-prb` — `test_policy_pipeline_correctness_prb.py` | Companion to `policy-eval-scenarios`/`policy-eval-robustness-consistency`, reusing the same 8-scenario corpus to score the PRB's raw grant/deny output (no OPA/PCE/k8s) against each scenario's truth table via a reusable, effect-aware scorer (`eval/correctness_scorer.py`): precision and recall tracked separately per gate and aggregated, plus a non-gating denial-precision figure for explicit `Deny` rules. `@pytest.mark.eval_correctness_prb`, zero-tolerance over-grant gate; under-grants/incorrect denials reported only. No Keycloak/`opa` needed — only `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`. | [eval/policy-eval-correctness-prb.md](eval/policy-eval-correctness-prb.md) |
+| `policy-eval-correctness-e2e` — `test_policy_pipeline_correctness_e2e.py` | Companion to `policy-eval-correctness-prb`, scoring the same 8-scenario corpus and the same reusable scorer one layer further downstream: real Keycloak provisioning → real Policy Rules Builder → real Policy Computation Engine → real `opa eval` against the rendered Rego, sourced from the rendered data maps (`subject_role_allow/deny_scopes`, `agent_role_scopes`) rather than per-pair decision probing. `@pytest.mark.eval_correctness_e2e`, same zero-tolerance over-grant gate. The shared `pipeline` fixture (`eval/test_policy_pipeline_eval.py`, also used by `eval_extended`) now provisions all 8 scenarios concurrently via `ProcessPoolExecutor`. Needs `KEYCLOAK_URL` + admin creds + `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`, plus `opa` on `PATH`. | [eval/policy-eval-correctness-e2e.md](eval/policy-eval-correctness-e2e.md) |
 
 Tracking issues: the live-Keycloak pytest integration tests in `testing/5.1-integration-tests.md`; the PDP Policy Writer integration test in `testing/5.2-pdp-writer-integration-test.md`; the policy-pipeline integration test in `testing/5.3-policy-pipeline-integration-test.md`; the UC-1 onboarding pipeline integration-test ladder in `testing/5.4-uc1-onboarding-integration-test.md` (epic) with rungs `testing/5.4.1`/`5.4.2`/`5.4.3` and the deferred two-policy `testing/5.4.4`.
 

aiac/docs/specs/eval/policy-eval-correctness-prb.md

@@ -24,6 +24,10 @@
   synthetic-`Role`/`Scope` builder `policy-eval-robustness-consistency.md`'s two suites use) and
   imports `SCENARIOS`, `orchestrate_prb`, `grant_sets`, `truth` from
   `eval.test_policy_pipeline_eval` unmodified.
+- `aiac/eval/best_effort_rules.py` — the best-effort fallback's own pure-logic helper
+  (`_best_effort_rules`, called from `orchestrate_prb`/`_invoke_graph`); `aiac/eval/
+  test_best_effort_rules.py` — its unmarked unit tests. See [Best-effort
+  proposals](#best-effort-proposals).
 
 ## Description
 
@@ -103,12 +107,46 @@ more precisely diagnosed under-grant, not a new failure class). `denial_precisio
 scenario and per gate via `record_property`, never blended into grant precision/recall, and never
 gates the test.
 
+## Best-effort proposals
+
+The PRB's generate→audit loop (`aiac.agent.policy_rules_builder.graph`'s `_audit`) can reject a
+scope/role decision outright — a genuine contradiction (`PolicyContradictionError`, fails closed
+immediately) or an exhausted retry budget (`PolicyRulesBuilderError`, after `MAX_AUDIT_RETRIES=3`).
+By default this aborts `orchestrate_prb()` entirely, discarding every other decision's
+already-approved rules along with it — one bad scope used to mean the whole scenario showed
+"setup failed" with no precision/recall at all.
+
+This suite calls `orchestrate_prb(..., best_effort=True)`: a rejected decision instead falls back
+to whatever was last proposed (before the auditor rejected it) — a real, if never-approved, guess
+at the grant/deny set for that one scope/role, built the same way the PRB's own `build` node would
+have (`aiac.agent.policy_rules_builder.graph._denied_names` reused; see
+`eval.best_effort_rules._best_effort_rules`, unit-tested unmarked in
+`eval/test_best_effort_rules.py`). Every other decision in the scenario proceeds normally.
+
+**This is an explicit, user-requested tradeoff, not free lunch**: a best-effort pair may not
+represent what a real deployment would ever contain — the auditor rejected it for a reason, and a
+real pipeline run would never emit it. `record_property("best_effort_notes", ...)` — a
+`{scope_or_role_name: reason}` dict — and the printed summary line name exactly which decisions
+this applies to, so a reader can tell which numbers are "real" and which are best-effort. The
+report (`eval/conftest.py`) renders this as an extra field with an explicit caveat whenever
+non-empty. `eval_consistency`/`eval_robustness` keep `best_effort=False` (the default, at their
+own direct `orchestrate_prb` call sites) — a rejected decision still aborts their scenario/repeat
+as before. `eval_extended` is the one exception: it shares the same session-scoped `pipeline`
+fixture `test_e2e_correctness` uses, and that fixture's `orchestrate_prb` call hardcodes
+`best_effort=True` unconditionally (confirmed with the user — see
+[policy-eval-correctness-e2e.md § Best-effort
+proposals](policy-eval-correctness-e2e.md#best-effort-proposals) and
+`eval.test_policy_pipeline_eval`'s module docstring for why this couldn't cleanly be made
+e2e-only), so `eval_extended`'s own per-cell tests are affected too, not just this suite or
+`test_e2e_correctness`.
+
 ## Expected output
 
 Parametrized over all 8 scenario names (`sorted(SCENARIOS)`); expects **all 8 to pass** (zero
 over-grants) given a well-behaved LLM endpoint. Each test case `record_property`s `precision`,
-`recall`, `denial_precision`, `over_grants`, `under_grants`, and `incorrectly_denied` (each of the
-latter three as `{gate: sorted(pairs)}`), and prints a one-line summary:
+`recall`, `denial_precision`, `over_grants`, `under_grants`, `incorrectly_denied` (each of the
+latter three as `{gate: sorted(pairs)}`), and `best_effort_notes` (see [Best-effort
+proposals](#best-effort-proposals)), and prints a one-line summary:
 
 ```text
 [correctness] wildcard_grant: precision=1.000 recall=1.000 denial_precision=1.000
@@ -255,6 +293,11 @@ This is **one** integration-test spec among several indexed by the master PRD
   tracked/reported via `record_property` and the printed summary line, never gated.
 - **New scenarios.** The taxonomy cross-check above confirms the existing 8-scenario corpus
   already covers every taxonomy theme; none is needed.
+- **Fixing the PRB audit/retry-convergence bug** that causes the auditor to reject a scope/role
+  decision (`PolicyContradictionError`/`PolicyRulesBuilderError`) for a variable subset of
+  scenarios depending on LLM sampling — see [Best-effort proposals](#best-effort-proposals), which
+  changes how a rejection is *reported*, not the underlying bug. Already deferred by the user as a
+  separate follow-up to `aiac.agent.policy_rules_builder.graph._audit` — not this ticket's job.
 
 ## Blocked-by
 

aiac/eval/best_effort_rules.py

@@ -0,0 +1,46 @@
+"""Pure-logic helper for ``test_policy_pipeline_eval.py``'s ``_invoke_graph`` — no LLM, no
+Keycloak, no ``opa``, no I/O. Mirrors ``correctness_e2e_helpers.py``'s split from its own test
+module: logic lives here (no ``test_`` prefix, not collected by pytest), unit tests live in
+``test_best_effort_rules.py`` (unmarked, runs in the default fast pass).
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from aiac.agent.policy_rules_builder.graph import _denied_names
+from aiac.policy.model.models import PolicyRule, RuleEffect
+
+
+def _best_effort_rules(entity: dict[str, Any], state: dict[str, Any]) -> list[PolicyRule]:
+    """Replicate ``graph.py``'s ``build`` node logic (role-focal or scope-focal, whichever
+    ``entity``'s shape indicates) against a proposal the auditor never approved — same
+    ``_denied_names()``-driven exclusivity-complement + explicit-prohibition set, same
+    ALLOW-then-DENY ``PolicyRule`` shape (``graph.py``'s ``build_role_graph``/``build_scope_graph``
+    closures, not importable — they're nested — hence replicated here rather than reused).
+
+    Only ever called from ``eval.test_policy_pipeline_eval._invoke_graph`` after catching a
+    rejection; the caller is responsible for flagging the result as best-effort (not a real,
+    auditor-approved decision) — see ``orchestrate_prb``'s ``best_effort_notes``.
+    """
+    selected = set(state.get("selected_names", []))
+    denied_explicit = state.get("denied_names", [])
+    exclusive = bool(state.get("exclusive", False))
+    if "role" in entity:  # ROLE_GRAPH shape: role-focal
+        role = entity["role"]
+        candidate_scopes = entity["scopes"]
+        denied = _denied_names(denied_explicit, exclusive, [sc.name for sc in candidate_scopes], selected)
+        allows = [
+            PolicyRule(role=role, scope=sc, effect=RuleEffect.ALLOW) for sc in candidate_scopes if sc.name in selected
+        ]
+        denies = [
+            PolicyRule(role=role, scope=sc, effect=RuleEffect.DENY) for sc in candidate_scopes if sc.name in denied
+        ]
+        return allows + denies
+    # SCOPE_GRAPH shape: scope-focal
+    scope = entity["scope"]
+    candidate_roles = entity["roles"]
+    denied = _denied_names(denied_explicit, exclusive, [r.name for r in candidate_roles], selected)
+    allows = [PolicyRule(role=r, scope=scope, effect=RuleEffect.ALLOW) for r in candidate_roles if r.name in selected]
+    denies = [PolicyRule(role=r, scope=scope, effect=RuleEffect.DENY) for r in candidate_roles if r.name in denied]
+    return allows + denies