parse LLM replies whose strings contain code fences

[teddytennant]

_parse_llm_response strips the fence unconditionally whenever ```json appears anywhere in the text, and takes the first ``` after the opening marker as the close. So a reply that is already valid JSON gets mangled when a justification quotes a ```json block, and a quoted ```python block truncates the slice.

It fails quietly, which is the part that matters: an unparseable reply isn't an error, it's scored IGNORED at evaluate_rubrics.py:138-140 and drops out of must_have_pass and agg_score.

The find to rfind change in the issue fixes the reported payload but not the other two shapes:

fenced + inner ```python          rfind-only: parses
bare JSON + inner ```json         rfind-only: None
fenced + inner + trailing prose   rfind-only: None

Rather than pick a different single delimiter rule, this collects candidate slices and lets json.loads decide: the untouched text first, so anything already valid survives whatever its strings contain, then closing-fence candidates outermost-first, then brace extraction. It returns the first candidate that is a dict containing "ratings" and falls back to the first merely-parseable one, which preserves the old behaviour on a top-level array.

Sweep over 16 payload shapes, scored on the caller's actual gate at line 139: 1 improvement, 0 regressions.

On scope: this is the 70 identical copies of evaluate_rubrics.py, which is what the issue names. All 70 pre-images share one md5 and all 70 post-images share one md5, so the fan-out is mechanical. evaluate_answer.py (124 copies) and evaluate_tests.py (90 copies) have the same bug, and I checked they still fail on the issue's payload after this change. I can extend it to those in one pass if you'd rather; evaluate_answer.py needs a slightly different shape since it normalizes the rating inline.

Closes #15.

Greptile Summary

The PR updates the RF rubric-response parser across 70 self-contained task bundles so valid JSON survives embedded code fences and fenced responses are evaluated through multiple candidate slices.

• Tries the untouched response before extracting fenced or embedded JSON.
• Considers closing fences from outermost to innermost.
• Prefers parsed objects containing ratings while retaining prior fallback behavior for other JSON values.

Confidence Score: 5/5

The PR appears safe to merge because no concrete changed-code failure remains.

The parser now preserves already-valid responses and tries progressively extracted candidates, while the identified edge-case limitations either predate the change or lack an established realistic trigger under the current response contract.

Important Files Changed

Filename	Overview
data/rf/task-69391d8d1ce51c407be1e531/tests/evaluate_rubrics.py	Representative mechanical update replaces single-slice parsing with ordered JSON candidates; no changed-code regression was established.
data/rf/task-69d196f015a150488265afc4/tests/evaluate_rubrics.py	Applies the same parser improvement to the documented RF task bundle without changing rubric aggregation behavior.

_{Reviews (1): Last reviewed commit: "parse LLM replies whose strings contain ..." | Re-trigger Greptile}

Context used:

• Knowledge Base — RF task bundles and patch validation