You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Example evaluators accept candidate-reported values without verification (matmul, function_minimization) #514
Two bundled example evaluators score values the candidate program reports about itself, so a candidate can score above the state of the art without solving the task. Verified at HEAD a1e9d91.
Suggested fix: treat an empty decomposition as a failed verification (score 0), take the rank from factor_matrix_1.shape[1] rather than the returned value, and check the returned n, m, p against the target sizes.
2. examples/function_minimization/evaluator.py
When run_search() returns three values, the evaluator uses the returned value; it computes f(x, y) only when two values are returned (lines ~105–113).
At the same point (0.5, 0.5): returning (0.5, 0.5) scores 0.449 (the evaluator computes f = 0.6931); returning (0.5, 0.5, -1.519) scores 0.793 (value_score 1.0).
Suggested fix: always recompute value = f(x, y) from the returned point.
In both examples these values are produced inside the EVOLVE-BLOCK, i.e. under the LLM's control during evolution. Happy to open a PR for either.
Found during an evaluation-integrity audit of LLM-driven program search. Prepared with Claude Code.
Activity