Skip to content

Example evaluators accept candidate-reported values without verification (matmul, function_minimization) #514

Open

Description

@iamgahr

Two bundled example evaluators score values the candidate program reports about itself, so a candidate can score above the state of the art without solving the task. Verified at HEAD a1e9d91.

1. examples/alphaevolve_math_problems/matmul/evaluator.py

  • verify_tensor_decomposition returns early, with only a printed warning, when any factor array is empty (lines 32–34).
  • The score is BENCHMARK / rank using the rank the candidate returns (lines 81, 98).
  • A candidate whose run() returns (np.zeros(0), np.zeros(0), np.zeros(0)), 2, 4, 5, 0.0, 1 gets combined_score = 32.0; a correct rank-32 decomposition gets 1.0.
  • Suggested fix: treat an empty decomposition as a failed verification (score 0), take the rank from factor_matrix_1.shape[1] rather than the returned value, and check the returned n, m, p against the target sizes.

2. examples/function_minimization/evaluator.py

  • When run_search() returns three values, the evaluator uses the returned value; it computes f(x, y) only when two values are returned (lines ~105–113).
  • At the same point (0.5, 0.5): returning (0.5, 0.5) scores 0.449 (the evaluator computes f = 0.6931); returning (0.5, 0.5, -1.519) scores 0.793 (value_score 1.0).
  • Suggested fix: always recompute value = f(x, y) from the returned point.

In both examples these values are produced inside the EVOLVE-BLOCK, i.e. under the LLM's control during evolution. Happy to open a PR for either.

Found during an evaluation-integrity audit of LLM-driven program search. Prepared with Claude Code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions