chore: reverts judge message stripping#205
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2e40f00. Configure here.
| (model, provider, messages, instructions, | ||
| tracker_factory, enabled, judge_configuration, variation) = self.__evaluate( | ||
| key, context, default.to_dict(), extended_variables | ||
| key, context, default.to_dict(), variables |
There was a problem hiding this comment.
Judge placeholders cleared at config
High Severity
_judge_config passes user variables into __evaluate, which Mustache-interpolates every message. Legacy judge templates like {{message_history}} and {{response_to_evaluate}} are not in that map, so chevron blanks them before Judge._construct_evaluation_messages runs, breaking legacy judge evaluations loaded from flags.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 2e40f00. Configure here.
There was a problem hiding this comment.
@andrewklatzke have you looked into this? This may not be an issue for the createJudge method but only if we switch it to the new template method that you added so that it does all interpolation later.
|
|
||
| evaluations_task = await self._track_judge_results(tracker, input_text, result.content) | ||
|
|
||
| self._messages.append(assistant_message) |
There was a problem hiding this comment.
Failed run appends assistant history
Medium Severity
ManagedModel.run always appends an assistant LDMessage after the runner returns, even when result.metrics.success is false. A failed call still leaves a user turn plus an empty or error assistant turn in _messages, which pollutes later all_messages sent to the model.
Reviewed by Cursor Bugbot for commit 2e40f00. Configure here.
jsonbailey
left a comment
jsonbailey
left a comment
There was a problem hiding this comment.
I added a follow up on one of the ai reviews about the interpolation for judge configs.
2 participants
Requirements
Related issues
https://launchdarkly.atlassian.net/jira/software/c/projects/AIC/boards/2045?selectedIssue=AIC-2848
Describe the solution you've provided
This is a reversion of functionality that was added.
Describe alternatives you've considered
Reverts behavior from #165 that stripped messages off of the judges and then re-created them.
Additional context
This was discussed and there are various concerns about this behavior. Since this is a semi-permanent solution once changes are applied on the server side we don't want to take this step yet.
Note
Medium Risk
Judge and managed-completion paths change how prompts are built and sent to providers, which can alter evaluation behavior for legacy judge configs; runner input typing is broader but list-input semantics are a contract change for direct runner callers.
Overview
Reverts the prior judge flow that stripped legacy template messages in
LDAIClientand sent a single formatted string to the model.judge_config()again returns the full message list (including{{message_history}}/{{response_to_evaluate}}placeholders), andJudge.evaluate()builds the prompt by Mustache-interpolating those templates into aList[LDMessage]passed toRunner.run(). Evaluations now fail early when judge config has no messages.The
Runnercontract is extended sorun()accepts either a string (still appended to multi-turn history) or a message list used as the complete payload without prepending runner history or config messages. OpenAI and LangChain factories no longer seed runners withconfig.messages.ManagedModeltakes over completion conversation management: it keeps turn history, prepends config system messages, invokes the runner with the full list, and feeds judges history as CRLF-joined content (notrole:lines).Reviewed by Cursor Bugbot for commit 2e40f00. Bugbot is set up for automated code reviews on this repo. Configure here.