Skip to content

fix(uipath-agents): forbid switching an existing coded agent's framework - #2883

Merged
andreibalas-uipath merged 1 commit into
mainfrom
chore/uipath-agents-failed-coder-eval-tasks
Sep 1, 2026
Merged

fix(uipath-agents): forbid switching an existing coded agent's framework#2883
andreibalas-uipath merged 1 commit into
mainfrom
chore/uipath-agents-failed-coder-eval-tasks

Conversation

@andreibalas-uipath

@andreibalas-uipath andreibalas-uipath commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Why

skill-agent-coded-antipattern-openai-agents-hitl failed in CI (codex / gpt-5.6-terra, score 0.125). The seed is an OpenAI Agents project that misuses LangGraph's interrupt(...). The agent correctly diagnosed the mismatch, then read the OpenAI integration note ("does not support HITL … use LangGraph or LlamaIndex instead") as a remediation and migrated the whole project to LangGraph — deleting openai_agents.json, swapping deps, rewriting main.py as a StateGraph.

The coded quickstart only stated framework selection for greenfield projects; for existing projects the "keep it" intent lived solely in a step-skip gate (Framework — skip if framework != none), not as a rule.

What

  • skills/uipath-agents/references/coded/quickstart.md § Critical Rules — new rule:

    Never switch an existing project's framework. When framework != none (a <framework>.json is already present), the framework is fixed: do not migrate to another framework, swap the <framework>.json, or change framework dependencies in pyproject.toml. Work within the existing framework's capabilities; if a request cannot be met within them, tell the user the limitation and let them decide.

    Deliberately generic — no worked example — so the agent pairs the rule with the capability facts already stated in the framework/capability references.

  • tests/.../check_antipattern_openai_agents_hitl.py — dropped the failure hint "or, if HITL is essential, migrate the project to LangGraph", which contradicted the task's own criteria.

Verification

  • npm run skills:validate OK (default + studioweb; rule sits outside all flavor markers).

  • Local coder-eval runs on this state, experiments/default.yaml, 3 replicates each:

    Harness / model Pass Duration
    codex / gpt-5.6-terra 3/3 105–131 s
    claude-code / claude-sonnet-4-6 3/3 80–106 s
  • CI brownfield sweep (Run Coder Eval, all 8 lifecycle:edit coded tasks × codex + claude): target task passes on both harnesses; 15/16 agent-graded runs pass with 0 framework migrations. The one non-pass (claude guardrails/byog_middleware) is a missing tenant fixture — byog-smoke-agent-pin isn't registered on the alpha codereval tenant and the agent correctly refused to fabricate it (skill Rule 18); separate follow-up.

The greenfield strict-input flake (strict_input_no_messages) seen on this PR's smoke run is addressed separately in #2888.

🤖 Generated with Claude Code

The coded quickstart only stated framework selection for greenfield
projects; for existing projects the "keep it" intent lived solely in a
step-skip gate. An agent fixing an OpenAI Agents project that misused
LangGraph's interrupt() read the OpenAI note ("use LangGraph instead for
HITL") as a remediation and migrated the whole project, failing
antipattern_openai_agents_hitl in CI.

Add an explicit Critical Rule: when a <framework>.json is present the
framework is fixed — work within its capabilities and surface
limitations to the user instead of migrating. Also drop the checker's
"or migrate to LangGraph" hint, which contradicted the task's criteria.

Local runs on this state: 3/3 codex (gpt-5.6-terra), 3/3 claude-code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Claude finished @andreibalas-uipath's task in 1m 37s —— View job


PR Review: fix(uipath-agents): forbid switching an existing coded agent's framework

  • Read review criteria and project conventions
  • Read full diff and changed files
  • Check skill-specific constraints
  • Post review findings

Summary

Adds a new Critical Rule to quickstart.md forbidding framework migration on existing coded agent projects, and removes a contradictory migration hint from the antipattern test's failure message. Motivated by a CI failure where the agent read "migrate the project to LangGraph" as remediation advice and rewrote the entire project.

Change-by-Change Review

1. skills/uipath-agents/references/coded/quickstart.md (line 39)

Severity: OK

New critical rule is well-placed — immediately after the "Select a framework before writing any code" rule, forming a natural pair (greenfield selection → existing-project lock). The rule is:

  • Prescriptive and unambiguous ("do not migrate", "do not swap", "do not change").
  • Scoped correctly — keyed on the framework != none detection variable already defined in the Project State Detection table (line 12).
  • Includes the right fallback: "tell the user the limitation and let them decide" — avoids a blanket refusal.
  • Consistent with the existing step 1 gate at line 85 (Skip if framework != none), which already skipped framework selection but lacked a rule forbidding active migration. The new rule closes that gap.

No token-optimization concerns — rule text is concise and follows the § 0 "keep full sentences for rules" boundary.

2. tests/.../check_antipattern_openai_agents_hitl.py (lines 59–61)

Severity: OK

Removed the parenthetical "(or, if HITL is essential, migrate the project to LangGraph)" from the failure message. This was the root cause — the test's own error text suggested migration, which contradicted both the task's success criteria (which assert openai_agents.json must still exist and Agent construction must be preserved) and the new critical rule. The remaining message is clear: "Remove the interrupt call."

The task YAML (antipattern_openai_agents_hitl.yaml) requires no changes — its initial_prompt and success_criteria were already correct (they grade framework preservation, not migration).

What's Missing

Nothing identified. The change is minimal and correctly scoped:

  • No other files in the skill or test suite reference "migrate to LangGraph" (verified via grep).
  • The task YAML's description and initial_prompt don't mention migration.
  • CODEOWNERS is not affected (no new paths).
  • No flavor overrides needed — the new rule sits outside all skill-flavor markers.

Area Ratings

Area Status Notes
Frontmatter OK Not modified
E2E Tests OK Existing test updated to align with the new rule; task YAML unchanged
Skill Body OK New critical rule is well-positioned and prescriptive
References & Assets OK No broken links; rule references existing detection variables
Repo Hygiene OK No secrets, no cross-skill deps, changes scoped to one skill

Issues for Manual Review

None found. The PR description includes passing local coder-eval runs (3/3 on two harness/model combos), confirming the fix prevents migration behavior.

Conclusion

Clean, well-scoped fix. The new rule closes a real gap (the "framework is fixed" intent existed only as a step-skip gate, not as an explicit prohibition), and the test change removes the contradictory advice that caused the CI failure. Approve.


@andreibalas-uipath
andreibalas-uipath merged commit b48f317 into main Sep 1, 2026
90 of 95 checks passed
@andreibalas-uipath
andreibalas-uipath deleted the chore/uipath-agents-failed-coder-eval-tasks branch September 1, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants