Skip to content

Add configurable agent model selection#253

Draft
isadeks wants to merge 4 commits into
mainfrom
configurable-model-selection
Draft

Add configurable agent model selection#253
isadeks wants to merge 4 commits into
mainfrom
configurable-model-selection

Conversation

@isadeks

@isadeks isadeks commented Jun 11, 2026

Copy link
Copy Markdown

Summary

  • add SSM-backed global model defaults and project-level model overrides for Kiro/OpenCode
  • resolve model at dispatch time and pass AGENT_MODEL through worker/ACP sessions
  • add verification logging across dispatcher, worker, ACP startup, and drivers
  • expose Admin and Project Settings UI for model defaults/overrides

Draft Status

This PR is intentionally still a draft. The implementation has unit/type/lint coverage and has been deployed to the dev stack for smoke checks, but it has not been fully end-to-end tested with a real agent run yet.

The remaining full smoke test requires Admin Agent Settings to have a valid Kiro API key or Bedrock bearer token, then starting a real agent run and verifying the model-selection log lines in CloudWatch.

Claude model override is intentionally not exposed yet. This draft enables runtime model selection for Kiro and OpenCode first because those drivers have clear, verifiable model configuration paths. Claude remains on its driver default until we validate a safe ACP/SDK override path; that can be handled later in this PR after Kiro/OpenCode are validated, or as a follow-up if it needs deeper driver work.

Verification

  • npm --prefix frontend run typecheck
  • npm run lint
  • terraform fmt -check -recursive
  • npx vitest run --config /tmp/vitest.no-setup.config.mjs --root lambda/shared test/mcp-validator.test.js test/cli-models.test.js
  • npx vitest run --config /tmp/vitest.no-setup.config.mjs --root lambda/agents-ecs test/pool-worker.test.js
  • npm test -- --project projects
  • deploy: ./scripts/deploy-terraform.sh dev
  • deploy: ./scripts/deploy-frontend.sh dev

Smoke Test Notes

  • CloudFront app responds with HTTP 200: https://dj82pet2n0iq2.cloudfront.net
  • Deployed agent image tag: c5f60d9ffe8328cc
  • /agents/settings via Lambda returns the expected default model settings
  • /agents/settings PUT successfully changed Kiro/OpenCode model defaults, GET read the temporary values back, and the originals were restored
  • /agents/settings PUT rejects unsupported Claude model override keys with a 400
  • /agents/capabilities reports runtime model override support for Kiro and OpenCode
  • Pool recycle/warm works on the new image, but workers cannot advertise runnable CLIs until credentials are configured

Closes #13

Comment thread frontend/src/pages/Admin.tsx Outdated
Comment thread frontend/src/pages/Admin.tsx
Comment thread frontend/src/pages/ProjectSettings.tsx
Comment thread lambda/shared/mcp-validator.js Outdated
Comment thread lambda/agents/index.js Outdated
return response(200, { available: [...cliSet] });
return response(200, {
available: [...cliSet],
runtimeModelOverride: { kiro: true, claude: false, opencode: true },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should move this as a constant on top of the file and explain why claude is false as a comment...

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Moved this into a top-level RUNTIME_MODEL_OVERRIDE constant and added a comment explaining why Claude is false for this draft.

expect(normalizeCliModels('{"opencode":"amazon-bedrock/model"}')).toEqual({
valid: true,
issues: [],
value: { opencode: 'amazon-bedrock/model' },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should test this also in the code, that models for opencode start with "amazon-bedrock/"? just to make sure they don't simply use the bedrock model id as-is

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added validation and a test for this. OpenCode model overrides now must start with amazon-bedrock/ so users do not save the raw Bedrock model ID by mistake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Configurable model selection

2 participants