flake: TestOpenAIResponsesNoStaleWebSearchReplay

[flake-investigator]

CI Run: https://github.com/coder/coder/actions/runs/25007375454
Failing Job: test-go-pg-17
Commit: 33ffedf411b858df81adb91043dca9157f0049a8 (Jeremy Ruppel)

Failure:

• coderd/x/chatd TestOpenAIResponsesNoStaleWebSearchReplay
• Error: expected 2 OpenAI requests but recorded 1.
• Log snippet:

=== FAIL: coderd/x/chatd TestOpenAIResponsesNoStaleWebSearchReplay (0.39s)
    integration_responses_test.go:97:
        Error: "[{[map[content:You are the Coder agent — ... [map[text:search for the latest Coder docs type:input_text]] role:user]] ...}]" should have 2 item(s), but has 1

• Context around failure shows chat interrupted during processing:

chatdebug: failed to load runtime admin chat debug logging setting  error="context canceled"
processor: chat interrupted  chat_id=f548844e-6e33-46e1-96a6-638cf0f05bb7

Root Cause Assessment:

• Likely flaky chatd integration test where the follow-up message does not trigger a second OpenAI request (chat interrupted / context canceled), leaving only the initial request recorded.
• No data race warnings, panics, or OOM indicators observed in logs.

Assignment Analysis:

• Failing test location: coderd/x/chatd/integration_responses_test.go lines ~23-80.
• github_list_commits for this file shows the most recent meaningful change is commit 62e9752acd3212991de1c4a5da3d43bdf6da1fb9 by @ibetitsmike ("fix: prevent malformed OpenAI Responses continuations"), which introduced/updated these OpenAI Responses replay tests.
• Assigning to @ibetitsmike based on most recent test-file changes in this area.

Related Issues:

• None found in coder/internal (searched for TestOpenAIResponsesNoStaleWebSearchReplay, integration_responses_test.go, and OpenAI Responses chatd flakes).

Reproduction:

• go test ./coderd/x/chatd -run TestOpenAIResponsesNoStaleWebSearchReplay -count=1

[flake-investigator]

New occurrence:

• CI run: https://github.com/coder/coder/actions/runs/25039551318
• Job: test-go-pg (ubuntu-latest) https://github.com/coder/coder/actions/runs/25039551318/job/73339190183
• Commit: bf66f63ac576bcf4b923a2b08657876430377173 ("docs: clarify PR body wrapping", author Michael Suchacz)

Failure:

=== FAIL: coderd/x/chatd TestOpenAIResponsesFullReplayPairsReasoningAndWebSearch (0.35s)
    integration_responses_test.go:180:
        Error: should have 2 item(s), but has 1

No panic/OOM or data race indicators observed. Looks like another OpenAI Responses replay variant failing with only one recorded request.

[flake-investigator]

New occurrence in https://github.com/coder/coder/actions/runs/25051098725 (job: test-go-pg-17) on commit 1c70c9638d490c2fc270491f110866b1633d1b4c.

Failing test:

• coderd/x/chatd TestOpenAIResponsesFullReplayPairsReasoningAndWebSearch
• Location: coderd/x/chatd/integration_responses_test.go (assert require.Len(t, requests, 2) around line ~180)

Error snippet:

=== FAIL: coderd/x/chatd TestOpenAIResponsesFullReplayPairsReasoningAndWebSearch (0.34s)
    integration_responses_test.go:180:
        Error: "[...]" should have 2 item(s), but has 1

Context shows the chat was interrupted before the second request was recorded:

processor: chat interrupted  chat_id=cc7f74f8-69c0-4f2b-b792-e75b802dfbc1

No data race warnings, panics, or OOM indicators found in the job logs.

[flake-investigator]

Recurrence detected in CI run: https://github.com/coder/coder/actions/runs/25060745771
Failing job: https://github.com/coder/coder/actions/runs/25060745771/job/73413841006 (test-go-pg (ubuntu-latest))

Failure:

=== FAIL: coderd/x/chatd TestOpenAIResponsesNoStaleWebSearchReplay (0.24s)
    integration_responses_test.go:97:
        Error: "[{[map[content:You are the Coder agent — ... [map[text:search for the latest Coder docs type:input_text]] role:user]] ...}]" should have 2 item(s), but has 1

Context:

chatdebug: failed to load runtime admin chat debug logging setting  error="context canceled"
processor: chat interrupted  chat_id=08d2b70c-6ecc-43db-82a8-2d2f46034866

[flake-investigator]

New occurrence on main run: https://github.com/coder/coder/actions/runs/25064499796
Failing job: https://github.com/coder/coder/actions/runs/25064499796/job/73427821816 (test-go-pg ubuntu-latest)
Commit: 3f0e015fe53dca6172ebf31feafa30756ae980f8 (George K)

Failure:

• coderd/x/chatd TestOpenAIResponsesNoStaleWebSearchReplay
• Error: expected 2 OpenAI requests but recorded 1.

Log snippet:

=== FAIL: coderd/x/chatd TestOpenAIResponsesNoStaleWebSearchReplay (0.11s)
    integration_responses_test.go:97:
        Error: "[{[map[content:You are the Coder agent — ... [map[text:search for the latest Coder docs type:input_text]] role:user]] ...}]" should have 2 item(s), but has 1
    chatdebug: failed to load runtime admin chat debug logging setting  error="context canceled"
    processor: chat interrupted  chat_id=5cd44b23-bb2d-4bff-8561-e9ff946cd3d8

No WARNING: DATA RACE, panic, or OOM indicators in the job logs.

[flake-investigator]

New occurrence in nightly gauntlet:

• CI run: https://github.com/coder/coder/actions/runs/25091958228
• Job: https://github.com/coder/coder/actions/runs/25091958228/job/73519781633 (test-go-pg windows-2022)
• Commit: 76242f82024d85f8603dcbaab38fdcfd49575f46 (Rowan Smith)

Failure:

=== FAIL: coderd/x/chatd TestOpenAIResponsesNoStaleWebSearchReplay (5.19s)
    integration_responses_test.go:97:
        Error: "[{[map[content:You are the Coder agent — ... [map[text:search for the latest Coder docs type:input_text]] role:user]] ...}]" should have 2 item(s), but has 1

[flake-investigator]

New occurrence:

• CI run: https://github.com/coder/coder/actions/runs/25120997694
• Job: https://github.com/coder/coder/actions/runs/25120997694/job/73621426260 (test-go-pg-17)
• Commit: 6ea9c61da035d175ea2035b19b9b4327507d4a0b (Paweł Banaszewski)

Failure excerpt:

chatdebug: failed to load runtime admin chat debug logging setting  error="pq: canceling statement due to user request"
processor: chat interrupted  chat_id=60fe6c90-1044-4c5f-9e28-f185e0a53f5d
integration_responses_test.go:97:
    Error: "[...]" should have 2 item(s), but has 1

No data race warnings, panic, or OOM indicators observed in the logs.

[flake-investigator]

New occurrence:

• CI run: https://github.com/coder/coder/actions/runs/25123844383
• Failing job: https://github.com/coder/coder/actions/runs/25123844383/job/73631511981 (test-go-pg (ubuntu-latest))
• Commit: 950660e392e560a84abf7c61a7190e21f87ce73c (Callum Styan)

Failure:

• Test: coderd/x/chatd TestOpenAIResponsesFullReplayPairsReasoningAndWebSearch
• File: coderd/x/chatd/integration_responses_test.go:180
• Error:

Error: "[{[map[content:You are the Coder agent — ... [map[text:search for the latest Coder docs type:input_text]] role:user]] ...}]" should have 2 item(s), but has 1

• Context:

processor: chat interrupted  chat_id=0d7423ec-b898-4481-a9c3-610598f0ca83

Looks consistent with the existing OpenAI Responses replay flakes where the follow-up prompt is missing expected items / second request isn’t recorded.

[flake-investigator]

New occurrence in https://github.com/coder/coder/actions/runs/25180400944

Failing jobs:

• https://github.com/coder/coder/actions/runs/25180400944/job/73823863516 (test-go-pg (ubuntu-latest))
• https://github.com/coder/coder/actions/runs/25180400944/job/73823863513 (test-go-pg-17)

Commit: 201de14a9d549f7f5179925c6da988d274945cef (Max Schwenk)

Failure snippet:

=== FAIL: coderd/x/chatd TestOpenAIResponsesNoStaleWebSearchReplay
integration_responses_test.go:97:
    Error: "..." should have 2 item(s), but has 1

Context from logs:

• chatdebug: failed to load runtime admin chat debug logging setting error="context canceled"
• chatdebug: failed to load runtime admin chat debug logging setting error="pq: canceling statement due to user request"
• processor: chat interrupted