Repository navigation
Twilio: speak with TwiML Say, add ConversationRelay, bound OpenAI TTS timeout - #1459
Merged
Merged
Conversation
GenerateSpeechAsync used the SDK defaults: 100s per attempt and 3 retries.
In production some /v1/audio/speech requests never return, so a failed call
blocked the caller for about 7 minutes ("Retry failed after 4 tries").
AudioSynthesisProvider now waits at most 30s per attempt and keeps 3 retries.
Both can be set per model under Audio.Synthesis (TimeoutSeconds, MaxRetries).
ProviderHelper.GetClient takes the timeout and retry count as optional
arguments; other callers keep the SDK defaults.
Verified against a local server that never answers: 4 attempts about 30s
apart, failing after 125.7s (was ~400s); a hang followed by a good response
recovers on the retry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Say mode (Twilio:SayGeneratedSpeech, off by default)
- Generated speech is saved as a text file and spoken with <Say voice=SayVoice>
instead of pre-generating an mp3 with the audio synthesizer: the opening line
(no longer synthesized before dialing), replies, hang-up, transfer, voicemail
and tool-call indications.
- Prerecorded prompts (twilio/{name}.mp3) are spoken from Twilio:PromptTexts
so the whole call uses one voice; prompts without text keep playing.
- TwilioService.SaveSpeechAsync / AppendSpeech are the single places that
write and play speech. With the switch off, behaviour is unchanged.
ConversationRelay (Twilio:ConversationRelay, off by default)
- Outbound calls for enabled agents (AgentIds, empty = all) are answered with
<Connect><ConversationRelay>: Twilio transcribes and speaks, and exchanges
text with TwilioConversationRelayMiddleware at /twilio/relay/{agent}/{conv}.
- ConversationRelaySession runs one agent turn at a time (later words are
merged into the next turn), speaks tool-call indications, re-prompts on
caller silence, hangs up after MaxIdlePrompts or AiResponseTimeoutSeconds,
and ends with handoffData hangup/transfer.
- twilio/relay/action dials the CSR on transfer, hangs up otherwise, and
reconnects a failed session up to MaxReconnects times.
- The WebSocket handshake is checked against X-Twilio-Signature.
- The caller-message handling used by the webhook queue moved to
TwilioConversationProcessor so both flows share it.
Tested with in-process harnesses: 24 TwiML checks for Say mode and 28 checks
driving the relay WebSocket (turns, merging, DTMF, indications, silence,
timeouts, signature, TwiML, action callback).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Code Review by QodoSorry, something went wrongWe weren't able to complete the code review on our side. Please try again manually by commenting/agentic_review on this PR.
|
Contributor
PR Summary by QodoAdd Twilio Say and ConversationRelay with bounded OpenAI TTS requests
AI Description
Diagram
High-Level Assessment
Files changed (20)
|
… them With Twilio:ConversationRelay:StreamReplies on (off by default), a relay turn asks BotSharp to stream (use_stream_message, and language English when unset) and forwards each OnReceiveLlmStreamMessage piece to Twilio as a text token with last=false, so Twilio starts speaking with the first words instead of after the whole reply. - Each model response is closed with last=true when the next one starts, before a tool-call indication is spoken, and at the end of the turn. - The final reply is not spoken again; only the part not streamed is (text set by a tool, or a reply rewritten after the model wrote it). - When the caller cuts in, the rest of the streamed reply is dropped; when the turn times out, nothing streamed afterwards reaches the call. - TwilioConversationProcessor takes AgentTurnCallbacks (indication, stream start, stream text); the webhook queue passes only the indication, so that flow does not stream. Needs a chat provider that streams (OpenAI, Azure OpenAI, Anthropic, Google); non-English conversations get the whole reply as before. Tested with in-process harnesses: relay harness 42/42 (13 new: pieces and close, first words before the model finishes, tool-call boundary, remainder, rewritten reply, tool-set reply, interrupt, timeout, and the processor's state and event wiring against stubbed BotSharp services); Say harness passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Stop dequeuing caller input once the call is gone: when the relay socket closes, a turn already running finishes (so the conversation is saved) but words queued during it no longer start another agent turn. - Keep each keypad entry's Digits: inputs are queued with their digits, and a keypad entry ends the merge of words said while the agent was busy, so two entries during one turn become two turns instead of losing the first. - Decide and end the silence hang-up atomically, so caller input arriving at that moment cannot start a turn on a call being hung up; skip the "say it again" prompt when the caller spoke since it was decided. - Drop the unused conversation lookup on every caller message (moved from the webhook queue, never read). - Escape agent and conversation ids in the relay action query and as single path segments of the WebSocket URL; the signature check rebuilds the same escaped URL. - Move ConversationRelayHandoff to its own file and wrap lines over 125 characters. Tested with the relay harness, 49/49: seven new checks for these fixes, six of which fail on the previous code (the hang-up race itself cannot be forced, its check guards the outcome). Say harness passes. Downstream app: build clean, unit and integration suites at their known baselines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous fix marked the session ended only after waiting for the idle watcher to stop. In that interval a turn that finished could still dequeue the next caller input and start an agent request for a call that was gone. Mark the session ended and clear the queue first, inside the finally that follows the receive loop, so this also holds when the loop fails with an unexpected exception; then stop the idle watcher and let the turn already running finish. Queue caller inputs in a Queue<PendingInput> instead of removing batches from the front of a List. Tested with the relay harness, 50/50 (new: closing at seven points around the end of a running turn, each run either runs the queued words while the call is up or drops them at close, never both or neither). The race itself is too narrow to force in-process. Say harness passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
|
reviewed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Outbound Twilio calls synthesize speech with OpenAI
/v1/audio/speechbefore dialing and on every turn. In production some of those requests never return.GenerateSpeechAsyncused the SDK defaults (100s per attempt, 3 retries), so one stuck request blocked the call for about 7 minutes (Retry failed after 4 tries), and when it was the opening line the call was never placed.Changes
1. OpenAI text-to-speech timeout (
bf039def7)AudioSynthesisProviderwaits at most 30s per attempt and keeps 3 retries.Audio.Synthesis.TimeoutSeconds/MaxRetriesinLlmModelSetting.ProviderHelper.GetClienttakes the timeout and retry count as optional arguments; other callers keep the SDK defaults.2. Twilio
<Say>mode (Twilio:SayGeneratedSpeech, off by default)<Say voice="Twilio:SayVoice">instead of a pre-generated mp3: opening line, replies, hang-up, transfer, voicemail, tool-call indications.twilio/{name}.mp3) can be spoken fromTwilio:PromptTextsso the whole call uses one voice; prompts without text keep playing.TwilioService.SaveSpeechAsync/AppendSpeechare now the single places that write and play speech. With the switch off, behaviour is unchanged.3. Twilio ConversationRelay (
Twilio:ConversationRelay, off by default;AgentIdslimits it to listed agents)<Connect><ConversationRelay>: Twilio transcribes and speaks, and exchanges text with the newTwilioConversationRelayMiddlewareat/twilio/relay/{agentId}/{conversationId}.ConversationRelaySessionhandles one call:#-terminated) and speaks tool-call indications;MaxIdlePrompts, or when the agent doesn't answer withinAiResponseTimeoutSeconds;handoffDatahanguportransfer.twilio/relay/actiondials the CSR number on transfer, hangs up otherwise, and reconnects a failed session up toMaxReconnectstimes.X-Twilio-Signature(ValidateSignature, on by default).TwilioMessageQueueServicemoved toTwilioConversationProcessorso both flows share it.TwilioStreamMiddleware:app.UseMiddleware<TwilioConversationRelayMiddleware>();. Before enabling it, the account must accept Twilio's AI/ML features addendum.Testing
<Say>mode: 24 TwiML checks throughTwilioServicepass: each speech type, escaping, prompt replacement, config binding, unchanged behaviour with the switch off.🤖 Generated with Claude Code