Repository navigation
Python: DevUI audio input and response playback - #9462
Draft
Fernando de Oliveira (fedeoliv) wants to merge 15 commits into
Draft
Fernando de Oliveira (fedeoliv) wants to merge 15 commits into
Fernando de Oliveira (fedeoliv) wants to merge 15 commits into
Conversation
Send microphone recordings and uploaded WAV/MP3 files as input_audio content without transcription.
Render agent audio outputs with preview, download, and optional autoplay while preserving DevUI media output items in final responses. Refine microphone startup behavior and audio settings copy for a less noisy chat experience.
Add push-to-talk, live input level, optional silence auto-stop, clearer audio settings, and tighter recording/preview layouts.
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 15:48 — with 
GitHub Actions
Active
GitHub Actions
Active
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 15:48 — with
GitHub Actions
Active
GitHub Actions
Active
agent-framework-automation
Bot
added
documentation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved recording lifecycle and audio-handling defects can leave microphone capture active, disable the composer, or lose audio content.
6 open findings
What changed in this PR
Adds audio input and response playback to Python DevUI, enabling developers to test audio-capable agents through the existing chat interface.
Changes:
- Adds microphone recording, WAV/MP3 uploads, previews, playback controls, and audio settings.
- Extends request mapping, response handling, and conversation persistence for audio.
- Adds backend tests and user-facing documentation.
| File | Description |
|---|---|
| python/packages/devui/tests/devui/test_multimodal_workflow.py | Tests audio input conversion and format rejection. |
| python/packages/devui/tests/devui/test_mapper.py | Tests streamed and aggregated audio output. |
| python/packages/devui/tests/devui/test_conversations.py | Tests audio persistence round trips. |
| python/packages/devui/README.md | Documents audio features and limitations. |
| python/packages/devui/frontend/src/types/openai.ts | Adds audio content and auto-play types. |
| python/packages/devui/frontend/src/types/agent-framework.ts | Adds structured audio input types. |
| python/packages/devui/frontend/src/stores/devuiStore.ts | Persists audio preferences. |
| python/packages/devui/frontend/src/hooks/use-audio-recorder.ts | Implements capture, WAV normalization, and analysis. |
| python/packages/devui/frontend/src/hooks/index.ts | Exports audio utilities. |
| python/packages/devui/frontend/src/components/ui/file-upload.tsx | Restricts audio selection to WAV/MP3. |
| python/packages/devui/frontend/src/components/ui/chat-message-input.tsx | Integrates recording and audio attachments. |
| python/packages/devui/frontend/src/components/ui/audio-message-preview.tsx | Adds recording and playback controls. |
| python/packages/devui/frontend/src/components/layout/settings-modal.tsx | Exposes audio settings. |
| python/packages/devui/frontend/src/components/features/agent/message-renderers/OpenAIMessageRenderer.tsx | Passes audio auto-play flags. |
| python/packages/devui/frontend/src/components/features/agent/message-renderers/OpenAIContentRenderer.tsx | Renders audio previews and players. |
| python/packages/devui/frontend/src/components/features/agent/agent-view.tsx | Integrates audio into chat messages. |
| python/packages/devui/agent_framework_devui/models/_openai_custom.py | Adds extension-compatible response models. |
| python/packages/devui/agent_framework_devui/models/__init__.py | Exports custom response models. |
| python/packages/devui/agent_framework_devui/_server.py | Serializes extended conversation items. |
| python/packages/devui/agent_framework_devui/_mapper.py | Preserves audio in mapped responses. |
| python/packages/devui/agent_framework_devui/_executor.py | Converts structured audio into agent content. |
| python/packages/devui/agent_framework_devui/_conversations.py | Adds audio conversation serialization. |
| python/packages/devui/agent_framework_devui/_audio.py | Centralizes audio format helpers. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 17:00 — with
GitHub Actions
Active
GitHub Actions
Active
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 17:07 — with
GitHub Actions
Active
GitHub Actions
Active
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Recording limits, audio persistence, and playback behavior still have functional defects.
11 open findings
🧠 Review effort: Balanced
Enhance audio content check to include additional conditions. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 17:35 — with
GitHub Actions
Active
GitHub Actions
Active
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 17:46 — with
GitHub Actions
Active
GitHub Actions
Active
Fernando de Oliveira (fedeoliv)
force-pushed
the
feature/devui-voice-input
branch
from
October 16, 2026 17:57
b1eed51 to
9b27221
Compare
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 17:57 — with
GitHub Actions
Active
GitHub Actions
Active
Restore the OpenAI response alias to the SDK Response type and remove custom completed/failed event models so the audio backend changes stay localized to DevUI extension output items. Keep typed input_audio parsing for pyright and preserve audio conversion behavior without changing the OpenAI proxy executor contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 259add50-0116-466a-9cbe-3ac3983977d4
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 19:12 — with
GitHub Actions
Active
GitHub Actions
Active
Restore the conversation test to assert input_audio round-trip behavior instead of relying on the internal item representation, and keep mapper assertions compatible with the stricter typing checks used in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 259add50-0116-466a-9cbe-3ac3983977d4
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 19:49 — with
GitHub Actions
Active
GitHub Actions
Active
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 19:52 — with
GitHub Actions
Active
GitHub Actions
Active
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 20:06 — with
GitHub Actions
Active
GitHub Actions
Active
Restore the existing arg-type ignore comments around DevUI output item constructors while keeping the custom output item event required for runtime validation of extension output items. Narrow the conversation audio round-trip test helper so CI typing checkers accept both dict and model-backed conversation items. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 259add50-0116-466a-9cbe-3ac3983977d4
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 20:32 — with
GitHub Actions
Active
GitHub Actions
Active
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 259add50-0116-466a-9cbe-3ac3983977d4
Fernando de Oliveira (fedeoliv)
deployed
to
github-app-auth
October 16, 2026 20:50 — with
GitHub Actions
Active
GitHub Actions
Active
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation & Context
DevUI can render text conversations and file attachments, but it did not provide a complete path for testing agents that accept audio input or return audio output. This change adds that path so developers can test audio-capable agents directly in DevUI without building a custom UI.
The feature adds browser microphone recording with push-to-talk, preview before send, audio upload handling, audio response playback, optional auto-stop on silence, and clearer audio settings.
Description & Review Guide
What are the major changes?
Adds microphone recording in DevUI chat with WAV normalization before sending. Supports WAV and MP3 uploads as agent audio input.
Recording UX: push-to-talk, live mic level feedback, fixed-size recording layout, and optional auto-stop on silence.
Adds reusable audio preview and playback UI with waveform rendering, scrubber support, removal, download, and optional auto-play for agent audio responses.
Adds audio settings under Settings, grouped into Input, Auto-stop on silence, and Output sections.
Adds backend mapping for audio request and response content through DevUI conversations, executor handling, and OpenAI-compatible response mapping.
Adds tests for audio content mapping, conversation persistence, and multimodal workflow behavior.
Updates DevUI documentation with user-facing voice input guidance.
What is the impact of these changes?
What do you want reviewers to focus on?
Related Issue
Fixes #9359
AI Assistance
AI assistance details: GitHub Copilot assisted with implementation, testing, and UI refinements.
AI assistance details:
Contribution Checklist
breaking changelabel or add "[BREAKING]" to the title prefix, before or after any language prefix. A workflow keeps the label and title prefix in sync automatically.