feat: Phase 3 — streaming TTS WebSocket & self-hosted distribution credentials by dg-coreylweathers · Pull Request #166 · deepgram/deepgram-rust-sdk

dg-coreylweathers · 2026-07-31T11:24:27Z

Phase 3 — Streaming TTS WebSocket + self-hosted credentials

Stacked on #165 (Phase 2). Base is feat/phase-2-read-stream-models; GitHub will retarget to main as the stack merges.

What's included

[Enhancement] Add Text-to-Speech WebSocket streaming support #148 / [Enhancement] Add Text-to-Speech WebSocket streaming support #147 / Text to Speech - Websocket API #95 — Streaming TTS over a WebSocket. Speak::speak_stream() → SpeakStreamBuilder (model / encoding / sample rate); handle() opens the connection → SpeakStreamHandle with speak / flush / clear / close, and receives audio + events. SpeakStreamHandle: futures::Stream<Item = Result<SpeakResponse>>. SpeakResponse (Audio / Metadata / Flushed / Cleared / Warning / Unknown) is #[non_exhaustive]; unknown message types are preserved rather than breaking the stream. The speak feature now pulls the WebSocket deps. Modeled on src/listen/websocket.rs. Example text_to_speech_websocket.
Self-hosted distribution credentials. Deepgram::self_hosted() → SelfHosted with list / get / create / delete_distribution_credentials + typed response models. Example self_hosted_credentials.

Verification (Rust 1.97 container)

build / clippy -D warnings / fmt / cargo test --all --all-features (142 tests, incl. TTS-WS protocol + self-hosted deserialization tests) ✅
per-feature cargo check incl. speak-only (now pulls WS deps) ✅
cargo semver-checks: no new breaking changes (only the inherited Phase 1 Extra metadata support #130 streaming breaks).
Live-verified against production: TTS WebSocket connected, emitted Metadata + Flushed, and streamed 219 KB of audio. Self-hosted list returns a correct 403 INSUFFICIENT_PERMISSIONS on accounts without the self-hosted scope (confirms the request path).

Closes #148, #147, #95

dg-coreylweathers · 2026-07-31T11:34:38Z

Addressed the review's BLOCKING finding: the TTS-WS worker busy-spun (and hung on current-thread runtimes) after input close because it kept selecting on the drained input channel. run_worker now stops selecting on message_rx once input is closed and only awaits server messages (via a shared handle_incoming helper). Verified: re-ran the example under #[tokio::main(flavor = "current_thread")] — it completes cleanly (219KB audio) where it previously would hang. Also documented the send-then-drain contract and the nil-request_id fallback on SpeakStreamHandle. The self-hosted create/get envelope matches list per the API docs (no secret token field in the create response).

GregHolmes

Review: deepgram-rust-sdk #166, feat: Phase 3 - streaming TTS WebSocket and self-hosted distribution credentials

Classification: mixed (code-led)
Verdict: request-changes

Intent

Add WebSocket streaming TTS and self-hosted distribution-credential management while keeping the SDK's existing async patterns. This PR is intentionally stacked on #165 and cannot merge until #164 and #165 are fixed, merged, and the stack is retargeted. Intent is based on the PR, issues #95, #147, and #148, and the author's live-verification comment; no launch or Slack context is linked.

Blocking

[B1] src/speak/websocket.rs:161, streaming TTS silently drops documented metadata.

Title: [B1] SpeakResponse::Metadata omits additional_model_uuids

Summary: The streaming TTS API's Metadata event includes the optional additional_model_uuids array, but neither TextEvent::Metadata nor the public SpeakResponse::Metadata carries it. Serde ignores the unknown field, so a successful request silently loses model provenance that a caller may need for auditing or billing.

Expected: All documented fields from a streaming TTS Metadata event are available in the typed response.

Observed: additional_model_uuids is discarded during deserialization and cannot be recovered by SDK users.

Recommended fix: Add additional_model_uuids: Option<Vec<Uuid>> to TextEvent::Metadata and SpeakResponse::Metadata, pass it through the conversion, and add a fixture that asserts the values survive parsing.

[B2] src/speak/websocket.rs:98, the streaming TTS builder does not enforce or expose the endpoint's option contract.

Title: [B2] Streaming TTS accepts invalid encodings and omits supported controls

Summary: The streaming endpoint supports only linear16, mulaw, and alaw, but SpeakStreamBuilder::encoding accepts the REST Encoding enum, including mp3, opus, flac, aac, and arbitrary custom values. Conversely, the endpoint documents speed and mip_opt_out, but the builder has no setters or query-parameter escape hatch, so callers cannot use them at all.

Expected: The streaming API rejects locally known-invalid encodings before connecting and exposes every documented streaming control.

Observed: Encoding::Mp3 is serialized into a request the server rejects, while speed and mip_opt_out cannot be sent through this client.

Recommended fix: Use a streaming-specific encoding enum or validate the existing enum in handle() with a clear DeepgramError::InvalidOptions; add speed and mip_opt_out to the builder, with the documented 0.7..=1.5 range for speed, and cover valid and invalid URLs in tests.

Should-fix

[S1] src/speak/websocket.rs:534, protocol tests do not cover every supported client and server message.

Title: [S1] Streaming TTS protocol coverage omits Clear, Cleared, and Warning paths

Summary: The unit tests assert Speak, Flush, Close, Metadata, Flushed, and Unknown behavior, but omit Clear transmission and Cleared and Warning parsing. These are public control paths; a future serde rename or refactor can break them without a test failure.

Expected: Each public protocol message has an exact serialization or parsing assertion.

Observed: The available tests do not exercise Clear, Cleared, or Warning.

Recommended fix: Extend the existing table-style tests with those three cases, including optional sequence_id, description, and code values.

Nits

None.

Verified (evidence)

The streaming TTS WebSocket path, ClientMessage variants, required metadata fields, Flush/Clear/Close messages, and supported encoding list align with the live API reference.
The self-hosted list and create paths, request body/query placement, defaults, and response envelope align with the live Self-Hosted API reference.
Local Rust 1.94.1: build, tests (79 unit plus 140 doc-tests), clippy, fmt, docs, and cargo check --no-default-features --features speak pass. The rust:1.97 image named in the PR contains neither cargo nor rustup, so it cannot run the declared container gate.
The prior busy-spin report is fixed in the current worker: once the input closes, it stops selecting on the drained input channel and drains server messages only.

Needs human

Greg: the stack depends on #164 and #165. Both have requested-changes reviews and #164 conflicts with main, so this PR must stay unmerged until their fixes are merged and the stack is retargeted. The inherited SemVer failures are a release-process decision for the planned 0.11.0 aggregation.

Developer-facing messaging

The changelog accurately says the speak feature now pulls WebSocket dependencies. It should not imply the typed Metadata event is complete until [B1] preserves additional_model_uuids.
Rejecting unsupported output encodings before opening the socket and exposing documented controls gives developers a clear local error instead of an avoidable server-side failure.

…credentials - TTS WebSocket (#148/#147/#95): Speak::speak_stream() -> SpeakStreamBuilder (model/encoding/sample_rate); handle() opens the connection and returns a SpeakStreamHandle for sending text (speak/flush/clear/close) and receiving audio + events. SpeakStreamHandle implements futures::Stream; SpeakResponse (Audio/Metadata/Flushed/Cleared/Warning/Unknown) is #[non_exhaustive] and unknown message types are preserved rather than breaking the stream. The speak feature now enables the WebSocket deps. New example text_to_speech_websocket. - Self-hosted distribution credentials: Deepgram::self_hosted() -> SelfHosted with list/get/create/delete_distribution_credentials, plus typed response models. New example self_hosted_credentials. All additive (cargo semver-checks: no new breaking changes beyond the Phase 1 #130 streaming change). Live-verified the TTS WebSocket against production; the self-hosted list path returns a correct 403 on accounts without the self-hosted scope. Closes #148, #147, #95 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… validation, speed/mip_opt_out, protocol test coverage - B1: `SpeakResponse::Metadata` and the internal `TextEvent::Metadata` now carry `additional_model_uuids: Option<Vec<Uuid>>` (absent -> None), passed through the conversion, with a parsing fixture asserting the values survive (present, absent, and empty array). - B2: `SpeakStreamBuilder::handle()` validates options locally before connecting. REST-only encodings (`mp3`, `opus`, `flac`, `aac`) and a `speed` outside `0.7..=1.5` (or non-finite) are rejected with the new `DeepgramError::InvalidOptions(String)`; `linear16` / `mulaw` / `alaw` pass and `CustomEncoding` is left as an escape hatch. New `speed(f32)` and `mip_opt_out(bool)` setters serialize as `speed=` / `mip_opt_out=` query params. Tests cover valid/invalid URLs, each rejected encoding, the speed boundaries, and that `handle()` short-circuits without a network attempt. - S1: protocol tests now cover `Clear` client serialization and table-style parsing of `Cleared` (with/without `sequence_id`) and `Warning` (all combinations of `description` / `code`), matching the live API reference. - CHANGELOG: Unreleased entry mentions `speed`, `mip_opt_out`, local encoding validation via `DeepgramError::InvalidOptions`, and `additional_model_uuids`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…shared TLS connector Route /v1/speak streaming through crate::tls like every other wss:// surface since 0.11.0, so rustls-tls-native-roots and Deepgram::tls_config apply to it and an untrusted certificate surfaces as UntrustedTlsCertificate. Adds the per-surface tls_config test alongside the existing live-transcription and Flux cases.

Uses the shared crate::websocket_host_header so a base URL on a non-default port is routed by host:port like every other WebSocket surface; the Host capture test covers the streaming TTS client.

…rward, and correct the streaming TTS docs Follow-ups from the #166 review: - `speak::options::Model` was missing `aura-2-perseo-it`, the 91st Aura-2 voice in the API specification. Add it in locale order and make the round-trip test assert the exact variant count (103) instead of a lower bound, so the next specification drift fails loudly. - Document that a hand-built `Model::CustomId` is not `==` to the named variant whose wire string it spells. - `src/tls.rs`, the `Deepgram::tls_config` rustdoc, and `README.md` each enumerated only three `wss://` surfaces; streaming text-to-speech goes through the same shared rustls connector, so name it. - AGENTS.md: mark streaming text-to-speech and self-hosted credentials as shipped with their real entry points (the documented self-hosted path was wrong too: it is `/self-hosted/distribution/credentials`, not `/onprem/`), add `src/speak/websocket.rs` and `self_hosted` to the repository map, and correct the default-features list. - The installable text-to-speech skill said the Aura `/v1/speak` WebSocket was not implemented. Correct every such claim and teach the real API. - `SpeakResponse::Metadata::additional_model_uuids` is now `Option<Vec<String>>`, matching `model_uuid`: one non-UUID element no longer downgrades a readable `Metadata` event to `Unknown`. - The worker's terminal write-error forward no longer waits for room on the response channel. With events undrained it parked the worker on a full channel while the caller parked on a full outbound channel, contradicting the rustdoc promise that an audio backlog never blocks `speak`. Same fix in the Flux text-to-speech worker, where it is a deadlock released in 0.10.1. - Tests: a wire assertion for `Encoding::CustomEncoding`, the `terminal_read_error_ends_worker_after_single_error` case the other two sockets already had, and a write-error-with-undrained-events regression test for both text-to-speech workers (each fails without the fix above). - `tests/connect_tls_config_local.rs` was gated `listen`-only, so its `speak`-only cases never compiled in CI's `speak` jobs. Relax the gate, per-test `cfg` the `listen` cases, and add a `speak`-only negative case.

…t the new public items Four follow-ups on the self-hosted and streaming-TTS work in this branch, all on API that is new here and not in 0.11.0: - `CreateDistributionCredentials::scopes` and the `scopes` on both response types are now `Vec<Scope>` instead of `Vec<String>`. `Scope` follows the SDK's open-enum convention: eight named variants plus `Scope::Unknown(String)`, which preserves an unrecognized wire value and re-serializes to it exactly, and is also how a caller sends a scope the SDK has not caught up with. `From<&str>` / `From<String>` keep the string form of the builder working. - All four `SelfHosted` methods build their URL against the client's configured base URL, the way `read::rest` does, so `Deepgram::with_base_url` is no longer ignored by the module self-hosted users are most likely to point at their own host. - `get_distribution_credentials` and `delete_distribution_credentials` take the `Uuid` the SDK hands back in `distribution_credentials_id` rather than `&str`, so the round trip through `.to_string()` in the example is gone. - The `#[allow(missing_docs)]` on the seven self-hosted response fields is replaced with real doc comments, and `speak_models!` now generates a doc comment per variant from the wire string and the language it is grouped under, so all 103 voices are documented from one place.

…2 voice Applies the pass-4 review of #166. B1: the streaming text-to-speech worker forwarded a terminal write error with `try_send` on the shared response channel, so the error was silently discarded whenever the 256-deep channel was full at that instant. With `SpeakStreamHandle::split()` a consumer drains events from its own task and never parks on the outbound channel, so a full response channel at failure time is reachable with the consumer very much alive: it saw the audio stream end and its next `speak()` fail, with no error value. That violates the written contract in AGENTS.md ("the worker forwards a terminal transport error exactly once"). The worker now keeps a dedicated clone of the sender for exactly that error: a `futures` mpsc channel's capacity is `buffer + num_senders`, so the clone carries its own guaranteed slot and the forward neither blocks (which is what deadlocked a non-draining caller before) nor drops. Same shape applied to the Flux text-to-speech worker's write-failure and post-loop close paths. New test `write_error_reaches_a_slow_split_consumer` asserts the error is *received*, not merely that nothing hangs — the existing `write_error_with_undrained_events_does_not_stall_the_worker` passed either way, which is how this slipped through. Against the previous code it fails deterministically with 0 terminal errors after 297 drained audio events. B2: remove `Model::Aura2PerseoIt`. The API specification lists it as the 91st Aura-2 voice, but the specification is wrong: production answers `400 {"err_msg":"No such model/version combination found."}` for `aura-2-perseo-it`, the other nine Italian Aura-2 voices all return `200 audio/mpeg`, and `GET /v1/models?include_outdated=true` lists 90 Aura-2 canonical names with `aura-2-perseo-it` the only specification entry missing. The round-trip test keeps its exact-count assertion — that is the right mechanism — pinned to 102 (12 Aura-1 + the 90 Aura-2 voices the API serves) with the reason recorded in the comment, and gains an assertion that `aura-2-perseo-it` resolves to `CustomId`. Prose that claimed coverage of every voice "in the API specification" now says what is true: every voice the API serves. S1: the changelog bullet for the Flux text-to-speech fix says what a developer now observes and drops the "no public signature changed" clause; a signature fact is not evidence about behavior. S2: `with_base_url` / `with_base_url_and_api_key` no longer claim that all admin features ignore the base URL. They name what honors it (transcription, text-to-speech, Text Intelligence, model listing, self-hosted credentials) and what does not (billing, usage, keys, members, invitations, projects, scopes, and the token grant). Nits: the host-header capture now also reports the upgrade target, giving the streaming text-to-speech escape hatches (`query_params`, `Encoding::CustomEncoding`) a wire-level guard; `output.raw` and `flux-tts-batch.mp3` are gitignored; the `Url::join` trailing-slash requirement is documented on both `with_base_url` constructors; and the stray space inside the `scopes` URL literal is removed (the URL parser stripped it, so no behavior change). Gates, all exit 0 with `RUSTFLAGS=-D warnings RUSTDOCFLAGS=-D warnings`: `cargo fmt --check --all`; `cargo clippy --all-targets --all-features`; `cargo test --all --all-features` (204 unit + 150 doc tests, every `*_local.rs` suite, 0 failed); `cargo doc --workspace --all-features` plus the five single-feature doc builds; `cargo check --all-targets --no-default-features` with `speak` and with `speak,rustls-tls-native-roots`; `cargo test --no-default-features --features speak`.

…delivery Applies the three should-fix findings from the final verification review of #166. [S1] The README's `speak` feature row named only "Aura REST, Flux TTS REST and streaming WebSocket" — true on the parent branch, but this branch ships Aura streaming over `wss /v1/speak`, so the row understated the feature. It now covers all four surfaces, and the table's column width is back in alignment. The README named the new entry point nowhere, so a short pointer after the table gives `dg.text_to_speech().speak_stream()` and the runnable examples for both streaming sockets. [S2] The shipped management-API skill omitted the self-hosted credentials surface and stated a base-URL rule this branch breaks. It now lists the four `self_hosted()` methods (with the Uuid credentials ID and the return-once secret), `src/manage/self_hosted.rs`, and `examples/manage/self_hosted_credentials.rs`; gotcha 3 now matches the `with_base_url` rustdoc exactly — `models()` and `self_hosted()` honor the configured base URL, while billing, usage, keys, members, invitations, projects, scopes, and `/v1/auth/grant` stay on the hosted site. [S3] Nothing tested that the Flux TTS terminal write error is *delivered*: the only covering test asserted that `speak()` stops hanging, so it passed with the error dropped. The new test in `tests/flux_speak_backpressure_local.rs` fills the response channel, breaks the transport, and then drains `handle.receive()`, asserting that at least 256 audio events were queued (the channel was provably full at the failure), that exactly one `Err` arrives, that it is the last item, and that the stream then ends. Reverting the `error_tx` forward to `response_tx.try_send(..)` fails it with 0 errors delivered after 257 queued audio events, while the pre-existing deadlock test still passes.

The skill pinned 0.10.1 while teaching `speak_stream()`, which this branch adds. The pin resolves, so it fails quietly: the reader gets a crate that cannot compile the streaming examples the skill is built around. Drop the version, matching the other skills.

dg-coreylweathers force-pushed the feat/phase-3-tts-ws-selfhosted branch from 5a60203 to 570125f Compare July 31, 2026 11:34

This was referenced Jul 31, 2026

feat(agent): Phase 4 — Voice Agent WebSocket client (#152) #167

Open

[Enhancement] Add Text-to-Speech WebSocket streaming support #148

Open

[Enhancement] Add Text-to-Speech WebSocket streaming support #147

Open

Text to Speech - Websocket API #95

Open

GregHolmes requested changes Sep 11, 2026

View reviewed changes

dg-coreylweathers and others added 9 commits September 15, 2026 09:32

fix(speak): send the full authority in the streaming TTS Host header

d26f6c5

Uses the shared crate::websocket_host_header so a base URL on a non-default port is routed by host:port like every other WebSocket surface; the Host capture test covers the streaming TTS client.

dg-coreylweathers force-pushed the feat/phase-2-read-stream-models branch from a5fa559 to 4c76c47 Compare September 15, 2026 14:02

dg-coreylweathers force-pushed the feat/phase-3-tts-ws-selfhosted branch from 570125f to 8052666 Compare September 15, 2026 14:04

dg-coreylweathers mentioned this pull request Sep 15, 2026

fix(listen)!: unblock and deliver the streaming terminal transport error #179

Open

4 tasks

Navigation Menu

Uh oh!

feat: Phase 3 — streaming TTS WebSocket & self-hosted distribution credentials - #166

feat: Phase 3 — streaming TTS WebSocket & self-hosted distribution credentials#166
dg-coreylweathers wants to merge 9 commits into
feat/phase-2-read-stream-modelsfrom
feat/phase-3-tts-ws-selfhosted

dg-coreylweathers commented Jul 31, 2026

Uh oh!

dg-coreylweathers commented Jul 31, 2026

Uh oh!

GregHolmes left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

dg-coreylweathers commented Jul 31, 2026

Phase 3 — Streaming TTS WebSocket + self-hosted credentials

What's included

Verification (Rust 1.97 container)

Uh oh!

dg-coreylweathers commented Jul 31, 2026

Uh oh!

GregHolmes left a comment

Choose a reason for hiding this comment

Review: deepgram-rust-sdk #166, feat: Phase 3 - streaming TTS WebSocket and self-hosted distribution credentials

Intent

Blocking

Should-fix

Nits

Verified (evidence)

Needs human

Developer-facing messaging

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants