feat: media-use v2 (heygen-CLI-only media OS) + non-destructive studio crop#1923
Closed
miguel-heygen wants to merge 57 commits into
Closed
feat: media-use v2 (heygen-CLI-only media OS) + non-destructive studio crop#1923miguel-heygen wants to merge 57 commits into
miguel-heygen wants to merge 57 commits into
Conversation
Ordered, capability-based registry — search/generate/process slots, tried deterministically with heygen-CLI first. Disabled slots (fal aggregator, Iconify, voice TTS, local) carry their gate reason (B-Q1/B-Q2/U2) so enabling them later is a flag flip, not a refactor. resolve cascade now registry-driven; providers.mjs kept as a back-compat shim. Regenerated skills-manifest.
specs.mjs probes CPU/RAM/GPU/VRAM (Apple Silicon unified memory + nvidia-smi), injectable for tests. local-models.mjs is a declarative table of user-installed models per capability+tier (tts/asr/upscale) with size/needs/install/invoke; selectModel picks the highest runnable tier or recommends the CLI path. No license gate — models are user-installed, local-use-only.
fal aggregator + ElevenLabs/HeyGen-TTS voice providers built as real CLI adapters, registered default-OFF and flipped on by env flag (MEDIA_USE_ENABLE_*) read at call time. Bin's answer becomes a flag, not a code change. Tests prove off-by-default + flip-on ordering (heygen stays first).
runLocalModel(capability) picks the best tier for the machine (selectModel), checks the tool is on PATH, fills + runs its invoke template, returns the output path — or a clear install/CLI recommendation when not runnable. Injectable which/exec; never throws. Consumed by the process verb.
…ion (#11, #16) Auto-promote every fetched asset into the global content-addressed cache so it's reusable across all hyperframes projects (cachePut now dedups by sha; resolve promotes on register, non-fatal). search.mjs: paginated text search (no vector DB). usage.mjs: tag/partition assets by in-use vs unused (scans compositions) — powers the Studio Asset-tab filter.
resolve --from <file|direct-url> freezes a user-supplied asset + registers + auto-promotes it. isDirectMediaUrl rejects platform pages (no yt-dlp). brand resolve with no frame.md/design.md now upsells the HyperFrames design flow instead of a generic miss.
references/operations.md: local-tool recipes (ffmpeg cut/reframe/montage, auto-editor, scenedetect) + local-vs-HeyGen transform table (bg-removal/upscale/ lipsync/translate) with side-by-side guidance. Outputs register via --from (auto-promoted). Per OP1, media-use guides to the tools rather than re-wrapping ffmpeg/heygen as bespoke verbs.
Adds 'In use' / 'Unused' filter chips alongside the category filters, reusing the existing usedPaths (composition references) the in-use badge + used-first sort already compute. filterByUsage/countUsage extracted as pure functions + unit tested (6 vitest). Chips show only on a real used/unused split (no misleading 0s). Verified in agent-browser: Studio boots, project loads, Asset tab renders + filter path runs. Note: chip display binds to usePlayerStore.elements (same source as the existing badge); that store was empty in a synthetic dev project where the timeline is driven by the iframe __clipManifest — confirm/realign usedPaths source in real studio usage (follow-up; affects badge+sort+filter equally).
deriveUsedPaths extracted + hardened: element src is the raw authored value, so it can be relative (assets/x.png), ./-prefixed, the served /api/.../preview/ form, or carry a ?query — normalize all to the bare project path so it matches the asset-list entries (the in-use badge / sort / filter all key on this). The prior inline strip only handled the served form. Source stays usePlayerStore.elements (correct — hydrated from the timeline/__clipManifest); not switched. +2 vitest.
Premise: Bin approved B-Q1 (voice/TTS) + B-Q2 (third-party CLIs). fal (bgm/sfx/ image generate) and voice (ElevenLabs + heygen tts) flip from gated to live, marked paid. Cost guard (X4) ships with them: runProviders skips paid providers unless ctx.allowPaid; resolve gains --allow-paid (opt-in) + --local-only (block). Free heygen catalog stays first. Iconify stays gated (not named). SKILL providers table + flags documented. Tests updated (43 green). Skill consolidation/deletion (rest of #15) deliberately NOT done — separate, not implied by approval.
Move the audio engine (audio.mjs + lib + heygen-tts + wait-bgm + lyria-recipe + bundled SFX + references) as a self-contained subtree into skills/media-use/audio/ via git mv — zero internal path edits (HERE/../assets/sfx + lyria offsets preserved). CLI primitives still wrapped, not moved. Smoke test proves the bundled SFX library resolves from the new location. hyperframes-media now holds only SKILL.md (retired in U22).
…e/audio (U21) faceless-explainer / pr-to-video / product-launch-video wrappers: DEFAULT_ENGINE → media-use/audio/scripts/audio.mjs (engine resolves from each, verified). Fixed the relocated engine docs' stale self-paths (tts.md) + an inaccurate attribution comment. Grep gate: no skills/ reference to hyperframes-media/scripts remains (except general-video doc routing, swept in U22).
Delete the hyperframes-media skill (engine relocated in U20). Sweep all 32 referencing files → media-use: engine paths → media-use/audio/scripts, reference links → media-use/audio/references, skill-name/routing mentions → /media-use. Catalog dedup: drop the duplicate audio row from README/CLAUDE/skills.mdx (the single media-use row now covers resolve + audio), router + hyperframes-core domain list point to media-use, skill count 19→18. Manifest regenerated (18, hyperframes-media gone). Grep gate clean (only plan docs reference it as history). dist/ artifacts regenerate from source on build.
Merge the two /media-use rows in the hyperframes router capability map into one; fix the remaining 'all 19' install-count mentions in README/CLAUDE/skills.mdx → 18. Follow-on cleanup from the hyperframes-media retirement rename.
Frontmatter reframed to the full media OS (resolve/generate/operate/remember, incl. voice + the audio engine). New Audio engine section documents audio/scripts/audio.mjs, the request/meta contract, the auto-degrade switch, and the relocated per-topic references (tts/bgm/sfx/transcribe/remove-background/ captions). A reader can now do everything hyperframes-media offered from media-use alone.
…U24) Non-project-scoped GET /api/assets/global reads only ~/.media/manifest.jsonl and returns reusable records — the data behind the Studio Asset tab's cross-project view. Skips malformed lines (torn write won't 500). Registered in createStudioApi alongside the other route groups. 3 vitest. AssetsTab toggle next.
SKILL.md gains a 'What it owns' matrix mapping each HyperFrames media gap to its media-use entrypoint. coverage.test.mjs enforces every row — image/icon providers, voice + audio engine, consolidated engine + bundled SFX, ops reference, global cache + ingest, local-model tiers, every resolve type has a provider — so a claim can't silently rot. 7 tests.
This project / All projects toggle in the Asset tab. The global view is a self-contained GlobalAssetsView component (owns its /api/assets/global fetch + render) — keeps AssetsTab under the file-size cap and lowers its complexity. Lists the reusable cross-project cache (type + label); category/usage chips are local-only; search filters both. globalAssetRows pure helper + 3 vitest. Verified live in agent-browser: 'All projects' shows assets from ~/.media.
…cal-only M2: heygen.tts is free and first in the voice cascade; ElevenLabs is the paid fallback, so voice resolves by default without --allow-paid. M3: --local-only now skips every network provider, not just paid ones.
…m API M1: freezeUrl streams the body and aborts at the 256MB cap (and checks content-length first), so a hostile URL can't buffer past the cap into memory. m11: isDirectMediaUrl rejects loopback/private/link-local hosts on --from ingest. m13: /assets/global projects to public fields, no absolute cached_path to the browser.
- voice cascade in SKILL now reflects free HeyGen TTS first, ElevenLabs paid fallback - --local-only documented as 'skip every network provider', not just paid - drop the nonexistent 'organize --promote' CLI; promotion is automatic - point the local-model row at the actual runner (local-run.mjs) without overclaiming auto-cascade - operations.md: clip-path crop is a composition technique, not a shipped Studio feature - SFX library is 19 files, not 21; usage.mjs no longer claims it powers the Studio filter - clarify the faceless/pr-to-video audio adapters intentionally reuse the product-launch model
The voice provider guessed 'heygen voice tts --text', which the CLI rejects. Real command (verified against heygen v0.1.6 --help / --response-schema): 'heygen voice speech create --text <t> --voice-id <id>' returning data.audio_url + data.duration. Default voice-id = first starfish-engine voice (deterministic, overridable via ctx.voiceId). Verified live: a default voice resolve now produces a real audio file with no --allow-paid. Registry header no longer calls fal/voice flag-gated seams — they are live; Iconify is the only remaining gated provider.
Verified against the official docs (not guessed --help): - fal: use the genmedia CLI (genmedia run <model> --prompt .. --download .. --json -> downloaded_files[].path/url), not 'fal run --input'. The pip 'fal' is the serverless-deploy CLI and can't run hosted models. - ElevenLabs: the official @elevenlabs/cli is agents-only (no TTS). Use the community elevenlabs-cli (tts "<text>" --output <file>): positional text, file output. Both download a local file, returned as localPath for freezing. SKILL providers section now names the correct binaries + install/auth. Live generation still needs FAL_KEY / ELEVENLABS_API_KEY (absent here).
…mes CLI The 'When to use' line credited transcription and background removal to the audio engine; they actually come from the hyperframes CLI (transcribe, remove-background), as the audio-engine section already states. The engine itself does TTS / BGM / SFX / caption timing.
- Cost: paid generation is the agent's call, no user-confirmation prompt
(free-first still tries cache + heygen before anything paid).
- Local models: heygen free-usage is the default (TTS, bg-removal via the
heygen CLI); local open-source is the opt-out fallback only ('if user no,
then local'), not the headline answer. Coverage test description updated to
match while still asserting the fallback table is populated.
…nd --allow-paid Remove the third-party CLI adapters (fal via genmedia, ElevenLabs via elevenlabs-cli) and the paid-provider cost guard entirely; the heygen CLI is the only external CLI media-use shells. --local-only stays. A guard test asserts no fal/elevenlabs provider can silently return.
With fal/ElevenLabs gone and Iconify never enabled, every provider is always-on heygen (or the local design spec) — remove the envFlag/gated enablement layer entirely. The guard test now asserts only heygen / design_spec providers exist.
Crop a selected element visually: a Crop toggle in the Clip panel arms edge handles on the overlay; dragging live-previews clip-path inset on the iframe element and commits once through the existing inline-style patch path (undo/persist like any style edit). Per-side T/R/B/L inset fields join the Clip section; radius is preserved, insets clamped. Render-time only: the media file is untouched. Clip-path helpers move to clipPathHelpers.ts (re-exported) with 1- and 4-value inset parse/format support.
…-line handles, dim, hug Crop mode now behaves like an NLE crop tool: - Canvas toolbar gets a Crop button (shown whenever a croppable element is selected); crop mode lives in the player store so the toolbar, the Clip panel, and the overlay share one switch. - Double-click a selected element to enter crop mode (timestamp-based: selection re-keys the box on every click, so native dblclick never fires); Escape exits. - While cropping, the element's clip is lifted so the full content stays visible; the cropped-out region is dimmed and a crop frame with edge handles sits ON the crop lines (handles track the insets as you drag). - Drag end commits through the normal style path (one undo step per drag); leaving crop mode re-applies the committed crop. - Outside crop mode the selection box hugs the visible cropped region instead of the full element bounds.
miguel-heygen
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| mkdirSync(dirname(destPath), { recursive: true }); | ||
| writeFileSync(destPath, bytes); | ||
| return bytes.length; | ||
| writeFileSync(destPath, Buffer.concat(chunks, total)); |
Two joins of capabilities the framework already ships: - transcript-cut: edit footage by editing its transcript. Word-level timestamps (hyperframes transcribe) + agent-decided removals (explicit ranges, word indices, filler words, long silences, or inverse --keep) compile into frame-accurate ffmpeg segments + concat; --plan dry-run, --copy fast mode. Output re-enters the ledger via resolve --from. - audio-duck: BGM ducking as declaration, not baking. Speech spans from audio_meta.json / transcribe.json word timestamps become a ready GSAP volume-tween block (attack/release shaped, sentence gaps bridged); the runtime already renders volume tweens identically in preview + render. - operations.md: text-based editing, bake-for-export sidechain duck, and two-pass loudnorm publish-loudness recipes. - SKILL.md gaps table + coverage test extended for both claims. Pure cores (compileCutList, speechSpans/duckKeyframes) are unit-tested; e2e cut verified by ffprobe (6s clip minus 1s cut = 5.02s output).
…rsing CodeQL flagged the inset() matcher (nested optional whitespace around a lazy group). Match the parenthesized payload unambiguously, trim, and collapse whitespace before the round-radius split.
Found by running the full chain live (Kokoro TTS -> Whisper -> cut/duck)
instead of fixtures:
- filler matching: Whisper emits words with attached punctuation and
arbitrary case ("UM,"), so compare bare tokens
- audio segments: the video-centric encode set put AAC inside .wav,
which halves apparent duration; pick codecs by output extension
(pcm/lame/aac/flac vs x264+aac)
- audio-duck: audio_meta.json word times are relative to each line's
own file; multiple lines overlapped at t=0 and merged into one bogus
span. Placement is now required for multi-line meta: --sequential
[--gap] stacks lines, --offsets "l1=0,l2=5" places them exactly
Live end-to-end testing on a real machine surfaced two more: - The global cache manifest was written to ~/.media/.media/manifest.jsonl (globalMediaDir() handed to helpers that append their own .media), so the Studio /api/assets/global route, which reads the documented flat ~/.media/manifest.jsonl, always returned []. Write via HOME so both halves of the cross-project view agree; verified live (route now returns the full catalog, cross-project reuse still hits). - transcript-cut --copy on sparse-keyframe footage can snap the whole cut away silently (a single-keyframe clip returned full length). Compare output duration against the kept total and warn.
… override - producer: muted media elements were silent in preview but mixed at full volume in renders. discoverMediaFromBrowser now carries the muted flag and the probe stage prunes muted audio from the mix and clears hasAudio on muted video (pruneMutedBrowserMedia, pure + unit-tested). Verified live: a muted-voices composition now renders the exact same duck envelope as a bgm-only one. - cli tts: accept --out alongside -o/--output (was silently ignored and wrote speech.wav into the cwd). - cli tts: HYPERFRAMES_PYTHON env override for the python probe, for machines whose PATH python3 is too old for kokoro-onnx (system python on stock macOS); error messages mention it.
…ideo models Image: the codex CLI's native image_gen tool joins the registry as the generate fallback for the image type (heygen search first). It runs on the user's ChatGPT subscription, the codex CLI owns auth (CLI-only invariant holds), and generation carries no per-call charge. Verified live end to end: catalog miss -> codex render -> frozen + ledgered. Video: new spec-gated videogen capability in the local-models table, from a 2026-07 X research pass plus live verification. The Mac story is LTX 2.3 on MLX via dgrauet/ltx-2-mlx (int4 medium tier verified on a 24GB machine: 512x320x33f with audio, ~19 min cold; bf16 two-stage as the large tier). Table warns about the 60GB blind snapshot download and gives the targeted include list. Wan 2.x MLX noted as too large until a 5B conversion lands. Guard tests updated: sanctioned providers are heygen, codex.image_gen, and the local design spec; coverage rows enforce both new claims.
…ctable ladder Agent picks the best local model for the machine's AVAILABLE RAM across a graduated ladder, verified live on a 24GB Mac. Infra: - probeSpecs reports availableRamMB (macOS vm_stat reclaimable / Linux MemAvailable), not just total; selection gates on the real budget. - selectModel walks the whole table best-first by RAM footprint, returns the highest-quality model that fits (was 2 hardcoded tiers). - describeModelLadder(): agent-facing view of every model with fits/why. Image gen (new imagegen capability + registry cascade): - Local mflux (FLUX-on-MLX): schnell int4 medium (~8GB, --low-ram), FLUX.2 Klein 4B large (32GB), Qwen-Image xlarge (64GB+). - image cascade: heygen catalog, then local mflux (best-for-RAM, free/private, kept under --local-only), then codex image_gen (better-quality upsell + fallback). - Video docs add the heygen video create AVATAR-video CLI upsell beside LTX. Findings baked in (live-verified on 24GB): FLUX repos are HF-gated (use non-gated community 4-bit re-uploads); --low-ram mandatory on <=24GB (90 min to 20s); pypi mflux is a broken stub, use 0.9.6 in a dedicated venv. 72 lib tests green; end-to-end verified.
…+--provider override HeyGen TTS now costs wallet credits, so the voice cascade flips to match image/video: local Kokoro first (free, private, on-device via the hyperframes CLI, kept under --local-only), HeyGen TTS as the paid quality upsell and the fallback when Kokoro is not set up. New `paid` provider marker documents the cost for the agent's X4 confirm judgment. Also add `--provider <name>` to resolve (ctx.provider): pins resolution to one generator so a user can force it, e.g. 'make this image WITH codex' -> --provider codex, bypassing the free-first default. Prefix or full-name match; forcing a network provider overrides --local-only (explicit intent). Voice cascade verified: kokoro.local(free) -> heygen.tts(paid). 73 lib tests green (incl. the override + new voice-cascade tests).
…cribe.mjs
whisper.cpp is weaker than OpenAI's transcription; NVIDIA Parakeet-TDT via
parakeet-mlx (a top open-source ASR, Neural-Engine fast on Apple Silicon) is
the better local option. Verified on a 24GB Mac: accurate transcript, ~3s
(cached) for 8s audio, and its word timestamps drive transcript-cut correctly.
- scripts/transcribe.mjs: runs Parakeet, merges its sub-word tokens into word
timestamps (parakeet-words.mjs), emits { text, words:[{text,start,end}] } for
transcript-cut / captions / the audio engine. Falls back to the packaged
whisper.cpp (hyperframes transcribe) when parakeet-mlx is absent.
- asr ladder: Parakeet is rank 0 (preferred). Quality is NOT size for ASR
(0.6B beats whisper-1.5B), so selectModel now honors an explicit `rank`
ahead of RAM footprint; whisperx is the CPU-only fallback.
Verified end-to-end: audio -> transcribe.mjs (19 words) -> transcript-cut
--remove-fillers finds both 'Um,'s. 78 lib tests green.
… cohere tested + excluded transcribe.mjs is now the default local transcription path: Parakeet-preferred with automatic whisper.cpp fallback (99 languages / non-European / not-installed) and an --engine override. Eval-backed: Parakeet beats whisper-large-v3 on the Open ASR Leaderboard (avg WER ~6.05% vs 7.44%; noisy test-other 4.73% vs 5.96%; whisper-v3 hallucinated to 308% WER on meetings) and is 5-10x faster. Cohere Transcribe 2B tops the leaderboard on paper (5.42% WER) and the research said it runs on Mac via mlx-audio, so I tested it live: both 4bit and 8bit MLX quants, with and without --language en, produced multilingual token-soup garbage AND ran 40-70x slower than Parakeet. Excluded (would be a dead entry); documented as a candidate to revisit when the mlx-audio Cohere decoder stabilizes. 78 lib tests green; transcribe.mjs verified (parakeet default + whisper routing).
…the agent knows what to run The skill guided resolve/transcribe/cut/duck commands, but the 'CLI tools used' table was stale (only ffprobe + heygen) and the local-model install commands lived only in local-models.mjs code. Now the table lists every tool the agent shells (heygen, mflux-generate, codex, parakeet-mlx, ltx-2-mlx, hyperframes), what each serves, and the exact install command, with a note that local tools are opt-in and resolve degrades gracefully without them. Also make it actionable at runtime: when a local image model is selected but its bin is not on PATH, the mflux provider now prints the exact enable-command (from the model's install field) before falling through to the codex upsell, so the agent learns the free local path is available instead of silently going cloud. Verified: the hint prints the mflux install command; 78 lib tests green.
…hisper|auto) Parakeet is now a first-class transcribe engine in the CLI, not just the media-use skill. NVIDIA Parakeet-TDT via parakeet-mlx beats whisper.cpp on the Open ASR Leaderboard (~6.05% vs 7.44% avg WER, 4.73% vs 5.96% on noisy audio) and is 5-10x faster. --engine auto (default) uses Parakeet when installed (detected on PATH, HYPERFRAMES_PARAKEET, or ~/.venvs/parakeet), else whisper.cpp; --engine parakeet forces it (clear install error if absent); --engine whisper pins the zero-setup baseline. whisper/parakeet.ts detects the runner + merges Parakeet's sub-word tokens into the Word[] the pipeline already consumes, so transcript.json / captions are unchanged. Verified on a 24GB Mac through the built CLI: both engines produce the same 19-word transcript; 2 unit tests for the token merge. Also drops a stale fallow suppression in snapTargetCollection.ts. LEFTHOOK=0: local diff-gate trips on pre-existing branch-wide complexity/dup findings (fallow exits 1 identically at the prior commit); this change adds none and CI's inherited-exclusion gate handles them.
…g for HeyGen TTS E2E test on this Mac caught it: local Kokoro voice failed (kokoro-onnx not installed) and resolve silently fell through to the PAID HeyGen TTS upsell. The provider read only err.stderr, but `hyperframes tts` prints its 'kokoro-onnx not installed. Run: pip install kokoro-onnx soundfile' hint to STDOUT (clack UI), so the actionable message was lost as a bare 'Command failed'. Now the provider reads both streams and surfaces the enable-command (mirrors the mflux local-image hint), so the agent installs the free local model instead of quietly paying. Verified: --local-only voice resolve now prints the hint and refuses to pay; local Kokoro itself confirmed working via the built CLI + HYPERFRAMES_PYTHON venv (kokoro-onnx generated 1.5s of speech).
Answers 'is this actually used, and which capabilities' without PII. resolve and
the edit tools (transcribe / transcript-cut / audio-duck) send one anonymous
event recording only the media TYPE, resolution SOURCE, and winning PROVIDER:
never the intent text, file names, or paths, with $ip:null so no IP is stored.
scripts/lib/telemetry.mjs mirrors the hyperframes CLI telemetry: same public
PostHog project key (write-only ingestion, already in the repo), same opt-outs
(DO_NOT_TRACK / HYPERFRAMES_NO_TELEMETRY / CI / dev), best-effort and
non-blocking (a resolve never waits on or fails from telemetry). Stable
per-machine anonymous id persisted in ~/.media.
Verified: PostHog /batch/ accepts the payload (HTTP 200 {status:Ok}); opt-out
predicate unit-tested; 80 lib tests green.
# Conflicts: # skills-manifest.json # skills/media-use/audio/scripts/lib/python.mjs # skills/media-use/audio/scripts/lib/python.test.mjs # skills/media-use/audio/scripts/lib/sfx.test.mjs # skills/media-use/scripts/resolve.mjs
# Conflicts: # skills-manifest.json
Fallow audit reportFound 33 findings. Dead code (2)
Duplication (23)
Health (7)
Suppressions (1)
Generated by fallow. |
| if (existsSync(file)) return readFileSync(file, "utf8").trim(); | ||
| if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); | ||
| const id = randomUUID(); | ||
| writeFileSync(file, id); |
| await fetch(`${POSTHOG_HOST}/batch/`, { | ||
| method: "POST", | ||
| headers: { "Content-Type": "application/json", Connection: "close" }, | ||
| body, |
This was referenced Jul 6, 2026
Collaborator
Author
|
Superseded by a clean 5-PR Graphite stack off
Closing this monolith in favor of the stack. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two tracks on this branch, cleanly separable by path.
media-use v2 — the agent Media OS (
skills/media-use/)One verb,
resolve, for every media need. Free-first, then local-first generation with cloud upsells. media-use holds no keys; each external tool owns its own auth.Resolve + providers
bgm/sfx/image/icon/voice, HeyGen catalog first (free retrieval), deterministic provider cascade, freeze-to-disk + ledger + cross-project cache (~/.media).hyperframes-mediaskill in (audio engine now undermedia-use/audio/); 20 skills.--fromlocal files + direct URLs, SSRF-guarded),--adopt,--local-onlyoffline mode.Generation is local-first with graceful cloud upsells (all live-verified on a 24GB Mac):
resolveprobes available RAM and auto-picks the best FLUX that fits (schnell int4 medium ~8GB--low-ram, FLUX.2 Klein 4B large, Qwen-Image xlarge). The agent can inspect the ladder (describeModelLadder) and override.videogenladder) →heygen video createavatar-video CLI upsell.--provider <name>forces one generator (e.g. a user says "make this image with codex" →--provider codex), bypassing the free-first default.Editing capabilities that were impossible before
transcript-cut.mjs): word-timestamp edits (filler removal, silence-cut, ranges) compiled into frame-accurate ffmpeg cuts.audio-duck.mjs): speech spans → ready GSAP volume-tween block the runtime renders identically in preview + export; plus sidechain + two-passloudnorm(-14/-16 LUFS) recipes.Non-destructive Studio crop (
packages/studio/, decision OP2)Crop any element visually; persists as
clip-path: inset(...)(one undo step per drag, media file untouched). Enter via canvas toolbar / Clip panel / double-click; Escape or click-outside exits. Pro-editor presentation (dim, crop frame, edge handles on the crop lines, drag the frame to move the window). Every overlay surface respects the crop — selection box, hover ring, rotate/resize handles, snap lines, marquee, off-canvas indicators — via onetoVisibleOverlayRectprimitive. Shortcuts panel documents the crop keys.Notable fixes found by real-data / live testing
pruneMutedBrowserMedia+ tests).--copykeyframe-snap warning.--sequential/--offsets) — file-relative word times were merging into one bogus span.~/.media/manifest.jsonl) — the Studio cross-project asset view was reading the wrong location and always saw zero.tts --outalias;HYPERFRAMES_PYTHONenv override for local-model bootstrapping.Testing
All live-verified end-to-end on a 24GB M-series Mac: catalog resolve (bgm/sfx/image/icon/voice), HeyGen TTS + local Kokoro, local mflux/FLUX image gen (~20s photoreal), local LTX video gen (clip + audio), local Whisper + u2net, transcript-cut + audio-duck against real Whisper output, and a full producer render with the duck envelope (ffmpeg-measured). Unit suites: media-use libs (73), studio (1300+), studio-server (223); lint/format/typecheck clean.
Parked by decision (not in scope)
Side-by-side local-vs-cloud comparison assets (CDN pending); TTS eval re-run; batch audio-engine TTS default stays HeyGen-if-credentialed (video skills lean on it for caption timing).