build: enable ProRes VideoToolbox encoder#693
Closed
Nottlespike wants to merge 1 commit into
Closed
Conversation
Nottlespike
added a commit
to RESMP-DEV/Sunshine
that referenced
this pull request
May 25, 2026
ProRes profiles are intrinsically 10-bit (proxy / lt / standard / hq)
or 12-bit (4444 / 4444 XQ); FFmpeg's prores_videotoolbox supported
input pix_fmt list does not include the 4:2:0 BiPlanar formats (NV12 /
P010) Sunshine has historically delivered for H.264 / HEVC / AV1. The
422 family wants 4:2:2 or higher chroma and the 4444 family wants
4:4:4 natively. Until now, the macOS capture path could only produce
4:2:0 buffers, so the ProRes encoder probe failed at avcodec_open2
with "Couldn't open" regardless of profile selection or colorspace
configuration.
Wire the 4:4:4 BiPlanar capture path end-to-end:
* Add nv24 / p410 to platf::pix_fmt_e and the from_pix_fmt switch.
* Map AV_PIX_FMT_NV24 / AV_PIX_FMT_P410 in video::map_pix_fmt.
* Populate the previously-NONE 4:4:4 pix_fmt slots on the
videotoolbox encoder declaration and add YUV444_SUPPORT to its
flags. H.264 and HEVC VideoToolbox don't gain anything new
functionally — they remain 4:2:0 only on Apple Silicon hardware —
but the 4:4:4 probe runs against them harmlessly and falls
through with their YUV444 capability bit set false, which is
correct.
* Route nv24 / p410 through nv12_zero_device in
display.mm::make_avcodec_encode_device, alongside the existing
nv12 / p010 paths.
* Set the matching CVPixelBufferType
(kCVPixelFormatType_444YpCbCr*BiPlanarVideoRange) per pix_fmt in
nv12_zero_device::init.
Update the ProRes encoder probe in test_prores to use a ProRes-shaped
config:
* dynamicRange = 1 (10-bit pix_fmt slot instead of 8-bit).
* encoderCscMode = 3 (full range, BT.709 instead of BT.601).
* chromaSamplingType = 1 (4:4:4 P410 instead of 4:2:0 P010).
Any ProRes probe that succeeds inherently uses 10-bit input, so
DYNAMIC_RANGE is promoted eagerly here rather than relying on
test_hdr_and_yuv444 below (which gates on PASSED and only sets
DYNAMIC_RANGE itself).
Verified end-to-end on M4 Max: the startup probe now produces
"Found ProRes encoder: prores_videotoolbox [videotoolbox]", where
previously the encoder failed to open at every config permutation
attempted.
This is the runtime-side companion to the build-deps change that
adds prores_videotoolbox to the FFmpeg configure's --enable-encoder
flag (LizardByte/build-deps#693). Without that, the encoder symbol
is missing from libavcodec.a's static codec list and
avcodec_find_encoder_by_name returns null regardless of the wiring
in this commit.
ReenigneArcher
force-pushed
the
feat/ffmpeg/enable-prores-videotoolbox-encoder
branch
from
c40fba6 to
543117b
Compare
|
Author
|
Closing — maintaining this work in the RESMP-DEV fork going forward rather than upstreaming. Branch will continue to live in Nottlespike/build-deps and RESMP-DEV/build-deps for anyone who wants to pull from it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
prores_videotoolboxto the comma-separated FFmpeg--enable-encoderconfigure flag in the macOS-specific branch ofcmake/ffmpeg/ffmpeg.cmake. The encoder is a hardware ProRes encode path that exists in Apple Silicon's media engine but isn't currently being built into the FFmpeg binary shipped to Sunshine. This one-line change enables it.One line of impact:
elseif(APPLE) list(APPEND FFMPEG_EXTRA_CONFIGURE - --enable-encoder=h264_videotoolbox,hevc_videotoolbox + --enable-encoder=h264_videotoolbox,hevc_videotoolbox,prores_videotoolbox --enable-videotoolbox )The flag is macOS-only (inside the
elseif(APPLE)branch). Linux, Windows, FreeBSD, and Flatpak build paths are entirely unaffected.Why this matters
This is the upstream-side dependency of LizardByte/Sunshine#5192 (experimental ProRes VideoToolbox encoder in Sunshine). The Sunshine PR's encoder probe path enumerates
prores_videotoolboxvia FFmpeg's avcodec; the encoder is silently absent from the FFmpeg binary unless this configure flag is set, which would make the Sunshine probe fail at runtime even though all the plumbing is correct.The encoder is part of FFmpeg's standard VideoToolbox encoder family alongside the already-enabled
h264_videotoolboxandhevc_videotoolbox. It is conditionally compiled in the FFmpeg source tree iff its name appears in--enable-encoder, so this change is opt-in by configure flag, not by code addition. The resulting libavcodec.a grows by a handful of KB; no new third-party dependency is introduced.Screenshot
N/A — build configuration change.
Issues Fixed or Closed
Roadmap Issues
Type of Change
Checklist
AI Usage