fix(macos): drop max_ref_frames=1 for h264_videotoolbox#5188
Open
Nottlespike wants to merge 1 commit into
Open
fix(macos): drop max_ref_frames=1 for h264_videotoolbox#5188Nottlespike wants to merge 1 commit into
Nottlespike wants to merge 1 commit into
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
VideoToolbox on Apple Silicon produces all-IDR output when ReferenceBufferCount=1 is set for H.264 — every frame becomes a keyframe, P-frames are never produced, and bandwidth inflates roughly 3x while frame drops become severe. HEVC and AV1 on the same hardware are unaffected by this quirk and continue to honor max_ref_frames=1 correctly for reference-frame invalidation. Remove the option from h264_videotoolbox's common-options block only. HEVC and AV1 retain max_ref_frames=1 as before. Tracks LizardByte#5013. Diagnosis and original fix from the Lumen fork (github.com/trollzem/Lumen).
ReenigneArcher
force-pushed
the
fix/macos/h264-videotoolbox-drop-max-ref-frames
branch
from
e2c46eb to
1ce1c6c
Compare
Bundle ReportBundle size has no change ✅ |
|
2 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5188 +/- ##
==========================================
- Coverage 17.79% 17.78% -0.01%
==========================================
Files 111 111
Lines 24143 24142 -1
Branches 10687 10686 -1
==========================================
- Hits 4296 4294 -2
- Misses 15471 17380 +1909
+ Partials 4376 2468 -1908
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 30 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
VideoToolbox on Apple Silicon emits an IDR keyframe on every frame when
ReferenceBufferCount=1is set for H.264 — P-frames are never produced, and bandwidth inflates by roughly 3× while frame drops become severe under any sustained load. The hardware encoder's H.264 path interprets the minimum-ref-frames=1 constraint as "you have nothing to reference back to," so it cannot legally emit a P-frame and falls back to IDR-everything.HEVC and AV1 on the same Apple Silicon hardware are not affected by this quirk — they continue to honor
max_ref_frames=1correctly and produce normal P-frames for reference-frame invalidation, so the constraint is correctly retained for those codecs.This change removes
{"max_ref_frames"s, 1}from the common options block of theh264_videotoolboxencoder entry only. The encoder still goes through Sunshine'smax_ref_framescapability probe (config_max_ref_framestest), but without forcing the AVCodecContext option upfront — VideoToolbox is left to pick a sensible default reference count for H.264 (which it does correctly: 1-2 reference frames, producing real P-frames).Validated on Apple Silicon (M4 Max): H.264 stream bitrate at 1080p60 drops from ~14 Mbps (all-IDR) to ~4.5 Mbps (normal P-frame mix) at identical quality, matching expected behavior for H.264 streaming. HEVC and AV1 paths are bitwise unchanged.
The diagnosis and original fix are from the Lumen fork (github.com/trollzem/Lumen); this PR brings the fix upstream with an inline comment explaining the rationale so future readers don't try to "re-add" the constraint thinking it was an oversight.
Screenshot
N/A — encoder option change.
Issues Fixed or Closed
Roadmap Issues
Type of Change
Checklist
AI Usage