Skip to content

feat(encoder): Add VideoToolbox hardware H.264/H.265 encoder for Apple platforms#5

Merged
jspanchu merged 5 commits into
wheelfrom
videotoolbox-encoder
Jul 11, 2026
Merged

feat(encoder): Add VideoToolbox hardware H.264/H.265 encoder for Apple platforms#5
jspanchu merged 5 commits into
wheelfrom
videotoolbox-encoder

Conversation

@jspanchu

@jspanchu jspanchu commented Jul 9, 2026

Copy link
Copy Markdown
Member

On Apple Silicon the only working encoder was software VP9; NVENC is NVIDIA-only. Add a VTKStreaming::VTEncode module whose vtkVideoToolboxEncoder subclasses vtkVideoEncoder and drives a VTCompressionSession on the hardware media engine with https://developer.apple.com/documentation/videotoolbox?language=objc, encoding H.264 and HEVC.

Details:

  • CPU-uploads the captured NV12/IYUV/RGBA32 frame into an IOSurface-backed CVPixelBuffer from the session pool, then hardware-encodes it.
  • Emits an Annex B elementary stream (start codes, parameter sets inlined on key frames), byte-compatible with the NVENC backend and directly playable by ffplay.
  • Reports a full RFC 6381 / WebCodecs codec string (avc1./hvc1.) parsed from the SPS, mirroring the NVENC backend's BuildCodecString.
  • Honors BitRateControlMode: CBR (ConstantBitRate on macOS 13+, else average bitrate + data-rate cap), VBR (average + ceiling). VideoToolbox has no constant-QP mode, so CQP/QP map QuantizationParameter to a quality hint plus allowed-QP bounds.
  • Module is only added to the build under if(APPLE); non-Apple builds are unaffected.

Adds a Python test (requires_videotoolbox helper), an example, and updates the README.

…e platforms

On Apple Silicon the only working encoder was software VP9; NVENC is
NVIDIA-only. Add a VTKStreaming::VTEncode module whose
vtkVideoToolboxEncoder subclasses vtkVideoEncoder and drives a
VTCompressionSession on the hardware media engine, encoding H.264 and
HEVC.

Details:
- CPU-uploads the captured NV12/IYUV/RGBA32 frame into an IOSurface-backed
  CVPixelBuffer from the session pool, then hardware-encodes it.
- Emits an Annex B elementary stream (start codes, parameter sets inlined
  on key frames), byte-compatible with the NVENC backend and directly
  playable by ffplay.
- Reports a full RFC 6381 / WebCodecs codec string (avc1.*/hvc1.*) parsed
  from the SPS, mirroring the NVENC backend's BuildCodecString.
- Honors BitRateControlMode: CBR (ConstantBitRate on macOS 13+, else
  average bitrate + data-rate cap), VBR (average + ceiling). VideoToolbox
  has no constant-QP mode, so CQP/QP map QuantizationParameter to a
  quality hint plus allowed-QP bounds.
- Module is only added to the build under if(APPLE); non-Apple builds are
  unaffected.

Adds a Python test (requires_videotoolbox helper), an example, and updates
the README.
@jspanchu jspanchu force-pushed the videotoolbox-encoder branch from 5daa266 to aae2138 Compare July 9, 2026 20:36
@jspanchu

jspanchu commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

newly added videotoolbox tests pass https://github.com/Kitware/VTKStreaming/actions/runs/29048565697/job/86223212457?pr=5#step:5:1105

../../../../../../../../../Users/runner/work/VTKStreaming/VTKStreaming/tests/test_videotoolbox_encode_simple.py::test_videotoolbox_encode_simple[iyuv] PASSED [ 60%]
../../../../../../../../../Users/runner/work/VTKStreaming/VTKStreaming/tests/test_videotoolbox_encode_simple.py::test_videotoolbox_encode_simple[nv12] PASSED [ 66%]
../../../../../../../../../Users/runner/work/VTKStreaming/VTKStreaming/tests/test_videotoolbox_encode_simple.py::test_videotoolbox_encode_simple[rgba32] PASSED [ 73%]

Comment thread examples/simple_videotoolbox_record.py Outdated
- it was needed in example that have dual window (encoder + decoder). These two examples do not have two windows, so MakeCurrent is not needed.
Comment thread tests/test_videotoolbox_encode_simple.py
Comment thread examples/simple_hardware_encoder_record.py
@jourdain

jourdain commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Next steps:

  • Make sure all classes exist regardless of platform. The CheckAvailability() can always return false when not supported on that platform.
  • Create factory implementation relying on vtk 9.7 factory preference mechanism with a VTK 9.6 path that just ignore the preference request.

@jspanchu

Copy link
Copy Markdown
Member Author

Next steps:

  • Make sure all classes exist regardless of platform. The CheckAvailability() can always return false when not supported on that platform.
  • Create factory implementation relying on vtk 9.7 factory preference mechanism with a VTK 9.6 path that just ignore the preference request.

On item 1: I ended up not needing this. The vtkEncoderFactory makes it moot. Consumers never import a concrete encoder class, they only go through the factory. Each backend self-registers into the factory's registry at load time, so on a given platform the registry simply contains whatever was built (VTEncode stays Apple-only, NvEncode/Vpx build everywhere). CheckAvailability(codec) and CreateEncoder() then degrade gracefully in two ways: a backend that isn't built is absent from the registry, and one that's built-but-unusable returns false from its own CheckAvailability() probe. So there's no missing-symbol risk and no need to force every class onto every platform.

On step 2 (9.6 path): this lives in vtkEncoderFactory. One tweak from what we sketched, instead of ignoring the preference request on 9.6, the interim factory actually honors it: it takes the same VTK 9.7 preference string ("Codec=H265,H264,VP9;Hardware=true", ; between keys and , between ranked values) and computes a lexicographic rank over the registered backends. It reuses the 9.7 attribute vocabulary I added throughout, so the migration will be simple: swap vtkEncoderFactory::SetPreferences/CreateEncoder for vtkObjectFactory::SetPreferences/vtkVideoEncoder::New() and promote each backend's attribute map into CreateOverrideAttributes(), then delete the helper.

@jourdain

Copy link
Copy Markdown
Collaborator

LGTM

@jspanchu jspanchu merged commit 6c71b60 into wheel Jul 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants