feat(encoder): Add VideoToolbox hardware H.264/H.265 encoder for Apple platforms#5
Conversation
…e platforms On Apple Silicon the only working encoder was software VP9; NVENC is NVIDIA-only. Add a VTKStreaming::VTEncode module whose vtkVideoToolboxEncoder subclasses vtkVideoEncoder and drives a VTCompressionSession on the hardware media engine, encoding H.264 and HEVC. Details: - CPU-uploads the captured NV12/IYUV/RGBA32 frame into an IOSurface-backed CVPixelBuffer from the session pool, then hardware-encodes it. - Emits an Annex B elementary stream (start codes, parameter sets inlined on key frames), byte-compatible with the NVENC backend and directly playable by ffplay. - Reports a full RFC 6381 / WebCodecs codec string (avc1.*/hvc1.*) parsed from the SPS, mirroring the NVENC backend's BuildCodecString. - Honors BitRateControlMode: CBR (ConstantBitRate on macOS 13+, else average bitrate + data-rate cap), VBR (average + ceiling). VideoToolbox has no constant-QP mode, so CQP/QP map QuantizationParameter to a quality hint plus allowed-QP bounds. - Module is only added to the build under if(APPLE); non-Apple builds are unaffected. Adds a Python test (requires_videotoolbox helper), an example, and updates the README.
5daa266 to
aae2138
Compare
|
newly added videotoolbox tests pass https://github.com/Kitware/VTKStreaming/actions/runs/29048565697/job/86223212457?pr=5#step:5:1105 ../../../../../../../../../Users/runner/work/VTKStreaming/VTKStreaming/tests/test_videotoolbox_encode_simple.py::test_videotoolbox_encode_simple[iyuv] PASSED [ 60%] |
- it was needed in example that have dual window (encoder + decoder). These two examples do not have two windows, so MakeCurrent is not needed.
|
Next steps:
|
- we can get rid of this after upgrading to vtk 9.7
On item 1: I ended up not needing this. The On step 2 (9.6 path): this lives in vtkEncoderFactory. One tweak from what we sketched, instead of ignoring the preference request on 9.6, the interim factory actually honors it: it takes the same VTK 9.7 preference string ("Codec=H265,H264,VP9;Hardware=true", |
|
LGTM |
On Apple Silicon the only working encoder was software VP9; NVENC is NVIDIA-only. Add a VTKStreaming::VTEncode module whose vtkVideoToolboxEncoder subclasses vtkVideoEncoder and drives a VTCompressionSession on the hardware media engine with https://developer.apple.com/documentation/videotoolbox?language=objc, encoding H.264 and HEVC.
Details:
Adds a Python test (requires_videotoolbox helper), an example, and updates the README.