Skip to content

[GStreamer][MediaCapabilities] missing validation of HdrMetadataType and TransferFunction during decodingInfo method #1453

Open
Open
[GStreamer][MediaCapabilities] missing validation of HdrMetadataType and TransferFunction during decodingInfo method#1453
Assignees
emutavchipgorszkowski-igalia
Labels
questionwpe-2.38wpe-2.46

Description

@pgorszkowski-igalia
pgorszkowski-igalia
opened on Jan 30, 2025

decodingInfo method: https://www.w3.org/TR/media-capabilities/#decodinginfo-method, in case of VideoConfiguration (https://www.w3.org/TR/media-capabilities/#videoconfiguration), should also validate HDRMetadataType (https://www.w3.org/TR/media-capabilities/#enumdef-hdrmetadatatype) and TransferFunction (https://www.w3.org/TR/media-capabilities/#transferfunction).

Available video decoders can provide HDRMetadataType and TransferFunction which they support.

Activity

  1. pgorszkowski-igalia
    added
    wpe-2.38
    on Jan 30, 2025
  2. pgorszkowski-igalia
    self-assigned this
    on Jan 30, 2025

  3. pgorszkowski-igalia commented on Jan 31, 2025

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Jan 31, 2025
    Author

    @emutavchi : is it, what for you need additional API to set HDR types and EOTF?

  4. pgorszkowski-igalia
    assigned
    emutavchi
    on Jan 31, 2025
  5. modeveci
    added
    question
    wpe-2.46
    on Jan 31, 2025

  6. pgorszkowski-igalia commented on Jan 31, 2025

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Jan 31, 2025
    Author

    Based on the release notes from the gstreamer: https://gstreamer.freedesktop.org/releases/1.18/

    Video library

    High Dynamic Range (HDR) video information representation and signalling enhancements:

        New APIs for HDR video information representation and signalling:
            GstVideoMasteringDisplayInfo: display color volume info as per SMPTE ST 2086
            GstVideoContentLightLevel: content light level specified in CEA-861.3, Appendix A.
            plus functions to serialise/deserialise and add them to or parse them from caps
            gst_video_color_{matrix,primaries,transfer}_{to,from}_iso(): new utilility functions for conversion from/to ISO/IEC 23001-8
            add ARIB STD-B67 transfer chracteristic function
            add SMPTE ST 2084 support and BT 2100 colorimetry
            define bt2020-10 transfer characteristics for clarity: bt707, bt2020-10, and bt2020-12 transfer characteristics are functionally identical but have their own unique values in the specification.

        h264parse, h265parse: Parse mastering display info and content light level from SEIs.
        matroskademux: parse HDR metadata
        matroskamux: Write MasteringMetadata and Max{CLL,FALL}. Enable muxing with HDR meta data if upstream provided it
        avviddec: Extract HDR information if any and map bt2020-10, PQ and HLG transfer functions

    we can assume that HDR types and EOTF things are supported by gstreamer >= 1.18.

  7. emutavchi commented on Feb 11, 2025

    @emutavchi
    emutavchi
    on Feb 11, 2025
    Collaborator

    @pgorszkowski-igalia we need a way to configure browser to accept HDR mime types MediaSource.isTypeSupported, for example to pass tests like:

    https://ytlr-cert.appspot.com/latest/main.html?&test_type=formatsupport-test&tests=10,11,12#1739287094977

  8. pgorszkowski-igalia commented on Feb 11, 2025

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Feb 11, 2025
    Author

    @pgorszkowski-igalia we need a way to configure browser to accept HDR mime types MediaSource.isTypeSupported, for example to pass tests like:

    https://ytlr-cert.appspot.com/latest/main.html?&test_type=formatsupport-test&tests=10,11,12#1739287094977

    Thanks @emutavchi . The failing test: "10. isTypeSupported EOTF Support" validates MIME types which also contain information about EOTF:
    "video/webm; codecs="vp09.02.51.10.01.09.16.09.00"; width=1280; height=720; framerate=30; eotf=smpte2084"
    "video/webm; codecs="vp09.02.51.10.01.09.18.09.00"; width=1280; height=720; framerate=30; eotf=arib-std-b67"
    "video/webm; codecs="vp09.02.51.10.01.09.18.09.00"; width=1280; height=720; framerate=30; eotf=strobevision"

    "eotf=smpte2084" and "eotf=arib-std-b67" are expected to be valid, while "eotf=strobevision" is expected to be invalid and the test expects that for invalid EOTF in MIME type, "isTypeSupported" should fail.

    MIME type specification does not specify what EOTF values are valid and also in https://w3c.github.io/media-source/#istypesupported-method there is nothing about "optional parameters" and EOTF valid values.

    I will try to find some more information about that but it seems that these tests are specific for some custom implementation (at least it is my assumption).
    NOTE: Chrome and FF also fail with that.

  9. pgorszkowski-igalia commented on Feb 12, 2025

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Feb 12, 2025
    Author

    EOTF parameter parser is implemented in Cobalt:
    https://github.com/youtube/cobalt/blob/main/starboard/shared/starboard/media/parsed_mime_info.cc#L37

    It bases on "YouTube TV HTML5 Technical Requirements" defined in https://developers.google.com/youtube/devices/living-room/files/pdf-guides/YouTube_TV_HTML5_Technical_Requirements_2018.pdf

  10. pgorszkowski-igalia commented on Feb 13, 2025

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Feb 13, 2025
    Author

    I have added in upstream (WebKit/WebKit#40539) the similar implementation as is in Cobalt. With this implementation the tests from https://ytlr-cert.appspot.com/latest/main.html?&test_type=formatsupport-test&tests=10,11,12#1739287094977 pass.

  11. pgorszkowski-igalia commented on Feb 14, 2025

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Feb 14, 2025
    Author

    And downstream PR: #1463

  12. pgorszkowski-igalia commented on Feb 17, 2025

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Feb 17, 2025
    Author

    @emutavchi : #1463 is merged so now the tests from https://ytlr-cert.appspot.com/latest/main.html?&test_type=formatsupport-test&tests=10,11,12#1739287094977 pass.

  13. pgorszkowski-igalia commented on Mar 3, 2025

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Mar 3, 2025
    Author

    @emutavchi : can we close this ticket?

  14. emutavchi commented on Mar 25, 2025

    @emutavchi
    emutavchi
    on Mar 25, 2025
    Collaborator

    Hi @pgorszkowski-igalia, sorry for the delayed reply. pr 1463 seems to always return that required eotf are supported. Which is not always true. We'd like to have an option to control that. Return not supported if connected display doesn't support HDR at all (i.e. ae22d69) or a particular transfer function is not supported.

  15. pgorszkowski-igalia commented on Apr 2, 2025

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Apr 2, 2025
    Author

    @emutavchi: Yes, currently, we check only if the HDR is supported by the decoder (GStreamer), which is always true. I will investigate how information about connected display capabilities can be used to verify HDR or transfer function support.

  16. philn commented on Sep 2, 2025

    @philn
    philn
    on Sep 2, 2025

    I don't think a setting (introduced in ae22d69) is the right approach for this. We should somehow probe the capabilities of the connected monitor.

  17. pgorszkowski-igalia commented on Mar 23, 2026

    @pgorszkowski-igalia
    pgorszkowski-igalia
    on Mar 23, 2026
    Author

    @emutavchi : sorry for long silence here, but do you need something on our side in this issue (I lost what is the final conclusion here)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

questionwpe-2.38wpe-2.46

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions