HEVC Roadmap?

[bchah]

Hello friends, can we please get an update on your roadmap for HEVC support? For me it is most needed in the WebRTC Publisher. But other requests like HEVC in RTSP & RTMP Enhanced have been pending for some time so it would be great to see some updates around HEVC. Thank you!

[getroot]

Enhanced RTMP is planned on our roadmap, and unfortunately, as far as I know, no browser supports HEVC in WebRTC yet.

[bchah]

Thank you for the response. To my delight I've seen that HEVC in WebRTC (WHIP/WHEP) is fully supported in iOS and macOS Safari, or in Chrome on Windows with a GPU and these flags:

chrome.exe --enable-features=PlatformHEVCEncoderSupport,WebRtcAllowH265Receive,WebRtcAllowH265Send --force-fieldtrials=WebRTC-Video-H26xPacketBuffer/Enabled

If others are doing it, that means OME can do it even better 😀 OvenLiveKit could also support HEVC transmission.

[getroot]

Oh... actually I didn't know this was built into chrome. Now it's our turn.

[bchah]

I see that HEVC in WebRTC has been added with commit b4b72a7! This is an excellent addition to OME and of course you know I have begun testing it a lot.

There is a new issue with the SRT provider since this commit - since SRT does not support the <BlockDuplicateStreamName> flag, OME will ingest two streams at the same time. This is OK but when you stop Stream 1 it will start rendering every frame of Stream 2 from the beginning, as fast as possible (up to 1000 fps!) until it reaches real-time. I think I will submit this as a real issue since this is not a problem in 0.18.0.

For H265 itself, the playback is working great but I notice it is only supporting SDR playback so far. When I provide a PQ/HLG stream in H265 Main10 or Main10 4:2:2 profiles, it is still required to use HLS to view these in the proper HDR color space. Do you know if it will be possible to play back the proper color space in WebRTC?

[getroot]

I don't know much about HDR. None of us have stepped foot on it yet.

But I know that metadata for HDR is provided at the VPS/SPS/PPS level. And OvenMediaEngine doesn't modify these NAL Units.

Did you do your WebRTC HEVC streaming on OBS? If so, most likely OBS doesn't support HDR?

If you find out about this, please let us know for everyone.

[getroot]

After looking into it a bit more, it seems that in order to properly decode HDR10, the decoder and renderer must also utilize the meta passed to SEI. OME doesn't even touch SEI in bypass encoding. So does this mean that libWebRTC can't properly decode HDR10? Or is it not passing the HDR10 meta information to the renderer? It seems worth investigating.

[bchah]

I do use OBS for this - there are also some hardware encoders that do it too. In OBS if you have a compatible encoder you can set the color format to P010 and the color space to PQ or HLG. I do this with the Apple HEVC Hardware Encoder which allows me to transmit either Main10 or Main10 4:2:2 profiles, and these play back perfectly over HLS. In WebRTC they are being tone-mapped to SDR (which is actually a good sign - if it was not interpreting the colour space at all it would just show wrong colours entirely). It's really amazing watching an HDR live stream over OME, so I hope we can find a way to bring this to WebRTC (or "real time"). I wish I knew more about how the HDR metadata is transmitted so I could be of more help.

[getroot]

Looking at the HLS spec, it says that in order to express HDR, you need to add meta to the hvcC box.

1.8. For HDR10 video, the SEI NAL units (that is, static metadata) SHOULD be in the HEVC Configuration Box ('hvcC') and not in the individual sample data.

https://developer.apple.com/documentation/http-live-streaming/hls-authoring-specification-for-apple-devices

According to this, HDR is not supported yet in OME HLS.

And how did you confirm that it is mapped to SDR in WebRTC? Did you distinguish it by sight?

[bchah]

Weird, I thought I replied to this but it appears I have not yet. HDR is definitely working in OME HLS on both Safari and Chrome, and it is beautiful. If HDR>SDR is not being properly tone-mapped it is very obvious by sight. So at least for now WebRTC is still compatible with playback of HDR sources, it just plays them in SDR (In Safari at least - I hope that v136 of Chrome will behave the same since it does not support the GPU flags on macOS).