Commit f618603
committed
[MSE][GStreamer] Initial support for multiple video tracks on MSE
https://bugs.webkit.org/show_bug.cgi?id=298245
Reviewed by Philippe Normand.
Multiple video tracks in MSE are (optionally) supported by the spec[1],
but have never been used in practice in WebKitGTK/WPE. Creating more
than one video SourceBuffer causes a not-linked pipeline error in
WebKitMediaSrc. It would be nice to have a basic support of that use
case, since it's apparently required for the YouTube EME tests.
See: #1549
This patch sanitizes the video track selection state on creation (only
one video track can be active at once, all of them appeared as active on
creation before), mirrors the selection state from the video tracks
known to the HTMLMediaElement to the ones knows by the player private
(they're a different set in the case of MSE), and adds support for
unregistered streams in WebKitMediaSrc (the ones causing the not-linked
error in the first place), just avoiding pushing buffers to pads with
unlinked steams.
This patch only provides an initial support. It doesn't address complex
scenarios such as the init segment and data reenqueues that should be
needed if the switch happens after the init segment and data have been
appended (and enqueued) and the track selection change happens after
that. In such cases, the pipeline is refactored but, since the needed
data hasn't reached the video decoder on time (because the decoder just
didn't exist at enqueue time), only future samples will be enqueued from
that time on and decoding artifacts will happen.
A layout test has been added to test multiple video track basic support,
ensuring that the track ids are properly generated and that only
the first track is selected.
[1] https://w3c.github.io/media-source/#dfn-sourcebuffer-configuration
Layout test based on code provided by: Filipe Norte <[email protected]>
* LayoutTests/media/media-source/content/video320x180-101-1.m4s: Added.
* LayoutTests/media/media-source/content/video320x180-101-2.m4s: Added.
* LayoutTests/media/media-source/content/video320x180-101-3.m4s: Added.
* LayoutTests/media/media-source/content/video320x180-101-init.mp4: Added.
* LayoutTests/media/media-source/content/video320x180-102-1.m4s: Added.
* LayoutTests/media/media-source/content/video320x180-102-2.m4s: Added.
* LayoutTests/media/media-source/content/video320x180-102-3.m4s: Added.
* LayoutTests/media/media-source/content/video320x180-102-init.mp4: Added.
* LayoutTests/media/media-source/content/video320x180-103-1.m4s: Added.
* LayoutTests/media/media-source/content/video320x180-103-2.m4s: Added.
* LayoutTests/media/media-source/content/video320x180-103-3.m4s: Added.
* LayoutTests/media/media-source/content/video320x180-103-init.mp4: Added.
* LayoutTests/media/media-source/media-source-multiple-video-tracks-expected.txt: Added.
* LayoutTests/media/media-source/media-source-multiple-video-tracks.html: Test that created 3 different SourceBuffers, loads videos with 3 different track ids, and checks that the proper ids are generated and that only the first video trac>
* Source/WebCore/Modules/mediasource/MediaSource.cpp:
(WebCore::MediaSource::addVideoTrackToElement): Select the first video track added and unselect the next ones to ensure that only one video track is active at a given time after adding several ones.
* Source/WebCore/platform/graphics/gstreamer/VideoTrackPrivateGStreamer.cpp:
(WebCore::VideoTrackPrivateGStreamer::setSelected): Propagate the "selected" state to the mirror track that the player private may have.
* Source/WebCore/platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.cpp:
(WebCore::MediaPlayerPrivateGStreamerMSE::mirrorEnabledVideoTrackIfNeeded): Propagate the "selected" state from a given VideoTrackPrivateGStreamer (typically one know by HTMLMediaElement) to the internal VideoTrackPrivateGStreamer managed by >
* Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamer.h: Added mirrorEnabledVideoTrackIfNeeded() empty implementation.
* Source/WebCore/platform/graphics/gstreamer/mse/MediaPlayerPrivateGStreamerMSE.h: Added mirrorEnabledVideoTrackIfNeeded().
* Source/WebCore/platform/graphics/gstreamer/mse/WebKitMediaSourceGStreamer.cpp:
(WebKitMediaSrcPrivate::streamById): Support asking for streams not registered in this element. Log an info instead of asserting when not present.
(webKitMediaSrcTearDownStream): Gracefully bail out if the Stream is not registered in this element.
(webKitMediaSrcLoop): Ditto. Just skip the buffer push without triggering an error.
(webKitMediaSrcFlush): Ditto.
Canonical link: https://commits.webkit.org/299700@main1 parent f5892cd commit f618603
File tree
20 files changed
+159
-10
lines changed- LayoutTests/media/media-source
- content
- Source/WebCore
- Modules/mediasource
- platform/graphics/gstreamer
- mse
20 files changed
+159
-10
lines changedBinary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
0 commit comments