RTX+simulcast motion artifacts

[jpietek]

I did some more testing with the setup described here: #6, so simulcast on with bwe turned off, and recv-video-height=720.

With ideal network conditions it works fine, but breaks (in a weird way) when introducing for example 5% packet loss between the client and JVB. I mean the pipeline is still going, frame pacing is fine, no stutters (so to some degree RTX is working?) and I do get lots of activity in GST_DEBUG logs like: rtpsession request NACK for SSRC (and here goes SSRC of 180p layer), append NACK with deadline and rtpjitterbuffer Add RTX timer. But after decoding the vp8 stream random motion, macroblock artifacts are clearly visible. After turning simulcast off the problem is gone.

I wonder technically why does it happen and if it's a known issue due to some RTX/simulcast functionality missing? Maybe gst-meet is receiving reransmissions for the wrong RTP layer, ssrc? It usually starts with the 180p quality with its ssrc and then switches up to 720p after a few seconds. The artifacts are persistent and never go away, unless the packet loss is 0%.

Pipeline as follows: ./gst-meet --verbose --web-socket-url=wss://srv/xmpp-websocket --last-n=1 --single-endpoint=<eid> --select-endpoints=<eid> --xmpp-domain=<domain>--focus-jid=focus.meet.jitsi --muc-domain=muc.meet.jitsi --recv-video-height=720 --room-name=<room> --recv-pipeline-participant-template="vp8dec name="video" ! videoconvert ! autovideosink"

[jpietek]

@jbg any luck with simulcast?

What I've found out:

• It breaks in a similar way if I turn off gst's re-transmissions for the non-simulcast scenario, even tough some NACKs are sent by the JVB itself, but it's not perfect without gst RTX even with 1000ms rtpbin latency
• JVB's AdaptiveVp8Projection projects all the incoming streams to a single SSRC for gst
• Jicofo has a stripSimulcast option, so it presents only the lowest layer to gst in XMPP
• The artifacts won't go away even if I select the lowest 180p layer (why?)

[jpietek]

@jbg as gstreamer 1.20 is out, do you plan to release some webrtc/simulcasts improvements? Thanks in advance for the info.

[jbg]

@jpietek Thanks for the report here and apologies for the slow response.

At the moment I'm working on trying to get bandwidth estimation working correctly. We can't disable BWE (since we need adaptation to different client connectivity), so getting this part working correctly is our priority. I suspect we may run into this issue once we get BWE working correctly, though, at which point I'll update this issue with what I can find.

It usually starts with the 180p quality with its ssrc and then switches up to 720p after a few seconds.

This is probably because the sender starts off sending 180p and then ramps up; you can control this with startBitrate in the sender's config.js. It defaults to 800 kbit/s which would not be enough for 720p.

Jicofo has a stripSimulcast option, so it presents only the lowest layer to gst in XMPP

Yes, and this is now enabled by default. It doesn't change anything for this issue but does make things simpler / easier to understand.

[jbg]

BTW, latest master is now working with gstreamer 1.20, but I don't think it will have fixed this issue.

[jbg]

As of 3557183, TWCC is now working, so the connection doesn't freeze after 10 seconds with BWE enabled. I do now see motion artifacts and will update this ticket with any findings.

[jbg]

@jpietek PLIs were not being sent when the decoder needed a keyframe, because we needed to add an undocumented field to the GStreamer caps. This was probably the cause of the persistent artifacts.

We've added this field, and configured the jitter buffers to send loss events downstream. You will need to add these properties to the video decoder element in your pipeline (e.g. vp8dec or vp9dec):

automatic-request-sync-points=true automatic-request-sync-point-flags=2

automatic-request-sync-points=true makes the decoder send the necessary event upstream, and automatic-request-sync-point-flags=2 makes it continue outputting (potentially corrupted) frames while waiting for the next keyframe, rather than discarding input.

With this configuration and latest master, you will still see some artifacts during packet loss (as expected), but the decoder should request a keyframe and recover the picture as soon as it receives it.

[jpietek]

@jbg great news, I've had this keyframe/PLI issue too, resulting in persistent artifacts.

As for the temporal artifacts on lossy/jittered internet, I believe the problem is with gst vp8 depayloader. In Chrome the temporal continuity is taken into account, so cur_tl0 == prev_tl0 + 1 is considered a valid sync frame (without KF request). This way the stream can recover by dropping tl1 and tl2 frames with no artifacts, only reduced fps.

https://chromium.googlesource.com/external/webrtc/stable/webrtc/+/21623309f2aedf9dee47b8917c9a96982091c8c7/modules/video_coding/main/source/decoding_state.cc#165

GST can't handle that properly, as it only takes pictureIds and seq numbers into account. The only workaround for me was to spawn electron app with a Jitsi client, that would handle the WebRTC signalling/NACKs etc. and then forward the packets directly from JVB's AdaptiveVp8Projection to Gstreamer plain udpsrc. I did the depayloading in Java and took the temporal layers into account, rewrote seq_nums and pictureIds to continuous numbers. This works but it's obviously not optimal for big conferences, as it doubles the network traffic and decoding power. The electron browser windows and Gstreamer are doing the same thing twice.

[jbg]

Thanks for the analysis, this is useful. We may need to look at improving the gst vp8 depayloader. Do you have the same issue with vp9?

[jpietek]

Haven't tried this with VP9 yet. BTW nvvp8dec behaves weirdly with simulcast, resulting in additional artifacting. For me the latest cpu-based vpx decoder from google worked best.

[jbg]

Hey @jpietek — not sure if you are still using gst-meet, but if you are, can you confirm if you are still seeing these issues with latest gstreamer version and latest gst-meet version?

[jpietek]

I plan to give it a try with the updated av1 enabled jitsi, I'll update on it.

[jbg]

FYI, right now using AV1 requires gstreamer from git master because the released version doesn't support the payload type Jitsi signals (<96)

[jpietek]

@jbg BTW is there any way to have simulcast in the sending pipeline from gstenc to webrtcbin?

Tested the sender pipeline with jitsi-9457-2, the sender with fixed 1080p30 works, but still only with trust-bwe true on jvb side. If I turn it on, the video is disabled from time to time on other conference participants due to insufficient bandwidth. I send only the 500kbit test pattern.