Mixed video codecs. H264 and VP8

[xBasov]

Making a video call from MicroSIP to my application. My application uses WindowsVideoEndPoint with FFmpegVideoEncoder and the media endpoints will be passed to VoIPMediaSession. However VoIPMediaSession.OnVideoSinkSample is not firing. That is because there is mixed video codecs as seen from logs that the video sink and source is setting the format to VP8, but Video depacketisasion codec is set to H264.

The issue is fixed when I remove from Helper class the VP8 codec as an supported video format. So everything will be set to H264.

[12:32:14 DBG] Setting audio sink and source format to 0:PCMU 8000 (RTP clock rate 8000).
[12:32:14 DBG] Setting video sink and source format to 96:VP8.
[12:32:14 DBG] Set remote track (audio - index=0) SSRC to 1470613415.
[12:32:15 INF] Video capture device Integrated Webcam successfully initialised: 1280x720 30fps pixel format NV12.
[12:32:15 WRN] Video source for capture device failure. Hardware MFT failed to start streaming due to lack of hardware resources.
[12:32:15 DBG] Successfully initialised ffmpeg based image encoder: CodecId:[AV_CODEC_ID_VP8] - 640:480 - 30 Fps
[12:32:15 DBG] Set remote track (video - index=0) SSRC to 1714766501.
[12:32:15 DBG] Video depacketisation codec set to H264 for SSRC 1714766501.
[12:32:19 DBG] [InitialiseDecoder] CodecId:[AV_CODEC_ID_VP8

[xBasov]

Tried to debug it deeper and I can see the issue might come from the RTPSession.cs at base SipSorcery project. From lines 1171 to 1179. Since the LocalTrack.Capabilites is not updated based on RemoteTrack capabilities. Meaning in the LocalTrack the VP8 will always used before H264, even if remote track only supports H264. Therefore in the VideoFormatsNegotiated event the .First() will return VP8 as a videoFormat that will be used as a videoSink and videoSource.

// As proved by Azure implementation, we need to send based on capabilities of remote track. Azure return SDP with only one possible Codec (H264 107)
// but we receive frames based on our LocalRemoteTracks, so its possiblet o receive a frame with ID 122, for exemple, even when remote annoucement only have 107
 // Thats why we changed line below to keep local track capabilities untouched as we can always do it during send/receive moment
capabilities = SDPAudioVideoMediaFormat.GetCompatibleFormats(currentMediaStream.RemoteTrack?.Capabilities, currentMediaStream.LocalTrack?.Capabilities);
 //Keep same order of LocalTrack priority to prevent incorrect sending format

SDPAudioVideoMediaFormat.SortMediaCapability(capabilities, currentMediaStream.LocalTrack?.Capabilities);
currentMediaStream.RemoteTrack.Capabilities = capabilities;
//currentMediaStream.LocalTrack.Capabilities = capabilities;