RTPPacket.Payload codec

[nicolastmaia]

Hi.
I am developing a webrtc SFU and so far I got 2 mobile devices to talk to each other using the same codec, everything with a sipsorcery server in the middle.
What I want to do now is get the audio received from one of the mobile devices and save it on the sipsorcery server computer's storage, but for that I have a few questions.
I'm using pc.OnRtpPacketReceived += (IPEndPoint rep, SDPMediaTypesEnum media, RTPPacket rtpPkt)=>{} to manipulate the RTPPacket received. I'd like to know if the rtpPkt.Payload suffers any decoding from the sipsorcery side (internally) or any kind of manipulation. In summary, if the codec agreed upon by the two devices is PCMU, can I expect the rtpPkt.Payload to still be PCMU encoded or will it be linear PCM?

[imtadashi]

Hi @nicolastmaia, can you please tell me how did you built a simple SFU server using Sipsorcery?

[nicolastmaia]

Sure!
All the code is in this repo.
If you have any questions feel free to contact me.

[imtadashi]

Thank you so much!

[ChristopheI]

RTPPacket objects from this event have not been decoded. So they are in the expected format defined in the Payload property

[nicolastmaia]

Got it. Thank you!

[imtadashi]

Hi @nicolastmaia, I wen through your code and hardcoded a POC. Basically what I did was, created a 2 PeerConncetions PC and PC2. PC for sender and PC2 for receiver. I used PC.OnRtpPacketReceived to send rtp pakcets to PC2 after adding it to a dictionary(also tried directly). But I have a small issue

Scenario 1:

1. Sender connects to the server first
2. Sender starts sending video feed
3. Receiver connects to the server

In the above scenario receiver only gets the stream approximately after 1 minute(very rarely not receiving).

Scenario 2:

1. Receiver connects to the server first
2. Receiver waits for video feed
3. Sender connects to the server
4. Sender sends feed to the server

In this scenario receiver gets feed as soon as the sender initiates the connection.

This is the code where I create PeerConnection for both sender and receiver

internal static RTCPeerConnection CreatePeerConnection()
{
    RTCConfiguration config = new RTCConfiguration
    {
        iceServers = new List<RTCIceServer> { new RTCIceServer { urls = STUN_URL }, new RTCIceServer { urls = TURN_URL, username = TURN_USER, credential = TURN_CRED } },
    };

    PC = new RTCPeerConnection(config);
    peerConnections.Add(PC);

    MediaStreamTrack videoTrack = new MediaStreamTrack(new VideoFormat(VideoCodecsEnum.VP8, 96));
    PC.addTrack(videoTrack);

    MediaStreamTrack audioTrack = new MediaStreamTrack(SDPWellKnownMediaFormatsEnum.PCMU);
    PC.addTrack(audioTrack);

    PC.OnRtpPacketReceived += (IPEndPoint rep, SDPMediaTypesEnum media, RTPPacket rtpPkt) =>
    {
        //PC2.SendRtpRaw(media, rtpPkt.Payload, rtpPkt.Header.Timestamp, rtpPkt.Header.MarkerBit, rtpPkt.Header.PayloadType);
        foreach (var pcs in peerConnections)
        {
            pcs.SendRtpRaw(media, rtpPkt.Payload, rtpPkt.Header.Timestamp, rtpPkt.Header.MarkerBit, rtpPkt.Header.PayloadType);
        }
    };

    PC.onconnectionstatechange += async (state) =>
    {
        logger.LogDebug($"Peer connection state change to {state}.");

        if (state == RTCPeerConnectionState.failed)
            PC.Close("ice disconnection");

        else if (state == RTCPeerConnectionState.closed)
        {
            Console.WriteLine("Stream Closed");
            SignalR.remoteAnswerB64 = null;
        }
        else if (state == RTCPeerConnectionState.connected)
        {
            Console.WriteLine("");
        }
    };

    // Diagnostics.
    PC.oniceconnectionstatechange += (state) => logger.LogDebug($"ICE connection state change to {state}.");

    return PC;
}

internal static RTCPeerConnection CreatePeerConnection2()
{
    RTCConfiguration config = new RTCConfiguration
    {
        iceServers = new List<RTCIceServer> { new RTCIceServer { urls = STUN_URL }, new RTCIceServer { urls = TURN_URL, username = TURN_USER, credential = TURN_CRED } },
    };

    PC2 = new RTCPeerConnection(config);
    peerConnections.Add(PC2);

    MediaStreamTrack videoTrack = new MediaStreamTrack(new VideoFormat(VideoCodecsEnum.VP8, 96));
    PC2.addTrack(videoTrack);

    MediaStreamTrack audioTrack = new MediaStreamTrack(SDPWellKnownMediaFormatsEnum.PCMU);
    PC2.addTrack(audioTrack);

    PC2.onconnectionstatechange += async (state) =>
    {
        logger.LogDebug($"Peer connection state change to {state}.");

        if (state == RTCPeerConnectionState.failed)
            PC.Close("ice disconnection");

        else if (state == RTCPeerConnectionState.closed)
        {
            Console.WriteLine("Stream Closed");
            SignalR.remoteAnswerB64 = null;
        }
        else if (state == RTCPeerConnectionState.connected)
        {
            Console.WriteLine("");
        }
    };

    // Diagnostics.
    PC2.oniceconnectionstatechange += (state) => logger.LogDebug($"ICE connection state change to {state}.");
    return PC2;
}

[imtadashi]

@nicolastmaia I think there is some issue related to handling Ice Candidate. With my code, the second participant is not able to get the stream as soon as the WebRTC session starts. After a minute or two, 2nd participant gets the stream. Is there a way to resolve this issue?