Ex_webrtc throwing error on answer SDP

[ponkii-1]

Hi,

I'm trying to implement a basic VOIP server with ex_webrtc but I'm running into an issue.
The client is using libdatachannel, and everything works so long as I'm only accepting tracks created by the server. If I try to create a track in response, then I get this error:

[debug] Closing peer connection with reason: {{:badkey, :attributes, nil}, [{ExWebRTC.SDPUtils, :get_media_direction, 1, [file: ~c"lib/ex_we
brtc/sdp_utils.ex", line: 96]}, {ExWebRTC.PeerConnection, :tr_negotiation_needed?, 2, [file: ~c"lib/ex_webrtc/peer_connection.ex", line: 2278]}, {ExWebRT
C.PeerConnection, :update_negotiation_needed, 1, [file: ~c"lib/ex_webrtc/peer_connection.ex", line: 2232]}, {ExWebRTC.PeerConnection, :apply_remote_descr
iption, 2, [file: ~c"lib/ex_webrtc/peer_connection.ex", line: 1896]}, {ExWebRTC.PeerConnection, :handle_call, 3, [file: ~c"lib/ex_webrtc/peer_connection.
ex", line: 864]}, {:gen_server, :try_handle_call, 4, [file: ~c"gen_server.erl", line: 2381]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", li
ne: 2410]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}

Here's the SDP from the client being applied, IPs removed:

v=0
o=rtc 2570525212 0 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 TESTINGTESTINGTESTING
a=group:LS 0 TESTINGTESTINGTESTING
a=msid-semantic:WMS *
a=ice-options:ice2,trickle
a=fingerprint:sha-256 CE:0B:83:77:6D:AD:30:63:27:E1:A5:D2:14:B1:C9:66:42:AB:60:4F:90:CA:2D:C0:DB:D4:ED:DD:32:B5:03:18
m=audio 45329 UDP/TLS/RTP/SAVPF 111 96
c=IN IP4 -----------
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=sendrecv
a=rtcp:9 IN IP4 0.0.0.0
a=msid:TAGOBKBL3GLMUTNL7YJ7MXNMVNVNLOCM
a=rtcp-mux
a=rtpmap:96 opus/48000/2
a=rtcp-fb:96 transport-cc
a=rtpmap:111 opus/48000
a=rtcp-fb:111 transport-cc
a=setup:actpass
a=ice-ufrag:opU0
a=ice-pwd:q7qS6cy3llprqdLsijsGZn
a=candidate:1 1 UDP 2122317823 ----------- 45329 typ host
m=audio 45329 UDP/TLS/RTP/SAVPF 96
c=IN IP4  -----------
a=mid:TESTINGTESTINGTESTING
a=recvonly
a=ssrc:42 cname:audio-send
a=rtcp-mux
a=rtpmap:96 opus/48000/2
a=fmtp:96 minptime=10;maxaveragebitrate=96000;stereo=1;sprop-stereo=1;useinbandfec=1
a=setup:actpass
a=ice-ufrag:opU0
a=ice-pwd:q7qS6cy3llprqdLsijsGZn

TESTINGTESTINGTESTING is the one created by the client, and I can clearly see that it's recvonly.
I think it's pretty likely I'm just misunderstanding the negotiating process, but what is going on here?

Thank you.

[mickel8]

Hi @x-kem0, I wrote a simple test and it passes

  test "" do
    sdp = """
    v=0
    o=rtc 2570525212 0 IN IP4 127.0.0.1
    s=-
    t=0 0
    a=group:BUNDLE 0 TESTINGTESTINGTESTING
    a=group:LS 0 TESTINGTESTINGTESTING
    a=msid-semantic:WMS *
    a=ice-options:ice2,trickle
    a=fingerprint:sha-256 CE:0B:83:77:6D:AD:30:63:27:E1:A5:D2:14:B1:C9:66:42:AB:60:4F:90:CA:2D:C0:DB:D4:ED:DD:32:B5:03:18
    m=audio 45329 UDP/TLS/RTP/SAVPF 111 96
    c=IN IP4 -----------
    a=mid:0
    a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:mid
    a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
    a=sendrecv
    a=rtcp:9 IN IP4 0.0.0.0
    a=msid:TAGOBKBL3GLMUTNL7YJ7MXNMVNVNLOCM
    a=rtcp-mux
    a=rtpmap:96 opus/48000/2
    a=rtcp-fb:96 transport-cc
    a=rtpmap:111 opus/48000
    a=rtcp-fb:111 transport-cc
    a=setup:actpass
    a=ice-ufrag:opU0
    a=ice-pwd:q7qS6cy3llprqdLsijsGZn
    a=candidate:1 1 UDP 2122317823 ----------- 45329 typ host
    m=audio 45329 UDP/TLS/RTP/SAVPF 96
    c=IN IP4  -----------
    a=mid:TESTINGTESTINGTESTING
    a=recvonly
    a=ssrc:42 cname:audio-send
    a=rtcp-mux
    a=rtpmap:96 opus/48000/2
    a=fmtp:96 minptime=10;maxaveragebitrate=96000;stereo=1;sprop-stereo=1;useinbandfec=1
    a=setup:actpass
    a=ice-ufrag:opU0
    a=ice-pwd:q7qS6cy3llprqdLsijsGZn
    """

    offer = %SessionDescription{type: :offer, sdp: sdp}

    {:ok, pid} = PeerConnection.start_link()
    assert :ok = PeerConnection.set_remote_description(pid, offer)
  end

Could you provide more detail? Or try to write an example test case that does not pass?

[mickel8]

Also, please note that while mid can be any string (as far as I remember), in WebRTC it is in most cases a number i.e. 0, 1, 2, etc.

[ponkii-1]

Good to know on the mid. I think that I'm probably confusing myself here, since this is supposed to be an answer.
This may be nonsense:

  test "" do
    sdp = """
      v=0
      o=rtc 2570525212 0 IN IP4 127.0.0.1
      s=-
      t=0 0
      a=group:BUNDLE 0 TESTINGTESTINGTESTING
      a=group:LS 0 TESTINGTESTINGTESTING
      a=msid-semantic:WMS *
      a=ice-options:ice2,trickle
      a=fingerprint:sha-256 CE:0B:83:77:6D:AD:30:63:27:E1:A5:D2:14:B1:C9:66:42:AB:60:4F:90:CA:2D:C0:DB:D4:ED:DD:32:B5:03:18
      m=audio 45329 UDP/TLS/RTP/SAVPF 111 96
      c=IN IP4 -----------
      a=mid:0
      a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:mid
      a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
      a=sendrecv
      a=rtcp:9 IN IP4 0.0.0.0
      a=msid:TAGOBKBL3GLMUTNL7YJ7MXNMVNVNLOCM
      a=rtcp-mux
      a=rtpmap:96 opus/48000/2
      a=rtcp-fb:96 transport-cc
      a=rtpmap:111 opus/48000
      a=rtcp-fb:111 transport-cc
      a=setup:actpass
      a=ice-ufrag:opU0
      a=ice-pwd:q7qS6cy3llprqdLsijsGZn
      a=candidate:1 1 UDP 2122317823 ----------- 45329 typ host
      m=audio 45329 UDP/TLS/RTP/SAVPF 96
      c=IN IP4  -----------
      a=mid:TESTINGTESTINGTESTING
      a=recvonly
      a=ssrc:42 cname:audio-send
      a=rtcp-mux
      a=rtpmap:96 opus/48000/2
      a=fmtp:96 minptime=10;maxaveragebitrate=96000;stereo=1;sprop-stereo=1;useinbandfec=1
      a=setup:actpass
      a=ice-ufrag:opU0
      a=ice-pwd:q7qS6cy3llprqdLsijsGZn
      """


    {:ok, pc} = PeerConnection.start_link()

    # act as offerer
    stream_id = MediaStreamTrack.generate_stream_id()
    audio_track = MediaStreamTrack.new(:audio, [stream_id])
    {:ok, _sender} = PeerConnection.add_track(pc, audio_track)

    {:ok, offer} = PeerConnection.create_offer(pc)
    :ok = PeerConnection.set_local_description(pc, offer)

    answer = %SessionDescription{type: :answer, sdp: sdp}
    assert :ok = PeerConnection.set_remote_description(pc, answer)
  end

[mickel8]

Yeah, that doesn't make a lot of sense 😅

If you have pc that you create offer from, you can't apply arbitrary answer back on pc.

You can do something like this:

    {:ok, pc} = PeerConnection.start_link()

    # act as offerer
    stream_id = MediaStreamTrack.generate_stream_id()
    audio_track = MediaStreamTrack.new(:audio, [stream_id])
    {:ok, _sender} = PeerConnection.add_track(pc, audio_track)

    {:ok, offer} = PeerConnection.create_offer(pc)
    :ok = PeerConnection.set_local_description(pc, offer)

    {:ok, pc2} = PeerConnection.start_link()
    assert :ok = PeerConnection.set_remote_description(pc2, offer)
    {:ok, answer} = PeerConnection.create_answer(pc2)
    :ok = PeerConnection.set_local_description(pc2, answer)
    :ok = PeerConnection.set_remote_description(pc, answer)

but this code does not use your SDP string.

[ponkii-1]

agh
Libdatachannel has a bit of a different API that I think is throwing me off.

So I guess my question is how does a client respond back with additional tracks?
The current flow is something like this, signalled with websockets:

Client requests offer
Server sends offer with a send track
Client adds a receive track <- broken here
Client applies remote description
Client sends answer
Server accepts answer
Done

Reading through the examples it seems that we should expect tracks from the remote, but is it more accurate to say that only the offerer may create tracks, and the answerer can only accept or reject them?

Thank you very much for your time, this particular info is really hard to find!

[mickel8]

but is it more accurate to say that only the offerer may create tracks, and the answerer can only accept or reject them?

Yes, exactly.

So if an offerer offers that it can both send and receive a single audio track, all the answerer can do is to accept or reject this track. Keep in mind that if offerer offers sendrecv direction, the answerer can reply with sendrecv, sendonly, recvonly or inactive.

If you want to have more tracks you have at least three options (random order):

• send an offer with all tracks, e.g. two outgoing tracks, 4 incoming tracks
• create a separate peer connection for sending and receiving
• perform renegotiation

If you describe your use case in more detail, maybe we can come up with an exact solution

[ponkii-1]

Sure

I'm writing a PoC VOIP system, really just to get a good understanding of WebRTC and some lower level concepts. I want to be able to connect and disconnect clients to channels similar to those that you would see in Discord. I figure the de facto standard would be to renegotiate the connection whenever anybody enters or leaves, adding (from clientside) recv only stream for each user and then audio mixing can be done in something like miniaudio.h.

What isn't clear to me is how the server and client each know which tracks they own. Given this is all being done outside of browser implementations I've got very low level control of everything, but the only thing I can imagine to do is to separate by mid and direction and, knowing that the server will send a recv track and x number of send tracks, sort those out in a callback and handle them appropriately.

Just knowing that only the offerer can create tracks helps a lot though. I've been piecing together half-solutions trying to figure this out and it seemed that I was supposed to create tracks on both sides.

[ponkii-1]

I should mention that I've got a P2P example up and running, so the particular thing I'm working on is inserting this server in the middle.

It also raises the question of how to identify tracks more generally. Should the server send something like 'mid 0 is username' via websocket, or is there some way to tag these tracks with metadata through webrtc itself?

[mickel8]

Should the server send something like 'mid 0 is username' via websocket

Yeap, I believe we were doing this in our SFU.

I would also recommend going for two peer connections. One that is responsible for sending and the other one that is responsible for receiving. The (re)negotiation process should be easier.

If you take a look at this from the server perspective, the connection on which the server receives something from a peer, once negotiated, it stays the same for the whole session duration. What changes is the connection on which the server sends something to the peer. And when the server needs to send a new track, it just adds it to this connection and sends a new SDP offer.

[ponkii-1]

I was kinda thinking that's how it'd work; I got my server working last night and had a 2-way VOIP call thanks to your help.

Next up is writing my own SFU to handle more than two peers. Thank you so much!