WebRTC

Web Real-Time Communication

What is it?

new RTCPeerConnection(configuration)

Real-time communication directly between browsers (P2P).

It is used for:

  • audio/video communication;
  • data channels;
  • file sharing.

Use cases

The existing use cases for WebRTC can get really diverse. The most promising spheres:

  • messengers;
  • smart home;
  • healthcare;
  • wearable devices;
  • Internet of Things.

Who use it?

Google Meet
Google Meet

WhatsApp
WhatsApp

Facebook Messenger
FB Messenger

Discord
Discord

Alternatives

  • SignalR - library for ASP.NET that allows server code to send asynchronous notifications to client-side web applications.
  • XMPP - open XML technology for real-time communication.
  • Zoom - SDK that allows to integrate all Zoom Client app features.
  • Twilio - SDK that allows to make voice calls.
  • Skype - SDK that allows to integrate a wide variety of real-time collaboration models.

Connection establishment

NAT

Network address translation

Both nodes in one network

Both nodes are in one network

One node in private and one in public network

One node in private network, the other is in public one

Both nodes in different private networks

Both nodes are in different private networks

Connection setup phase

WebRTC has no protocol for transferring connection data. For this purpose additional server (it is called signalling) is needed:

  • WebSockets;
  • HTTP;
  • SMTP.

All data is transmitted as text and is divided into two types - SDP and ICE Candidate.

Caller steps

const peerConnection = new RTCPeerConnection()

const stream = await navigator.getUserMedia({ video: false, audio: true })
for (const track of stream.getTracks()) {
  peerConnection.addTrack(track, stream)
}

const description = await this.peerConnection.createOffer()

await peerConnection.setLocalDescription(description)

peerConnection.onicecandidate = event => { /// }

peerConnection.ontrack = event => { /// }

Callee steps

const peerConnection = new RTCPeerConnection()

const stream = await navigator.getUserMedia({ video: false, audio: true })
for (const track of stream.getTracks()) {
  peerConnection.addTrack(track, stream)
}

peerConnection.setRemoteDescription(new RTCSessionDescription(remoteDescription))
const description = await this.peerConnection.createAnswer()

await peerConnection.setLocalDescription(description)

peerConnection.onicecandidate = event => { /// }

peerConnection.ontrack = event => { /// }

Basic entities

MediaStream

height:350px

Microphone

const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false })

Microphone and camera

const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: true })

Screen sharing

const stream = await navigator.mediaDevices.getDisplayMedia()

Session Description Protocol (SDP)

Codec negotiation Codec negotiation

Interactive Connectivity Establishment (ICE) candidates

ICE lets two peers find and establish a connection with one another in such order:

  1. Direct UDP connection.
  2. Direct TCP connection, via the HTTP port.
  3. Direct TCP connection, via the HTTPS port.
  4. Indirect connection via a relay/TURN server.

Types of ICE candidates

candidate:3511422883 1 udp 2113937151 47e5106c-1c95-4262-b1cd-e917fb8a51b1.local 60843 typ host generation 0 ufrag 3rtW network-cost 999
  1. host.
  2. srflx.
  3. prflx
  4. relay

Both nodes are in one network

p1 - [10.50.200.5, 531, udp] - p2
p2 - [10.50.150.3, 531, udp] - p1

width:600px

p1 - [10.50.200.5, 531, udp] - r1
r1 - [10.50.200.5, 531, udp] - p2
p2 - [10.50.150.3, 531, udp] - r1
r1 - [10.50.150.3, 531, udp] - p1

STUN/TURN servers

STUN, TURN

STUN server operation

  1. r1_nat empty table
Internal IP Internal PORT External IP External PORT
  1. p1 generates package
Src IP Src PORT Dest IP Dest PORT
192.168.0.200 35777 12.62.100.200 6000
  1. r1 changes package
Src IP Src PORT Dest IP Dest PORT
10.50.200.5 888 12.62.100.200 6000

r1_nat table

Internal IP Internal PORT External IP External PORT
192.168.0.200 35777 10.50.200.5 888
  1. s1 receives package
Src IP Src PORT Dest IP Dest PORT
10.50.200.5 888 12.62.100.200 6000
  1. s1 generates an answer
Src IP Src PORT Dest IP Dest PORT Content
12.62.100.200 6000 10.50.200.5 888 10.50.200.5:888
  1. r1 receives an answer and choose between ports
Internal IP Internal PORT External IP External PORT
192.168.0.200 35777 10.50.200.5 888
192.168.0.173 35777 10.50.200.5 889
  1. r1 changes an answer
Src IP Src PORT Dest IP Dest PORT
12.62.100.200 6000 192.168.0.200 35777

TURN advantages:

  • relay mode;
  • ability to work with symmetric NAT.
new RTCPeerConnection({
  iceServers: [
    {
      urls: 'stun:stun.server.com: 13773',
    },
    {
      urls: 'turn:turn.server.com:19403',
      username: 'user',
      credentials: 'credentials'
    }
  ]
})

WebRTC codecs

Audio codecs

  • Opus
  • G.711
  • G.722
  • iLBC
  • iSAC
{
  "channels": 2,
  "clockRate": 48000,
  "mimeType": "audio/opus",
  "payloadType": 111,
  "sdpFmtpLine": "minptime=10;useinbandfec=1"
}

Video codecs

  • VP8
  • H.264
  • VP9
  • H.265
  • AV1
{
  "clockRate": 90000,
  "mimeType": "video/VP8",
  "payloadType": 96
}

Codecs priority reordering

const senders = peerConnection.getSenders()
for (const sender of senders) {
  const params = sender.getParameters()
  for (const codec of params.codecs) {
    ///
  }

  sender.setParameters(params)
}

WebRTC Topologies

Mesh

height:500px

Selective Forwarding (SFU)

height:500px

Multipoint Control (Mixing)

height:500px

Comparison between topologies (this study)

height:500px

height:500px

Data channels

const channel = peerConnection.createDataChannel(label, options)
peerConnection.ondatachannel = event => {
  const channel = event.channel
}

channel.onmessage = event => {
  const data = event.data
}
  • text chat
  • file transfer
  • gaming
  • IoT/Streaming Data

Text chat

const data = {
  from: id,
  date: Date.now(),
  message: 'text',
}
channel.send(JSON.stringify(data))

...

channel.onmessage = event => {
  const data = JSON.parse(data.event)
}

File transfer

const file = event.target.files[0]

channel.binaryType = 'arraybuffer'
const arrayBuffer = await file.arrayBuffer()
channel.send(arrayBuffer)

...

channel.onmessage = event => {
  const data = event.data
  const blob = new Blob([data])
}

WebRTC Security

  • Browser Protection
  • Media Access
  • Encryption

adapter.js

npm install webrtc-adapter

import adapter from 'webrtc-adapter'

adapter.browserDetails.browser

adapter.browserDetails.version

Possible problems

creating offer without streams or data channel

if (stream) {
  for (const track of stream.getTracks()) {
    peerConnection.addTrack(track, stream)
  }
} else {
  peerConnection.createDataChannel('call')
}

await peerConnection.createOffer()

getting ice candidates before offer

if (peerConnection.remoteDescription) {
  await peerConnection.addIceCandidate(new RTCIceCandidate(candidate))
} else {
  pendingIceCandidates.push(candidate)
}

///

for (const iceCandidate of pendingIceCandidates) {
  await peerConnection.addIceCandidate(new RTCIceCandidate(iceCandidate))
}
  
pendingIceCandidates = []

adding new streams while the previous negotiation is in progress

if (['have-remote-offer', 'stable'].includes(peerConnection.signalingState)) {
  for (const track of stream.getTracks()) {
    peerConnection.addTrack(track, stream)
  }
} else {
  pendingStreams.push(stream)
}

Debugging

Testing device connectivity

https://test.webrtc.org/

Overview of a device's network and media capabilities.

WebRTC Internals

chrome://webrtc-internals

width:900px

getStats function

peerConnection.getStats().then(stats => {
  for (const report of stats) {
    /* [
      "RTCAudioSource_2",
      {
        "id": "RTCAudioSource_2",
        "timestamp": 1614698646082.791,
        "type": "media-source",
        "trackIdentifier": "5a897ba2-630e-453a-b164-d9e193a9391c",
        "kind": "audio",
        "audioLevel": 0.6677449873348186,
        "totalAudioEnergy": 11.904587962536198,
      }
    ] */
  }
});

Network packet sniffer (Wireshark)

width:900px

Useful links

Thank you for attention