Skip to main content

Simulcast

Simulcast is a technique used in WebRTC to optimize video streaming by sending multiple versions of the same video stream at different resolutions. These streams are then sent to VPaaS service simultaneously. The receiver can select the most appropriate stream based on network conditions and device capabilities, ensuring the best possible video quality and minimizing buffering or latency issues.

A powerful device with a fast internet connection can display high-resolution video, while a less capable device or one on a slower connection might only be able to handle lower resolution video.

Simulcast is particularly useful in scenarios where network conditions may vary or where we have devices with different capabilities.

You can download the starter code from Step.06-Exercise-Simulcast.

RTPStreamId interface​

The first step is to open the file ./client/src/types/RTPStreamId.ts and define the different video stream qualities. Our application will define two qualities: High and Low:

export enum RTPStreamId {
  Low = 'l',
  High = 'h',
}

StreamInfo interface​

Once we have defined all the possible RTPStreamId, we will open the file ./client/src/types/StreamInfo.ts and add two new parameters to our StreamInfo interface:

  • layers: Array with all possible layers or qualities to which we can subscribe to.
  • rid: The second parameter is called rid and represents the current quality to which we are subscribed to.
export interface StreamInfo {
  ...
  layers: Layer[]
  rid?: RTPStreamId
}

Define LocalStorage key​

In the previous lesson we have define a file with all possible localStorage keys in which we save the selected media devices. We will do the same for the rid. Open the file ./client/src/types/LocalStorageKey.ts and add the new parameter:

export enum LocalStorageKey {
  ...
  IncomingVideoQualityKey = `${prefix}:incomingVideoQuality`
}

Settings component​

Now we need a way to change the preferred rid. We will add a new selector in the Settings component that we have created for changing the media device. Open the file ./client/src/Meeting/Settings/Settings.tsx to make these modifications.

The first step is to add Select to the list of imports from @pexip/components:

import {Bar, Button, Modal, Select} from '@pexip/components';

Next now we import the RTPStreamId that we have previously defined:

import {RTPStreamId} from '../../types/RTPStreamId';

Now we will jump to the Settings function and define a state for incomingVideoQuality. Here we will store the current preferred quality for the incoming streams:

const [incomingVideoQuality, setIncomingVideoQuality] = useState<RTPStreamId>(
  RTPStreamId.High,
);

Now we will save the current value in the localStorage before calling props.onSave():

const handleSave = (): void => {
  ...
  localStorage.setItem(
    LocalStorageKey.IncomingVideoQualityKey,
    incomingVideoQuality
  )
  localStream?.getTracks().forEach((track) => {
    track.stop()
  })
  props.onSave()
}

Next we need to add a couple of lines to retrieve the last value of the incomingVideoQuality from the localStorage each time the Settings panel is displayed:

useEffect(() => {
  if (props.isOpen) {
    ...

    const quality: RTPStreamId = localStorage.getItem(
      LocalStorageKey.IncomingVideoQualityKey
    ) as RTPStreamId
    if (quality != null) {
      setIncomingVideoQuality(quality)
    }
  }
}, [props.isOpen])

Next, we render the Select component that will allow us to change the quality:

<Select
  className="IncomingVideoQuality"
  label="Incoming Video Quality"
  value={incomingVideoQuality}
  onValueChange={(id) => {
    setIncomingVideoQuality(id as RTPStreamId)
  }}
  options={[
    { id: RTPStreamId.High, label: 'High' },
    { id: RTPStreamId.Low, label: 'Low' }
  ]}
/>

Meeting component​

Finally, we need to edit the file ./client/src/Meeting/Meeting.tsx and modify how the meetings are created and managed. Our new functionality will:

  • Save the layers offered by other participants.
  • Include the rid when the user subscribes to a new stream.
  • Define the different rid that the user offers when they connect to the meeting.
  • Unsubscribe and subscribe again when the user changes the preferred rid.

We start by importing MediaEncodingParameters from @pexip/peer-connection:

import type {
  MediaInit,
  TransceiverConfig,
  MediaEncodingParameters,
} from '@pexip/peer-connection';

We also import RTPStreamId and LocalStorageKey:

import {RTPStreamId} from '../types/RTPStreamId';
import {LocalStorageKey} from '../types/LocalStorageKey';

We modify the handler for the roster event and save the layers in streamsInfo:

const handleRosterUpdate = (roster: Record<string, RosterEntry>): void => {
  ...
      if (!found) {
        let layers = []
        if ((stream as any)?.layers != null) {
          layers = (stream as any)?.layers
        }

        streamsInfo.push({
          streamId,
          type: stream.type,
          participantId,
          layers
        })
      }
    }
  }
  ...
}

We modify the subscribeStream function that now will receive an optional parameter preferredRid. If the preferredRid exists in the available layers, we add the rid in the call to requestStream and save this value in streamInfo.rid to indicate to which quality we are subscribed:

const subscribeStream = async (
  streamInfo: StreamInfo,
  preferredRid?: RTPStreamId,
): Promise<void> => {
  const requestedRid = streamInfo.layers?.some(
    layer => layer.rid === preferredRid,
  )
    ? preferredRid
    : undefined;
  const response = await vpaas.requestStream({
    producer_id: streamInfo.participantId,
    stream_id: streamInfo.streamId,
    rid: (requestedRid as string) ?? null,
    receive_mid: null,
  });

  streamInfo.mid = response.receive_mid;
  streamInfo.rid = requestedRid;
  setStreamsInfo([...streamsInfo]);
};

Now we modify connectMeeting to send the video with two different qualities. For that we add the sendEncodings property and build it in the getEncodings(mediaStream) that we create in the following step:

const connectMeeting = async (apiAddress: string, participant: Participant, mediaStream: MediaStream): Promise<void> => {
  ...

  const mediaInits: MediaInit[] = [{
    content: 'main',
    direction: 'sendonly',
    kindOrTrack: mediaStream.getAudioTracks()[0],
    streams: [mediaStream]
  }, {
    content: 'main',
    direction: 'sendonly',
    kindOrTrack: mediaStream.getVideoTracks()[0],
    streams: [mediaStream],
    sendEncodings: getEncodings(mediaStream)
  },
  ...createRecvTransceivers('audio', RECV_AUDIO_COUNT),
  ...createRecvTransceivers('video', RECV_VIDEO_COUNT)
  ]

  ...
}

We will define the function getEncodings that returns the different resolutions that user will deliver to VPaaS. We will define a low resolution that will be a quarter of the original resolution and a high resolution that will be the original resolution captured by the camera:

const getEncodings = (
  mediaStream: MediaStream,
): MediaEncodingParameters[] | undefined => {
  const [videoTrack] = mediaStream.getVideoTracks();
  const {width, height} = videoTrack?.getSettings() ?? {};
  if (width != null && height != null) {
    return [
      {
        rid: RTPStreamId.Low,
        scaleResolutionDownBy: 4.0,
        maxWidth: Math.trunc(width / 4),
        maxHeight: Math.trunc(height / 4),
      },
      {
        rid: RTPStreamId.High,
        maxWidth: width,
        maxHeight: height,
      },
    ];
  } else {
    return undefined;
  }
};

Now we modify the callback function for when the user changes a parameter in the settings. We will retrieve the value of incomingVideoQuality from the localStorage and, if the rid was changed, we unsubscribe from the previous stream and subscribe again with the new value of rid:

const handleSaveSettings = async (): Promise<void> => {
  ...

    streamsInfo.forEach((streamInfo) => {
      if (streamInfo.type === 'video') {
        const preferredRid =
          (localStorage.getItem(
            LocalStorageKey.IncomingVideoQualityKey
          ) as RTPStreamId) ?? RTPStreamId.High

        if (streamInfo.mid != null && streamInfo.rid !== preferredRid) {
          unsubscribeStream(streamInfo)
            .then(() => {
              subscribeStream(streamInfo, preferredRid).catch((e) => {
                console.error(e)
              })
            })
            .catch((e) => {
              console.error(e)
            })
        }
      }
    })
  }
}

Finally, we change the subscribeStream() call in the useEffect() hook to include the preferredRid:

useEffect(() => {
  ...

        const preferredRid =
          (localStorage.getItem(
            LocalStorageKey.IncomingVideoQualityKey
          ) as RTPStreamId) ?? RTPStreamId.High
        subscribeStream(streamInfo, preferredRid).catch((e) => {
          console.error(e)
        })
      }
    })
  }
}, [streamsInfo, remoteTransceiversConfig])

Run the app​

If you want to run the application, you will need to launch the server and the client at the same time. We start by launching the server:

$ cd server
$ npm start

And in another terminal we launch the client:

$ cd client
$ npm start

The browser will be opened automatically with the URL https://localhost:4000.

If you click on the settings button during a meeting, you will see a new selector to modify the incoming video quality:

Simulcast

You can compare your code with the solution in Step.06-Solution-Simulcast. You can also check the differences with the previous lesson in the git diff.