Skip to main content

Join a meeting

In the previous lesson we have learnt how to create a React component to generate a meeting. Now we will learn how to join that meeting.

You can download the starter code from Step.03-Exercise-Join-a-meeting.

StreamInfo interface​

We will start by creating another file ./client/src/types/StreamInfo.ts and defining our new interface there.

The StreamInfo interface will be used to store information about each available stream. This is important because, unlike other solutions, we won't receive any MediaStream directly by default. We get a list of available remote streams and we can subscribe (request_stream) and unsubscribe (disconnect) on demand.

The advantage of this selective approach is that it can save bandwidth and processor power in the end user device. Also, we will be able to choose the quality that we want to receive (simulcast), but we will learn about this in a future lesson.

This is our interface for now, although we will extend it in subsequent lessons:

export interface StreamInfo {
  streamId: string;
  type: 'audio' | 'video';
  participantId: string;
  mid?: string;
}

We have three mandatory properties:

  • streamId: Unique identifier for the stream.
  • type: Indicates if it is an audio or video stream.
  • participantId: Unique identifier for the participant that emits the stream.

We have one property that is optional:

  • mid: It stands for Media Identification and it's used to get the right transceiver for the streamId. We get this value once we request a stream.

Participant component​

The Participant component is located in the file ./client/src/Meeting/RemoteParticipants/Participant.tsx and it will contain the video and audio HTML elements to render the MediaStreams for a remote participant.

We will start by importing all the dependencies that we will use:

import {Cell, Icon, IconTypes, Video, Audio} from '@pexip/components';
import {type Cells} from '@pexip/components/dist/types/src/components/foundation/Grid/Grid.types';

Now we will define the input properties for our component:

interface ParticipantProps {
  participantId: string;
  md: Cells | undefined;
  audioStream: MediaStream | null;
  videoStream: MediaStream | null;
}

The properties for this component include:

  • participantId: Unique identifier for the user. It will be used as a title for the video.
  • md: It is Bootstrap parameter that corresponds to a medium size device. It will be used to adapt the component width, since it specifies the number of columns that the component will take in a 12-column layout. If we have a lot of participants in the screen, we will want to see each participant in a smaller section.
  • audioStream: This is the MediaStream used for the audio.
  • videoStream: This is the MediaStream used for the video.

And now we define the component itself and pass the ParticipantProps as input:

export const Participant = (props: ParticipantProps): JSX.Element => {
  return (
    <Cell className="Cell" xs={12} md={props.md}>
      {props.videoStream != null && (
        <Video
          className="RemoteVideo"
          title={props.participantId}
          srcObject={props.videoStream}
          style={{ objectFit: 'cover' }}
        />
      )}
      {props.videoStream == null && (
        <div className="NoStreamContainer">
          <Icon
            className="ParticipantIcon"
            source={IconTypes.IconParticipant}
          />
        </div>
      )}
      {props.audioStream != null && (
        <Audio srcObject={props.audioStream} autoPlay={true} />
      )}
    </Cell>
  )
}

We can divide the component into three sections:

  • RemoteVideo: Contains a Video tag and it's used to display the MediaStream for video.
  • NoStreamContainer: This element will be displayed if the video is not active.
  • Audio: This element will be the container for the audio MediaStream.

RemoteParticipants component​

Next we will implement the RemoteParticipants component. This component is located in ./client/src/Meeting/RemoteParticipants/RemoteParticipants.tsx and it's responsible for displaying the view with all the participants.

In our implementation, each participant will have only one stream, but we could have chosen to have several streams per participant if we would have liked to have that.

import {type StreamInfo} from '../../types/StreamInfo';
import {type TransceiverConfig} from '@pexip/peer-connection';
import {Participant} from './Participant';

We will start by defining the properties for the component:

interface RemoteParticipantsProps {
  remoteParticipantsIds: string[];
  streamsInfo: StreamInfo[];
  remoteTransceiversConfig: TransceiverConfig[];
}

In this component we receive three properties:

  • remoteParticipantsIds: An array of strings with the IDs of all remote participants. This excludes the local participant ID.
  • streamsInfo: This object will help us to match the remote stream of the transceiversConfig with the participant ID.
  • transceiversConfig: Information about all the available transceivers. It contains the MediaStreams for audio and video for all the participants in the call.

We define the properties for the component:

export const RemoteParticipants = (props: RemoteParticipantsProps): JSX.Element => {

And inside the RemoteParticipant we deconstruct the props:

const {remoteParticipantsIds, streamsInfo, remoteTransceiversConfig} = props;

Now we define the function getMediaStreams in which, using the participantId, we get the audio and video streams for a specific participant:

const getMediaStreams = (
  participantId: string,
): {
  audioStream: MediaStream | null;
  videoStream: MediaStream | null;
} => {
  const audioMid = streamsInfo.find(streamInfo => {
    return (
      streamInfo.participantId === participantId && streamInfo.type === 'audio'
    );
  })?.mid;
  const videoMid = streamsInfo.find(streamInfo => {
    return (
      streamInfo.participantId === participantId && streamInfo.type === 'video'
    );
  })?.mid;

  const audioTransceiverConfig =
    audioMid != null
      ? remoteTransceiversConfig.find(
          transceiverConfig => transceiverConfig.transceiver?.mid === audioMid,
        )
      : null;
  const videoTransceiverConfig =
    videoMid != null
      ? remoteTransceiversConfig.find(
          transceiverConfig => transceiverConfig.transceiver?.mid === videoMid,
        )
      : null;

  const audioStream =
    audioTransceiverConfig?.remoteStreams != null
      ? audioTransceiverConfig.remoteStreams[0]
      : null;
  const videoStream =
    videoTransceiverConfig?.remoteStreams != null
      ? videoTransceiverConfig.remoteStreams[0]
      : null;

  return {
    audioStream,
    videoStream,
  };
};

In our front end, we will display the participants in a matrix. This means that all the participants will have the same size. You could define a different layout if you wished; for instance, you could choose to display a main participant filling almost the whole screen and the rest of participants in smaller thumbnails.

For our simple matrix implementation, we will get total number of participants and use that to calculate the width of each participant. To do this, we define the value of md (medium size device in Bootstrap) that refers to the number of columns that it will take in a 12-column layout:

const totalStreams = remoteParticipantsIds.length;
const columns = Math.ceil(Math.sqrt(totalStreams));
const md = Math.max(Math.round(12 / columns), 1) as any;

Next we build the Participant elements for each remote participant:

const participants = remoteParticipantsIds.map((participantId) => {
  const { audioStream, videoStream } = getMediaStreams(participantId)
  return (
    <Participant
      participantId={participantId}
      md={md}
      audioStream={audioStream}
      videoStream={videoStream}
      key={participantId}
    />
  )
})

Finally, we put all together and render the component:

return (
  <Grid
    className="RemoteParticipants"
    style={{
      display: totalStreams === 1 ? 'block' : 'flex',
      flexGrow: totalStreams < 3 ? '1' : 'initial'
    }}
  >
    {participants}
  </Grid>
)

Meeting component​

The Meeting component is the main component, it will fill the whole page and will perform the following tasks:

  • Initialize the SDK.
  • Attach of the handlers to events.
  • Subscribe to streams.
  • Create a one-time participant for a meeting.
  • Connects to a meeting.

The file for this changes is located in ./client/src/Meeting/Meeting.tsx.

We start by importing all the dependencies that we will use:

import {useEffect, useState} from 'react';
import {useParams} from 'react-router-dom';
import {
  createVpaas,
  createVpaasSignals,
  createRecvTransceivers,
  type Vpaas,
  type RosterEntry,
} from '@pexip/vpaas-sdk';
import type {MediaInit, TransceiverConfig} from '@pexip/peer-connection';
import {type Participant} from '@pexip/vpaas-api';
import {type StreamInfo} from '../types/StreamInfo';
import {config} from '../config';
import {RemoteParticipants} from './RemoteParticipants/RemoteParticipants';
import {Selfview} from '@pexip/media-components';

Define constants and variables​

At the top of the file we will define some global constants and the vpaas variable:

const RECV_AUDIO_COUNT = 9;
const RECV_VIDEO_COUNT = 9;

let vpaas: Vpaas;

The two previous constants define the maximum number of incoming streams that our application will be able to manage. The vpaas variable will be initialized later and it will contain all the methods from the VPaaS SDK.

Now, inside the Meeting function, we define some other constants that will store the component state:

const {meetingId} = useParams();

const [participant, setParticipant] = useState<Participant>();
const [localStream, setLocalStream] = useState<MediaStream>();

const [remoteTransceiversConfig, setRemoteTransceiversConfig] = useState<
  TransceiverConfig[]
>([]);
const [roster, setRoster] = useState<Record<string, RosterEntry>>();
const [streamsInfo, setStreamsInfo] = useState<StreamInfo[]>([]);

Here is a brief description of each of them, but we will explain them more deeply throughout this tutorial:

  • meetingId: We use useParams to obtain the meetingId from the URL. This is the id that we obtained after using the CreateMeeting component.
  • participant: In this state we save the current participant info. We will need the participant's credentials to join the meeting.
  • localStream: Stores the MediaStream with the feed obtained from the getUserMedia API. It contents the audio and video.
  • remoteTransceiversConfig: Array with information about all the remote transceivers.
  • roster: Object that contains all the participants and their streams properties.
  • streamsInfo: Array that we will build with information about all the available streams. We will update this array each time we subscribe to a stream.

Initialize the SDK​

Now it's time to initialize the SDK. We will create the function initVpaas that will return an object that we can use to interact with VPaaS:

const initVpaas = (): Vpaas => {
  const vpaasSignals = createVpaasSignals();
  vpaasSignals.onRosterUpdate.add(handleRosterUpdate);
  vpaasSignals.onRemoteStreams.add(handleRemoteStreams);

  return createVpaas({vpaasSignals, config: {}});
};

In the function we also attach two function to events:

  • onRosterUpdate: This event contains the list of participants and the streams that each participant is emitting.
  • onRemoteStreams: This event contains information about the transceiver configuration.

We will define this handler for these events in the next section.

Handler for events​

We have created the vpaas object and we have attached handlers to the events. In this section we implement the handler functions for onRosterUpdate and onRemoteStreams.

Handler for Roster Update event​

The first handler that we will implement is onRosterUpdate. This event is triggered each time a participant connects to the meeting, adds streams, removes streams or leaves the meeting.

In our function we will do several things:

  • Save the roster in the state.
  • Add new streams to the streamsInfo array.
  • Unsubscribe streams and remove the element from streamsInfo when a stream is not longer available.

Here is the code to perform these tasks:

const handleRosterUpdate = (roster: Record<string, RosterEntry>): void => {
  setRoster(roster);

  // Check if we have a new stream
  const activeStreamsIds: string[] = [];
  for (const [participantId, rosterEntry] of Object.entries(roster)) {
    for (const [streamId, stream] of Object.entries(rosterEntry.streams)) {
      activeStreamsIds.push(streamId);
      const found = streamsInfo.some(
        streamInfo => streamInfo.streamId === streamId,
      );
      if (!found) {
        streamsInfo.push({
          streamId,
          type: stream.type,
          participantId,
        });
      }
    }
  }

  // Check if we should disconnect old streams
  streamsInfo.forEach(streamInfo => {
    const found = activeStreamsIds.includes(streamInfo.streamId);
    if (!found) {
      // Disconnect stream
      unsubscribeStream(streamInfo).catch(e => {
        console.error(e);
      });
      const index = streamsInfo.findIndex(
        stream => stream.streamId === streamInfo.streamId,
      );
      streamsInfo.splice(index, 1);
    }
    return found;
  });

  setStreamsInfo([...streamsInfo]);
};

The above code is easier to understand given an example roster message:

{
  "64p36hxloui65drtbkg7qyzfgi": {
    "streams": {
      "7c2gsjhloui65hdi22bzhn6sre": {
        "codec": "opus",
        "semantic": "main",
        "type": "audio"
      },
      "7c7fpqxloui65hdi22bzhn6sre": {
        "codec": "vp8",
        "semantic": "main",
        "type": "video",
        "layers": []
      }
    }
  },
  "6pyoukxloui65ke5mlrbm5vycy": {
    "streams": {
      "6xmbophloui65d5ttyzcb7sle4": {
        "codec": "opus",
        "semantic": "main",
        "type": "audio"
      },
      "6xq6ejxloui65d5ttyzcb7sle4": {
        "codec": "vp8",
        "semantic": "main",
        "type": "video",
        "layers": []
      }
    }
  }
}

Handler for Remote Streams event​

Now we will jump to the second event, onRemoteStreams. In this case the handler will do the following tasks:

  • Add the transceiver config to the array if it's need.
  • Update the transceiver if it changed.
  • Finally, we update the remoteTransceiversConfig state:
const handleRemoteStreams = (config: TransceiverConfig): void => {
  const index = remoteTransceiversConfig.findIndex(transceiverConfig => {
    return transceiverConfig.transceiver?.mid !== config.transceiver?.mid;
  });
  if (index > 0) {
    remoteTransceiversConfig[index] = config;
  } else {
    remoteTransceiversConfig.push(config);
  }
  setRemoteTransceiversConfig([...remoteTransceiversConfig]);
};

The parameter that the handler will receive will be similar to the following:

{
    "content": "main",
    "dirty": false,
    "kind": "video",
    "allowAutoChangeOfDirection": true,
    "relativeDirection": true,
    "direction": "recvonly",
    "streams": [],
    "remoteStreams": [...],
    "transceiver": {...},
    ...
}

Functions to subscribe/unsubscribe​

By default we don't receive any MediaStream from VPaaS, so we need to define a couple of functions to subscribe and unsubscribe to the available streams.

Subscribe to a stream​

In this section we learn how to subscribe to streams. You should take into account that a stream could be an audio or video, so we will need to subscribe to two streams per participant.

For this task, we will make use of the SDK function requestStream(). This function will send a message to VPaaS to request the stream and return the receive_mid. The unique Media ID (MID) is part of the SDP in WebRTC and helps to distinguish between different streams. During this lesson we will learn how to use the mid to obtain the corresponding MediaStream.

const subscribeStream = async (streamInfo: StreamInfo): Promise<void> => {
  const response = await vpaas.requestStream({
    producer_id: streamInfo.participantId,
    stream_id: streamInfo.streamId,
    rid: null,
    receive_mid: null,
  });

  streamInfo.mid = response.receive_mid;
  setStreamsInfo([...streamsInfo]);
};

Once we get the response from the request, we save it into the streamsInfo state. Later, in the useEffect hook, we will use this info to obtain the requested MediaStream.

Unsubscribe from a stream​

Now we will do the opposite. We need to unsubscribe from a stream to which we had previously subscribed. This function will perform the following tasks:

  • Disconnect the stream using the SDK function disconnectStream() and pass the value of the mid.
  • Set the mid = undefined in the streamInfo. This way we annotate that we aren't subscribed to the stream.
  • Finally, we save these changes in the streamsInfo array.
const unsubscribeStream = async (streamInfo: StreamInfo): Promise<void> => {
  const mid = streamInfo.mid;
  if (mid != null) {
    await vpaas.disconnectStream({
      receive_mid: mid,
    });
  }
  streamInfo.mid = undefined;
  setStreamsInfo([...streamsInfo]);
};

Get the VPaaS API address​

We have defined the VPaaS address in the server configuration. The client is agnostic to this URL. Here we create a simple request to obtain this value:

const getApiAddress = async (): Promise<string> => {
  const response = await fetch(`${config.server}/api-address`);
  const url = await response.text();
  return url;
};

Create a one-time participant​

In the previous lesson we created a meeting. Now we need to create a participant with access to that meeting.

To achieve this, we will define a simple function that makes a request to the endpoint that we previously created:

const createParticipant = async (): Promise<Participant> => {
  const response = await fetch(
    `${config.server}/meetings/${meetingId}/participants`,
    {
      method: 'POST',
    },
  );
  const participant = await response.json();
  return participant;
};

Connect to a meeting​

Finally, we arrived to the key function in this lesson. This function is the one that will connect us to a meeting. We first need to define some input parameters:

  • apiAddress: This is the VPaaS URL that was retrieved from the server.
  • participant: This object contains the participant credentials to authenticate into the meeting.
  • mediaStream: This is a MediaStream that contains the local audio and video.

The first thing that we need to do in the function is to define the media requirements. In this array we define the following:

  • Track for audio from the MediaStream.
  • Track for video from the MediaStream.
  • Define the maximum audio streams that we can receive.
  • Define the maximum video streams that we can receive.

::info

Take into account that if we reached the transceivers limit and we want to subscribe to another stream, we need to unsubscribe first from another stream.

::

Then we join the meeting using the meetingId, participant credentials and the MediaStream with audio and video.

Finally, we connect to the meeting with the media requirements.

const connectMeeting = async (
  apiAddress: string,
  participant: Participant,
  mediaStream: MediaStream,
): Promise<void> => {
  if (meetingId == null) {
    throw new Error('meetingId not defined');
  }

  const mediaInits: MediaInit[] = [
    {
      content: 'main',
      direction: 'sendonly',
      kindOrTrack: mediaStream.getAudioTracks()[0],
      streams: [mediaStream],
    },
    {
      content: 'main',
      direction: 'sendonly',
      kindOrTrack: mediaStream.getVideoTracks()[0],
      streams: [mediaStream],
    },
    ...createRecvTransceivers('audio', RECV_AUDIO_COUNT),
    ...createRecvTransceivers('video', RECV_VIDEO_COUNT),
  ];

  await vpaas.joinMeeting({
    meetingId,
    participantId: participant.id,
    participantSecret: participant.participant_secret,
    apiAddress,
  });

  vpaas.connect({mediaInits});
};

useEffect hooks​

Now we will make use of the useEffect hook to perform some actions depending on the state of our component. We will have two different hooks:

  • Component mounted: We run this code when the component is mounted. This happens each time we try to join a meeting. In this case we get the access to the media devices and use the VPaaS API to join the meeting.
  • Stream Info or Transceiver changed: In this case something happened. Maybe a new transceiver was received or a new stream was added. When this happens we will check if we should subscribe to a new stream.

Component mounted​

This hook runs when the component is mounted and it performs the following tasks:

  • Init the SDK if it wasn't done before.
  • Create the one-time participant with access to that meeting.
  • Get access the microphone and camera.
  • Get the VPaaS URL.
  • Connect to the meeting.

This process is repeated each time the Meeting component is mounted.

useEffect(() => {
  const bootstrap = async (): Promise<void> => {
    if (vpaas == null) {
      vpaas = initVpaas();
    }

    const participant = await createParticipant();

    const localStream = await navigator.mediaDevices.getUserMedia({
      audio: true,
      video: true,
    });

    setParticipant(participant);
    setLocalStream(localStream);

    const apiAddress = await getApiAddress();
    await connectMeeting(apiAddress, participant, localStream);
  };
  bootstrap().catch(e => {
    console.error(e);
  });
  return () => {
    vpaas.disconnect();
  };
}, []);

Stream Info or Transceiver changed​

This hook is triggered when streamsInfo or remoteTransceiversConfig change. The goal is to subscribe to a stream if the following conditions are fulfilled:

  • We have received all the remoteTransceiversConfig.
  • We don't have a mid for that stream yet.

So, in our case, we will subscribe to all available streams.

useEffect(() => {
  // Check if it's connected (we received all the transceivers)
  if (remoteTransceiversConfig.length === RECV_AUDIO_COUNT + RECV_VIDEO_COUNT) {
    streamsInfo.forEach(streamInfo => {
      if (
        streamInfo.mid == null &&
        streamInfo.participantId !== participant?.id
      ) {
        subscribeStream(streamInfo).catch(e => {
          console.error(e);
        });
      }
    });
  }
}, [streamsInfo, remoteTransceiversConfig]);

Render the Meeting component​

Now we move to the visual component part. We start by obtaining a list of all remote participants Ids:

let remoteParticipantsIds: string[] = [];
if (roster != null) {
  remoteParticipantsIds = Object.keys(roster);
  remoteParticipantsIds = remoteParticipantsIds.filter(
    id => id !== participant?.id,
  );
}

Inside the Meeting element, we define the RemoteParticipants. This element will contain the audio and video for all the participants:

{remoteParticipantsIds.length > 0 &&
  <RemoteParticipants
    remoteParticipantsIds={remoteParticipantsIds}
    streamsInfo={streamsInfo}
    remoteTransceiversConfig={remoteTransceiversConfig}
  />
}

Next we define the component that will be displayed if there isn't any available participant:

{remoteParticipantsIds.length === 0 &&
  <h2 className='NoParticipants'>Waiting for other participants...</h2>
}

Finally, we define the component that will contain our local video. Take into account the isMirrored property. This allows us to display the self-view video flipped, which improves user experience.

<div className='PipContainer'>
  <Selfview
    className='SelfView'
    isVideoInputMuted={false}
    shouldShowUserAvatar={false}
    username='User'
    localMediaStream={localStream}
    isMirrored={true}
  />
</div>

Run the app​

If you want to run the application, you need to launch the server and the client at the same time. We start by launching the server:

$ cd server
$ npm start

And in another terminal we launch the client:

$ cd client
$ npm start

The browser will open automatically with the URL https://localhost:4000.

Once you have joined a meeting, you will see the following interface:

Join a meeting

You can compare your code with the solution in Step.03-Solution-Join-a-meeting. You can also check the differences with the previous lesson in the git diff.