Presentation

An important capability in any videoconferencing app is the ability to share the screen. In this lesson we will learn how to develop this feature from both the perspective of sending and reception of screen sharing.

One difference between Pexip VPaaS and other video platforms and solutions is that in Pexip VPaaS several participants can share their screen at the same time. This can be a key feature when the users want to collaborate more efficiently in more demanding workflows.

You can download the starter code from Step.07-Exercise-Presentation.

StreamInfo interface

We start by editing the file ./client/src/types/StreamInfo.ts. We will define a new property called semantic. This property will be used to differentiate between the camera feed and the screen sharing feed. It will have the following values:

• main: Camera feed for a remote user.
• presentation: Screen sharing for a remote user.

export interface StreamInfo {
  ...
  semantic: 'main' | 'presentation' | 'misc'
}

Participant component

Now we will edit the file ./client/src/Meeting/remoteParticipants/Participant.tsx. We will make a small modification in this component and adapt the video proportions depending on the semantic (main or presentation).

In case the stream is from a camera feed we want it to fit the whole region for the video. It doesn't matter if the video is cropped slightly. We achieve this by assigning the CSS property objectFit: cover.

In screen sharing we don't want to crop the video. We shouldn't forget that we can share the whole screen, or just a web browser tab or a window with different aspect ratio. In this case we will assign the CSS property objectFit: contain.

First, we will add the semantic property to the ParticipantProps:

interface ParticipantProps {
  ...
  semantic: 'main' | 'presentation' | 'misc'
}

Now we will modify how the component is rendered and add a different objectFit CSS property depending on the semantic:

{props.videoStream != null && !videoMuted && (
  <Video
    ...
    style={{ objectFit: props.semantic === 'main' ? 'cover' : 'contain' }}
  />
)}

Remote Participants component

We will need to make some modifications to the file ./client/src/Meeting/RemoteParticipants/RemoteParticipants.tsx.

We will add a new condition to the getMediaStreams function. We will check that streamInfo.semantic === 'main to only return the video stream that come from the camera feed:

const getMediaStreams = (participantId: string): {
  audioStream: MediaStream | null
  videoStream: MediaStream | null
} => {
  ...
  const videoMid = streamsInfo.find((streamInfo) => {
    return (
      streamInfo.participantId === participantId &&
      streamInfo.type === 'video' &&
      streamInfo.semantic === 'main'
    )
  })?.mid
  ...
}

Now we filter the streams to obtain only the ones that come from screen sharing:

const presentationsInfo = streamsInfo.filter(streamInfo => {
  return streamInfo.semantic === 'presentation' && streamInfo.mid != null;
});

And we consider the also the presentations to calculate the total number of streams:

const totalStreams = remoteParticipantsIds.length + presentationsInfo.length;

Now we will modify the section in which we create the Participant components for the camera feed and we will add the property value semantic='main':

const participants = remoteParticipantsIds.map((participantId) => {
  const { audioStream, videoStream } = getMediaStreams(participantId)
  return (
    <Participant
      ...
      semantic="main"
    />
  )
})

And we create the Participant components for presentations:

const presentations = presentationsInfo.map((streamInfo) => {
  const { participantId, mid } = streamInfo

  const presentationTransceiverConfig =
    mid != null
      ? remoteTransceiversConfig.find((transceiverConfig) => {
          return transceiverConfig.transceiver?.mid === mid
        })
      : null

  const presentationStream =
    presentationTransceiverConfig?.remoteStreams != null
      ? presentationTransceiverConfig.remoteStreams[0]
      : null

  return (
    <Participant
      participantId={participantId}
      md={md}
      audioStream={null}
      videoStream={presentationStream}
      key={`${participantId}-presentation`}
      sinkId={props.sinkId}
      semantic="presentation"
    />
  )
})

Finally, we return also the presentations inside the RemoteParticipants:

return (
  <Grid
    className="RemoteParticipants"
    style={{
      display: totalStreams === 1 ? 'block' : 'flex',
      flexGrow: totalStreams < 3 ? '1' : 'initial'
    }}
  >
    {participants}
    {presentations}
  </Grid>
)

Toolbar component

We will need to make some modifications in the file ./client/src/Meeting/Toolbar/Toolbar.tsx. We will add here the button to start and stop the presentation.

We start by adding the current presentationStream and the onPresentationStreamChange callback in the Toolbar properties:

interface ToolbarProps {
  ...
  presentationStream: MediaStream | undefined
  onPresentationStreamChange: (stream: MediaStream | undefined) => void
}

And now we define the function that will be triggered when the user click on the presentation button:

const handlePressShareScreen = async (): Promise<void> => {
  let stream;
  if (props.presentationStream != null) {
    props.presentationStream.getVideoTracks().forEach(track => {
      track.stop();
    });
    props.vpaas.stopPresenting();
  } else {
    stream = await navigator.mediaDevices.getDisplayMedia();
    stream.getVideoTracks()[0].onended = handleEndShareScreen;
    props.vpaas.present(stream);
  }
  props.onPresentationStreamChange(stream);
};

The first thing that we will do is to check if the presentation is active:

• Presentation enabled: In this case we stop the presentation locally and send the stop presenting message to VPaaS.
• Presentation disabled: We use the getDisplayMedia() function that provides the browser API to get the MediaStream.

Now we define the function that will be called when the user stops the presentation through the native browser button:

const handleEndShareScreen = (): void => {
  props.vpaas.stopPresenting();
  props.onPresentationStreamChange(undefined);
};

Finally, we add the button that will be used to start/stop the presentation:

<Tooltip text="Share screen">
  <Button
    variant="translucent"
    modifier="square"
    onClick={() => {
      handlePressShareScreen().catch((e) => {
        console.error(e)
      })
    }}
    isActive={props.presentationStream != null}
  >
    <Icon source={IconTypes.IconPresentationOn} />
  </Button>
</Tooltip>

Meeting component

Now we need to make some other modifications in the file ./client/src/Meeting/Meeting.tsx to finish to wire up this feature.

We start by importing the Video component that we will use to display the own screen sharing `stream:

import {Video} from '@pexip/components';

Next we define the state to store the presentation stream:

const [presentationStream, setPresentationStream] = useState<MediaStream>();

We need to save also the semantic when we receive the information about the roster:

const handleRosterUpdate = (roster: Record<string, RosterEntry>): void => {
  ...
        streamsInfo.push({
          streamId,
          type: stream.type,
          participantId,
          layers,
          semantic: stream.semantic
        })
  ...
}

Next we define the Video when the local participant is sharing their screen. With this modification, the user that is sharing the screen will see their own feed on the top right corner:

<div className='PipContainer'>
  ...
  {isStreamActive(presentationStream) && (
    <Video
      className="PresentationSelfView"
      srcObject={presentationStream}
    />
  )}
</div>

Now we will pass two new properties to the Toolbar:

{vpaas != null && (
  <Toolbar
    ...
    presentationStream={presentationStream}
    onPresentationStreamChange={setPresentationStream}
  />
)}

Run the app

If you want to run the application, you will need to launch the server and the client at the same time. We start by launching the server:

$ cd server
$ npm start

And in another terminal we launch the client:

$ cd client
$ npm start

The browser will be opened automatically with the URL https://localhost:4000.

During a conference you will be able to share your screen and receive the screen sharing from other participants. If both actions happens at the same time, you will see the following interface:

You can compare your code with the solution in Step.07-Solution-Presentation. You can also check the differences with the previous lesson in the git diff.