How to get forward stream id when clicking on mixed stream video in client UI

In video conference, there is a common scenario: When entering room, a mixed stream of all the participants will display and then you double click on one participant in the mixed stream and hope to see this specified stream in a large scale. We will introduce how to implement this scenario in OWT in this article and take web app as an example in the following introduction.

1. Mixed stream view layout

1.1 How to set mixed stream view layout

OWT provides a default layout for video view, each stream in the view will occupy the same area. If you want to customize your own view layout, set it in https://serverip:3300/console or use restful API to customize, refer to restful API doc and customized video layout for more information.

1.2 How to get mixed stream view layout

We can define different layouts for mixed stream view, but how do get the forward stream region info in the mixed view? OWT provides following ways to get mixed stream view layout with stream info:

Layout info is included in mixed stream, you can get the stream info by:

GET ${host}/v1.1/rooms/{roomId}/streams/{streamId}

And layout info is listed in the restful response response.info.layout

{id: "5e69f6c44894ad25c79ab42c-common", type: "mixed", media: {…}, data: null, info: {…}}
    data: null
    id: "5e69f6c44894ad25c79ab42c-common"
    info:
      activeInput: "unknown"
      label: "common"
      layout: Array(4)
        0:
          region:
            area: {left: "0/2", top: "0/2", width: "1/2", height: "1/2"}
            id: "1"
            shape: "rectangle"
          stream: "5cafce081a3e4d84bb5b7151ab5bb00d"
       1:
         region:
           area: {left: "1/2", top: "0/2", width: "1/2", height: "1/2"}
           id: "2"
           shape: "rectangle"
         stream: "ca8e50d736e9430093dd73bfaca108d3"
      2:
        region:
          area: {left: "0/2", top: "1/2", width: "1/2", height: "1/2"}
          id: "3"
          shape: "rectangle"
     3:
       region:
         area: {left: "1/2", top: "1/2", width: "1/2", height: "1/2"}
         id: "4"
         shape: "rectangle"
     length: 4

You can check this info once participant joins the room, and the layout info will change when there is new stream added/deleted from the mixed view. In this case, you need to listen on layoutchange event to get the layout info updated.

let layoutchangeListener = (event) => {
    console.log('layoutchange event triggered: ', event);
    videolayout = event.layout;
};

stream.addEventListener('layoutchange', layoutchangeListener);

2. Calculate forward stream id of clicking point

Assume that mixVideo is the div element id for the remote mixed stream video display in the web page, and the layout info is:

layout: Array(1)
        0:
          region:
            area: {left: "0/2", top: "0/2", width: "1/2", height: "1/2"}

First you can get the width and height of the div element for the video display:

var videoElement = document.getElementById("mixVideo");
var displayWidth = videoElement.clientWidth;
var displayHeight = videoElement.clientHeight;

Then you can get the mixed stream layout info of pixel info:

var position.left = videoElement.clientWidth * 0/2; 
var position.right = region0.left + videoElement.clientWidth * 1/2;
var position.top = videoElement.clientHeight * 0/2;
var position.down = region0.top + videoElement.clientHeight * 1/2;
positions.push(position);

Get click cursor position by adding div click event listen as:

$(`#${elementID}`).click(function(e) {
   var x = e.offsetX;
   var y = e.offsetY;
   var index = 0;
   console.log("Click on position x:", x, " y:", y);
   //====================
   //Compare position with layout position and get the layout index of the clicking point
   for ( var i in positions) {
     if( x > positions[i].left && y < positions[i].top && x < position[i].right && y > position[i].down){
       index = i;
     }
   }   
   //====================
});

We can know which stream participant is clicking:

var clickStreamId = layout[index].stream

Now you can hide your mixed stream view div and subscribe the forward stream clickStreamId and full screen the subscribed forward stream view.