How to display a Video element over a virtual sensor in iTwin Viewer UI?

[gabschettino]

Hey,

I'm working on an iTwin Viewer integration where I need to display a live video stream (e.g., from an IP camera, or hosted anywhere on the web) as a UI element on top of a virtual sensor within the 3D model. The idea is to visually link the camera feed to the corresponding sensor's position in the model.

I've looked through the samples and found nothing similar to it. Would it be the best approach to use Widgets?

It would be similar to this demo, but instead of a text display, a video would appear.

[marcoherbst]

The bit that can be hard here is to get access to the camera feed in the right format. Most cameras will produce jpegs or rtsp , rtsp wont play directly in most browsers ... this means you generally need to do a middle step.

Our platform, www.evercam.io , supports this, we focus on the construction industry and we integrate with iTwin. Feel free to connect if you think this sounds like a solution that could help. Happy to share our knowledge.

[ben-polinsky]

Hi @gabschettino,

The simplest way would be to display video in a marker via the marker's HTMLElement property.

I've put together a very quick example with 2d markers to help you get started: https://www.itwinjs.org/sandboxes/BenPolinsky/quickvideomarkers2d

Take a look at MPDecorator.ts:

   // setup video
    const videlem = document.createElement("video");
    let sourceMP4 = document.createElement("source");
    sourceMP4.type = "video/mp4";
    sourceMP4.src = "https://storage.googleapis.com/gtv-videos-bucket/sample/TearsOfSteel.mp4";
    videlem.appendChild(sourceMP4);
    
    const ratio = 16/9
    videlem.height = 100
    videlem.width = videlem.height*ratio
    
    this.htmlElement = videlem

Then in the onMouseButton method:

 /** This method will be called when the user clicks on a marker */
  public onMouseButton(ev: BeButtonEvent): boolean {
    if (BeButton.Data !== ev.button || !ev.isDown || !ev.viewport || !ev.viewport.view.isSpatialView())
      return true;

    if (this._onMouseButtonCallback) {
      this._onMouseButtonCallback(this.data);
      return true;
    }

    this.showPopupMenu({ x: ev.viewPoint.x, y: ev.viewPoint.y });
    this.playVideo() // triggers the video

    return true; // Don't allow clicks to be sent to active tool
  }