[MM-63430] Calls 1-1 video support (MVP) :video_camera: by streamer45 · Pull Request #997 · mattermost/mattermost-plugin-calls

streamer45 · 2025-03-24T20:08:12Z

Summary

PR implements initial video support in DMs. The functionality is expected to be behind a new experimental EnableVideo config setting, which defaults to false.

The only slightly more involved logic is the image segmentation that powers the background blur functionality. This required adding an external dependency (@mediapipe/tasks-vision) plus some WASM assets and a model file (see under public/). We can consider removing this from the initial MVP if we think it's too much.

UX is a real work in progress, so don't focus too much on it. It will most certainly change before we merge as we are waiting for some designs.

Screenshots

Widget view

Popout view

Screen sharing

User settings

Related PRs

mattermost/rtcd#173
mattermost/calls-common#47
mattermost/mattermost-mobile#8716

Ticket Link

https://mattermost.atlassian.net/browse/MM-63430

streamer45 · 2025-03-24T23:15:01Z

standalone/src/widget/index.tsx

        }
    }

+    window.callsClient?.on('mute', () => {


This is an inconsistency I found when testing video on desktop. On the web side we do this to keep the redux state in sync with the local client. This avoids the issue of a perceivable lag between pressing mute/unmute and getting the UX to react, since otherwise we'd be waiting for a server side websocket event.

streamer45 · 2025-03-24T23:16:06Z

webapp/src/client.ts

+    frameRate: {
+        ideal: 30,
+    },
+    aspectRatio: {
+        ideal: 4 / 3,
+    },
+    width: {
+        ideal: 640,
+    },


This is temporary, for testing things. In the absence of user preferences, we should respect the device's default aspect ratio.

Did you get the aspect ratio fixed (from the test we did)? (is that issue actually these settings)

Most likely, although this is just about the source.

I still need to review aspect ratios in general in the UI, was waiting for some designs to understand what we want to do with them. I see a few options:

Clipping to cover all the available space without stretching (what we do today through object-fit: cover).

Show the full video content but add bands (lateral or vertical) to maintain the original ratio.

Dynamically modify widget dimensions (either height or width) to match the original ratio.

Stretch to cover the entire space (I wouldn't do this).

streamer45 · 2025-03-24T23:17:58Z

webapp/src/components/call_widget/component.tsx

+                                onToggle={noVideoInputDevices ? undefined : this.onVideoToggle}
+                                ariaLabel={videoTooltipText}
+
+                                //shortcut={} TODO: add shortcut


TBD, waiting for UX input.

@streamer45 is @asaadmahmood looking at this for you?

@matthewbirtch I haven't explicitly asked for it as I had to park this effort for a few weeks. It would be great to get a more formal UX pass on #1008, which is the tip of this work and should include most UX improvements to match against the designs you guys provided.

Sounds good. I'll add @asaadmahmood to that PR then

streamer45 · 2025-03-24T23:19:23Z

webapp/src/components/call_widget/loading_overlay.tsx

-    const onAnimationEnd = () => {
-        setAnimationEnded(true);
-    };


This would never fire and leave the component mounted for the whole duration of the call, still spinning but not visible (zero opacity). I am fixing it as part of this PR.

streamer45 · 2025-03-24T23:23:56Z

webapp/src/components/expanded_view/component.tsx

+                                tooltipText={videoTooltipText}
+                                tooltipSubtext={videoTooltipSubtext}
+                                // eslint-disable-next-line no-undefined
+                                // shortcut={noVideoInputDevices || noVideoPermissions ? undefined : reverseKeyMappings.popout[MUTE_UNMUTE][0]}


Same as for widget, TBD.

streamer45 · 2025-03-24T23:29:18Z

webapp/src/components/user_settings/video_devices_settings_section.tsx

+                                />
+                                <label htmlFor='mirror'>{formatMessage({defaultMessage: 'Mirror video'})}</label>
+                            </CheckBoxContainer>
+                            {!isFirefox() &&


Unfortunately, GPU acceleration isn't working yet on Firefox (see google-ai-edge/mediapipe#5879). I tested CPU only but it's just not going to work. Anyhow, this is experimental on top of an already experimental feature.

streamer45 · 2025-03-24T23:29:48Z

webapp/src/index.tsx

+        const theme = getTheme(store.getState());
+        setCallsGlobalCSSVars(theme.sidebarBg);


Another small bug I found when testing on Desktop.

ah, that's why it was up above. What was the bug?

--calls-bg CSS variable wouldn't be defined when running widget on desktop.

streamer45 · 2025-03-25T23:00:58Z

/cc @matthewbirtch for visibility.

matthewbirtch · 2025-03-27T17:46:21Z

/cc @matthewbirtch for visibility.

@streamer45 do you want me to review this from a UX standpoint, or are you just keeping me in the loop?

streamer45 · 2025-03-27T23:11:25Z

/cc @matthewbirtch for visibility.

@streamer45 do you want me to review this from a UX standpoint, or are you just keeping me in the loop?

@matthewbirtch The latter, for the time being. Of course early feedback is always welcome, but ideally I'd be waiting to have some Figma designs ready before asking for UX review because I've mostly improvised so far :)

cpoile

🎉 Amazing work.
Just lots of questions. 🧑‍🎓

server/db/calls_sessions_store.go

standalone/src/init.ts

standalone/tsconfig.json

cpoile · 2025-04-01T21:24:13Z

webapp/src/client.ts

+    frameRate: {
+        ideal: 30,
+    },
+    aspectRatio: {
+        ideal: 4 / 3,
+    },
+    width: {
+        ideal: 640,
+    },


Did you get the aspect ratio fixed (from the test we did)? (is that issue actually these settings)

webapp/src/client.ts

webapp/src/components/user_settings/video_devices_settings_section.tsx

cpoile · 2025-04-01T22:00:44Z

webapp/src/components/user_settings/video_devices_settings_section.tsx

+                                }}
+                                ref={videoInputsRef}
+                            />
+                            <VideoPreview


is there a way to reuse components, or is it not worth the trouble?

From the widget you mean? They serve very different purposes. Maybe a shared subcomponent but I am not sure.

cpoile · 2025-04-01T22:01:59Z

webapp/src/index.tsx

+        const theme = getTheme(store.getState());
+        setCallsGlobalCSSVars(theme.sidebarBg);


ah, that's why it was up above. What was the bug?

webapp/src/local_storage.ts

webapp/src/segmenter/segmenter.worker.ts

mattermost-build · 2025-08-02T01:06:12Z

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!

Calls video support

7463b33

streamer45 added the Do Not Merge Should not be merged until this label is removed label Mar 24, 2025

streamer45 self-assigned this Mar 24, 2025

Fixes

ccf5c3b

streamer45 mentioned this pull request Mar 24, 2025

[MM-63431] Calls 1-1 video support (MVP) mattermost/mattermost-mobile#8716

Open

streamer45 added 1: UX Review Requires review by ux 2: Dev Review Requires review by a core committer 2: QA Review Requires review by a QA tester 3: Security Review labels Mar 24, 2025

streamer45 commented Mar 24, 2025

View reviewed changes

streamer45 requested a review from cpoile March 24, 2025 23:31

streamer45 marked this pull request as ready for review March 25, 2025 01:35

Update calls-common

2395df7

cpoile approved these changes Apr 1, 2025

View reviewed changes

streamer45 requested a review from cpoile April 2, 2025 18:01

cpoile approved these changes Apr 2, 2025

View reviewed changes

streamer45 added 2 commits April 8, 2025 16:53

Merge remote-tracking branch 'origin/main' into MM-63430

b0864be

Update calls-common

3c01e9a

streamer45 mentioned this pull request Apr 18, 2025

[MM-63430] UX polish #1008

Draft

Merge remote-tracking branch 'origin/main' into MM-63430

48e44f2

mattermost-build added the Lifecycle/1:stale label Aug 2, 2025

		const theme = getTheme(store.getState());
		setCallsGlobalCSSVars(theme.sidebarBg);

Conversation

streamer45 commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Screenshots

Related PRs

Ticket Link

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

streamer45 Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

streamer45 commented Mar 25, 2025

Uh oh!

matthewbirtch commented Mar 27, 2025

Uh oh!

streamer45 commented Mar 27, 2025

Uh oh!

cpoile left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mattermost-build commented Aug 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

streamer45 commented Mar 24, 2025 •

edited

Loading

streamer45 Mar 24, 2025 •

edited

Loading