Virtual Background - Add temporal smoothing to reduce segmentation flicker

[solomon17705cs]

What problem are you trying to solve?

The virtual background processes each video frame independently, without referencing previous frames. This lack of temporal consistency causes visible artifacts during motion:

• Jitter: When raising a hand or moving quickly, the segmentation mask fluctuates frame-to-frame, creating a "ghostly" or "shimmering" effects
• Flicker on Small Movements: Subtle movements cause the mask to alternately classify pixels as foreground/background, resulting in a strobing effect where the user and background appear to toggle rapidly.

The segmentation model outputs a per-frame alpha mask with no temporal smoothing. Small variations in inference (body movements, due to lighting, or model uncertainty) manifest as visible jitter when composited.

What solution would you like to see?

Implementation of lightweight temporal smoothing by blending the current frame's alpha mask with the previous frame's mask would reduce the flickering

Is there an alternative?

No response

[solomon17705cs]

I'd like to take this on

[saghul]

Go for it!

[solomon17705cs]

The segmentation mask alpha is currently written directly per frame:

```javascript
this._segmentationMask.data[(i * 4) + 3] = 255 * person;

Reference.mov

I tested this with a temporal smoothing approach that blends the current frame's alpha with the previous frame's alpha:

const newAlpha = 255 * person;
const prevAlpha = this._prevMaskData[(i * 4) + 3];
this._segmentationMask.data[(i * 4) + 3] = 0.5 * newAlpha + 0.5 * prevAlpha;

This reduces flicker while keeping the mask responsive.

Before preparing a PR, I wanted to ask for feedback on the smoothing factor. For example:

Note: i used an image from my mobile to show it here

• 0.5 * new + 0.5 * prev → More Smoothing. i see a white space in quick body movements. also sometimes it is not producing consistent output

5.5.mov

• 0.7 * new + 0.3 * prev → almost current-frame dominant. not much lag or white space, also flickering is reduced

Screen.Recording.2026-03-05.at.10.19.38.AM.mov

Happy to make the PR based on your guidance.

Thanks!

[saghul]

Oh, exciting! How about waving a hand over? How does that behave?

[solomon17705cs]

Yeah, By now after implementing the temporal smoothing, the jittering is reduced. But still its not so perfect as I expected.
Still seeing slight ghosting on fast motion, some times jittering.
I'll show you the result of the current implementation.

Old:

Old.mp4

New: With the smoothing applied (0.7 * new + 0.3 * prev)

New.mp4

• flicker is reduced
• looks more stable

[solomon17705cs]

I experimented with a few post-processing approaches to improve the mask edges.

I tried adjusting the alpha mapping and confidence processing:

1. Smoothstep alpha mapping (3t² - 2t³) to create smoother edge transitions.
2. Confidence threshold tuning
   • Original: 0.1 – 0.9
   • Tested: 0.05 – 0.95
3. Edge confidence boost by multiplying confident scores by 1.1.

However, these approaches still produced large visible box-like pixels around the segmentation edges. Also there was a notable gap between the Background and the foreground.

Then experimented with some mask post-processing:

1. Eroding the mask by - 2px before blurring
2. Applying Gaussian blur to the mask

This improved the foreground and background slightly and produced a cleaner separation, but the output still shows noticeable blocky artifacts, likely due to the low segmentation resolution.

And i think the main limitation might be the model's 256×144 segmentation mask resolution, which becomes visible when upscaled to the full video frame.

I wanted to ask whether this is mostly an inherent limitation of the current segmentation resolution/model

Happy to experiment further if there are specific directions that would be useful.

[saghul]

Good research!

My understanding is that, yeah, the currently limiation is due to the segmentation model we use. I think experimenting with MediaPipe and a WebGL pipeline is worth a shot, that's one of the projects for GSoC this year we hope to get a student for :-)

[solomon17705cs]

Thanks!

Yeah, Experimenting with a MediaPipe-based segmentation model and a WebGL pipeline sounds really interesting. I’ll take some time to explore how MediaPipe’s segmentation performs compared to the current model and how it might integrate into the existing pipeline.

And, I'd actually love to take this on as a GSoC project