Add masks support to all Gaussian render methods

[swahtz]

Add masks and backgrounds support to all Gaussian splatting render methods

Closes #469

Summary

• Plumb masks and backgrounds through all render paths. The masks (per-tile [C, tileH, tileW] bool) and backgrounds (per-camera color) optional parameters were only supported on a subset of render methods. This PR adds them to all dense, sparse, and jagged render functions across the full stack: CUDA dispatch, autograd, C++ API, Python bindings, and Python wrappers.

• Expose a per-pixel mask API on dense render methods. Users now pass a per-pixel boolean mask of shape [C, H, W] to the 5 dense render methods (render_images, render_depths, render_images_and_depths, render_images_from_world, render_from_projected_gaussians). Internally, the Python wrapper derives a conservative per-tile mask via F.max_pool2d (a tile is rendered if any of its pixels are unmasked) and passes it to the C++ kernel for the existing tile-level early-out optimization. After rendering, a differentiable per-pixel mask is applied to the output so masked pixels receive the background color and zero alpha, with correct gradient flow.

• Add comprehensive tests. A new TestGaussianRenderMasks test class covers all render paths: all-ones mask matches unmasked output, all-zeros mask produces background, backward pass with masks produces nonzero gradients, and all-zeros mask produces zero gradients. Tests cover dense (per-pixel masks), sparse (per-tile masks), and jagged (per-tile masks) paths.

Test plan

python -m pytest tests/unit/test_gaussian_splat_3d.py -v -k TestGaussianRenderMasks

Verifies:

• render_images with all-ones pixel mask matches no-mask output
• render_images with all-zeros pixel mask produces background
• render_images backward with masks produces nonzero gradients
• render_images with all-zeros mask produces zero gradients
• render_depths with all-zeros pixel mask produces background
• render_images_and_depths with all-zeros pixel mask produces background
• render_from_projected_gaussians with all-ones and all-zeros pixel masks
• sparse_render_images with tile masks and backgrounds
• sparse_render_images backward with tile masks
• sparse_render_depths with tile masks and backgrounds
• sparse_render_images_and_depths with tile masks and backgrounds
• gaussian_render_jagged with all-ones and all-zeros tile masks

[Copilot]

Pull request overview

This PR adds comprehensive masks and backgrounds support to all Gaussian splatting render methods, enabling per-pixel masking for dense renders and per-tile masking for sparse/jagged renders. Previously, only render_images_from_world supported these parameters.

Changes:

• Extended all render APIs (dense, sparse, jagged) to accept optional masks and backgrounds parameters
• Implemented a two-stage masking approach for dense methods: conservative per-tile masking for early-out optimization, followed by precise per-pixel masking
• Added comprehensive test coverage verifying correct rendering behavior and gradient flow with masks

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit/test_gaussian_splat_3d.py	Added 13 new tests covering all render paths with masks: dense (pixel masks), sparse (tile masks), and jagged (tile masks)
src/python/GaussianSplatBinding.cpp	Added masks parameter bindings to 8 render methods (dense, sparse, jagged)
src/fvdb/detail/ops/gsplat/GaussianRasterizeForward.h	Added masks parameter to sparse forward dispatch signature
src/fvdb/detail/ops/gsplat/GaussianRasterizeForward.cu	Threaded masks through sparse forward dispatch implementations (CUDA, CPU, PrivateUse1)
src/fvdb/detail/ops/gsplat/GaussianRasterizeBackward.h	Added masks parameter to both dense and sparse backward dispatch signatures
src/fvdb/detail/ops/gsplat/GaussianRasterizeBackward.cu	Threaded masks through all backward dispatch implementations
src/fvdb/detail/autograd/GaussianRasterizeSparse.h	Added masks parameter to sparse rasterize autograd forward signature
src/fvdb/detail/autograd/GaussianRasterizeSparse.cpp	Implemented save/restore for optional masks and backgrounds in sparse autograd
src/fvdb/detail/autograd/GaussianRasterize.h	Added masks parameter to dense rasterize autograd forward signature
src/fvdb/detail/autograd/GaussianRasterize.cpp	Implemented save/restore for optional masks and backgrounds in dense autograd
src/fvdb/GaussianSplat3d.h	Added masks parameter to all public render methods and internal implementation methods
src/fvdb/GaussianSplat3d.cpp	Threaded masks through all render implementations; removed unused renderImpl method
fvdb/gaussian_splatting.py	Added helper functions for mask conversion and application; updated all render methods to support per-pixel masks (dense) or per-tile masks (sparse)
fvdb/__init__.py	Added masks parameter to gaussian_render_jagged wrapper

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.