Skip to content

plumb sparse rendering functions #348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fwilliams merged 23 commits into from
Dec 9, 2025
Merged

plumb sparse rendering functions #348

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
23c99dd
plumb sparse rendering function
fwilliams Nov 20, 2025
fab048d
unskip tests
fwilliams Nov 20, 2025
6eb8434
Update fvdb/gaussian_splatting.py
fwilliams Nov 20, 2025
570cf95
Update src/fvdb/GaussianSplat3d.cpp
fwilliams Nov 20, 2025
881672b
Oops
fwilliams Nov 20, 2025
1af8db7
Merge branch 'main' into fw/expose_sparse_gs
fwilliams Nov 20, 2025
d426130
fix tests
fwilliams Nov 20, 2025
8161b7d
rename
fwilliams Nov 20, 2025
486843b
Update fvdb/gaussian_splatting.py
fwilliams Nov 20, 2025
ac6c041
Update fvdb/gaussian_splatting.py
fwilliams Nov 20, 2025
aa0a83a
Update fvdb/gaussian_splatting.py
fwilliams Nov 20, 2025
ae4613c
fix naming
fwilliams Nov 20, 2025
b16eb20
rm synchronize and sh grads
fwilliams Nov 20, 2025
406d3d2
rm commented
fwilliams Nov 20, 2025
676d9aa
typevar is named like a typevar
fwilliams Nov 20, 2025
399fef6
revert naming changes. Will move to another PR
fwilliams Nov 21, 2025
cb43e4d
renames
fwilliams Nov 21, 2025
42f7e9e
more rename fix
fwilliams Nov 21, 2025
d0f1a51
Merge branch 'main' into fw/expose_sparse_gs
fwilliams Nov 21, 2025
c7e1e56
fix
fwilliams Nov 21, 2025
01b68c7
privateuse1
fwilliams Nov 21, 2025
d0d826f
revert names
fwilliams Dec 3, 2025
6e1f08a
Merge branch 'main' into fw/expose_sparse_gs
fwilliams Dec 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions fvdb/__init__.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ __all__ = [
"GaussianSplat3d",
"ProjectedGaussianSplats",
"ConvolutionPlan",
"ProjectionType",
"ShOrderingMode",
"Grid",
# JaggedTensor operations
"jcat",
Expand Down
47 changes: 47 additions & 0 deletions fvdb/_fvdb_cpp.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,21 @@ class GaussianSplat3d:
antialias: bool = ...,
backgrounds: Optional[torch.Tensor] = ...,
) -> tuple[torch.Tensor, torch.Tensor]: ...
def sparse_render_depths(
self,
pixels_to_render: JaggedTensor,
world_to_camera_matrices: torch.Tensor,
projection_matrices: torch.Tensor,
image_width: int,
image_height: int,
near: float,
far: float,
projection_type: ProjectionType = ...,
tile_size: int = ...,
min_radius_2d: float = ...,
eps_2d: float = ...,
antialias: bool = ...,
) -> tuple[JaggedTensor, JaggedTensor]: ...
def render_from_projected_gaussians(
self,
projected_gaussians: ProjectedGaussianSplats,
Expand All @@ -193,6 +208,22 @@ class GaussianSplat3d:
antialias: bool = ...,
backgrounds: Optional[torch.Tensor] = ...,
) -> tuple[torch.Tensor, torch.Tensor]: ...
def sparse_render_images(
self,
pixels_to_render: JaggedTensor,
world_to_camera_matrices: torch.Tensor,
projection_matrices: torch.Tensor,
image_width: int,
image_height: int,
near: float,
far: float,
projection_type: ProjectionType = ...,
sh_degree_to_use: int = ...,
tile_size: int = ...,
min_radius_2d: float = ...,
eps_2d: float = ...,
antialias: bool = ...,
) -> tuple[JaggedTensor, JaggedTensor]: ...
def render_images_and_depths(
self,
world_to_camera_matrices: torch.Tensor,
Expand All @@ -209,6 +240,22 @@ class GaussianSplat3d:
antialias: bool = ...,
backgrounds: Optional[torch.Tensor] = ...,
) -> tuple[torch.Tensor, torch.Tensor]: ...
def sparse_render_images_and_depths(
self,
pixels_to_render: JaggedTensor,
world_to_camera_matrices: torch.Tensor,
projection_matrices: torch.Tensor,
image_width: int,
image_height: int,
near: float,
far: float,
projection_type: ProjectionType = ...,
sh_degree_to_use: int = ...,
tile_size: int = ...,
min_radius_2d: float = ...,
eps_2d: float = ...,
antialias: bool = ...,
) -> tuple[JaggedTensor, JaggedTensor]: ...
def render_num_contributing_gaussians(
self,
world_to_camera_matrices: torch.Tensor,
Expand Down
314 changes: 301 additions & 13 deletions fvdb/gaussian_splatting.py
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion src/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,10 @@ set(FVDB_CPP_FILES
fvdb/detail/autograd/Attention.cpp
fvdb/detail/autograd/AvgPoolGrid.cpp
fvdb/detail/autograd/EvaluateSphericalHarmonics.cpp
fvdb/detail/autograd/GaussianRender.cpp
fvdb/detail/autograd/Inject.cpp
fvdb/detail/autograd/GaussianProjection.cpp
fvdb/detail/autograd/GaussianRasterize.cpp
fvdb/detail/autograd/GaussianRasterizeSparse.cpp
fvdb/detail/autograd/JaggedReduce.cpp
fvdb/detail/autograd/MaxPoolGrid.cpp
fvdb/detail/autograd/ReadFromDense.cpp
Expand Down
131 changes: 130 additions & 1 deletion src/fvdb/GaussianSplat3d.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@
// SPDX-License-Identifier: Apache-2.0
//
#include <fvdb/GaussianSplat3d.h>
#include <fvdb/detail/autograd/GaussianRasterizeSparse.h>
#include <fvdb/detail/io/GaussianPlyIO.h>
#include <fvdb/detail/utils/Nvtx.h>

// Autograd headers
#include <fvdb/detail/autograd/EvaluateSphericalHarmonics.h>
#include <fvdb/detail/autograd/GaussianRender.h>
#include <fvdb/detail/autograd/GaussianProjection.h>
#include <fvdb/detail/autograd/GaussianRasterize.h>

// Ops headers
#include <fvdb/detail/ops/gsplat/GaussianRasterizeContributingGaussianIds.h>
Expand Down Expand Up @@ -443,6 +445,47 @@ GaussianSplat3d::renderImpl(const torch::Tensor &worldToCameraMatrices,
state, settings.tileSize, settings.imageWidth, settings.imageHeight, 0, 0);
}

std::tuple<JaggedTensor, JaggedTensor>
GaussianSplat3d::sparseRenderImpl(const JaggedTensor &pixelsToRender,
const torch::Tensor &worldToCameraMatrices,
const torch::Tensor &projectionMatrices,
const fvdb::detail::ops::RenderSettings &settings) {
FVDB_FUNC_RANGE();

const ProjectedGaussianSplats &state =
projectGaussiansImpl(worldToCameraMatrices, projectionMatrices, settings);

const auto [activeTiles, activeTileMask, tilePixelMask, tilePixelCumsum, pixelMap] =
fvdb::detail::ops::computeSparseInfo(settings.tileSize,
state.tileOffsets.size(2),
state.tileOffsets.size(1),
pixelsToRender);

auto rasterizeResult =
detail::autograd::RasterizeGaussiansToPixelsSparse::apply(pixelsToRender,
state.perGaussian2dMean,
state.perGaussianConic,
state.perGaussianRenderQuantity,
state.perGaussianOpacity,
settings.imageWidth,
settings.imageHeight,
0,
0,
settings.tileSize,
state.tileOffsets,
state.tileGaussianIds,
activeTiles,
tilePixelMask,
tilePixelCumsum,
pixelMap,
false);
auto renderedPixelsJData = rasterizeResult[0];

auto renderedAlphasJData = rasterizeResult[1];
return {pixelsToRender.jagged_like(renderedPixelsJData),
pixelsToRender.jagged_like(renderedAlphasJData)};
}

std::tuple<torch::Tensor, torch::Tensor>
GaussianSplat3d::renderNumContributingGaussiansImpl(
const torch::Tensor &worldToCameraMatrices,
Expand Down Expand Up @@ -778,6 +821,92 @@ GaussianSplat3d::renderNumContributingGaussians(const torch::Tensor &worldToCame
return renderNumContributingGaussiansImpl(worldToCameraMatrices, projectionMatrices, settings);
}

std::tuple<JaggedTensor, JaggedTensor>
GaussianSplat3d::sparseRenderDepths(const fvdb::JaggedTensor &pixelsToRender,
const torch::Tensor &worldToCameraMatrices,
const torch::Tensor &projectionMatrices,
const size_t imageWidth,
const size_t imageHeight,
const float near,
const float far,
const ProjectionType projectionType,
const size_t tileSize,
const float minRadius2d,
const float eps2d,
const bool antialias) {
RenderSettings settings;
settings.imageWidth = imageWidth;
settings.imageHeight = imageHeight;
settings.nearPlane = near;
settings.farPlane = far;
settings.projectionType = projectionType;
settings.shDegreeToUse = 0;
settings.tileSize = tileSize;
settings.radiusClip = minRadius2d;
settings.eps2d = eps2d;
settings.renderMode = RenderSettings::RenderMode::DEPTH;

return sparseRenderImpl(pixelsToRender, worldToCameraMatrices, projectionMatrices, settings);
}

std::tuple<JaggedTensor, JaggedTensor>
GaussianSplat3d::sparseRenderImages(const fvdb::JaggedTensor &pixelsToRender,
const torch::Tensor &worldToCameraMatrices,
const torch::Tensor &projectionMatrices,
const size_t imageWidth,
const size_t imageHeight,
const float near,
const float far,
const ProjectionType projectionType,
const int64_t shDegreeToUse,
const size_t tileSize,
const float minRadius2d,
const float eps2d,
const bool antialias) {
RenderSettings settings;
settings.imageWidth = imageWidth;
settings.imageHeight = imageHeight;
settings.nearPlane = near;
settings.farPlane = far;
settings.projectionType = projectionType;
settings.shDegreeToUse = shDegreeToUse;
settings.tileSize = tileSize;
settings.radiusClip = minRadius2d;
settings.eps2d = eps2d;
settings.renderMode = RenderSettings::RenderMode::RGB;

return sparseRenderImpl(pixelsToRender, worldToCameraMatrices, projectionMatrices, settings);
}

std::tuple<JaggedTensor, JaggedTensor>
GaussianSplat3d::sparseRenderImagesAndDepths(const fvdb::JaggedTensor &pixelsToRender,
const torch::Tensor &worldToCameraMatrices,
const torch::Tensor &projectionMatrices,
const size_t imageWidth,
const size_t imageHeight,
const float near,
const float far,
const ProjectionType projectionType,
const int64_t shDegreeToUse,
const size_t tileSize,
const float minRadius2d,
const float eps2d,
const bool antialias) {
RenderSettings settings;
settings.imageWidth = imageWidth;
settings.imageHeight = imageHeight;
settings.nearPlane = near;
settings.farPlane = far;
settings.projectionType = projectionType;
settings.shDegreeToUse = shDegreeToUse;
settings.tileSize = tileSize;
settings.radiusClip = minRadius2d;
settings.eps2d = eps2d;
settings.renderMode = RenderSettings::RenderMode::RGBD;

return sparseRenderImpl(pixelsToRender, worldToCameraMatrices, projectionMatrices, settings);
}

std::tuple<fvdb::JaggedTensor, fvdb::JaggedTensor>
GaussianSplat3d::sparseRenderNumContributingGaussians(const fvdb::JaggedTensor &pixelsToRender,
const torch::Tensor &worldToCameraMatrices,
Expand Down
64 changes: 61 additions & 3 deletions src/fvdb/GaussianSplat3d.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -934,6 +934,50 @@ class GaussianSplat3d {
const bool antialias = false,
const std::optional<torch::Tensor> &backgrounds = std::nullopt);

std::tuple<JaggedTensor, JaggedTensor>
sparseRenderImages(const fvdb::JaggedTensor &pixelsToRender,
const torch::Tensor &worldToCameraMatrices,
const torch::Tensor &projectionMatrices,
const size_t imageWidth,
const size_t imageHeight,
const float near,
const float far,
const ProjectionType projectionType = ProjectionType::PERSPECTIVE,
const int64_t shDegreeToUse = -1,
const size_t tileSize = 16,
const float minRadius2d = 0.0,
const float eps2d = 0.3,
const bool antialias = false);

std::tuple<JaggedTensor, JaggedTensor>
sparseRenderDepths(const fvdb::JaggedTensor &pixelsToRender,
const torch::Tensor &worldToCameraMatrices,
const torch::Tensor &projectionMatrices,
const size_t imageWidth,
const size_t imageHeight,
const float near,
const float far,
const ProjectionType projectionType = ProjectionType::PERSPECTIVE,
const size_t tileSize = 16,
const float minRadius2d = 0.0,
const float eps2d = 0.3,
const bool antialias = false);

std::tuple<JaggedTensor, JaggedTensor>
sparseRenderImagesAndDepths(const fvdb::JaggedTensor &pixelsToRender,
const torch::Tensor &worldToCameraMatrices,
const torch::Tensor &projectionMatrices,
const size_t imageWidth,
const size_t imageHeight,
const float near,
const float far,
const ProjectionType projectionType = ProjectionType::PERSPECTIVE,
const int64_t shDegreeToUse = -1,
const size_t tileSize = 16,
const float minRadius2d = 0.0,
const float eps2d = 0.3,
const bool antialias = false);

/// @brief Render the number of contributing Gaussians for each pixel in the image.
/// @param worldToCameraMatrices [C, 4, 4] Camera-to-world matrices
/// @param projectionMatrices [C, 4, 4] Projection matrices
Expand Down Expand Up @@ -1241,9 +1285,9 @@ class GaussianSplat3d {
/// The mask must have the same length as the number of Gaussians in this scene.
GaussianSplat3d tensorIndexGetImpl(const torch::Tensor &indexOrMask) const;

/// @brief Render the gaussian splatting scene
/// This function returns a single render quantity (RGB, depth, RGB+D) and
/// single alpha value per pixel.
/// @brief Render the scene described by the Gaussian splats from the specified views.
/// This function returns a single render quantity (RGB, depth, RGB+D) and
/// single alpha value per pixel.
/// @param worldToCameraMatrices [C, 4, 4]
/// @param projectionMatrices [C, 3, 3]
/// @param settings
Expand All @@ -1253,6 +1297,20 @@ class GaussianSplat3d {
const torch::Tensor &projectionMatrices,
const fvdb::detail::ops::RenderSettings &settings);

/// @brief Render the scene described by the Gaussian splats at the specified pixels in the
/// specified views. This function returns a single render quantity (RGB, depth, RGB+D)
/// and single alpha value per pixel.
/// @param pixelsToRender [P1 + P2 + ..., 2] JaggedTensor of pixels per camera to render.
/// @param worldToCameraMatrices [C, 4, 4]
/// @param projectionMatrices [C, 3, 3]
/// @param settings
/// @return Tuple of (render quantity, alpha value)
std::tuple<JaggedTensor, JaggedTensor>
sparseRenderImpl(const JaggedTensor &pixelsToRender,
const torch::Tensor &worldToCameraMatrices,
const torch::Tensor &projectionMatrices,
const fvdb::detail::ops::RenderSettings &settings);

/// @brief Render the number of contributing Gaussians for each pixel in the image.
/// @param worldToCameraMatrices [C, 4, 4]
/// @param projectionMatrices [C, 3, 3]
Expand Down
Loading
Loading