Gaussian Projection with the Unscented Transform

[fwilliams]

Summary

This PR implements and hardens the Unscented Transform (UT) variant of Gaussian projection, and adds comprehensive native unit tests.

Key highlights:

• UT Gaussian projection implementation (GaussianProjectionUT.{h,cu}) with shared-memory camera parameters.
• Camera model refactor: consolidate OpenCV distortion + intrinsics into OpenCVCameraModel, including a constructor-based initialization path.
• New camera model support: CameraModel::ORTHOGRAPHIC (no perspective divide; distortion is not supported for orthographic).
• Correctness fixes in the UT projection path:
  • Split “depth-valid” vs “in-image” semantics so behind-camera points don’t leave outputs uninitialized.
  • Hard reject a Gaussian if any UT sigma point is behind the camera.
  • Rolling-shutter consistency: for RollingShutterType::NONE, depth/visibility decisions use the start pose (u=0.0) instead of the center pose.
  • Avoid undefined behavior when mNumDistortionCoeffs == 0 (e.g. coeff tensor shape [C,0]): pass nullptr instead of forming &nullptr[0].
• Numerical robustness: stabilize rotationMatrixToQuaternion for degenerate inputs.
• Documentation: add/refresh Doxygen docstrings and comments for readability.

Tests

Added/updated GTests covering:

• B + D policy (separating behind-camera vs out-of-bounds and hard-rejecting behind-camera sigma points)
• Rolling shutter depth pose selection
• Multi-camera per-camera intrinsics + distortion coefficients
• [C,0] distortion coeff tensors for pinhole projection (null coeff pointer path)
• Quaternion conversion robustness on degenerate rotation matrices

Test plan

• conda run -n fvdb ./build.sh install verbose -C cmake.define.FVDB_BUILD_TESTS=ON
• conda run -n fvdb ./build.sh ctest

[harrism]

What issues does this close?

[Copilot]

Pull request overview

Implements 3D Gaussian → 2D screen-space projection using an Unscented Transform (UT), including OpenCV-style distortion support and a new test suite.

Changes:

• Added a UT-based CUDA projection path (GaussianProjectionUT.*) with OpenCV distortion variants and rolling-shutter handling.
• Extended gsplat math utilities with rotation-matrix↔quaternion conversion and pose interpolation helpers.
• Added unit tests for the UT projection and new Gaussian math utilities; wired them into the test CMake.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/fvdb/detail/ops/gsplat/GaussianProjectionUT.h	Declares UT projection API, distortion models, and UT parameters.
src/fvdb/detail/ops/gsplat/GaussianProjectionUT.cu	Implements UT projection kernel, distortion application, sigma point generation, and CUDA dispatch.
src/fvdb/detail/ops/gsplat/GaussianUtils.cuh	Adds quaternion/pose utilities used by UT projection and tests.
src/fvdb/detail/utils/AccessorHelpers.cuh	Adjusts tensor accessor includes/types for C++/CUDA compatibility.
src/tests/GaussianProjectionUTTest.cpp	Adds UT projection correctness tests (pinhole + OpenCV distortion cases).
src/tests/GaussianUtilsTest.cu	Adds tests for new quaternion/pose math helpers.
src/tests/CMakeLists.txt	Registers the new UT and utils tests.
src/fvdb/detail/ops/gsplat/GaussianProjectionForward.cu	Minor constructor cleanup (init-list only).
src/fvdb/detail/GridBatchImpl.cu	Adds casts to avoid size/type mismatches in checks.
src/CMakeLists.txt	Adds the new UT CUDA source file to the build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/fvdb/detail/GridBatchImpl.cu:1344

• GridBatchImpl::deserializeV0 trusts the numGrids and totalBytes fields from the serialized header without fully validating that the metadata region fits inside the provided tensor, which can lead to out-of-bounds reads on attacker-controlled input. If numGrids is large or totalBytes is inconsistent with the implied metadata size, the gridMetadata[i] loop and subsequent memcpy(buf.data(), gridBuffer, sizeofGrid) can read past the end of the serialized buffer and copy adjacent process memory into the grid handle, potentially exposing sensitive data when the grid is later re-serialized or inspected. To fix this, add strict bounds checks that serialized.numel() (and header->totalBytes) are at least sizeof(V01Header) + sizeof(GridBatchMetadata) + numGrids * sizeof(GridMetadata) before accessing batchMetadata, gridMetadata[i], or computing sizeofGrid, and reject inputs where these invariants do not hold.

    TORCH_CHECK(serialized.scalar_type() == torch::kInt8, "Serialized data must be of type int8");
    TORCH_CHECK(serialized.numel() >= static_cast<int64_t>(sizeof(V01Header)),
                "Serialized data is too small to be a valid grid handle");

    const int8_t *serializedPtr = serialized.data_ptr<int8_t>();

    const V01Header *header = reinterpret_cast<const V01Header *>(serializedPtr);
    TORCH_CHECK(header->magic == 0x0F0F0F0F0F0F0F0F,
                "Serialized data is not a valid grid handle. Bad magic.");
    TORCH_CHECK(header->version == 0, "Serialized data is not a valid grid handle. Bad version.");
    TORCH_CHECK(static_cast<uint64_t>(serialized.numel()) == header->totalBytes,
                "Serialized data is not a valid grid handle. Bad total bytes.");

    const uint64_t numGrids = header->numGrids;

    const GridBatchMetadata *batchMetadata =
        reinterpret_cast<const GridBatchMetadata *>(serializedPtr + sizeof(V01Header));
    TORCH_CHECK(batchMetadata->version == 1,
                "Serialized data is not a valid grid handle. Bad batch metadata version.");

    const GridMetadata *gridMetadata = reinterpret_cast<const GridMetadata *>(
        serializedPtr + sizeof(V01Header) + sizeof(GridBatchMetadata));
    for (uint64_t i = 0; i < numGrids; i += 1) {
        TORCH_CHECK(gridMetadata[i].version == 1,
                    "Serialized data is not a valid grid handle. Bad grid metadata version.");
    }
    const int8_t *gridBuffer = serializedPtr + sizeof(V01Header) + sizeof(GridBatchMetadata) +
                               numGrids * sizeof(GridMetadata);

    const uint64_t sizeofMetadata =
        sizeof(V01Header) + sizeof(GridBatchMetadata) + numGrids * sizeof(GridMetadata);
    const uint64_t sizeofGrid = header->totalBytes - sizeofMetadata;

    auto buf = TorchDeviceBuffer(sizeofGrid, torch::kCPU);
    memcpy(buf.data(), gridBuffer, sizeofGrid);

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[harrism]

@fwilliams This looks great the added documentation really helps me follow the code. I made a number of suggestions around the design regarding pointers vs. reference / array parameters. I think these could make the code more robust.

[harrism]

Nice, thanks for the cleanup!

[harrism]

BTW you don't need to commit in order to rerun CI. Go to the failed workflow and click "rerun failed jobs" (little circle arrows) in the top right.