Add gaussianSplatsRaytracing.js

[soundform]

This is a basic raytracer that finds 8 nearest splats per frame, blends them and repeats that until color.a >= 1.0 or there no more splats along the ray. Each splat is represented by an ABC triangle: A = the splat center, BC = its bounding box. While this example uses spherical splats only, the ABC representation allows to deal with elliptical splats, with gaussian bezier arcs or even with gaussian NURBS surfaces.

In the example's UI:

• Bottom right: cost of rendering per pixel.
• Bottom left: pixels not yet rendered.

It's pretty fast on 1-2 million splats scenes:

However the scene below with 16M splats takes ~1 minute to render, while sparkjs.dev renders it at 30-50 fps with much better quality, as its rasterizer includes sub-pixel splats. However a rasterizer cannot do shadows or other lighting effects, while the BVH-based renderer can.

A few things that can be improved:

• The new file src/webgl/glsl/bvh_gsplat_ray_functions.glsl.js can be moved to the example. The idea of this file is to interpret BVH queries as "map reduce" operations: (1) enumerate splats that match a condition, (2) apply a function to each marching splat and (3) use another function to combine the results. Arguably, it's too specific to my use case.
• BVH needs to be updated with the .refit() function when splat parameters are changed.
• Sub-pixel splats can be handled this way: when ray intersection tests are performed, a small margin can be added to BVH boxes and to the splats, the margin corresponding to the pixel size that grows with distance from the camera. When a ray misses the splat, but hits the thus padded splat, its contribution to the pixel color can be added with a certain scale factor.
• A more advanced version of the above: when a sub-pixel BVH box is found to contain too many splats, there is no point in inspecting them all, as they only impact one pixel. Instead, the entire BVH box can be counted as one solid block with a color that equals the sum of all splat colors inside the box.

[gkjohnson]

Thanks for making the PR - I'm starting to see how the BVH can be adjusted to support this kind of splat structure by making the class a bit more generalized. The recent NVidia paper seems to be demonstrated with the help of modern raytracing hardware (though it's not required) so it's likely not possible to match that performance with WebGL but hopefully there'd be a way to get closer in the web. It would also be good to be able to do a more comparison with the open source implementation to get a more direct comparison on the same hardware. There are probably more optimizations in the APIs that aren't being done in this project (yet?) like raybundling, too.

Have you considered using WebGPU and compute shaders for this? Raytracing support was recently added / demonstrated in #756 and gets a bit of a boost from switching to compute shaders alone. This seems like a problem that might benefit from the more flexible API.

[gkjohnson]

Note: Samplers technically are not allowed to be defined within a struct definition. Some devices support this but it will break in other cases.

[soundform]

There is a couple more cosmetic improvements that could be done:

• inverse(modelWorldMatrix) * cameraWorldMatrix should be computed once on the JS side.
• All splats must be scaled up slightly so they cover at least 1 pixel on the screen:

    float z = distance(camera.origin, splat.position);
    splat.radius *= 1.0 + z*eps;
    splat.opacity /= 1.0 + z*eps;

It's also occurred to me that the problem with too many splats congregating in small volumes can be solved by voxelizing those splats before the render pass. The idea is to find BVH nodes that have too many splats in a small volume, voxelize them into a 4x4x4 grid, and save that grid to an auxiliary texture that's passed along with BVH to the shader. BVH nodes, in fact, have 16 bits of unused space that could store an index to the voxel grids:

bool isLeaf = bool( boundsInfo.x & 0xffff0000u ); // only 1 bit is used here

I haven't looked into WebGPU yet, mainly because it hasn't arrived to Firefox ESR on Debian yet.