Mesh -> Gaussian splats

[jitesh3023]

Is there a way to use the reconstructed mesh to create Gaussian splatting? I believe that using the images captured during mesh scanning may not be ideal for training Gaussians. During mesh scanning, we tend to scan meticulously by moving closer to objects, whereas for Gaussian splatting, it is better to scan from a distance to capture more information in a single view. This wider coverage helps improve overlap detection when running COLMAP. Is there any way to generate Gaussian splatting from the data collected during mesh scanning? I tried using the camera poses and RGB images from db file for training gaussians, but the results were bad. I would appreciate it if anyone knows a method to achieve this or is aware of any open-source repository that supports this workflow.

[matlabbe]

I didn't try myself running Gaussian Splatting on databases taken by rtabmap. Based on this, it seems requiring the RGB images, the camera poses, the camera calibrations and sparse 3D features:

<location>
|---images
|   |---<image 0>
|   |---<image 1>
|   |---...
|---sparse
    |---0
        |---cameras.bin
        |---images.bin
        |---points3D.bin

I am not sure how to generate these bin files though. If you know a reference for that, can you share it?

[elidiot18]

Hi, all variants of Gaussian Splatting accept COLMAP workspaces with pinhole camera models. There are two formats for COLMAP workspaces: txt and bin. The original 3dgs, for example, accepts both, but converts txt to bin during the first run.

You can run 3dgs from an RTAB database as follows:

• from the RTAB GUI, export as Bundler (.out) keeping 3D points
• convert the Bundler export into a COLMAP workspace. One way to do that is to use Kapture (which can be installed with pip), in which case you can first convert Bundler to Kapture, and then Kapture to COLMAP
• if you do as above, you will get a COLMAP workspace in txt, with radial camera model in cameras.txt. If there is nontrivial distorsion, you will have to undistort the images first, and then update the cameras.txt using pinhole model instead of radial (radial is not accepted for Gaussian splatting). If the distorsion is always 0, you can do a simple regex to replace "RADIAL" with "PINHOLE", remove the last 3 distorsion coeffs (all equal to 0), and duplicating the focal (because PINHOLE requires fx and fy). I believe that 3dgs also accepts SIMPLE_PINHOLE (to be checked) in which case you don't have to duplicate the focal.
• once the cameras.txt is modified you can run 3dgs, and it will convert the *.txt to *.bin because that's what they work with.

Best
Pierre

[jitesh3023]

Thank you so much @elidiot18 for the detailed workflow and clear explanation — I will definitely give it a try.

I just had a quick question: have you tried using RGB images captured during mesh scanning with RTAB-Map for Gaussian Splatting? I'm unsure if they provide enough viewpoint diversity and parallax for high-quality GS results. Since mesh scanning typically involves slower, close-range motion, I’m concerned it might impact the reconstruction quality. If you have tried this, I’d love to know how the results turned out.

[elidiot18]

Yes I have tried, and I would say that it depends how thorough you were while scanning in the first place. If you scan quickly in order to get an ok mesh, the images will be too blury and probably not good enough for a perfect result. But it depends on the camera you're using. Also you cannot expect to magically reconstruct all details if you didn't take pictures of them from different angles.
Depends strongly on your scene and how you scanned.

The results I had were pretty nice but also not "perfect", but that can be due to the fact that I'm working on dark environments. I'm still experimenting with it right now.

[jitesh3023]

Thanks for sharing! I’ll try doing this anyways—thanks for the insight.