Recommendations for retrieving 3D mesh

[kajc10]

Hi, my main goal is to retrieve the underlying geometry of wild scenes, texture is not needed. This implementation produces fantastic results when the task is to render new views, but I haven't managed to export good meshes.

I understand that NeRF focuses on returning good views, not accurate geometry. Could i modify a nerf to get better meshes? E.g. discard color loss, focus on density.

Also, if you know, could you recommend other architecures/techniques that are appropiate for this task? -retrieving 3d geometry of wild scenes, like streets with cars. (I'm mostly looking for options that already have implementations)

[Tom94]

Hello, there are a number of cool approaches to make NeRF reconstructions more plausible. RegNeRF and RefNeRF impose surface priors that are quite good at preventing floaters -- especially when only few training viewpoints are available. Depth supervision in general (e.g. when a depth camera is used) can also help a ton. Another option is to directly optimize for a mesh such as done in https://github.com/NVlabs/nvdiffrec. This latter approach might not scale to scene sizes you're interested in, but probably worth a closer look regardlessly. Cheers!

[kajc10]

Thanks. Another thought regarding the topic: as I understand the voxel grid's color depends on the view dir, however the voxels' densites are only dependent of their position. So, could I extract this point cloud somehow? Without colors, just the points' density values.

[Tom94]

Yes, in the source code, you can call nerf_network->density() on arbitrary 3D positions of your choosing.

[leastbad]

Thank you so much for your amazing work on this, @Tom94. At least on-screen, I'm seeing results far better than my old team was able to achieve with a photogrammetry -> {human ZBrush expert spending 3h} pipeline.

This dataset was captured in a custom-built 70 camera rig with LED panel lighting:

testbed_tlafvy6qW5.mp4

As you can see, it does a remarkable job, once you get past the junk cloud surrounding the subject.

I understand that you're not able to give us an A -> B -> C to a riggable avatar, so I won't ask. 😉

However, one of the things that would get me/us a lot closer to a 👍 is if we could somehow specify a bounding box within the cloud volume so that I could dispel the junk. Is that possible? It seems like this has to be a set of parameters that could be passed in to COLMAP, no?

Is there a way to switch the mouse movement from rotate to drag? Rotate + zoom works, but gimbal lock isn't optimal.

Is there a way to tell testbed to maintain highest quality at the sacrifice of frame rate? I have to shrink the window to a postage stamp to see the subject in high detail; if I maximize the window, it drops the quality. Makes sense as a default, but would be A+ if we can control it.

Happy to share some human datasets if it would be helpful!

[Tom94]

Really cool reconstruction! I suspect that the junk comes from background content of your dataset, which the model can't properly fit because its aabb_scale is set too small. Have you tried using "aabb_scale": 16, in your dataset json?

Other than that:

Is there a way to switch the mouse movement from rotate to drag? Rotate + zoom works, but gimbal lock isn't optimal.

You can hold the middle mouse button and drag to translate.

Is there a way to tell testbed to maintain highest quality at the sacrifice of frame rate?

You can uncheck the "dynamic resolution" checkbox.

However, one of the things that would get me/us a lot closer to a 👍 is if we could somehow specify a bounding box within the cloud volume so that I could dispel the junk. Is that possible? It seems like this has to be a set of parameters that could be passed in to COLMAP, no?

Not to colmap, but you can use the "crop aabb" settings of the GUI to render only a sub-region of the trained model.

[leastbad]

testbed_anP9FyJ73h.mp4

Much better. Thanks Thomas! Amazing tool.

I admit that I am heartbroken that there's no way to export this with textures projected. I guess the issue is that since your pipeline ultimately generates a dense cloud, it doesn't really understand what's a part of the subject so any attempt to create a texture map would include tons of junk.

Still, if it wasn't very clear from the majority of the posts on this forum, if you or others can connect the dots (sorry) you would have a home run on your hands, in terms of people urgently wanting to create full-body, life-like avatars. Nothing you don't already know. 😸