How to export the point cloud (like .ply file) at inference step?

[zydmtaichi]

I take a look to given introduction, there is only png photo exportion at inference step. Is it possible to convert and export 3D mesh file by using SceneDreamer?

[zydmtaichi]

hi @FrozenBurning,
I notice the output for inference step contains a .npy file. and project nvdiffrast can take the npz file as input to render 3d.
Is it a applicable operation to convert npy as npz then feed to concern script of nvdiffrast?

the nvdiffrast project link pasted below:
https://github.com/NVlabs/nvdiffrast

[FrozenBurning]

Thanks for your interest in our work. Some possible ways to get 3D mesh by using our model: 1) run Marching Cubes at the output of our volumetric renderer (

SceneDreamer/imaginaire/generators/scenedreamer.py

Line 373 in bf635c7

weights = mc_utils.volum_rendering_relu(net_out_s, new_dists * self.dists_scale, dim=-2)

) just like any of NeRF-like method (related material shown here); 2) run poisson reconstruction using our output images;

It would be great if you could specify the .npy file you've mentioned. AFAIK, we do not save .npy for 3D representation at inference time. Thanks!

[zydmtaichi]

@FrozenBurning, glad to see your reply, per method 1 you mentioned, can we regard the variable weights in line 373 of scenedreamer.py as the variable sigma in extract_mesh.ipynb of this project?
And there is a style.npy file in output per instruction, I'm not sure if this .npy can be used to extract mesh,refer to below file structures:
├── ... └── test └── camera_{:02d} # camera mode for trajectory ├── rgb_render # per frame RGB renderings ├── 00000.png ├── 00001.png └── ... ├── rgb_render.mp4 # rendered video ├── height_map.png # height map ├── semantic_map.png # semantic map └── style.npy # sampled style code

[FrozenBurning]

I feel like we should use net_out_s:

SceneDreamer/imaginaire/generators/scenedreamer.py

Line 365 in bf635c7

net_out_s, net_out_c = self._forward_perpix_sub(blk_feats, worldcoord2, raydirs_in, z, mc_masks_onehot, global_enc)

Regarding the style.npy you've mentioned, they are style codes that are dumped for reproducing the same style in case you want to sample the same style with different scene layouts. Therefore, they are not capable for mesh extraction.

[zydmtaichi]

I feel like we should use net_out_s:

SceneDreamer/imaginaire/generators/scenedreamer.py

Line 365 in bf635c7

net_out_s, net_out_c = self._forward_perpix_sub(blk_feats, worldcoord2, raydirs_in, z, mc_masks_onehot, global_enc)

Regarding the style.npy you've mentioned, they are style codes that are dumped for reproducing the same style in case you want to sample the same style with different scene layouts. Therefore, they are not capable for mesh extraction.

thanks a lot, i will give it a try after i get enough hardware resource to run.

[zydmtaichi]

hey @FrozenBurning ,
I'm back for more questions, per your intro, the net_out_s can be used to reconstruct mesh. I check the code and find the net_out_s is opacity data.
In my opinion, the point cloud data should be required for constructing mesh like using create_from_point_cloud_alpha_shape in open3d lib.
I have some difficulty in converting a opacity data to mesh. please give me more detailed guidance, thanks

[zydmtaichi]

BTW, I also try to perform the poisson reconstruction using output images per your method 2, but the recon process using img also require RGBD dataset(per open3d intro) while our output only have the RGB img and miss the depth one.