Reconstruction Error Evaluation

[yulgi]

HI,
Thank you for your excellent work. Recently I run your code to reproduce the experiment results.
But I found ATE metric is very good in Replica datasets, but the reconstruction error is quite large. And I found codes in these lines are inconsistent with the method described in paper.

MUTE_SLAM/src/networks/decoders.py

Lines 77 to 82 in 14b2983

	for submap in submap_list:
	pts_mask = torch.logical_and((pts[..., :] > submap.boundary[0]).all(dim=-1),
	(pts[..., :] < submap.boundary[1]).all(dim=-1))
	pts_mask = torch.logical_and(pts_mask, torch.logical_xor(pre_mask, pts_mask))
	#pre_mask = torch.logical_or(pre_mask, pts_mask)
	pre_mask = pts_mask

Therefore I makes pre_mask = torch.logical_or(pre_mask, pts_mask) , and the reconstruction error performs better but not enough. The result of reconstruction error evaluation in Replica office0 scene is not well and is showed below .

python src/tools/eval_recon.py --rec_mesh $OUTPUT_FOLDER/mesh/final_mesh_eval_rec_culled.ply --gt_mesh $GT_MESH -2d -3d
accuracy:  2.4982285506690265
completion:  1.2146457996535176
completion ratio:  99.44733333333333
Depth L1:  1.6115719452500343

I am confused and want to know what factors caused such a result. Could you please help me check the result and figure out the reason?

[lumennYan]

HI, Thank you for your excellent work. Recently I run your code to reproduce the experiment results. But I found ATE metric is very good in Replica datasets, but the reconstruction error is quite large. And I found codes in these lines are inconsistent with the method described in paper.

MUTE_SLAM/src/networks/decoders.py

Lines 77 to 82 in 14b2983

for submap in submap_list:
pts_mask = torch.logical_and((pts[..., :] > submap.boundary[0]).all(dim=-1),
(pts[..., :] < submap.boundary[1]).all(dim=-1))
pts_mask = torch.logical_and(pts_mask, torch.logical_xor(pre_mask, pts_mask))
#pre_mask = torch.logical_or(pre_mask, pts_mask)
pre_mask = pts_mask

Therefore I makes pre_mask = torch.logical_or(pre_mask, pts_mask) , and the reconstruction error performs better but not enough. The result of reconstruction error evaluation in Replica office0 scene is not well and is showed below .

python src/tools/eval_recon.py --rec_mesh $OUTPUT_FOLDER/mesh/final_mesh_eval_rec_culled.ply --gt_mesh $GT_MESH -2d -3d
accuracy: 2.4982285506690265
completion: 1.2146457996535176
completion ratio: 99.44733333333333
Depth L1: 1.6115719452500343
I am confused and want to know what factors caused such a result. Could you please help me check the result and figure out the reason?

Thank you for your question! The code here is to extract features for currently observed points, thus I traverse all submaps. For the point features in the overlapped region, they are already extracted and stored in the c_feat_all，so I use pre_mask to exclude them from other submaps. pts_mask = torch.logical_and(pts_mask, torch.logical_xor(pre_mask, pts_mask)) I think here only the points in pts_mask and are not in pre_mask are used. For the result of reconstruction error, I may have modified the code several times after the paper was written to enhance the robustness on real world scenarios, where the degration might come from. But it is a long time ago and it takes sometime to find out. I'll manage to find time to try to determine what caused this. Thank you again for your interest on this work!

[yulgi]

Thank you for your reply! I found that "only the oldest map is used when points fall into the area where multiple sub-maps overlap" in the paper. But line 82 in below codes makes pre_mask = pts_mask, which makes pre_mask could not mask the points that have used in older map correctly.

MUTE_SLAM/src/networks/decoders.py

Lines 77 to 82 in 14b2983

	for submap in submap_list:
	pts_mask = torch.logical_and((pts[..., :] > submap.boundary[0]).all(dim=-1),
	(pts[..., :] < submap.boundary[1]).all(dim=-1))
	pts_mask = torch.logical_and(pts_mask, torch.logical_xor(pre_mask, pts_mask))
	#pre_mask = torch.logical_or(pre_mask, pts_mask)
	pre_mask = pts_mask

For the degradation of reconstruction error result, I found it is caused by the line 168 in the below codes. Line 168 gets bound from submaps and it may be too large for the reconstruction error evaluation. It leads to degradation of reconstruction error evaluation though the visualization of reconstruction mesh looks great. After I use the marching_cube_bound to replace the line 168 as eslam used in get_grid_uniform(). The reconstruction error results in Replica scenes get better and is approximately consistent with the paper. Thanks for your excellent work.

MUTE_SLAM/src/utils/Mesher.py

Lines 158 to 170 in 14b2983

	def get_grid_uniform(self, resolution, submap_list):
	"""
	Get query point coordinates for marching cubes.
	Args:
	resolution (int): marching cubes resolution.
	Returns:
	(dict): points coordinates and sampled coordinates for each axis.
	"""
	bound = self.get_bound_from_submaps(submap_list) * self.scale
	bound = bound.cpu()
	padding = 0.05

[lumennYan]

Thank you for your reply! I found that "only the oldest map is used when points fall into the area where multiple sub-maps overlap" in the paper. But line 82 in below codes makes pre_mask = pts_mask, which makes pre_mask could not mask the points that have used in older map correctly.

MUTE_SLAM/src/networks/decoders.py

Lines 77 to 82 in 14b2983

for submap in submap_list:
pts_mask = torch.logical_and((pts[..., :] > submap.boundary[0]).all(dim=-1),
(pts[..., :] < submap.boundary[1]).all(dim=-1))
pts_mask = torch.logical_and(pts_mask, torch.logical_xor(pre_mask, pts_mask))
#pre_mask = torch.logical_or(pre_mask, pts_mask)
pre_mask = pts_mask
For the degradation of reconstruction error result, I found it is caused by the line 168 in the below codes. Line 168 gets bound from submaps and it may be too large for the reconstruction error evaluation. It leads to degradation of reconstruction error evaluation though the visualization of reconstruction mesh looks great. After I use the marching_cube_bound to replace the line 168 as eslam used in get_grid_uniform(). The reconstruction error results in Replica scenes get better and is approximately consistent with the paper. Thanks for your excellent work.

MUTE_SLAM/src/utils/Mesher.py

Lines 158 to 170 in 14b2983

 def get_grid_uniform(self, resolution, submap_list): 
     """ 
     Get query point coordinates for marching cubes. 

     Args: 
         resolution (int): marching cubes resolution. 

     Returns: 
         (dict): points coordinates and sampled coordinates for each axis. 
     """ 
     bound = self.get_bound_from_submaps(submap_list) * self.scale 
     bound = bound.cpu() 
     padding = 0.05

Yes, you're right, a logical_or like line 81 should be correct. It seems marching cubes introduce more reconstruction error with a larger bound, given the same TSDF field. I think those error may come from the border between the seen and unseen region of a submap. I'll try to figure out how to fix this.