mismatch in the pointcloud video rendering

[GeleiSG]

Thank you for your excellent work on TTT3R and for making your code and pretrained models available to the community.
Recentlly I encountered a mismatch issue when rendering pointcloud videos. I ran inference on some RealEstate10K videos to obatin the camera and depth information and directly used it to render a pointcloud video based on its first frame with the complete camera trajectory. But the output video have shown a mismatch issue when compared to the gt video. There is a discrepancy in its translation compared to the original.
If you know some possible reasons for this issue, could you please provide some ideas on the potential reasons for this mismatch? If you need more specific information, I can provide them accordingly.
Thank you for your time and for your valuable contributions to the field.

[rover-xingyu]

Hi @GeleiSG, thanks for your interest! To better understand the mismatch issue you encountered in pointcloud rendering, could you please share a visual comparison of the results (GT video vs. rendered video)?

[GeleiSG]

Thanks for your rapid reply! Here is some examples:

0_video_origin_ttt3r_render.mp4

0_video.mp4

1_video_origin_ttt3r_render.mp4

1_video.mp4

Will it be the consequence of inconsistent camera and depth scales? For some other examples the results are ok, but in several cases there seems to be a mismatch issue.

tip: I rendered the videos in 16 fps, 81 frames based on the first frame.

[rover-xingyu]

Thanks for the examples!
It seems the rendering might not cover the full trajectory – for instance, the bed in the GT video doesn't appear in the TTT3R rendering at all. Could you verify if the frame count matches (81 frames of prediction vs. the GT video)?
Regarding scale inconsistency: this shouldn't be an issue since TTT3R predicts both camera and depth from the same network, so their scales are inherently aligned.

[GeleiSG]

Thanks for the reply! That helps eliminate a possible cause.
Here are the gt video and rendered video both in 24fps, 81frames:

0_video_first_81frames_24fps.mp4

0_video_origin_ttt3r_render.mp4

Will it be an adjustment issue? Like the depth of the first frame haven't been fully optimized when saved or for some other reason.

[rover-xingyu]

Glad to hear the suggestion helped! Could you please share the viser visualization results? That would help us determine whether the scene reconstruction was successful.