Reproduce Tab. 1 of your paper

[abhi1kumar]

Hi authors,

Thank you for releasing the code and models for your work; it is greatly appreciated by the community. I am currently reproducing the results of your Table 1 and would be grateful if you could clarify some information regarding the evaluation dataset used for those metrics to ensure my setup is correct.

Specifically, I would need the following details to make your results fully reproducible:

• Video IDs for Movement Subsets: Could you please share the specific video IDs that constitute the small movement and large movement subsets of your evaluation set?

• Dataset Split Confirmation: Since the evaluation section doesn't specify a split, could you please confirm if your 100 evaluation videos were drawn exclusively from the test split of the RealEstate10K dataset?

PS: I did check the provided Google Drive link mentioned in the Inference section of your README. However, since it only contains 8 videos instead of the 100 used for evaluation, I suspect the full list of IDs is not located there.

Thank you again for your time and assistance!

[skygoo2000]

Hi,

Thank you for your interest in our work and for reaching out. We apologize for the inconvenience, but due to a recent server cleanup, we have lost the specific video IDs that were used to generate the metrics for Table 1. The videos available on the Google Drive link are examples only and do not constitute the full test set. You should be able to get similar results by using any 100 videos from the RealEstate10K dataset.

To clarify the evaluation setup:

• Small movement: The test examples are from RealEstate10K, and their camera trajectories use the dataset's ground truth.
• Large movement: The test examples are also from RealEstate10K, but their camera trajectories were manually specified. You can approximate this behavior by setting the camera trajectory to "spiral" in demo.py. Please note that the open-source code is slightly different from the version we used for our initial testing, so the numerical results may not be an exact match. We suggest you manually specify a camera trajectory that mimics the spiral's behavior.

We hope this information helps with your reproduction efforts.

[abhi1kumar]

Thank @skygoo2000 for your prompt and detailed reply.

We apologize for the inconvenience, but due to a recent server cleanup, we have lost the specific video IDs that were used to generate the metrics for Table 1.

The open-source code is slightly different from the version we used for our initial testing, so the numerical results may not be an exact match.

It is sad that the video IDs were lost. I also understand the slight differences between the open-source code and the initial version. So, I will do my best to reproduce your results for Table 1. To ensure the best possible reproduction, I would be grateful if you could clarify a few more points regarding the data and the simulation setup:

• Frame Selection: For the small and large movement subsets, did you evaluate performance on the entire video duration, or did you select specific frames from the videos?

• Dataset Size: What was the split between small movement and large movement videos that you used to construct the random 100 video data?

• Video Uniqueness: Since you were randomly sampling the RealEstate10K videos, can the same video ID appear in both the small movement subset and the large movement subset, or should the videos be mutually exclusive?

• Movement Definition: What is the precise definition or numerical threshold (e.g., a specific value for camera rotation or translation) that you used to distinguish between a "small movement" and a "large movement"?

• Spiral Radius: Your suggestion to manually specify a camera trajectory that mimics a spiral is very helpful. Your demo.py uses a spiral with a radius of argument 2. Should we use this radius of 2 for all videos when simulating the trajectory, or did you vary the radius based on the content or scale of different videos?

I look forward to your insights on these important details. Once again, a sincere thank you for both releasing your code and for your continued assistance in helping me with the reproduction efforts.