Path Configuration Clarification for DataLoader

[Lzy-dot]

Thank you for your excellent work! I have some questions regarding the dataloader.

1. In demo_dataloader.py, paths like “demo_result/step0/videos/dataset1_cluster” and “demo_result/step5/final_output/dataset1_cluster/foreground” are used. Where do these map in the downloaded dataset directory?
2. When using total1-total13 data, should we: Pass only corresponding paths in DataLoader(like video_base_paths = ["OpenS2V-5M/Videos/total_part1" ])? and set is_cross_frame=False? If is_cross_frame=True, does this add extra cross-frame data?
3. Could you kindly share a ​​minimal configuration example​​ for using only the 0.35M cross-frame data? (if convenient)?
4. Purpose of background path? (Given S2V tasks typically shouldn't require background images)

[sunjian2015]

The same question, and how to use cross_video_pairs? I don't understand how to find the video by "cur_id" or "aft_id" in those json files?

[SHYuanBest]

Thank you for your excellent work! I have some questions regarding the dataloader.

1. In demo_dataloader.py, paths like “demo_result/step0/videos/dataset1_cluster” and “demo_result/step5/final_output/dataset1_cluster/foreground” are used. Where do these map in the downloaded dataset directory?

@Lzy-dot Thanks for your interest!

---

1. mask_and_bbox_json_paths refers to the path_of_mask_and_bbox_total_partX.json; video_base_paths refers to the path_of_Videos_total_partX; and background_base_paths refers to the path_of_Background_total_partX.

2. Yes, you only need to set video_base_paths, mask_and_bbox_json_paths, and background_base_paths to total_part1. Also, there's no need to set is_cross_frame to False. Regardless of how many parts are being used as input, the paths cross_frames_cluster_path and cross_frames_base_path should remain unchanged, as they store cross-frame information across all 13 parts—not just within a single part. If you'd like to use cross-frame data while inputting just one part, you can add a filtering mechanism in the dataloader to exclude keys from other parts in the files corresponding to cross_frames_cluster_path and cross_frames_base_path.

3. I haven't tried training with only the 0.35M cross-frame data yet. This 0.35M cross-frame dataset is sourced from parts 1 through 13, so to use it, you simply need to input parts 1 to 13 together. (Admittedly, the current method is a bit brute-force, since it also loads non-cross data.) Of course, you can modify the dataloader to filter out non-cross data based on cross_frames_cluster_path and cross_frames_base_path, so that only the cross-frame portions from parts 1 to 13 are loaded.

4. The "background" is used for the Object Insertion task in S2V. For more details, you can refer to Figure 1 in [OpenS2V-Nexus](https://arxiv.org/pdf/2505.20292), as well as the showcases in [Phantom](https://phantom-video.github.io/Phantom/) and [VACE](https://ali-vilab.github.io/VACE-Page/).

---

1、mask_and_bbox_json_paths指代path_of_mask_and_bbox_total_partX.json；video_base_paths 指代；path_of_Videos_total_partX；background_base_paths指代path_of_Background_total_partX；

2、是的，只需要保持video_base_paths、mask_and_bbox_json_paths和background_base_paths都是输入total_part1即可；此外，is_cross_frame不需要设置为False，无论输入多少个part，cross_frames_cluster_path和cross_frames_base_path的路径始终保持不变，因为其记录着13个part的cross信息，不是只在part内做cross；如果你想在输入一个part的同时利用上cross data，你可以在dataloader中加个逻辑，把其他part的key从cross_frames_cluster_path和cross_frames_base_path对应的文件中剔除。

3、我还没尝试过只使用0.35M cross-frame data进行训练。0.35M cross-frame data来源于part1~part13，使用时只需要同时输入part1~part13即可（我承认现在方法有点暴力，因为这会把非cross data也加载进来）。当然你也可以修改dataloader，加载时根据cross_frames_cluster_path和cross_frames_base_path剔除非cross data，只加载part1~part13中的cross data部分。

4、“background”的作用在于完成S2V tasks中的Object Insertion任务，具体可以参考OpenS2V-Nexus的Figure 1以及Phantom和VACE的showcases。

---

[SHYuanBest]

The same question, and how to use cross_video_pairs? I don't understand how to find the video by "cur_id" or "aft_id" in those json files?

@sunjian2015 Sorry for the confusion.

1. For how to use cross_video_pairs, you can refer to [R2 and R3](Path Configuration Clarification for DataLoader #16 (comment)).

2. To understand how to obtain cur_id and aft_id, you can check out this [code](https://github.com/PKU-YuanGroup/OpenS2V-Nexus/blob/main/data_process/step6-4_get_gpt-frame.py).

[Lzy-dot]

@SHYuanBest 。感谢回复，我把video_base_paths、mask_and_bbox_json_paths和background_base_paths都设置为total_part1的时候，出现报错：No such file or directory:'/OpenS2V-5M/Background/total_part1/part1/aKAianF3-zQ_segment_2_step1-166-511_step2-166-345_step4_step5_step6.png'。看起来似乎background/total_part1/part1里面的文件和mask_and_bbox/total_part1.json中的key没有对应。（把下载的total_part1.tar解压后是一个part1的文件夹里面存储了各种背景图片，所以这里的路径是**/total_part1/part1）

[SHYuanBest]

是只有这张图片不存在吗？还是说所有？前者可能是传的时候文件损坏了，后者大概率是路径错误。

[SHYuanBest]

这图片里面只下载了part1，应该都存在的，只不过有些损坏

[Lzy-dot]

这图片里面只下载了part1，应该都存在的，只不过有些损坏

Background/total_part1/part1下的文件似乎都是step7结尾（是处理的时候标错名字了嘛），是不是需要把step6换成step7呢

[SHYuanBest]

这图片里面只下载了part1，应该都存在的，只不过有些损坏

Background/total_part1/part1下的文件似乎都是step7结尾（是处理的时候标错名字了嘛），是不是需要把step6换成step7呢

我们的锅...是的，得改成step7

[SHYuanBest]

We have fixed background image paths by change "step7" to "step6", and have update here.