Download commonpool dataset get stuck

[ffahmed]

Hi,
I am trying to download datacomp small dataset and medium dataset (python download_upstream.py --scale small --data_dir /mnt/video/data/datacomp/commonpool/small/ --resize_mode no --processes_count 64), for both it somehow get stuck towards end. Tried restarting it. It resumes and get stuck.

Last few lines from my log from downloading small dataset when it get stuck

File sharded in 2 shards
Downloading starting now, check your bandwidth speed (with bwm-ng)your cpu (with htop), and your disk usage (with iotop)!
Sharding file number 25 of 26 called /mnt/video/data/datacomp/commonpool/small/metadata/0343f539d27054fb89b80cc6cd232aad.parquet
File sharded in 3 shards
Downloading starting now, check your bandwidth speed (with bwm-ng)your cpu (with htop), and your disk usage (with iotop)!
Sharding file number 26 of 26 called /mnt/video/data/datacomp/commonpool/small/metadata/036d6b9ae87a00e738f8fc554130b65b.parquet
File sharded in 0 shards
Downloading starting now, check your bandwidth speed (with bwm-ng)your cpu (with htop), and your disk usage (with iotop)!