[Question]: InternData-N1 是否存在数据文件损坏的问题

[XieWeikai]

Question

我把huggingface上InternData-N1的数据都下载下来了，但是很多.tar.gz在解压的时候会碰到类似如下的问题:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
[fail] 00791-JPMDv7zL4bF.tar.gz (exit=2)

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
[fail] 00793-NRsmXFcVTbN.tar.gz (exit=2)

请问是否存在上传数据集文件时文件损坏的情况？另外我在处理数据时发现各个子集的数据似乎并没有完全统一格式？有的是一个目录多个episodes，有的是一个。task有的是纯用字符串来存json，有的不是。是否统一一下会好一些呢？

[XieWeikai]

以下是以hm3d_d435i这个部分为例子失败的案例

00003-NtVbfPCkBFy.tar.gz
00024-XNoaAZwsWKk.tar.gz
00017-oEPjPNSPmzL.tar.gz
00032-jTTGECZYKRA.tar.gz
00046-UQ5EhY5wve1.tar.gz
00088-z9w4aD7JsiQ.tar.gz
00093-HneuS3CYDFG.tar.gz
00098-MVStfLiYYQv.tar.gz
00102-r77mpaAYUEc.tar.gz
00116-xp4FyfQ6Wr5.tar.gz
00134-Mre6deDcPCP.tar.gz
00122-QDtpZSqaeyW.tar.gz
00141-iigzG1rtanx.tar.gz
00157-3PiKdwyfEkX.tar.gz
00198-eZrc5fLTCmi.tar.gz
00227-bvdRzJBgJyg.tar.gz
00236-FcUgaJv6JHA.tar.gz
00273-8DDKELpgD99.tar.gz
00281-mj5sKX44BmS.tar.gz
00274-NcK5aACg44h.tar.gz
00300-GcfUJ79xCZc.tar.gz
00324-DoSbsoo4EAg.tar.gz
00329-asrq4PFvdvF.tar.gz
00366-fxbzYAGkrtm.tar.gz
00373-WPzCkWEorzk.tar.gz
00414-77mMEyxhs44.tar.gz
00415-rBmEe6ab5VP.tar.gz
00467-mggziYKSc6S.tar.gz
00487-erXNfWVjqZ8.tar.gz
00530-C7xtw9uhYFn.tar.gz
00563-Bnq6SeZGL5b.tar.gz
00593-m17UDpW3tHm.tar.gz
00618-T4G9hTR5WSv.tar.gz
00660-Pmv1pdeirDT.tar.gz
00654-VZy9kKQJcUF.tar.gz
00688-SgkmkWjjmDJ.tar.gz
00757-LVgQNuK8vtv.tar.gz
00779-3YSDRj9kTU7.tar.gz
00783-J9adB1bm54A.tar.gz
00791-JPMDv7zL4bF.tar.gz
00793-NRsmXFcVTbN.tar.gz

我是上周下载的，我看最近有更新，不知道是否有修改

比如我截图的情况，这个0 bytes显然有问题吧

[XieWeikai]

关于数据格式的问题 大概是这样，比如下面是episodes.jsonl的一行例子

{"episode_index": 0, "tasks": [{"sub_instruction": "Walk straight ahead, passing the staircase on your left and the white radiator on your right, until you reach the end of the hallway where the doors are. Stop at the last door on your right.", "sub_indexes": [0, 91], "revised_sub_instruction": "Set your sights on the hallway ahead. Glide forward, noting the wooden staircase gently rising to your left and a white radiator humming quietly on your right. When you arrive at the end of this corridor, pause by the final door on your right-hand side."}, {"sub_instruction": "Turn right at the wooden door frame, proceed to the bed with yellow bedding on your left, and stop at the nightstand next to it.", "sub_indexes": [92, 112], "revised_sub_instruction": "Swing a right at that wooden door frame, stroll on until you spot the bed with its sunny yellow sheets on your left, then come to a halt right by the nightstand hugging it."}, {"sub_instruction": "Continue straight ahead, passing the wooden door frame on your left and the white wall on your right, until you reach the bed with the white sheets and wooden headboard. Stop at the foot of the bed.", "sub_indexes": [113, 177], "revised_sub_instruction": "Proceed directly forward, keeping the rustic wooden door frame to your left and the pristine white wall to your right. Walk until you arrive at the bed, characterized by its crisp white sheets and sturdy wooden headboard. Halt when you reach the foot of this bed."}, {"sum_instruction": "Starting at the beginning of the hallway, head forward while keeping the gently rising wooden staircase on your left and the quiet white radiator on your right. Upon reaching the end of the corridor, pause beside the final door on your right-hand side. Turn right at that wooden door frame and continue walking until you encounter a bed with sunny yellow sheets on your left—halt just beside the nightstand next to it. Then proceed directly forward, with the rustic wooden door frame to your left and the pristine white wall on your right. As you advance, you'll soon arrive at another bed, this one marked by crisp white sheets and a sturdy wooden headboard; come to a stop when you reach the foot of this bed.", "sum_indexes": [0, 177]}]}

这里的tasks直接就是很正常的形式，但还有一些如下

{"episode_index": 0, "tasks": ["{\"sum_instruction\": \"Stroll onward, keeping the bed to your right and the textured wall brushing your left. Halt when you meet a gray cabinet with its distinct circular handle right ahead of you.\", \"sum_indexes\": [0, 151]}", "{\"sub_instruction\": \"Walk straight ahead, passing the bed on your right and the textured wall on your left. Stop at the gray cabinet with the circular handle, which is directly in front of you.\", \"sub_indexes\": [0, 151], \"revised_sub_instruction\": \"Stroll onward, keeping the bed to your right and the textured wall brushing your left. Halt when you meet a gray cabinet with its distinct circular handle right ahead of you.\"}"], "length": 304}

这个的tasks里面每一条都是一个纯str，我在处理数据时先是都按照后一种情况处理的，结果处理时碰到了前一种情况，出错了，不知道整个数据集里还有没有其他的情况出现。

[XieWeikai]

另外目录结构上也看到了不太一样的两种组织，不清楚是否有别的情况