Parse multi GB list of jsons

[adriansev]

Hi! I'm trying to parse a 60+ GB file with the structure of [{}....{}]
I'm not able to figure out how to use ijson for this .. does anyone have some pointers for me?
Thanks a lot!!

[rtobar]

@adriansev assuming you want to loop through the individual objects inside the top-level array you'd want to do:

# f is your file/socket/etc.... that is already opened
for element in ijson.items(f, "item"):
    # do something with element

The "Prefix" section in the documentation explains how the prefix works.

On the command line you can also inspect what the prefix looks at different parts of the parsing process, which can help you determine what prefix you want to use:

$> echo '[{"a": 1}, {"a": 2}]' | python -mijson.dump -m parse
#: path, name, value
--------------------
0: , start_array, None
1: item, start_map, None
2: item, map_key, a
3: item.a, number, 1
4: item, end_map, None
5: item, start_map, None
6: item, map_key, a
7: item.a, number, 2
8: item, end_map, None
9: , end_array, None

[adriansev]

Thanks a lot! It worked without problem!!