Alternatives to large COCO JSON and DistributedDataParallel

[austinmw]

Hi, I have a very large object detection dataset. Currently I'm using COCO JSON annotation format, but the annotation json file is 80GB. When I try to train on a 8xV100 instance with 400GB RAM, it tries to load the full dataset in each distributed GPU process and runs out of memory. So COCO is obviously the wrong annotation format for large datasets.

Is Detectron2 compatible with an annotation format for very large datasets or is this not supported?