[Data Transfer] memory footprint #164
Description
The Data Transfer service partitions data by class or time. As we poll by batch, this leads to many (potentially small) files. It has been reported that after doing post-processing the data size on disk gets down by a factor of 3 (thanks @bregeon !). We need to inspect to see if the gain comes from the compression or the merge (my guess is compression -- which is uncompress by default).