Data formats (examples of many different ways) #55
Description
In our garage today, an idea came up: take one dataset, put it in as many different forms as possible:
- apache arrow for speed of reading
- hdf5 for similar
- csv for archival
- sqlite3 for incrementally updating and querying
- column-orineted database
- zip/tar file
- many small files
- etc
and then discuss the advantages and disadvantages of each of them. maybe profile them. We had a long discussion about different data formats, and having these kind of examples could make things concrete.