sqlite

Author: Karsten1987

articles/rosbags.md

@@ -86,14 +86,15 @@ data (in hex): 0x00 0x01 0x00 0x00 0x0f 0x00 0x00 0x00 0x68 0x65 0x6c 0x6c 0x6f
 Given the API proposed in the above section, we need a data storage format which allows to sufficiently store and replay this serialized data.
 There are a few requirements for writing to such a data storage format:
 
-- Scalability: The chosen format has to scale up to multiple terabytes of file size
-- Parallelism: It must be possible to write to the file from multiple instances at the same time
+- Scalability: The chosen format has to scale up to multiple terabytes of file size.
+- Parallelism: It must be possible to write to the file from multiple instances at the same time.
 - Compression (might be relaxed): Ideally, when the file size becomes larger, it should be possible to compress the file for long time archiving.
 
 In terms of reading from the file, there are similar requirements:
 
-- Random access: It must be possible to grant random read access to the file and extract specific individual messages
-- Range access: It further should be possible to only replay/read a section of the record, specified by a range of time
+- Random access: It must be possible to grant random read access to the file and extract specific individual messages.
+- Range access: It further should be possible to only replay/read a section of the record, specified by a range of time.
+- Chunk sizes: The chunk sizes must be configurable to fit various large message types.
 
 A general requirement is to be backwards compatible with existing ROS1 bags.
 This compatibility can either be via a conversion script, which permanently converts ROS1 bags into ROS2 bags or a bridge API which allows to manually open existing ROS1 bags and publish them into the ROS2 system.
@@ -105,12 +106,44 @@ In the following, we are iterating over a couple of data formats, which may be s
 We hereby iterate over existing third party software as well as examining of maintaining a self-made format.
 
 
-## HDF5
+### HDF5
 
 One very popular framework for storing scientific data is [HDF5](https://support.hdfgroup.org/HDF5/).
 It basically has all the necessary requirements listed above such as random access, parallelism and compression.
 It is further designed for highly complex data with an extensive amount of data.
 HDF5 is open source and its source can be freely obtained from [bitbucket](https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse) and is under a [permissive license](https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse/COPYING) of the HDF group.
 Multiple language wrapper or bindings are available, namely c/c++, python, fortran or java.
 
-## Existing ROS1 format
+#### Pros
+- Open Source and standard specification for file format
+- Fullfils the requirements given
+- Multi language support
+- Large community of users
+
+#### Cons
+- Depending on a slowly developing standard
+
+There is a popular [blog post](http://cyrille.rossant.net/moving-away-hdf5/) by Cyrille Rossant, which gives a short introduction, but also discusses some controversy with HDF5.
+
+
+### Existing ROS1 format
+
+Alternatively, the existing ROS1 format can be continued to be used. The format description can be found [here](http://wiki.ros.org/Bags/Format/2.0).
+
+#### Pros
+- Already ROS specific and evaluated for ROS messages, existing code could be reused
+
+#### Cons
+- No random access
+
+
+### SQLite
+
+A third alternative which provides capabilities for data logging, and is used as such, is [SQLite](https://www.sqlite.org/about.html)
+Despite other relational database systems, it doesn't require any external SQL Server, but is self-contained.
+It is also open source, [extensively tested](https://www.sqlite.org/testing.html) and known to have a large [community](https://www.sqlite.org/support.html).
+The Gazebo team created a nicely composed [comparison](https://osrfoundation.atlassian.net/wiki/spaces/GAZ/pages/99844178/SQLite3+Proposal) between SQLite and the existing rosbag format.
+
+#### Pros
+
+#### Cons