docs: Improved section describing the protocol

Author: amunra

BUILD.md

@@ -37,7 +37,7 @@ Visual Studio Code should also work well provided you have the "C/C++",
 
 ## Build outputs
 
-The build will generate libraries compiled to `.build/`
+The build will generate libraries compiled to `./build`
 (or your otherwise selected build directory).
 
 You will find one dynamic library and depending on

README.md

@@ -1,4 +1,5 @@
-# QuestDB - Line Protocol - Ingestion Client Library for C and C++
+# c-questdb-client
+**QuestDB - InfluxDB Line Protocol - Ingestion Client Library for C and C++**
 
 This library makes it easy to insert data into [QuestDB](https://questdb.io/).
 
@@ -16,7 +17,7 @@ to get notified of new releases.
 
 ## Protocol
 
-This client library implements the input line protocol (ILP) over TCP.
+This client library implements the InfluxDB Line Protocol (ILP) over TCP.
 
 To understand the benefits and limitations of using the protocol, you may want
 to consult the
@@ -27,24 +28,27 @@ it can't report errors that happen in the server as the protocol is not designed
 to report them.
 
 For example, the client library will report that a supplied string isn't encoded
-in UTF-8, but will not report that a column previously used to record `BOOLEAN`
-data in a previous row is used in a later row to record a `STRING`.
+in UTF-8, but will not report that a column previously used to record a
+`BOOLEAN` is later incorrectly used to record a `STRING`.
 
-Unreported errors such as this one usually result in the row not being written
-and skipped silently to the client.
+Unreported errors such as this one usually result in skipped rows that don't
+get saved.
 
-Other errors, such as the database running out of disk space or crashing, mean
-that.
+Other errors, such as the database running out of disk space or crashing will
+result in a TCP connection drop, but also uncertainty from the client's
+perspective on which records were written.
 
 With these caveats out of the way, you can however expect well-formed data to be
-written so long as it has been received by the database.
+written so long as it has been received by the database, caveat bugs and system
+errors.
 
-The input line protocol is currently the fastest way to insert data into
+The line protocol is currently the fastest way to insert data into
 QuestDB, but if the lack of confirmation of written data is a concern to you,
 you may want to consider alternatives such as inserting via CSV uploads or
-through the PostgreSQL interface.
+through the PostgreSQL interface. Those alternatives are slower but more robust.
+The choice of input mechanism will depend on your use case.
 
-When using the input line protocol, data will be written in the order it is
+When using the line protocol, data will be written in the order it is
 sent, unless you configured the
 [out-of-order feature](https://questdb.io/docs/guides/out-of-order-commit-lag/#how-to-configure-out-of-order-ingestion)
 in QuestDB which will reorder records based on timestamp for a given insertion
@@ -64,13 +68,40 @@ silently resulting in rows not getting inserted.
   `,`, `'`, `"`, `\`, `/`, `:`, `(`, `)`, `+`, `-`, `*`, `%`, `~`,
   `' '` (space), `\0` (nul terminator),
   [ZERO WIDTH NO-BREAK SPACE](https://unicode-explorer.com/c/FEFF).
+* Each row should contain, *in order*:
+  * table name
+  * at least one of:
+    * symbols, zero or more
+    * columns, zero or more
+  * timestamp, optionally
 
 ### Non-validated rules
 
-TODO: document me.
+The following is a non-exhaustive of guidelines to follow:
 
-* Different types across rows.
-* Reused column names and symbols.
+* For a given row, a column name should not be repeated.
+  If it's repeated, only the first value will be kept.
+  This also applies to symbols.
+* Values for a given colum should always have the same type.
+  If changing types the whole row will be dropped (unless we can cast).
+* If supplying timestamps these need to be at least equal to
+  previous ones in the same table, unless using the out of order
+  feature. Such rows would be dropped.
+* The timestamp column should be written out through the provided
+  `line_sender_at` function (in C) or or `.at()` method in (C++).
+  It is also possible to write out additional timestamps values
+  as columns.
+
+*Refer to the
+[protocol reference docs](https://questdb.io/docs/reference/api/ilp/overview/)
+for details and more usage guidelines.*
+
+### Data type conversions
+
+The ILP protocol has its own set of data types which is smaller
+that the set supported by QuestDB.
+We map these types into QuestDB types and perform conversions
+as necessary wherever possible.
 
 ## Building this library
 
@@ -203,6 +234,19 @@ and you can obtain an error message description by calling `.what()`.
 If you intend to retry, you must create a new sender object: The same sender
 object can't be reused.
 
+## Symbols or Strings
+
+SYMBOLs are strings with which are automatically
+[interned](https://en.wikipedia.org/wiki/String_interning) by the database on a
+per-column basis.
+You should use this type if you expect the string to be re-used over and over.
+This is common for identifiers, etc.
+
+For one-off strings use STRING columns.
+
+For more details see our
+[datatypes](https://questdb.io/docs/reference/sql/datatypes) page.
+
 ## If you don't see any data
 
 You may be experiencing one of these three issues.
@@ -211,7 +255,7 @@ You may be experiencing one of these three issues.
 
 If you can't initially see your data through a `select` query straight away,
 his is normal: by default the database will only commit data it receives
-though the input line protocol periodically to maximize throughput.
+though the line protocol periodically to maximize throughput.
 
 For dev/testing you may want to tune the following database configuration
 parameters as so:

TODO.md

@@ -146,7 +146,7 @@ bool line_sender_name_init(
 /////////// Connecting and disconnecting.
 
 /**
- * Insert data into QuestDB via the input line protocol.
+ * Insert data into QuestDB via the InfluxDB Line Protocol.
  *
  * Batch up rows, then call `line_sender_flush` to send.
  */

include/questdb/line_sender.h

@@ -182,7 +182,7 @@ namespace questdb
     }
 
     /**
-     * Insert data into QuestDB via the input line protocol.
+     * Insert data into QuestDB via the InfluxDB Line Protocol.
      *
      * Batch up rows, then call `.flush()` to send.
      */

include/questdb/line_sender.hpp

@@ -87,7 +87,15 @@ def check_table():
     return retry(check_table, timeout_sec=timeout_sec)
 
 
-class TestSomething(unittest.TestCase):
+def ns_to_qdb_date(at_ts_ns):
+    # We first need to match QuestDB's internal microsecond resolution.
+    at_ts_us = int(at_ts_ns / 1000.0)
+    at_ts_sec = at_ts_us / 1000000.0
+    at_td = datetime.datetime.fromtimestamp(at_ts_sec)
+    return at_td.isoformat() + 'Z'
+
+
+class TestLineSender(unittest.TestCase):
     def test_insert_three_rows(self):
         table_name = uuid.uuid4().hex
         with qls.LineSender('localhost', QDB_FIXTURE.line_tcp_port) as sender:
@@ -244,16 +252,36 @@ def test_at(self):
                 .symbol('a', 'A')
                 .at(at_ts_ns))
 
-        # We first need to match QuestDB's internal microsecond resolution.
-        at_ts_us = int(at_ts_ns / 1000.0)
-        at_ts_sec = at_ts_us / 1000000.0
-        at_td = datetime.datetime.fromtimestamp(at_ts_sec)
-        at_str = at_td.isoformat() + 'Z'
+        resp = retry_check_table(table_name)
+        exp_dataset = [['A', ns_to_qdb_date(at_ts_ns)]]
+        self.assertEqual(resp['dataset'], exp_dataset)
+
+    def test_bad_at(self):
+        if QDB_FIXTURE.version <= (6, 0, 7, 1):
+            self.skipTest('No support for user-provided timestamps.')
+            return
+        table_name = uuid.uuid4().hex
+        at_ts_ns1 = 1648032959100000000
+        at_ts_ns2 = 1648032958100000000  # A second before `at_ts_ns1`.
+        with qls.LineSender('localhost', QDB_FIXTURE.line_tcp_port) as sender:
+            (sender
+                .table(table_name)
+                .symbol('a', 'A')
+                .at(at_ts_ns1))
+            (sender
+                .table(table_name)
+                .symbol('a', 'B')
+                .at(at_ts_ns2))
 
         resp = retry_check_table(table_name)
-        exp_dataset = [['A', at_str]]
+        exp_dataset = [['A', ns_to_qdb_date(at_ts_ns1)]]
         self.assertEqual(resp['dataset'], exp_dataset)
 
+        # The second time stamp is dropped and will not appear in results.
+        with self.assertRaises(TimeoutError):
+            retry_check_table(table_name, min_rows=2, timeout_sec=1)
+
+
     def test_underscores(self):
         table_name = f'_{uuid.uuid4().hex}_'
         with qls.LineSender('localhost', QDB_FIXTURE.line_tcp_port) as sender: