Partial copy data from InfluxDB2 to CrateDB

[ZillKhan]

Hello @amotl

Thank you for providing this patch. #182
I have tested this and it is working without getting a timeout error in 10sec.

But running this command to get data directly from InfluxDB2 is partial. It doesn't fetch the complete record.
The total record available is around 83K, but crateDB fetches only 46 records.

influxio copy \
  "http://bedm:MyToken@192.168.200.63:8087/demmel/sys_alarmlogging" \
  "crate://192.168.200.55:4200/doc/sys_alarmlogging"

[amotl]

Hi @ZillKhan,

thanks for reporting this. Well, receiving 46 records out of 83K is certainly underwhelming. We will look into this issue right away.

With kind regards,
Andreas.

[amotl]

Hi again,

those commands exercise importing 5288 records, originally from an ILP file. ¹

# Copy from ILP to InfluxDB.
influxio copy \
  "https://github.com/influxdata/influxdb2-sample-data/raw/refs/heads/master/air-sensor-data/air-sensor-data.lp" \
  "http://example:token@localhost:8086/testdrive" \

# Copy from InfluxDB to CrateDB.
influxio copy \
  "http://example:token@localhost:8086/testdrive/airSensors" \
  "crate://localhost:4200/testdrive/airSensors?if-exists=replace"

crash -c 'select count(*) from testdrive."airSensors";'

CONNECT OK
+-------+
| count |
+-------+
|  5288 |
+-------+

This at least demonstrates the machinery can import more than just 46 records. 🍀

With kind regards,
Andreas.

Footnotes

1. https://github.com/influxdata/influxdb2-sample-data/blob/master/air-sensor-data/air-sensor-data.lp ↩

[amotl]

How do you think we could proceed here? Could you provide your dataset, or a subset thereof, big enough it can be used to reproduce the problem you are observing? Please reach out at andreas.motl [at] crate.io if this would be an option for you.

[amotl]

Hi Zill, and a happy new year! We just released influxio v0.0.7, and hope it will work better out of the box for production data like yours.