TinyETL: Investigate `Failed to insert batch: error returned from database`

[amotl]

Backlog

Investigate and dispatch this error accordingly. Is it originating from TinyETL or from CrateDB?

Problem

When attempting to import the file churn-dataset.csv into CrateDB, the procedure fails.

tinyetl \
  "https://cdn2.crate.io/downloads/datasets/cratedb-datasets/machine-learning/automl/churn-dataset.csv" \
  "postgresql://crate:crate@localhost/testdrive#churn"

ERROR tinyetl: Transfer failed: Data transfer error: Failed to insert batch: error returned from database:
readerIndex: 1315907160, writerIndex: 215335 (expected: 0 <= readerIndex <= writerIndex <= capacity(215335))

References

• TinyETL: Basic example (import CSV and Parquet, export to DuckDB) #1285 (review)

[amotl]

When attempting to import the file into PostgreSQL, the procedure also fails, but with a different error.

docker run --rm -it --publish=5432:5432 \
  --env "POSTGRES_HOST_AUTH_METHOD=trust" postgres:18 \
  postgres -c log_statement=all

tinyetl   "https://cdn2.crate.io/downloads/datasets/cratedb-datasets/machine-learning/automl/churn-dataset.csv"   "postgresql://postgres:postgres@localhost/postgres#churn"

ERROR tinyetl: Transfer failed: Data transfer error: Failed to insert batch: error returned from database: invalid byte sequence for encoding "UTF8": 0x00

[amotl]

On the other hand, converting to Avro, DuckDB, JSON, Parquet, and SQLite seems to work well.

tinyetl \
  "https://cdn2.crate.io/downloads/datasets/cratedb-datasets/machine-learning/automl/churn-dataset.csv" \
  "churn-dataset.avro"

tinyetl \
  "https://cdn2.crate.io/downloads/datasets/cratedb-datasets/machine-learning/automl/churn-dataset.csv" \
  "churn-dataset.duckdb"

tinyetl \
  "https://cdn2.crate.io/downloads/datasets/cratedb-datasets/machine-learning/automl/churn-dataset.csv" \
  "churn-dataset.json"

tinyetl \
  "https://cdn2.crate.io/downloads/datasets/cratedb-datasets/machine-learning/automl/churn-dataset.csv" \
  "churn-dataset.parquet"

tinyetl \
  "https://cdn2.crate.io/downloads/datasets/cratedb-datasets/machine-learning/automl/churn-dataset.csv" \
  "churn-dataset.db"

[amotl]

I've reported the problem to the upstream repository. /cc @matriv

• Import of particular CSV file fails for PostgreSQL, but works for file and embedded db outputs alrpal/TinyETL#24