AirByte connection for large file too slow

[jsrgdc]

Hello everybody, please help with the following situation:
I have an AirByte connection already running for 72 hours trying to export a 277GB .tsv data set from an S3 Bucket into PostGress. All, the S3 Bucket, AirByte, and Postgress are installed in the same AWS instance which is powerful enough to process that data including 8 Virtual Processors and 1TB of disk space. The Airbyte connection is set to work with a block size of 10.000, if we try with a lower number the exporting process fails if we try with a higher number the process also fails. On top of that, after 72 hours and having exported 166GB the process failed and Airbyte initiated a new exporting attempt starting from 0. Is it normal that it takes so long to move the data from source to destination even though they al are in the same cloud instance?

[bk521234]

jsrgdc , did you discover the cause of the slowness ?

I've been seeing source-s3 connector being crazy slow syncing a large set of small files to my destination ( snowflake destination ). I wasn't quite sure if the slowness was because of the s3 source connector or if it was because of the destination-snowflake connector. But I'd guess that it's likely because of the source-s3 connector because I've altered my destination config and removed deduplication step from my data sync and things are still extremely slow ( even when the destination does not have to de-duplicate records in snowflake ).

so, seems like the s3 connector is just slow with the default k8s / job configuration.