Replies: 1 comment
-
The Postgres connector is single threaded for reads. How much throughput does a SELECT * over JDBC give you on the source database? I'd assume it's 50MBps that you area observing. In other words how much time does it take to do a |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I am exporting data from PostgreSQL to another database using the following query:
INSERT INTO db_name.public.example_table (id, name, content, boolean, timestamp) SELECT id, name, content, CAST(boolean AS tinyint), CAST(timestamp AS timestamp(0)) FROM postgresql.public.example_table;
However, the transfer speed is always limited to around 50 MB per second. Here are the details of my setup:
Ping between hosts: 0.6 ms
Network bandwidth: 1 Gbps
PostgreSQL server specs: Intel Cascade Lake, 8 CPUs, 8 GB RAM
Target database server specs: Similar to PostgreSQL server.
Trino server: Intel Cascade Lake 16 CPU 32 GB RAM.
I have set the following configurations in Trino:
SET SESSION task_writer_count = 128; SET SESSION task_concurrency = 128; SET SESSION query_max_memory = '10GB'; SET SESSION query_max_total_memory = '10GB'; SET SESSION query_max_memory_per_node = '4GB';
Despite these settings, the transfer speed does not improve. I am 100% sure the issue is not with the target database, as I have tried multiple different setups. Additionally, during the export process, resource utilization never exceeds 50%. What could be causing this bottleneck and how can I optimize the data transfer speed?"
Beta Was this translation helpful? Give feedback.
All reactions