Skip to content

Commit 277c33b

Browse files
authored
ingest benchmark: after effective_io_concurrency = 100 we can increase compute side parallelism (#9904)
## Problem ingest benchmark tests project migration to Neon involving steps - COPY relation data - create indexes - create constraints Previously we used only 4 copy jobs, 4 create index jobs and 7 maintenance workers. After increasing effective_io_concurrency on compute we see that we can sustain more parallelism in the ingest bench ## Summary of changes Increase copy jobs to 8, create index jobs to 8 and maintenance workers to 16
1 parent 2b788cb commit 277c33b

File tree

1 file changed

+5
-5
lines changed

1 file changed

+5
-5
lines changed

test_runner/performance/test_perf_ingest_using_pgcopydb.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -60,13 +60,13 @@ def build_pgcopydb_command(pgcopydb_filter_file: Path, test_output_dir: Path):
6060
"--no-acl",
6161
"--skip-db-properties",
6262
"--table-jobs",
63-
"4",
63+
"8",
6464
"--index-jobs",
65-
"4",
65+
"8",
6666
"--restore-jobs",
67-
"4",
67+
"8",
6868
"--split-tables-larger-than",
69-
"10GB",
69+
"5GB",
7070
"--skip-extensions",
7171
"--use-copy-binary",
7272
"--filters",
@@ -136,7 +136,7 @@ def run_command_and_log_output(command, log_file_path: Path):
136136
"LD_LIBRARY_PATH": f"{os.getenv('PGCOPYDB_LIB_PATH')}:{os.getenv('PG_16_LIB_PATH')}",
137137
"PGCOPYDB_SOURCE_PGURI": cast(str, os.getenv("BENCHMARK_INGEST_SOURCE_CONNSTR")),
138138
"PGCOPYDB_TARGET_PGURI": cast(str, os.getenv("BENCHMARK_INGEST_TARGET_CONNSTR")),
139-
"PGOPTIONS": "-c maintenance_work_mem=8388608 -c max_parallel_maintenance_workers=7",
139+
"PGOPTIONS": "-c maintenance_work_mem=8388608 -c max_parallel_maintenance_workers=16",
140140
}
141141
# Combine the current environment with custom variables
142142
env = os.environ.copy()

0 commit comments

Comments
 (0)