Skip to content

Conversation

@rafiss
Copy link
Collaborator

@rafiss rafiss commented Jan 7, 2026

Add a new variant of the large-schema-benchmark roachtest that creates 1 million tables and benchmarks introspection query performance. Unlike the existing benchmark variants, this test:

  • Uses minimal data import (warehouses=1) instead of full TPCC data
  • Skips the TPCC workload and runs only ORM introspection queries
  • Tests at a much larger scale (1M tables vs max 40K)
  • Single-region only configuration
  • 48-hour timeout to accommodate the larger scale

This serves as a smoke test to determine how introspection queries perform with 1 million tables, as part of our efforts to improve introspection query performance.

Resolves: #159557
Epic: CRDB-54999

Release note: None

Add a new variant of the large-schema-benchmark roachtest that creates
1 million tables and benchmarks introspection query performance. Unlike
the existing benchmark variants, this test:

- Uses minimal data import (warehouses=1) instead of full TPCC data
- Skips the TPCC workload and runs only ORM introspection queries
- Tests at a much larger scale (1M tables vs max 40K)
- Single-region only configuration
- 48-hour timeout to accommodate the larger scale

This serves as a smoke test to determine how introspection queries
perform with 1 million tables, as part of our efforts to improve
introspection query performance.

Resolves: cockroachdb#159557
Epic: CRDB-54999

Release note: None
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@cockroach-teamcity
Copy link
Member

⚪ Sysbench [SQL, 3node, oltp_read_write]
Metric Old Commit New Commit Delta Note
sec/op 10.80m ±1% 10.84m ±1% ~ p=0.486 n=15
allocs/op 8.020k ±0% 8.042k ±2% ~ p=0.442 n=15
Reproduce

benchdiff binaries:

mkdir -p benchdiff/19a2374/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/19a23749f4e265442c58d8d76bb5a40775fc20c2/bin/pkg_sql_tests benchdiff/19a2374/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/19a2374/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
mkdir -p benchdiff/a214566/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/a214566cc04655e11ff731c95e08c994dc8c64e9/bin/pkg_sql_tests benchdiff/a214566/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/a214566/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests

benchdiff command:

benchdiff --run=^BenchmarkSysbench/SQL/3node/oltp_read_write$ --old=a214566 --new=19a2374 ./pkg/sql/tests
⚪ Sysbench [KV, 3node, oltp_read_only]
Metric Old Commit New Commit Delta Note
sec/op 3.571m ±2% 3.609m ±1% ~ p=0.026 n=15
allocs/op 2.084k ±1% 2.064k ±0% ~ p=0.071 n=15
Reproduce

benchdiff binaries:

mkdir -p benchdiff/19a2374/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/19a23749f4e265442c58d8d76bb5a40775fc20c2/bin/pkg_sql_tests benchdiff/19a2374/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/19a2374/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
mkdir -p benchdiff/a214566/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/a214566cc04655e11ff731c95e08c994dc8c64e9/bin/pkg_sql_tests benchdiff/a214566/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/a214566/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests

benchdiff command:

benchdiff --run=^BenchmarkSysbench/KV/3node/oltp_read_only$ --old=a214566 --new=19a2374 ./pkg/sql/tests
🔴 Sysbench [KV, 3node, oltp_write_only]
Metric Old Commit New Commit Delta Note
🔴 sec/op 3.927m ±0% 3.971m ±1% +1.13% p=0.000 n=15
allocs/op 4.158k ±0% 4.159k ±0% ~ p=0.288 n=15
Reproduce

benchdiff binaries:

mkdir -p benchdiff/19a2374/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/19a23749f4e265442c58d8d76bb5a40775fc20c2/bin/pkg_sql_tests benchdiff/19a2374/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/19a2374/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
mkdir -p benchdiff/a214566/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/a214566cc04655e11ff731c95e08c994dc8c64e9/bin/pkg_sql_tests benchdiff/a214566/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/a214566/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests

benchdiff command:

benchdiff --run=^BenchmarkSysbench/KV/3node/oltp_write_only$ --old=a214566 --new=19a2374 ./pkg/sql/tests
Artifacts

download:

mkdir -p new
gcloud storage cp gs://cockroach-microbench-ci/artifacts/19a23749f4e265442c58d8d76bb5a40775fc20c2/20772861632-1/\* new/
mkdir -p old
gcloud storage cp gs://cockroach-microbench-ci/artifacts/a214566cc04655e11ff731c95e08c994dc8c64e9/20772861632-1/\* old/

built with commit: 19a23749f4e265442c58d8d76bb5a40775fc20c2

@cockroach-teamcity cockroach-teamcity added the X-perf-check Microbenchmarks CI: Added to a PR if a performance regression is detected and should be checked label Jan 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

X-perf-check Microbenchmarks CI: Added to a PR if a performance regression is detected and should be checked

Projects

None yet

Development

Successfully merging this pull request may close these issues.

roachtest: create variant of large-schema-benchmark that tests 1 million tables without a workload

2 participants