Skip to content

Conversation

@gabotechs
Copy link
Collaborator

Reports some relevant data about the environment in which the TPCH benchmarks are running:

==== Comparison with the previous benchmark from 2025-09-08 13:28:37 UTC ====
os:        macos
arch:      aarch64
cpu cores: 16
threads:   2 -> 4
workers:   3 -> 6
=============================================================================
 Query 1: prev= 231 ms, new= 120 ms, diff=1.93 faster ✅
 Query 2: prev= 159 ms, new= 142 ms, diff=1.12 faster ✔
 Query 3: prev=  58 ms, new=  52 ms, diff=1.12 faster ✔
 Query 4: prev= 114 ms, new=  60 ms, diff=1.90 faster ✅
 Query 5: prev= 297 ms, new= 183 ms, diff=1.62 faster ✅
 Query 6: prev=  26 ms, new=  14 ms, diff=1.86 faster ✅
 Query 7: prev= 298 ms, new= 194 ms, diff=1.54 faster ✅
 Query 8: prev= 136 ms, new= 122 ms, diff=1.11 faster ✔
 Query 9: prev= 203 ms, new= 149 ms, diff=1.36 faster ✅
Query 10: prev= 399 ms, new= 313 ms, diff=1.27 faster ✅
Query 11: prev=  72 ms, new=  59 ms, diff=1.22 faster ✅
Query 12: prev= 134 ms, new=  88 ms, diff=1.52 faster ✅
Query 13: prev=  99 ms, new=  70 ms, diff=1.41 faster ✅
Query 14: prev=  77 ms, new=  45 ms, diff=1.71 faster ✅
Query 15: prev=  71 ms, new=  72 ms, diff=1.01 slower ✖
Query 16: prev=  87 ms, new=  57 ms, diff=1.53 faster ✅
Query 17: prev= 211 ms, new= 141 ms, diff=1.50 faster ✅
Query 18: prev= 586 ms, new= 297 ms, diff=1.97 faster ✅
Query 19: prev= 111 ms, new= 101 ms, diff=1.10 faster ✔
Query 20: prev=  92 ms, new=  66 ms, diff=1.39 faster ✅
Query 21: prev= 409 ms, new= 170 ms, diff=2.41 faster ✅
Query 22: prev=  57 ms, new=  34 ms, diff=1.68 faster ✅

Copy link
Collaborator

@jayshrivastava jayshrivastava left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Copy link
Collaborator

@NGA-TRAN NGA-TRAN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Th ePR looks good. Some questions regarding cores & threads

// Print machine information
println!("os: {}", std::env::consts::OS);
println!("arch: {}", std::env::consts::ARCH);
println!("cpu cores: {}", get_available_parallelism());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the number of cores per worker or total of cpu cores of all workers?
Is there any constraints between this and number of threads per worker? Should number of threads per worker < CPU cores per worker? Is there any enforcement or unexpected behavior if it is not enforced

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the number of cores per worker or total of cpu cores of all workers?

Total CPU cores available in the machine that runs the benchmarks

Is there any constraints between this and number of threads per worker?

Only the ones specified by the people running the commands, it's possible to do --threads 1000 in a 2 core CPU machine. It does not make much sense to do that though...

Should number of threads per worker < CPU cores per worker?

Assuming get_available_parallelism() returns the number of CPU cores in the machine, I think so yes. As these are just benchmarks meant to be run by us, I think it's fine to not do any enforcement in the code, hopefully people running the benchmarks know what they are doing if the do --threads 1000 in a 2 core CPU machine.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree people who run the benchmark should know what we are doing. Maybe we just help remind them that. How about we add a brief message saying number of threads should be < number of cores and number of partitions should be < number of threads?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a doc comment in the CLI

Base automatically changed from gabrielmusat/tpch-threads-and-workers to main September 11, 2025 14:55
@gabotechs gabotechs merged commit 1e61d51 into main Sep 11, 2025
4 checks passed
@gabotechs gabotechs deleted the gabrielmusat/report-host-stats-on-benchmark branch September 11, 2025 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants