Skip to content

Conversation

@gabotechs
Copy link
Collaborator

@gabotechs gabotechs commented Aug 12, 2025

Adds a delta report that compares the last run benchmarks with the current ones, something like this:

==== Comparison with the previous benchmark from 2025-08-12 14:38:33 UTC ====
 Query 1: prev=  74 ms, new=  70 ms, diff=1.06 faster ✔
 Query 2: prev=  25 ms, new=  25 ms, diff=1.00 slower ✖
 Query 3: prev=  37 ms, new=  36 ms, diff=1.03 faster ✔
 Query 4: prev=  25 ms, new=  28 ms, diff=1.12 slower ✖
 Query 5: prev=  46 ms, new=  54 ms, diff=1.17 slower ✖
 Query 6: prev=  22 ms, new=  21 ms, diff=1.05 faster ✔
 Query 7: prev=  64 ms, new=  61 ms, diff=1.05 faster ✔
 Query 8: prev=  47 ms, new=  43 ms, diff=1.09 faster ✔
 Query 9: prev=  63 ms, new=  53 ms, diff=1.19 faster ✔
Query 10: prev=  66 ms, new=  53 ms, diff=1.25 faster ✅
Query 11: prev=  34 ms, new=  21 ms, diff=1.62 faster ✅
Query 12: prev=  58 ms, new=  41 ms, diff=1.41 faster ✅
Query 13: prev=  34 ms, new=  32 ms, diff=1.06 faster ✔
Query 14: prev=  35 ms, new=  30 ms, diff=1.17 faster ✔
Query 15: prev=  53 ms, new=  48 ms, diff=1.10 faster ✔
Query 16: prev=  23 ms, new=  19 ms, diff=1.21 faster ✅
Query 17: prev=  77 ms, new=  70 ms, diff=1.10 faster ✔
Query 18: prev= 103 ms, new=  92 ms, diff=1.12 faster ✔
Query 19: prev=  58 ms, new=  51 ms, diff=1.14 faster ✔
Query 20: prev=  52 ms, new=  41 ms, diff=1.27 faster ✅
Query 21: prev= 112 ms, new=  89 ms, diff=1.26 faster ✅
Query 22: prev=  22 ms, new=  17 ms, diff=1.29 faster ✅

The first time the benchmarks are run, no report will be printed to stdout, the second time, a report will be produced comparing it with the first run, the third time will be compared to the second one, etc...

This can be tested running this command twice:

cargo run -p datafusion-distributed-benchmarks --release -- tpch --path benchmarks/data/tpch_sf1 

Copy link
Collaborator

@robtandy robtandy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but it will have conflicts for port number if #89 is merged first. I suggest we do it in that order to accommodate our staggered working hours.

@gabotechs gabotechs force-pushed the gabrielmusat/add-delta-benchmark-report branch from c1be7cc to 47e0b92 Compare August 13, 2025 10:48
@gabotechs gabotechs merged commit bae85ca into main Aug 13, 2025
3 checks passed
@gabotechs gabotechs deleted the gabrielmusat/add-delta-benchmark-report branch August 13, 2025 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants