Skip to content

Conversation

ldionne
Copy link
Member

@ldionne ldionne commented Sep 8, 2025

[libc++] Add a job that runs the benchmarks as part of the CI

This will make it easier to see the performance impact of individual
patches. This is a first step towards better performance tracking, but
many things still need to be figured out:

  • We should get a dedicated set of machines to run these tests on
  • We could probably reuse regularly-generated baseline results published
    to LNT instead of re-generating the baselines in this job every time.

@ldionne ldionne force-pushed the review/add-benchmarking-job branch from dfaeca9 to c7306f4 Compare September 11, 2025 16:21
@ldionne ldionne changed the title [libc++][WIP] Add a job that runs the benchmarks as part of the CI [libc++] Add a job that runs the benchmarks as part of the CI Sep 11, 2025
@ldionne ldionne force-pushed the review/add-benchmarking-job branch from d3f3bbf to 99ea18c Compare September 11, 2025 19:49
This will make it easier to see the performance impact of individual
patches. This is a first step towards better performance tracking, but
many things still need to be figured out:
- We should get a dedicated set of machines to run these tests on
- We could probably reuse regularly-generated baseline results published
  to LNT instead of re-generating the baselines in this job every time.
@ldionne ldionne force-pushed the review/add-benchmarking-job branch from 99ea18c to cf5f82e Compare September 11, 2025 22:20
@ldionne
Copy link
Member Author

ldionne commented Sep 11, 2025

test edit commit via API

@ldionne
Copy link
Member Author

ldionne commented Sep 11, 2025

/libcxx-bot benchmark libcxx/test/benchmarks/join_view.bench.cpp libcxx/test/benchmarks/hash.bench.cpp

Benchmark results:
Benchmark                              Baseline    Candidate    Difference    % Difference
-----------------------------------  ----------  -----------  ------------  --------------
BM_Hash/uint32_random_std_hash/1024      363.18       362.45         -0.74           -0.20
BM_Hash/uint32_top_std_hash/1024         363.50       364.26          0.76            0.21
BM_join_view_deques/0                     17.34        17.29         -0.05           -0.29
BM_join_view_deques/1                     23.45        23.43         -0.02           -0.10
BM_join_view_deques/1024                7354.99      7622.90        267.91            3.64
BM_join_view_deques/2                     28.33        29.24          0.92            3.23
BM_join_view_deques/4000               38507.79     39075.92        568.12            1.48
BM_join_view_deques/4096               40568.47     39302.25      -1266.22           -3.12
BM_join_view_deques/512                 3372.40      3252.08       -120.32           -3.57
BM_join_view_deques/5500               52833.82     55038.01       2204.19            4.17
BM_join_view_deques/64                   371.55       364.72         -6.84           -1.84
BM_join_view_deques/64000            1132596.67   1114455.72     -18140.95           -1.60
BM_join_view_deques/65536            1213556.56   1257778.02      44221.46            3.64
BM_join_view_deques/70000            1354353.37   1435964.08      81610.71            6.03
BM_join_view_in_vectors/0                  7.17         7.19          0.02            0.27
BM_join_view_in_vectors/1                 11.84        11.81         -0.03           -0.22
BM_join_view_in_vectors/1024            3698.59      3638.25        -60.33           -1.63
BM_join_view_in_vectors/2                 10.31        10.32          0.01            0.08
BM_join_view_in_vectors/4000           15672.18     18722.86       3050.68           19.47
BM_join_view_in_vectors/4096           15971.69     18405.45       2433.76           15.24
BM_join_view_in_vectors/512             1811.43      1807.01         -4.42           -0.24
BM_join_view_in_vectors/5500           22703.60     21713.82       -989.78           -4.36
BM_join_view_in_vectors/64               228.15       227.82         -0.33           -0.15
BM_join_view_in_vectors/64000         333670.88    286360.85     -47310.03          -14.18
BM_join_view_in_vectors/65536         309820.17    289218.06     -20602.11           -6.65
BM_join_view_in_vectors/70000         339185.03    329616.72      -9568.31           -2.82
BM_join_view_out_vectors/0                 0.93         0.93          0.00            0.02
BM_join_view_out_vectors/1                 4.99         4.99         -0.00           -0.05
BM_join_view_out_vectors/1024           4032.52      4115.79         83.28            2.07
BM_join_view_out_vectors/2                 8.20         8.12         -0.09           -1.05
BM_join_view_out_vectors/4000          18296.18     16280.00      -2016.19          -11.02
BM_join_view_out_vectors/4096          18476.16     16397.04      -2079.12          -11.25
BM_join_view_out_vectors/512            2022.21      1973.60        -48.61           -2.40
BM_join_view_out_vectors/5500          22438.25     22383.05        -55.21           -0.25
BM_join_view_out_vectors/64              245.87       245.85         -0.02           -0.01
BM_join_view_out_vectors/64000        280803.15    278943.67      -1859.48           -0.66
BM_join_view_out_vectors/65536        345450.46    285769.10     -59681.36          -17.28
BM_join_view_out_vectors/70000        314685.18    313704.59       -980.59           -0.31

@ldionne
Copy link
Member Author

ldionne commented Sep 11, 2025

Closing this since after a whole lot of trial and error, I managed to make it work with a comment-triggered job instead.

@ldionne ldionne closed this Sep 11, 2025
@ldionne ldionne deleted the review/add-benchmarking-job branch September 11, 2025 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant