Skip to content

perf(profiling)!: use zstd for compression #1065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft

perf(profiling)!: use zstd for compression #1065

wants to merge 9 commits into from
+322 −62

Conversation

morrisonlevi
Copy link
Contributor

@morrisonlevi morrisonlevi commented May 19, 2025 •
edited
Loading

What does this PR do?

Converts from lz4 compression to zstd for file uploads in the profiler.

Breaking Change

This is a breaking change in the sense that tooling and tests may have previously assumed that the compression is lz4. Using an lz4 decompressor on zstd input will obviously fail.

Motivation

tl;dr Generally faster while creating smaller files.

zstd may have a better tradeoff for performance to compressed size.

Here are some benchmarks on compressors for a CPU profile which I cannot share publicly yet. We're working on anonymizing it so we can commit these to the repository. Throughput is in MiB/s. Utility is the compression ratio multiplied by throughput. This is on an Apple M1 Max.

Algo.      Size     Ratio    Throughput     Utility
None    2363410     1.000     11,746.     11,746.
lz4     1215720     1.944        459.23      892.74
zstd-1   778653     3.035        518.39    1,573.31
zstd-2   745217     3.171        350.07    1,110.07
zstd-3   698466     3.384        277.35      938.55
zstd-4   694724     3.402        270.86      921.47

I tested zstd-5 and zstd-6 informally, and compression rations were incrementally better, but you lose throughput steadily. Sticking to something in the level 1-4 range seems better.

And here's from another, also not public set of data:

Algo.      Size     Ratio    Throughput     Utility
None    5804495     1.0000    11.901.     11,901.
lz4     1467466     3.9554       760.52    3,008.1
zstd-1   746324     7.7774       728.48    5,665.6
zstd-2   727673     7.9768       639.62    5,102.1
zstd-3   696252     8.3368       585.82    4,883.9
zstd-4   696241     8.3369       569.99    4,751.9

My personal opinion is that zstd-1 is a safe upgrade. The performance is roughly similar to lz4, but the compression is much better.. Moving to a higher level is something that could be discussed.

Additional Notes

Currently lacks tests, pushed to see if this will cause build issues or significant artifact size bloat.

How to test the change?

As noted in Breaking Change, this adjusts the outputs to be zstd compressed instead of lz4 compressed. Tests which work on compressed files need updated.

@github-actions github-actions bot added the profiling Relates to the profiling* modules. label May 19, 2025
@morrisonlevi morrisonlevi added the enhancement New feature or request label May 19, 2025
@r1viollet
Copy link
Contributor

r1viollet commented May 19, 2025 •
edited by dd-octo-sts bot
Loading

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 7.19 MB 7.44 MB +3.47% (+256.00 KB) ⚠️
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 70.99 MB 82.42 MB +16.08% (+11.42 MB) 🚨
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 83.30 MB 95.61 MB +14.77% (+12.30 MB) 🚨
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 9.25 MB 9.53 MB +3.04% (+288.82 KB) ⚠️
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 18.39 MB 18.78 MB +2.12% (+400.00 KB) ⚠️
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 65.77 KB 65.77 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 125.10 MB 125.75 MB +.51% (+664.00 KB) 🔍
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 654.42 MB 656.23 MB +.27% (+1.80 MB) 🔍
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 5.88 MB 6.26 MB +6.61% (+398.50 KB) 🚨
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 65.77 KB 65.77 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 17.37 MB 18.89 MB +8.72% (+1.51 MB) 🚨
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 32.21 MB 36.60 MB +13.63% (+4.39 MB) 🚨
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 15.68 MB 15.93 MB +1.62% (+261.50 KB) ⚠️
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 66.79 KB 66.79 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 127.41 MB 128.13 MB +.57% (+744.00 KB) 🔍
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 644.54 MB 645.40 MB +.13% (+889.31 KB) 🔍
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 4.49 MB 4.81 MB +6.94% (+320.00 KB) 🚨
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 66.79 KB 66.79 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 18.50 MB 20.05 MB +8.35% (+1.54 MB) 🚨
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 30.25 MB 34.34 MB +13.50% (+4.08 MB) 🚨
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 63.64 MB 71.42 MB +12.22% (+7.77 MB) 🚨
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 8.51 MB 8.85 MB +4.08% (+356.00 KB) ⚠️
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 78.07 MB 90.43 MB +15.83% (+12.35 MB) 🚨
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 9.85 MB 10.20 MB +3.56% (+359.82 KB) ⚠️

@pr-commenter
Copy link

pr-commenter bot commented May 19, 2025 •
edited
Loading

Benchmarks

Comparison

Benchmark execution time: 2025-10-09 23:10:52

Comparing candidate commit f28efa6 in PR branch PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog with baseline commit d9b1999 in branch main.

Found 5 performance improvements and 6 performance regressions! Performance is the same for 42 metrics, 2 unstable metrics.

scenario:benching serializing traces from their internal representation to msgpack

  • 🟩 execution_time [-741.740µs; -728.415µs] or [-4.892%; -4.804%]

scenario:credit_card/is_card_number/ 378282246310005

  • 🟩 execution_time [-4.103µs; -3.906µs] or [-5.467%; -5.205%]
  • 🟩 throughput [+733136.704op/s; +771180.677op/s] or [+5.502%; +5.788%]

scenario:credit_card/is_card_number/378282246310005

  • 🟥 execution_time [+6.133µs; +6.697µs] or [+8.493%; +9.274%]
  • 🟥 throughput [-1164668.034op/s; -1079754.756op/s] or [-8.410%; -7.797%]

scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...

  • 🟥 execution_time [+38.591µs; +38.984µs] or [+7.777%; +7.856%]
  • 🟥 throughput [-146895.294op/s; -145345.705op/s] or [-7.289%; -7.212%]

scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters

  • 🟥 execution_time [+24.282µs; +24.418µs] or [+14.296%; +14.377%]
  • 🟥 throughput [-740165.703op/s; -736283.419op/s] or [-12.572%; -12.506%]

scenario:normalization/normalize_service/normalize_service/[empty string]

  • 🟩 execution_time [-2.019µs; -1.986µs] or [-5.296%; -5.207%]
  • 🟩 throughput [+1440928.735op/s; +1466225.909op/s] or [+5.495%; +5.591%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 86.781µs 87.163µs ± 0.135µs 87.145µs ± 0.058µs 87.213µs 87.317µs 87.452µs 88.505µs 1.56% 5.012 47.580 0.15% 0.010µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [87.144µs; 87.181µs] or [-0.021%; +0.021%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 17.660µs 25.626µs ± 9.229µs 18.084µs ± 0.318µs 33.337µs 42.676µs 51.584µs 63.492µs 251.09% 0.869 0.308 35.92% 0.653µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [24.347µs; 26.905µs] or [-4.991%; +4.991%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 34.121µs 34.598µs ± 0.829µs 34.225µs ± 0.060µs 34.312µs 36.353µs 36.414µs 37.883µs 10.69% 1.784 1.545 2.39% 0.059µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [34.483µs; 34.713µs] or [-0.332%; +0.332%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 10.509ms 10.546ms ± 0.016ms 10.545ms ± 0.009ms 10.555ms 10.572ms 10.585ms 10.638ms 0.89% 1.004 4.348 0.15% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [10.544ms; 10.549ms] or [-0.021%; +0.021%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
receiver_entry_point/report/2597 execution_time 6.229ms 6.306ms ± 0.049ms 6.294ms ± 0.021ms 6.318ms 6.426ms 6.470ms 6.537ms 3.87% 1.825 4.005 0.78% 0.003ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
receiver_entry_point/report/2597 execution_time [6.299ms; 6.313ms] or [-0.109%; +0.109%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 162.702µs 163.715µs ± 0.349µs 163.654µs ± 0.116µs 163.785µs 164.321µs 164.517µs 166.797µs 1.92% 3.800 29.934 0.21% 0.025µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [163.667µs; 163.764µs] or [-0.030%; +0.030%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 3.894µs 3.912µs ± 0.003µs 3.912µs ± 0.002µs 3.914µs 3.916µs 3.917µs 3.918µs 0.16% -1.288 8.480 0.07% 0.000µs 1 200
credit_card/is_card_number/ throughput 255212882.763op/s 255620049.990op/s ± 173999.354op/s 255621461.190op/s ± 114579.226op/s 255735036.908op/s 255864037.653op/s 255901636.710op/s 256785439.804op/s 0.46% 1.306 8.616 0.07% 12303.612op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 76.881µs 78.462µs ± 0.722µs 78.464µs ± 0.499µs 78.917µs 79.614µs 80.510µs 81.533µs 3.91% 0.558 1.117 0.92% 0.051µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 12265042.979op/s 12746056.335op/s ± 116728.636op/s 12744761.655op/s ± 81304.955op/s 12826767.876op/s 12924946.457op/s 12969341.304op/s 13007195.520op/s 2.06% -0.482 0.905 0.91% 8253.961op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 69.899µs 71.050µs ± 0.679µs 70.968µs ± 0.408µs 71.388µs 72.239µs 72.587µs 75.684µs 6.65% 1.828 9.665 0.95% 0.048µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 13212874.400op/s 14075919.998op/s ± 132260.445op/s 14090919.137op/s ± 80475.967op/s 14154195.797op/s 14250459.282op/s 14302616.327op/s 14306431.796op/s 1.53% -1.607 7.896 0.94% 9352.226op/s 1 200
credit_card/is_card_number/37828224631 execution_time 3.892µs 3.912µs ± 0.003µs 3.912µs ± 0.002µs 3.913µs 3.916µs 3.918µs 3.921µs 0.23% -1.400 11.718 0.07% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 255069264.722op/s 255641628.238op/s ± 181046.512op/s 255646403.733op/s ± 109866.725op/s 255755953.009op/s 255857500.763op/s 255929850.600op/s 256933847.724op/s 0.50% 1.425 11.909 0.07% 12801.922op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 76.860µs 78.627µs ± 2.024µs 78.182µs ± 0.649µs 78.850µs 82.376µs 88.075µs 88.862µs 13.66% 3.184 11.162 2.57% 0.143µs 1 200
credit_card/is_card_number/378282246310005 throughput 11253395.180op/s 12725999.997op/s ± 303686.744op/s 12790680.919op/s ± 107102.072op/s 12890054.669op/s 12985445.013op/s 12995318.175op/s 13010680.744op/s 1.72% -2.958 9.804 2.38% 21473.896op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 52.146µs 52.211µs ± 0.067µs 52.202µs ± 0.024µs 52.227µs 52.277µs 52.343µs 53.000µs 1.53% 8.529 97.078 0.13% 0.005µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 18867804.009op/s 19153220.129op/s ± 24181.931op/s 19156447.816op/s ± 8980.349op/s 19164310.785op/s 19170970.933op/s 19176400.237op/s 19176778.839op/s 0.11% -8.429 95.445 0.13% 1709.921op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 6.028µs 6.034µs ± 0.006µs 6.034µs ± 0.003µs 6.036µs 6.040µs 6.070µs 6.074µs 0.66% 4.050 23.384 0.10% 0.000µs 1 200
credit_card/is_card_number/x371413321323331 throughput 164639785.636op/s 165718743.286op/s ± 162676.233op/s 165725422.281op/s ± 75094.287op/s 165824069.569op/s 165865870.026op/s 165895003.273op/s 165900341.252op/s 0.11% -4.023 23.159 0.10% 11502.947op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 3.893µs 3.911µs ± 0.003µs 3.911µs ± 0.001µs 3.913µs 3.916µs 3.920µs 3.923µs 0.30% -0.325 9.482 0.07% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 254931686.882op/s 255657529.749op/s ± 186393.123op/s 255686039.542op/s ± 97538.971op/s 255774993.228op/s 255849192.063op/s 255898590.147op/s 256869320.521op/s 0.46% 0.350 9.608 0.07% 13179.984op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 63.836µs 64.377µs ± 0.171µs 64.369µs ± 0.098µs 64.463µs 64.678µs 64.859µs 65.035µs 1.04% 0.572 1.367 0.26% 0.012µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 15376320.663op/s 15533536.956op/s ± 41082.336op/s 15535468.794op/s ± 23714.028op/s 15560585.815op/s 15588305.053op/s 15625068.669op/s 15665195.410op/s 0.84% -0.548 1.338 0.26% 2904.960op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 58.122µs 58.347µs ± 0.159µs 58.310µs ± 0.098µs 58.435µs 58.648µs 58.792µs 58.937µs 1.08% 1.039 0.879 0.27% 0.011µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 16967256.746op/s 17139012.484op/s ± 46500.185op/s 17149773.473op/s ± 28871.185op/s 17175120.979op/s 17195041.552op/s 17199046.760op/s 17205276.858op/s 0.32% -1.024 0.831 0.27% 3288.060op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 3.891µs 3.912µs ± 0.003µs 3.912µs ± 0.002µs 3.914µs 3.917µs 3.921µs 3.921µs 0.24% -0.819 9.419 0.08% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 255036878.612op/s 255617404.707op/s ± 205798.785op/s 255637675.537op/s ± 112416.482op/s 255741180.726op/s 255861530.503op/s 255884391.572op/s 256999983.014op/s 0.53% 0.845 9.588 0.08% 14552.172op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 54.546µs 54.852µs ± 0.213µs 54.801µs ± 0.137µs 54.988µs 55.271µs 55.447µs 55.684µs 1.61% 1.008 0.892 0.39% 0.015µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 17958598.071op/s 18231041.241op/s ± 70613.337op/s 18247885.905op/s ± 45768.940op/s 18288598.435op/s 18311872.053op/s 18324342.994op/s 18333140.819op/s 0.47% -0.987 0.812 0.39% 4993.117op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 52.143µs 52.202µs ± 0.032µs 52.195µs ± 0.019µs 52.219µs 52.266µs 52.297µs 52.320µs 0.24% 0.867 0.661 0.06% 0.002µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 19112984.316op/s 19156406.532op/s ± 11844.897op/s 19158792.312op/s ± 6895.651op/s 19164722.059op/s 19172462.018op/s 19175403.595op/s 19178000.915op/s 0.10% -0.863 0.652 0.06% 837.561op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 6.028µs 6.034µs ± 0.006µs 6.033µs ± 0.003µs 6.036µs 6.042µs 6.068µs 6.074µs 0.67% 3.855 19.863 0.10% 0.000µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 164636693.912op/s 165717352.957op/s ± 166540.079op/s 165741402.207op/s ± 71450.026op/s 165818514.638op/s 165864225.492op/s 165886343.614op/s 165897966.043op/s 0.09% -3.833 19.696 0.10% 11776.162op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [3.912µs; 3.912µs] or [-0.009%; +0.009%] None None None
credit_card/is_card_number/ throughput [255595935.353op/s; 255644164.627op/s] or [-0.009%; +0.009%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [78.362µs; 78.562µs] or [-0.128%; +0.128%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [12729878.869op/s; 12762233.801op/s] or [-0.127%; +0.127%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [70.956µs; 71.144µs] or [-0.132%; +0.132%] None None None
credit_card/is_card_number/ 378282246310005 throughput [14057589.972op/s; 14094250.024op/s] or [-0.130%; +0.130%] None None None
credit_card/is_card_number/37828224631 execution_time [3.911µs; 3.912µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/37828224631 throughput [255616536.932op/s; 255666719.543op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number/378282246310005 execution_time [78.347µs; 78.908µs] or [-0.357%; +0.357%] None None None
credit_card/is_card_number/378282246310005 throughput [12683911.935op/s; 12768088.059op/s] or [-0.331%; +0.331%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [52.201µs; 52.220µs] or [-0.018%; +0.018%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [19149868.746op/s; 19156571.512op/s] or [-0.017%; +0.017%] None None None
credit_card/is_card_number/x371413321323331 execution_time [6.034µs; 6.035µs] or [-0.014%; +0.014%] None None None
credit_card/is_card_number/x371413321323331 throughput [165696197.924op/s; 165741288.647op/s] or [-0.014%; +0.014%] None None None
credit_card/is_card_number_no_luhn/ execution_time [3.911µs; 3.912µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ throughput [255631697.454op/s; 255683362.043op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [64.354µs; 64.401µs] or [-0.037%; +0.037%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [15527843.339op/s; 15539230.572op/s] or [-0.037%; +0.037%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [58.325µs; 58.369µs] or [-0.038%; +0.038%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [17132568.006op/s; 17145456.962op/s] or [-0.038%; +0.038%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [3.912µs; 3.913µs] or [-0.011%; +0.011%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [255588882.975op/s; 255645926.439op/s] or [-0.011%; +0.011%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [54.823µs; 54.882µs] or [-0.054%; +0.054%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [18221254.911op/s; 18240827.570op/s] or [-0.054%; +0.054%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [52.197µs; 52.206µs] or [-0.009%; +0.009%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [19154764.943op/s; 19158048.120op/s] or [-0.009%; +0.009%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [6.034µs; 6.035µs] or [-0.014%; +0.014%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [165694272.104op/s; 165740433.810op/s] or [-0.014%; +0.014%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 239.611ns 251.833ns ± 13.792ns 245.040ns ± 2.138ns 250.472ns 281.337ns 286.835ns 288.515ns 17.74% 1.508 0.667 5.46% 0.975ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [249.922ns; 253.745ns] or [-0.759%; +0.759%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching serializing traces from their internal representation to msgpack execution_time 14.350ms 14.428ms ± 0.037ms 14.424ms ± 0.018ms 14.442ms 14.490ms 14.544ms 14.626ms 1.40% 1.506 5.145 0.26% 0.003ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching serializing traces from their internal representation to msgpack execution_time [14.423ms; 14.433ms] or [-0.036%; +0.036%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 60.586ms 61.049ms ± 1.916ms 60.777ms ± 0.081ms 60.884ms 61.055ms 73.251ms 80.903ms 33.11% 8.451 73.956 3.13% 0.135ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [60.784ms; 61.315ms] or [-0.435%; +0.435%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.210µs 3.334µs ± 1.461µs 3.084µs ± 0.025µs 3.118µs 3.785µs 14.362µs 15.118µs 390.18% 7.240 53.996 43.72% 0.103µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [3.131µs; 3.536µs] or [-6.075%; +6.075%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
ip_address/quantize_peer_ip_address_benchmark execution_time 4.981µs 5.053µs ± 0.041µs 5.039µs ± 0.027µs 5.091µs 5.127µs 5.131µs 5.133µs 1.87% 0.433 -1.106 0.81% 0.003µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark execution_time [5.047µs; 5.058µs] or [-0.113%; +0.113%] None None None

Group 13

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 534.274µs 534.999µs ± 0.443µs 534.931µs ± 0.224µs 535.176µs 535.670µs 536.086µs 538.775µs 0.72% 3.361 24.954 0.08% 0.031µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1856062.161op/s 1869161.951op/s ± 1543.160op/s 1869400.801op/s ± 783.298op/s 1870108.892op/s 1871012.920op/s 1871553.382op/s 1871697.967op/s 0.12% -3.322 24.522 0.08% 109.118op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 379.563µs 380.196µs ± 0.325µs 380.136µs ± 0.190µs 380.364µs 380.785µs 381.155µs 381.220µs 0.29% 0.871 0.868 0.09% 0.023µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2623156.335op/s 2630222.734op/s ± 2248.406op/s 2630634.748op/s ± 1318.665op/s 2631827.074op/s 2633063.269op/s 2634296.149op/s 2634609.049op/s 0.15% -0.865 0.856 0.09% 158.986op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 193.727µs 194.198µs ± 0.424µs 194.152µs ± 0.139µs 194.305µs 194.513µs 194.810µs 199.412µs 2.71% 9.424 112.855 0.22% 0.030µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5014745.993op/s 5149394.781op/s ± 11014.706op/s 5150599.811op/s ± 3676.381op/s 5154029.704op/s 5157644.805op/s 5159950.396op/s 5161913.608op/s 0.22% -9.255 110.045 0.21% 778.857op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 35.880µs 36.130µs ± 0.106µs 36.120µs ± 0.081µs 36.210µs 36.295µs 36.349µs 36.595µs 1.31% 0.426 0.683 0.29% 0.007µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 27326412.108op/s 27677690.962op/s ± 81122.445op/s 27685347.437op/s ± 62415.499op/s 27736967.154op/s 27797182.347op/s 27830561.886op/s 27870930.043op/s 0.67% -0.404 0.613 0.29% 5736.223op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 44.957µs 45.144µs ± 0.167µs 45.119µs ± 0.141µs 45.266µs 45.363µs 45.431µs 46.373µs 2.78% 2.119 12.851 0.37% 0.012µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 21564313.649op/s 22151843.176op/s ± 81162.246op/s 22163549.044op/s ± 69504.243op/s 22232986.007op/s 22235436.660op/s 22242441.041op/s 22243372.649op/s 0.36% -2.008 11.790 0.37% 5739.037op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [534.938µs; 535.061µs] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1868948.084op/s; 1869375.818op/s] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [380.151µs; 380.241µs] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2629911.127op/s; 2630534.342op/s] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [194.140µs; 194.257µs] or [-0.030%; +0.030%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5147868.249op/s; 5150921.314op/s] or [-0.030%; +0.030%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [36.116µs; 36.145µs] or [-0.041%; +0.041%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [27666448.171op/s; 27688933.752op/s] or [-0.041%; +0.041%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [45.120µs; 45.167µs] or [-0.051%; +0.051%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [22140594.870op/s; 22163091.483op/s] or [-0.051%; +0.051%] None None None

Group 14

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 186.630µs 186.985µs ± 0.271µs 186.954µs ± 0.109µs 187.057µs 187.308µs 188.306µs 188.754µs 0.96% 3.762 19.929 0.14% 0.019µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 5297914.304op/s 5348025.486op/s ± 7712.357op/s 5348906.496op/s ± 3110.414op/s 5352136.641op/s 5355786.863op/s 5358023.032op/s 5358198.231op/s 0.17% -3.728 19.662 0.14% 545.346op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 17.598µs 17.729µs ± 0.059µs 17.726µs ± 0.039µs 17.763µs 17.830µs 17.876µs 17.992µs 1.50% 0.809 1.714 0.33% 0.004µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 55579297.412op/s 56404035.228op/s ± 187321.836op/s 56415269.534op/s ± 125647.847op/s 56544973.810op/s 56663775.466op/s 56760910.871op/s 56824476.355op/s 0.73% -0.779 1.608 0.33% 13245.654op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 10.001µs 10.095µs ± 0.028µs 10.094µs ± 0.019µs 10.115µs 10.143µs 10.162µs 10.186µs 0.92% 0.205 0.367 0.28% 0.002µs 1 200
normalization/normalize_name/normalize_name/good throughput 98169550.054op/s 99056284.873op/s ± 275266.994op/s 99073586.368op/s ± 187003.993op/s 99233647.327op/s 99471347.820op/s 99581240.513op/s 99989848.359op/s 0.92% -0.186 0.365 0.28% 19464.316op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [186.948µs; 187.023µs] or [-0.020%; +0.020%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [5346956.628op/s; 5349094.345op/s] or [-0.020%; +0.020%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [17.721µs; 17.738µs] or [-0.046%; +0.046%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [56378074.223op/s; 56429996.233op/s] or [-0.046%; +0.046%] None None None
normalization/normalize_name/normalize_name/good execution_time [10.091µs; 10.099µs] or [-0.039%; +0.039%] None None None
normalization/normalize_name/normalize_name/good throughput [99018135.515op/s; 99094434.231op/s] or [-0.039%; +0.039%] None None None

Group 15

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f28efa6 1760050670 PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.439µs 2.472µs ± 0.023µs 2.471µs ± 0.015µs 2.482µs 2.525µs 2.538µs 2.546µs 3.05% 1.053 1.071 0.91% 0.002µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.469µs; 2.476µs] or [-0.127%; +0.127%] None None None

Baseline

Omitted due to size.

@codecov-commenter
Copy link

codecov-commenter commented May 19, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 69.93007% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.88%. Comparing base (e741848) to head (f28efa6).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1065      +/-   ##
==========================================
+ Coverage   71.68%   71.88%   +0.19%     
==========================================
  Files         355      357       +2     
  Lines       56312    56798     +486     
==========================================
+ Hits        40369    40830     +461     
- Misses      15943    15968      +25
Components Coverage Δ
datadog-crashtracker 49.30% <ø> (-0.03%) ⬇️
datadog-crashtracker-ffi 5.93% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 87.85% <ø> (-2.61%) ⬇️
data-pipeline-ffi 88.19% <ø> (ø)
ddcommon 84.54% <ø> (+0.24%) ⬆️
ddcommon-ffi 73.84% <ø> (ø)
ddtelemetry 59.73% <ø> (-0.30%) ⬇️
ddtelemetry-ffi 21.24% <ø> (ø)
dogstatsd-client 83.26% <ø> (ø)
datadog-ipc 82.39% <ø> (ø)
datadog-profiling 76.67% <69.93%> (-0.23%) ⬇️
datadog-profiling-ffi 62.12% <ø> (ø)
datadog-sidecar 37.53% <ø> (+0.34%) ⬆️
datdog-sidecar-ffi 13.54% <ø> (+1.67%) ⬆️
spawn-worker 55.35% <ø> (ø)
tinybytes 92.22% <ø> (ø)
datadog-trace-normalization 98.24% <ø> (ø)
datadog-trace-obfuscation 94.17% <ø> (ø)
datadog-trace-protobuf 59.65% <ø> (ø)
datadog-trace-utils 89.99% <ø> (+0.22%) ⬆️
datadog-tracer-flare 56.08% <ø> (+1.55%) ⬆️
datadog-log 76.31% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch from 80c8f07 to 0d6cabc Compare May 19, 2025 22:56
felixge
felixge reviewed May 20, 2025
View reviewed changes
datadog-profiling/src/serializer/compressed_streaming_encoder.rs Outdated
UploadCompression::Zstd => Self {
buffer: Vec::with_capacity(TEMPORARY_BUFFER_SIZE),
compressor: Compressor::Zstd {
// A level of 0 uses zstd's default (currently 3).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that the level 3 from the upstream zstd C library? If yes, level 3 is good. I'm asking because there are different zstd implementations out there which assign different meanings to different levels.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, at least as far as I understood it.

danielsn
danielsn reviewed May 21, 2025
View reviewed changes
@morrisonlevi morrisonlevi force-pushed the PROF-11830-Add-upload-compression-configuration-to-libdatadog branch 2 times, most recently from d301705 to 8fa167e Compare May 25, 2025 19:25
@morrisonlevi morrisonlevi changed the title feat(profiling): add zstd as a compression option feat(profiling): zstd as a compression option May 25, 2025
@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch 2 times, most recently from 8788621 to 714ecc0 Compare May 27, 2025 16:33
@morrisonlevi morrisonlevi force-pushed the PROF-11830-Add-upload-compression-configuration-to-libdatadog branch from 5d54a41 to 9cafb27 Compare May 27, 2025 16:42
@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch from 714ecc0 to 5ed02e4 Compare May 27, 2025 16:42
@morrisonlevi morrisonlevi force-pushed the PROF-11830-Add-upload-compression-configuration-to-libdatadog branch from 9cafb27 to c4c8301 Compare May 28, 2025 21:18
@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch 2 times, most recently from 064a512 to 6f72248 Compare May 28, 2025 21:23
@morrisonlevi morrisonlevi force-pushed the PROF-11830-Add-upload-compression-configuration-to-libdatadog branch 2 times, most recently from 49d50fa to 9fa834b Compare May 29, 2025 02:32
@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch 2 times, most recently from a13e0dd to b751d09 Compare May 29, 2025 17:49
@morrisonlevi morrisonlevi force-pushed the PROF-11830-Add-upload-compression-configuration-to-libdatadog branch from 9fa834b to 35933df Compare May 29, 2025 17:49
@github-actions GitHub Actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. To override this behavior, add the keep-open label or update the PR.

@github-actions github-actions bot added the stale Used by actions/stale to identify PRs that have been inactive for 90+ days label Aug 28, 2025
@morrisonlevi morrisonlevi added the keep-open Overrides actions/stale auto-closing stale PRs label Aug 28, 2025
@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch from b751d09 to 958b210 Compare September 30, 2025 18:01
@morrisonlevi morrisonlevi changed the base branch from PROF-11830-Add-upload-compression-configuration-to-libdatadog to main September 30, 2025 20:43
@morrisonlevi morrisonlevi removed the stale Used by actions/stale to identify PRs that have been inactive for 90+ days label Sep 30, 2025
@morrisonlevi morrisonlevi changed the title feat(profiling): zstd as a compression option perf(profiling): use zstd for compression Sep 30, 2025
@morrisonlevi morrisonlevi changed the title perf(profiling): use zstd for compression perf(profiling)!: use zstd for compression Sep 30, 2025
@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch from 6630172 to c2a9233 Compare October 9, 2025 01:10
@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch from c2a9233 to 868f0dc Compare October 9, 2025 04:19
@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch from 6370af5 to d06b17b Compare October 9, 2025 22:06
@morrisonlevi morrisonlevi force-pushed the PROF-11832-Add-zstd-to-compression-configuration-to-libdatadog branch from d06b17b to f28efa6 Compare October 9, 2025 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@felixge felixge felixge left review comments

@danielsn danielsn danielsn left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
enhancement New feature or request keep-open Overrides actions/stale auto-closing stale PRs profiling Relates to the profiling* modules.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@morrisonlevi @r1viollet @codecov-commenter @felixge @danielsn