Make stress tests independent of batch size

[utpilla]

Changes

Current implementation:

• The current implementation of stress test reports the progress made in batches instead of reporting it on the fly.
• Each thread executes the function to be stress tested 1000 times (BATCH_SIZE is set to 1000) and then updates its stats by updating the AtomicU64 value in WorkerStats vec.
• The motivation to report the progress in batches is mainly to lower the number of fetch_add atomic operations so that it does not interfere with the measurements reported for the function under test.

Motivation for this PR

• Reporting measurements in batches makes the stress test results dependent on the BATCH_SIZE. You have to choose an optimal BATCH_SIZE for the function being stress tested. If the BATCH_SIZE is too low, then you make a lot more fetch_add calls for WorkerStats which would influence the final throughput. If the BATCH_SIZE is too high, then you don't report the actual progress made in a timely manner.
• If the whole idea, is to not have the stress test's own bookkeeping affect the throughput results, then we can consider avoiding making fetch_add calls altogether.
• This PR makes use of some unsafe code to have the update threads report progress by making a simple + operation instead of the atomic fetch_add operation.
• Another change in the PR is to use scoped threads which simplifies things a bit by not having to join on the update thread handles.

Note for reviewers:

Please hide whitespace from the diff for easier reviewing:

Merge requirement checklist

• CONTRIBUTING guidelines followed
• Unit tests added/updated (if applicable)
• Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
• Changes in public API reviewed (if applicable)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.6%. Comparing base (3ac2d9f) to head (f442e41).
Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #2306   +/-   ##
=====================================
  Coverage   79.6%   79.6%           
=====================================
  Files        123     123           
  Lines      21263   21263           
=====================================
+ Hits       16938   16940    +2     
+ Misses      4325    4323    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[lalitb]

Can this read operation conflict with the concurrent writes, as there is no safety with UnsafeCell operation?

[cijothomas]

at worse, we'll underreport the numbers. ?

[utpilla]

At worst, we won't get the most up-to-date sum which is okay?

[lalitb]

Yes under-reporting should be fine at worse. Though this can result in data corruption due to reading of partially updated values in some 32-bit machines (64-bit writes may not atomic in all 32-bit machines), but we needn't worry about that for stress test.

[cijothomas]

Won't we be now forced to check this AtomicBool for each iteration? I think that still be avoided and instead can be checked every BATCH_SIZE?

[utpilla]

Since the STOP is only ever changed when exiting the stress test, the cache line where STOP resides wouldn't have to be updated during the test. So, I don't expect it to have much of a perf implication. (don't think it suffers from false sharing here as STOP is static).

[utpilla]

If we still want to avoid it, I'd prefer if remove the STOP variable altogether and simply exit the process on Ctrl + C.

I ran the stress test with the above-mentioned change, and there isn't much of a difference in the results. I would prefer removing it as the existing code for using STOP doesn't add much value.

[cijothomas]

not a blocker for this PR, lets revisit removing it alltogether in a future pr, if needed.

[cijothomas]

@utpilla open-telemetry/opentelemetry-dotnet#5985 Can run an empty fun here too and share the resutls from same hardware? Curious to know the stress test in .NET,Rust is comparable when doing empty fn...

[utpilla]

@utpilla open-telemetry/opentelemetry-dotnet#5985 Can run an empty fun here too and share the resutls from same hardware? Curious to know the stress test in .NET,Rust is comparable when doing empty fn...

Yes, it's comparable. In fact, the reason I sent the PR on the .NET repo is that our stress test numbers for an empty function were way better than .NET's. That's when I realized that their stress test suffers from false sharing 😄

For Rust, I stress tested the empty function on the same hardware (WSL though), and the throughput is around 7.7 B iterations per second. For .NET it was ~7B loops per second.

[cijothomas]

perf numbers from my laptop ( 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz 4 cores with 48 GB RAM)

tracing with no subscriber 4B iterations/sec (.NET is ~35 M/sec)

^ Something I found in my personal note, a long time ago. 4B /sec was with tracing + no subscriber, which is pretty close to empty fn()!