Fix flaky CpuAndWallTimeWorker sampling test on macOS

[p-datadog]

What does this PR do?

Fixes a flaky profiling test by increasing the sampling window from 100ms to 200ms.

Motivation:

The test CpuAndWallTimeWorker#start when main thread is sleeping but a background thread is working failed on Test (macos-15, 3.0) in PR #5481 with sample_count: 4 (threshold: 5). The profiler only managed 4 trigger_sample_attempts in the 100ms window due to startup overhead on macOS ARM64 + Ruby 3.0 runners.

How I Reproduced the Issue

The CI failure on PR #5481 shows the test got exactly 4 samples in 100ms instead of the required 5. The stats (trigger_sample_attempts=>4) confirm the profiler didn't even attempt enough samples — it's not a signal delivery issue, but insufficient time.

Root Cause

The test sleeps for only 100ms and expects ≥5 samples at a target rate of 100 samples/sec. On macOS-15 ARM64 CI runners with Ruby 3.0, profiler startup overhead and thread scheduling variability reduce the effective sampling window below what's needed for 5 samples. The margin between expected (10 samples) and threshold (5) is too thin for this environment.

Fix

Increase sleep from 0.1s to 0.2s, matching the duration used by similar tests in the same file (lines 474 and 605). At 100 samples/sec, 200ms gives ~20 expected samples — well above the threshold of 5.

Change log entry

None.

How to test the change?

CI should pass on Test (macos-15, 3.0) which was previously failing.

[datadog-official]

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 95.11% (-0.03%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f0d4d56 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-03-20 19:24:00

Comparing candidate commit f0d4d56 in PR branch investigate/flaky-cpu-wall-time-worker-spec with baseline commit e87e284 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

[ivoanjo]

Hmmmmmm thanks for looking into this. To be honest, I think it's cleaner to skip this test on macOS with a "TODO: This was flaky on macOS".

Why skip?

1. We don't support profiling on macOS anyway
2. A bunch of us don't have macs to debug these
3. Once we're ready to officially support macOS (if ever), we can spend the time investment to fix these

---

Also

where profiler startup overhead can eat into the sampling window.

this comment the AI added is an hallucination, the wait_until_running already includes almost all of the initialization ;)

[p-datadog]

this comment the AI added is an hallucination, the wait_until_running already includes almost all of the initialization ;)

Thanks for confirming - I was not sure the proposed fix was kosher with doubling of sleep time. Let's skip the test as you suggested.

[p-datadog]

Done — skipped on macOS with a TODO and restored the original comment 😅 Fixed in:

• spec/datadog/profiling/collectors/cpu_and_wall_time_worker_spec.rb:412 (added skip "TODO: This test is flaky on macOS" if PlatformHelpers.mac?)
• spec/datadog/profiling/collectors/cpu_and_wall_time_worker_spec.rb:419 (reverted sleep 0.2 → sleep 0.1)
• spec/datadog/profiling/collectors/cpu_and_wall_time_worker_spec.rb:436-439 (restored original comment)