Fix flaky gauge tests in TelemetryMetricsEnabledSanityIT#21020
Fix flaky gauge tests in TelemetryMetricsEnabledSanityIT#21020andrross wants to merge 1 commit intoopensearch-project:mainfrom
Conversation
github-actions
bot
added
>test-failure
|
Failed to generate code suggestions for PR |
|
❌ Gradle check result for bcd28b9: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for bcd28b9: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Failed to generate code suggestions for PR |
|
❌ Gradle check result for 9cf800a: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for 9cf800a: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Both testGauge and testGaugeWithValueAndTagSupplier used a hardcoded
metric name ("test-gauge") and relied on fixed Thread.sleep durations
for synchronization, making them flaky.
The shared metric name caused cross-test pollution through the shared
InMemorySingletonMetricsExporter, which is the likely cause of the
failure in build 73291 (expected 3.0 but was 5.0). Each test now uses
a randomized metric name to isolate its metrics.
Replace the initial Thread.sleep with assertBusy polling to wait for
gauge values to be published. For the post-close assertion, use
assertBusy to retry a check that snapshots the callback counter,
waits longer than the publish interval, and verifies it has not
changed. This handles the case where an in-flight collection that
started before close() is still draining.
Resolves opensearch-project#19422
Signed-off-by: Andrew Ross <andrross@amazon.com>
andrross
force-pushed
the
fix/flaky-gauge-tests
branch
from
9cf800a to
bba7756
Compare
|
Failed to generate code suggestions for PR |
|
❌ Gradle check result for bba7756: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for bba7756: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
1 participant
Both
testGaugeandtestGaugeWithValueAndTagSupplierused a hardcoded metric name (test-gauge) and relied on fixedThread.sleepdurations for synchronization, making them flaky.The shared metric name caused cross-test pollution through the shared
InMemorySingletonMetricsExporter, which is the likely cause of the failure in build 73291 (expected 3.0 but was 5.0). Each test now uses a randomized metric name to isolate its metrics.Replace the initial
Thread.sleepwithassertBusypolling to wait for gauge values to be published. For the post-close assertion, useassertBusyto retry a check that snapshots the callback counter, waits longer than the publish interval, and verifies it has not changed. This handles the case where an in-flight collection that started beforeclose()is still draining.Resolves #19422