Skip to content

Commit c1fc21c

Browse files
authored
Fix timing-related flakiness in telemetry acceptance test (#3721)
## Why The test checks that at least 5 mutators recorded execution times, but on fast machines, many mutators complete in under 1ms and are not recorded (per the threshold in bundle/mutator.go). This caused intermittent failures when exactly 5 mutators were recorded. The test's purpose is to verify that the telemetry mechanism works, not to enforce a specific count. We changed the assertion to check for at least one recorded mutator execution time, making the test robust across all machine speeds. ## Tests This failed here https://github.com/databricks/cli/actions/runs/18225329786/job/51894965321
1 parent 09d5952 commit c1fc21c

File tree

2 files changed

+6
-5
lines changed

2 files changed

+6
-5
lines changed

acceptance/bundle/telemetry/deploy/output.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,6 @@ Deployment complete!
77

88
>>> cat out.requests.txt
99

10-
=== Assert that there are atleast 5 mutators for which the execution time is recorded
10+
=== Assert that mutator execution times are being recorded
1111
>>> cat telemetry.json
1212
true

acceptance/bundle/telemetry/deploy/script

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,11 @@ trace $CLI bundle deploy
22

33
trace cat out.requests.txt | jq 'select(has("path") and .path == "/telemetry-ext") | .body.protoLogs[] | fromjson' > telemetry.json
44

5-
# Note that the number 5 here is arbitrary. In practice I saw 16 mutators be recorded in acceptance test runs so 5 seems
6-
# like a reasonably safe number to assert that mutator execution times are being recorded.
7-
title "Assert that there are atleast 5 mutators for which the execution time is recorded"
8-
trace cat telemetry.json | jq ' .entry.databricks_cli_log.bundle_deploy_event.experimental.bundle_mutator_execution_time_ms | length > 5'
5+
# Assert that the telemetry mechanism is working and recording mutator execution times.
6+
# We only check that at least one mutator execution time is recorded to avoid flakiness on fast machines
7+
# where many mutators may complete in less than 1ms (the threshold for recording in bundle/mutator.go).
8+
title "Assert that mutator execution times are being recorded"
9+
trace cat telemetry.json | jq ' .entry.databricks_cli_log.bundle_deploy_event.experimental.bundle_mutator_execution_time_ms | length > 0'
910

1011
# bundle_mutator_execution_time_ms can have variable number of entries depending upon the runtime of the mutators. Thus we omit it from
1112
# being asserted here.

0 commit comments

Comments
 (0)