Bug: Flaky tests in bindings and tracing

[mikehaller]

Description

Summary

The following test cases fail sporadically with a timeout error:

• //score/mw/com/impl/bindings/lola:event_subscription_control_test
• //score/mw/com/impl/bindings/lola:proxy_test
• //score/mw/com/impl/bindings/lola/tracing:tracing_runtime_test
• //score/mw/com/impl/tracing:proxy_event_tracing_test
• //score/mw/com/impl/tracing:skeleton_event_tracing_test

Example output of bazel test //...:

//score/mw/com/impl/bindings/lola:event_subscription_control_test       TIMEOUT in 60.0s
  /var/cache/bazel/.../execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/bindings/lola/event_subscription_control_test/test.log
...

Error Occurrence Rate

Sporadic

How to reproduce

Run all tests using bazel test --flaky_test_attempts 10 --nocache_test_results //... shows flaky tests:

//score/mw/com/impl/tracing:skeleton_event_tracing_test                   FLAKY, failed in 1 out of 2 in 60.0s
  Stats over 2 runs: max = 60.0s, min = 53.2s, avg = 56.6s, dev = 3.4s
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/tracing/skeleton_event_tracing_test/test_attempts/attempt_1.log
//score/mw/com/impl/tracing:tracing_runtime_test                          FLAKY, failed in 2 out of 3 in 60.0s
  Stats over 3 runs: max = 60.0s, min = 33.0s, avg = 51.0s, dev = 12.7s
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/tracing/tracing_runtime_test/test_attempts/attempt_1.log
  /var/cache/bazel/d130cfea8d610fd5cbbe7bb32e05decb/execroot/_main/bazel-out/k8-fastbuild/testlogs/score/mw/com/impl/tracing/tracing_runtime_test/test_attempts/attempt_2.log

Supporting Information

Environment:

• Bazel 8.3.0
• Up-to-date main branch
• S-CORE DevContainer with local modifications (corporate proxy settings) running on WSL2
• CPU and memory load are low

Logfiles

comm_flaky_tests_log.tar.gz

Observations

I observed the following behavior on my local environment:

• Running bazel test a second time may succeed. Test execution time is usually 30 to 40 seconds for each of those four tests, so the 60 seconds timeout seems to be a good value.
• Running ONLY the four tests will always succeed, which indicates an unwanted dependency / side effect from another test:

  # Always succeeds:
  bazel test --flaky_test_attempts 10 --nocache_test_results //score/mw/com/impl/bindings/lola:proxy_test //score/mw/com/impl/bindings/lola/tracing:tracing_runtime_test //score/mw/com/impl/tracing:skeleton_event_tracing_test //score/mw/com/impl/tracing:tracing_runtime_test
  

Classification

minor

Affected Version

0.5

Expected Closure Version

0.5

Category

• Safety Related
• Security Related