-
Notifications
You must be signed in to change notification settings - Fork 16
Closed
Labels
test suite reliabilityIssue concerns tests that should behave more predictablyIssue concerns tests that should behave more predictably
Description
In #409 a FreeBSD CI run on 5.0.0 failed to trigger a counterexample in STM Sys test parallel within 1000 attempts:
https://ocaml-multicoretests.ci.dev:8100/job/2023-11-09/084832-ci-ocluster-build-542efa
random seed: 4002063
generated error fail pass / total time test name
[ ] 0 0 0 0 / 1000 0.0s STM Sys test sequential
[ ] 0 0 0 0 / 1000 0.0s STM Sys test sequential (generating)
[✓] 1000 0 0 1000 / 1000 4.8s STM Sys test sequential
[ ] 0 0 0 0 / 1000 0.0s STM Sys test parallel
[ ] 512 0 0 512 / 1000 55.3s STM Sys test parallel
[✗] 1000 0 0 1000 / 1000 115.0s STM Sys test parallel
--- Failure --------------------------------------------------------------------
Test STM Sys test parallel failed:
Negative test STM Sys test parallel succeeded but was expected to fail
================================================================================
failure (1 tests failed, 0 tests errored, ran 2 tests)
File "src/sys/dune", line 4, characters 7-16:
4 | (name stm_tests)
^^^^^^^^^
(cd _build/default/src/sys && ./stm_tests.exe --verbose)
Command exited with code 1.
Metadata
Metadata
Assignees
Labels
test suite reliabilityIssue concerns tests that should behave more predictablyIssue concerns tests that should behave more predictably