Skip to content

Commit 2d4d4ec

Browse files
authored
fix: deflake //rs/tests/consensus/tecdsa:tecdsa_key_rotation_test (#9177)
## Root Cause Analysis In `await_pre_signature_stash_size` (`rs/tests/consensus/tecdsa/utils/src/lib.rs`), line 1141 had: ```rust assert_eq!(sizes.len(), subnet.nodes().count()); ``` This runs inside a `retry_with_msg!` loop. When `MetricsFetcher::fetch()` returns metrics, not all nodes may have started reporting the `execution_pre_signature_stash_size` metric yet. For example, only 1, 2, or 3 out of 4 nodes might be reporting. The `assert_eq!` **panics** in this case, killing the test immediately instead of allowing the retry loop to continue. This caused 5 out of 7 flaky failures in the last week with errors like: - `assertion left == right failed: left: 1, right: 4` - `assertion left == right failed: left: 2, right: 4` - `assertion left == right failed: left: 3, right: 4` The remaining 2 failures were timeouts (600s), likely caused by the same timing sensitivity. ## Fix Replaced `assert_eq!` with `bail!` so the retry loop continues when not all nodes have reported the metric yet, giving them time to catch up. ## Verification All 3 runs passed with `--runs_per_test=3 --jobs=3` (avg 271.5s). --- This PR was created following the steps in `.claude/skills/fix-flaky-tests/SKILL.md`.
1 parent 3856ad3 commit 2d4d4ec

File tree

1 file changed

+7
-1
lines changed
  • rs/tests/consensus/tecdsa/utils/src

1 file changed

+7
-1
lines changed

rs/tests/consensus/tecdsa/utils/src/lib.rs

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1138,7 +1138,13 @@ pub fn await_pre_signature_stash_size(
11381138
let Some(sizes) = val.get(metric) else {
11391139
bail!("Metric {metric} not found in {val:?}");
11401140
};
1141-
assert_eq!(sizes.len(), subnet.nodes().count());
1141+
if sizes.len() != subnet.nodes().count() {
1142+
bail!(
1143+
"Metric {metric} only reported by {} out of {} nodes",
1144+
sizes.len(),
1145+
subnet.nodes().count()
1146+
);
1147+
}
11421148
for size in sizes {
11431149
if *size != expected_size {
11441150
bail!(

0 commit comments

Comments
 (0)