Skip to content

Commit a31c79a

Browse files
committed
test: move "doing A/B-Test" print statement below ignore-check
Move the print statement that states which A/B-Tests we're performing below the check that filters out ignored test cases. This way, we avoid claiming to do an A/B-Test for things that are later actually filtered out (meaning no A/B-Test actually happens). Signed-off-by: Patrick Roy <[email protected]>
1 parent 0f306e5 commit a31c79a

File tree

1 file changed

+2
-3
lines changed

1 file changed

+2
-3
lines changed

tools/ab_test.py

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -211,9 +211,6 @@ def analyze_data(
211211
), "A and B run produced incomparable data. This is a bug in the test!"
212212

213213
for metric, (values_a, unit) in metrics_a.items():
214-
print(
215-
f"Doing A/B-test for dimensions {dimension_set} and property {metric}"
216-
)
217214
result = check_regression(
218215
values_a, metrics_b[metric][0], n_resamples=n_resamples
219216
)
@@ -269,6 +266,8 @@ def analyze_data(
269266
if is_ignored(dict(dimension_set)):
270267
continue
271268

269+
print(f"Doing A/B-test for dimensions {dimension_set} and property {metric}")
270+
272271
values_a = processed_emf_a[dimension_set][metric][0]
273272
baseline_mean = statistics.mean(values_a)
274273

0 commit comments

Comments
 (0)