[CI] Ideas to reduce PR build and test time

[whitneywhtsang]

The build and test time has increased to 1.5 hours recently. Can we add features in CI to reduce unnecessary testing?
Below are some ideas:

1. Sometime only E2E step is failing, due to huggingface connectivity, and we would need to rerun the action, which rerun the whole workflow, from building the compiler, lit test, core test, etc, all the way to E2E testing. Suggest to only rerun failed steps if the PR is not changed, i.e., no new commits.
2. Cache the build step, and parallel testing steps.
3. Categorize different testing and trigger different portion by manually setting PR labels or PR comments. (Cache build step if we do this)

[whitneywhtsang]

FYI @pbchekin @Egor-Krivov

[pbchekin]

I think we will start with no. 2, it is simpler to implement.

[alexbaden]

I think we need to understand why our test_core run takes 1 hour 6 minutes while upstream can run the entire python test suite in under 10 minutes.

[pbchekin]

I think we need to understand why our test_core run takes 1 hour 6 minutes while upstream can run the entire python test suite in under 10 minutes.

Good point. I will collect durations of the tests for the beginning.

[alexbaden]

I pulled the slowest 100 from test_core using --durations=100:

============================================================================================================================================ slowest 100 durations =============================================================================================================================================
49.85s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-1-src_layout5-128-128]
48.41s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-1-src_layout5-128-128]
48.20s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-1-src_layout5-128-128]
47.05s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-expand_reduce2d-1-src_layout5-128-128]
46.78s call     test/unit/language/test_core.py::test_scan_layouts[True-1-src_layout10-64-32]
36.83s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-1-src_layout5-128-128]
35.61s call     test/unit/language/test_core.py::test_dot[1-128-128-64-4-False-False-chain-dot-ieee-float8e4nv-float32-1-None]
30.86s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-1-src_layout5-128-128]
28.99s call     test/unit/language/test_core.py::test_scan_layouts[True-0-src_layout10-64-32]
25.51s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-0-src_layout5-128-128]
22.47s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-0-src_layout5-128-128]
19.22s call     test/unit/language/test_core.py::test_dot3d[8-1-64-64-64-32-32-float32-float32]
18.06s call     test/unit/language/test_core.py::test_dot3d[8-1-64-64-64-32-32-float16-float32]
17.74s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e2m1-e4m3-4-16-1]
17.40s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e2m1-e4m3-4-16-1]
17.15s call     test/unit/language/test_core.py::test_dot[1-128-128-64-4-False-False-chain-dot-ieee-float8e5-float32-1-None]
16.43s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e2m1-e4m3-4-16-1]
16.37s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e2m1-e5m2-4-16-1]
16.34s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e2m1-e5m2-4-16-1]
15.93s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e4m3-e4m3-4-16-1]
15.77s call     test/unit/language/test_core.py::test_dot3d[8-1-64-64-64-32-32-float16-float16]
15.65s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-True-e2m1-e4m3-4-16-1]
15.54s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e5m2-e4m3-4-16-1]
15.48s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e5m2-e4m3-4-16-1]
15.43s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e4m3-e5m2-4-16-1]
15.13s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e4m3-e4m3-4-16-1]
15.06s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e2m1-e5m2-4-16-1]
15.00s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-1-src_layout5-128-16]
14.79s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e2m1-e4m3-4-16-1]
14.68s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-expand_reduce2d-0-src_layout5-128-128]
14.45s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e5m2-e5m2-4-16-1]
14.37s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e4m3-e5m2-4-16-1]
14.36s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e5m2-e5m2-4-16-1]
14.34s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e2m1-fp16-4-16-1]
14.34s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-0-src_layout5-128-128]
14.25s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e2m1-fp16-4-16-1]
14.04s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-1-src_layout5-128-16]
14.03s call     test/unit/language/test_core.py::test_tensor_atomic_add_non_exclusive_offset[128-1-float16]
13.96s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e2m1-bf16-4-16-1]
13.83s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e2m1-fp16-4-16-1]
13.57s call     test/unit/language/test_core.py::test_scan_layouts[True-1-src_layout10-32-64]
13.17s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-False-False-False-e2m1-e4m3-4-16-1]
12.96s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-True-e2m1-e5m2-4-16-1]
12.92s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e2m1-bf16-4-16-1]
12.92s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-False-False-False-e2m1-e5m2-4-16-1]
12.89s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e4m3-bf16-4-16-1]
12.80s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e4m3-bf16-4-16-1]
12.77s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e2m1-e5m2-4-16-1]
12.71s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e2m1-bf16-4-16-1]
12.65s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-False-True-e2m1-e4m3-4-16-1]
12.57s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e5m2-bf16-4-16-1]
12.51s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-True-True-e2m1-e4m3-4-16-1]
12.48s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-True-True-e5m2-e4m3-4-16-1]
12.47s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-False-e2m1-e4m3-4-16-1]
12.40s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-0-src_layout5-128-128]
12.28s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-0-src_layout5-128-128]
12.16s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-True-False-False-e2m1-e4m3-4-16-1]
11.99s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e4m3-fp16-4-16-1]
11.93s call     test/unit/language/test_core.py::test_scan_layouts[True-1-src_layout10-32-32]
11.93s call     test/unit/language/test_core.py::test_dot3d[8-1-64-64-64-32-32-int8-int8]
11.64s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-False-False-False-e5m2-e4m3-4-16-1]
11.61s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e4m3-fp16-4-16-1]
11.55s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-False-True-e2m1-e5m2-4-16-1]
11.50s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-True-True-e4m3-e4m3-4-16-1]
11.46s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-False-e2m1-e5m2-4-16-1]
11.43s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-1-src_layout5-32-128]
11.42s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e4m3-e4m3-4-16-1]
11.36s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-True-True-e2m1-e5m2-4-16-1]
11.22s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-False-False-False-e4m3-e5m2-4-16-1]
11.21s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e5m2-bf16-4-16-1]
11.14s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-True-True-e2m1-e4m3-4-16-1]
11.10s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-True-True-e4m3-e5m2-4-16-1]
11.09s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e4m3-e5m2-4-16-1]
11.08s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-1-src_layout5-128-16]
11.04s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-False-False-False-e5m2-e5m2-4-16-1]
10.99s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-True-False-False-e2m1-e5m2-4-16-1]
10.97s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-1-src_layout5-32-128]
10.93s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-True-True-e5m2-e5m2-4-16-1]
10.93s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-True-e4m3-e4m3-4-16-1]
10.86s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-True-e2m1-bf16-4-16-1]
10.84s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-1-src_layout5-128-16]
10.82s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-True-e5m2-e4m3-4-16-1]
10.78s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-False-False-False-e2m1-fp16-4-16-1]
10.77s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-True-e2m1-fp16-4-16-1]
10.76s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-1-src_layout5-32-128]
10.76s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-False-False-False-e2m1-bf16-4-16-1]
10.60s call     test/unit/language/test_core.py::test_scaled_dot[128-64-128-False-False-False-e4m3-e4m3-4-16-1]
10.57s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-False-False-e2m1-e5m2-4-16-1]
10.45s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-expand_reduce2d-1-src_layout5-32-128]
10.35s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e2m1-fp16-4-16-1]
10.34s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e5m2-e5m2-4-16-1]
10.34s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e5m2-e4m3-4-16-1]
10.34s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-True-False-e2m1-e4m3-4-16-1]
10.31s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-False-False-e2m1-e4m3-4-16-1]
10.28s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-True-e4m3-e5m2-4-16-1]
10.22s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e2m1-bf16-4-16-1]
10.18s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-True-True-e4m3-e4m3-4-16-1]
10.18s call     test/unit/language/test_core.py::test_scaled_dot[64-128-128-True-True-True-e4m3-bf16-4-16-1]
10.14s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-1-src_layout5-128-16]
10.14s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e5m2-e4m3-4-16-1]

somewhat surprised to see reduce_layouts at the top of the list. I will see if I can get a similar list for NVIDIA/H100.

[pbchekin]

First 100 slowest tests tor the whole run:

136.95 test/unit/tools/test_aot.py::test_compile_link_autotune_matmul
119.84 test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-256-128-2-4-32]
118.99 test/unit/language/test_matmul.py::test_mxfp[0-4-1-128-256-128-2-4-32]
114.92 test/unit/language/test_matmul.py::test_mxfp[0-4-1-128-256-128-2-4-64]
114.39 test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-256-128-2-4-64]
89.78  test/unit/language/test_matmul.py::test_mxfp[0-4-1-256-128-128-2-4-32]
86.73  test/unit/language/test_matmul.py::test_mxfp[0-4-3-256-128-128-2-4-32]
85.91  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-2-128-128-256-998-111-512]
82.18  test/unit/language/test_matmul.py::test_mxfp[0-4-1-256-128-128-2-4-64]
81.08  test/unit/language/test_matmul.py::test_mxfp[0-4-3-256-128-128-2-4-64]
78.07  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[True-1-128-128-256-998-111-512]
77.51  test/unit/language/test_matmul.py::test_mxfp[0-4-1-256-128-128-128-128-128]
77.22  test/unit/language/test_matmul.py::test_mxfp[0-4-3-256-128-128-1024-512-256]
75.44  test/unit/language/test_matmul.py::test_mxfp[0-4-3-256-128-128-128-128-128]
75.05  test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-256-128-128-128-128]
74.95  test/unit/language/test_matmul.py::test_mxfp[0-4-1-128-256-128-1024-512-256]
74.54  test/unit/language/test_matmul.py::test_mxfp[0-4-1-256-128-128-256-16-32]
74.42  test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-256-128-256-16-32]
73.83  test/unit/language/test_matmul.py::test_mxfp[0-4-1-128-256-128-256-16-32]
73.06  test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-256-128-1024-512-256]
72.11  test/unit/language/test_matmul.py::test_mxfp[0-4-3-256-128-128-256-16-32]
70.03  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-2-128-128-256-63-128-512]
63.22  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[True-1-128-128-256-1024-512-512]
55.62  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-4-128-256-128-998-111-512]
53.87  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[True-1-128-256-128-998-111-512]
52.73  test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-128-128-1024-512-256]
52.70  test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-128-128-2-4-64]
52.43  test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-128-128-2-4-32]
50.64  test/unit/language/test_matmul.py::test_mxfp[0-4-1-128-256-128-128-128-128]
47.46  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-4-256-128-128-998-111-512]
46.61  test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-1-src_layout5-128-128]
46.37  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-256-128-2-4-64]
45.74  test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-128-128-128-128-128]
45.04  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[True-1-256-128-128-998-111-512]
44.30  test/unit/language/test_matmul.py::test_mxfp[0-4-3-128-128-128-256-16-32]
43.54  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-256-128-2-4-32]
43.10  test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-1-src_layout5-128-128]
42.59  test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-expand_reduce2d-1-src_layout5-128-128]
42.09  test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-1-src_layout5-128-128]
41.75  test/unit/language/test_core.py::test_dot[1-128-128-64-4-False-False-chain-dot-ieee-float8e4nv-float32-1-None]
40.91  test/unit/language/test_core.py::test_scan_layouts[True-1-src_layout10-64-32]
40.59  test/unit/language/test_matmul.py::test_mxfp[0-8-1-256-128-128-2-4-64]
40.57  test/unit/tools/test_aot.py::test_compile_link_matmul[False]
40.52  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-4-256-128-128-1024-512-512]
39.71  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[True-1-128-256-128-1024-512-512]
39.49  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-4-128-256-128-63-128-512]
38.91  test/unit/language/test_pipeliner.py::test_pipeline_matmul[True]
38.81  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[True-1-128-256-128-63-128-512]
38.41  test/unit/language/test_matmul.py::test_mxfp[0-8-1-256-128-128-2-4-32]
38.30  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-4-128-256-128-1024-512-512]
36.75  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[True-1-256-128-128-1024-512-512]
35.29  test/unit/language/test_matmul.py::test_mxfp[0-8-1-256-128-128-256-16-32]
34.60  test/unit/language/test_matmul.py::test_mxfp[0-8-1-256-128-128-128-128-128]
34.58  test/unit/language/test_matmul.py::test_mxfp[0-8-1-256-128-128-1024-512-256]
34.00  test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-1-src_layout5-128-128]
33.67  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-256-128-256-16-32]
33.34  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-256-128-1024-512-256]
32.79  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-256-128-128-128-128]
30.69  test/unit/language/test_core.py::test_dot3d[8-1-64-64-64-32-32-float32-float32]
30.52  test/unit/language/test_core.py::test_dot3d[8-1-64-64-64-32-32-float16-float32]
30.31  test/unit/tools/test_aot.py::test_compile_link_matmul[True]
30.00  test/unit/language/test_core.py::test_dot3d[8-1-64-64-64-32-32-float16-float16]
28.42  test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-1-src_layout5-128-128]
25.56  test/unit/language/test_core.py::test_dot[1-128-128-64-4-False-False-chain-dot-ieee-float8e5-float32-1-None]
25.34  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-128-128-2-4-32]
25.19  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-4-128-128-128-998-111-512]
25.07  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-128-128-2-4-64]
24.90  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[True-1-128-128-128-998-111-512]
24.36  test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-0-src_layout5-128-128]
24.10  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e5m2-e4m3-4-16-1]
23.78  test/unit/language/test_core.py::test_scan_layouts[True-0-src_layout10-64-32]
23.37  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e4m3-e5m2-4-16-1]
23.35  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e2m1-e5m2-4-16-1]
23.16  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e5m2-e5m2-4-16-1]
23.07  test/unit/language/test_matmul.py::test_mxfp[0-4-1-256-128-128-1024-512-256]
23.06  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e2m1-e4m3-4-16-1]
22.90  test/unit/language/test_core.py::test_dot3d[8-1-64-64-64-32-32-int8-int8]
22.83  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-1-256-128-128-63-128-512]
22.27  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e2m1-e5m2-4-16-1]
21.88  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e2m1-e5m2-4-16-1]
21.70  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-4-128-128-128-1024-512-512]
21.61  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e4m3-e4m3-4-16-1]
21.18  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-128-128-128-128-128]
21.16  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e2m1-e4m3-4-16-1]
20.83  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e2m1-bf16-4-16-1]
20.76  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-128-128-1024-512-256]
20.75  test/unit/language/test_matmul.py::test_simple_matmul[4-1-64-512-32-2-float16-float8e5]
20.72  test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e2m1-e5m2-4-16-1]
20.67  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e2m1-e4m3-4-16-1]
20.61  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[False-1-128-256-128-1024-512-512]
20.37  test/unit/language/test_matmul.py::test_blocked_scale_mxfp[True-1-128-128-128-1024-512-512]
20.35  test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e2m1-e4m3-4-16-1]
20.12  test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-0-src_layout5-128-128]
20.11  test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e4m3-e5m2-4-16-1]
20.04  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e2m1-fp16-4-16-1]
19.95  test/unit/language/test_matmul.py::test_mxfp[0-8-1-128-128-128-256-16-32]
19.67  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e2m1-bf16-4-16-1]
19.64  test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e4m3-bf16-4-16-1]
19.55  test/unit/tools/test_aot.py::test_launcher_has_no_available_kernel[True]
19.32  test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-True-False-e4m3-e5m2-4-16-1]

[pbchekin]

With 32 pytest workers, the duration is ~1h:
https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/14253475294/job/39951453745

After splitting the workflow into two parallel workflows (core tests, everything else) i think it is possible to optimize the runtime further to ~35 min.

[alexbaden]

heavy hitters list from H100:

============================ slowest 100 durations =============================
37.44s call     test/unit/language/test_core.py::test_convert2d[dst_layout0-interm_layout3-src_layout14-float16-128-128]
25.79s call     test/unit/language/test_core.py::test_convert2d[dst_layout0-interm_layout3-src_layout15-float16-128-128]
9.56s call     test/unit/language/test_core.py::test_convert2d[dst_layout15-interm_layout1-src_layout14-float16-128-128]
9.50s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout3-src_layout15-float16-128-128]
8.49s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-1-src_layout7-128-128]
7.99s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-1-src_layout7-128-128]
7.94s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-1-src_layout6-128-128]
7.90s call     test/unit/language/test_core.py::test_convert2d[dst_layout1-interm_layout1-src_layout14-float16-128-128]
7.85s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-expand_reduce2d-1-src_layout7-128-128]
7.81s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-1-src_layout7-128-128]
7.78s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-1-src_layout6-128-128]
7.73s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-0-src_layout6-128-128]
7.50s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-0-src_layout6-128-128]
7.48s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout4-src_layout15-float16-128-128]
7.47s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-expand_reduce2d-0-src_layout6-128-128]
7.39s call     test/unit/language/test_core.py::test_convert2d[dst_layout2-interm_layout1-src_layout14-float16-128-128]
7.27s call     test/unit/language/test_core.py::test_convert2d[dst_layout0-interm_layout1-src_layout14-float16-128-128]
7.26s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-1-src_layout6-128-128]
7.25s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-1-src_layout6-128-128]
7.23s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-1-src_layout6-128-128]
7.16s call     test/unit/language/test_core.py::test_convert2d[dst_layout2-interm_layout3-src_layout15-float16-128-128]
7.16s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-expand_reduce2d-1-src_layout6-128-128]
7.07s call     test/unit/language/test_core.py::test_convert2d[dst_layout5-interm_layout3-src_layout15-float16-128-128]
7.04s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-1-src_layout7-128-128]
7.01s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-1-src_layout7-128-128]
6.90s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-0-src_layout6-128-128]
6.89s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-0-src_layout6-128-128]
6.88s call     test/unit/language/test_core.py::test_convert2d[dst_layout15-interm_layout3-src_layout14-float16-128-128]
6.84s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-0-src_layout6-128-128]
6.79s call     test/unit/language/test_core.py::test_convert2d[dst_layout1-interm_layout3-src_layout15-float16-128-128]
6.78s call     test/unit/language/test_core.py::test_convert2d[dst_layout5-interm_layout1-src_layout14-float16-128-128]
6.56s call     test/unit/language/test_core.py::test_convert2d[dst_layout5-interm_layout3-src_layout14-float16-128-128]
6.42s call     test/unit/language/test_core.py::test_convert2d[dst_layout2-interm_layout3-src_layout14-float16-128-128]
6.41s call     test/unit/language/test_core.py::test_convert2d[dst_layout1-interm_layout3-src_layout14-float16-128-128]
6.40s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout1-src_layout15-float16-128-128]
6.16s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout2-src_layout15-float16-128-128]
6.05s call     test/unit/language/test_core.py::test_convert2d[dst_layout15-interm_layout4-src_layout14-float16-128-128]
5.93s call     test/unit/language/test_core.py::test_convert2d[dst_layout1-interm_layout4-src_layout15-float16-128-128]
5.79s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-0-src_layout7-128-128]
5.78s call     test/unit/language/test_core.py::test_convert2d[dst_layout9-None-src_layout14-float16-128-128]
5.77s call     test/unit/language/test_core.py::test_convert2d[dst_layout0-interm_layout4-src_layout15-float16-128-128]
5.64s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-0-src_layout7-128-128]
5.56s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-expand_reduce2d-0-src_layout7-128-128]
5.49s call     test/unit/language/test_core.py::test_convert2d[dst_layout13-None-src_layout14-float16-128-128]
5.37s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce2d-1-src_layout8-128-128]
5.33s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-0-src_layout7-128-128]
5.30s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-0-src_layout7-128-128]
5.29s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-0-src_layout7-128-128]
5.24s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-expand_reduce2d-1-src_layout8-128-128]
5.22s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce2d-1-src_layout8-128-128]
5.21s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-expand_reduce2d-1-src_layout8-128-128]
5.18s call     test/unit/language/test_core.py::test_convert2d[dst_layout2-interm_layout4-src_layout15-float16-128-128]
5.17s call     test/unit/language/test_core.py::test_convert2d[dst_layout15-interm_layout2-src_layout14-float16-128-128]
5.15s call     test/unit/language/test_core.py::test_convert2d[dst_layout5-interm_layout4-src_layout15-float16-128-128]
5.05s call     test/unit/language/test_core.py::test_convert2d[dst_layout1-interm_layout4-src_layout14-float16-128-128]
4.91s call     test/unit/language/test_core.py::test_convert2d[dst_layout5-interm_layout4-src_layout14-float16-128-128]
4.87s call     test/unit/language/test_core.py::test_convert2d[dst_layout2-interm_layout4-src_layout14-float16-128-128]
4.64s call     test/unit/language/test_core.py::test_convert2d[dst_layout0-interm_layout2-src_layout15-float16-128-128]
4.63s call     test/unit/language/test_core.py::test_convert2d[dst_layout0-interm_layout4-src_layout14-float16-128-128]
4.59s call     test/unit/language/test_core.py::test_convert2d[dst_layout1-interm_layout2-src_layout15-float16-128-128]
4.49s call     test/unit/language/test_core.py::test_convert2d[dst_layout0-interm_layout1-src_layout15-float16-128-128]
4.48s call     test/unit/language/test_core.py::test_convert2d[dst_layout1-interm_layout1-src_layout15-float16-128-128]
4.47s call     test/unit/language/test_core.py::test_convert2d[dst_layout5-interm_layout2-src_layout15-float16-128-128]
4.47s call     test/unit/language/test_core.py::test_convert2d[dst_layout15-None-src_layout14-float16-128-128]
4.44s call     test/unit/language/test_core.py::test_convert2d[dst_layout2-interm_layout2-src_layout15-float16-128-128]
4.33s call     test/unit/language/test_core.py::test_convert2d[dst_layout2-interm_layout2-src_layout14-float16-128-128]
4.24s call     test/unit/language/test_core.py::test_reduce_layouts[sum-int32-True-reduce1d-1-src_layout8-128-128]
4.22s call     test/unit/language/test_core.py::test_reduce_layouts[max-int32-True-reduce1d-1-src_layout8-128-128]
4.14s call     test/unit/language/test_core.py::test_convert2d[dst_layout5-interm_layout1-src_layout15-float16-128-128]
4.11s call     test/unit/language/test_core.py::test_convert2d[dst_layout2-interm_layout1-src_layout15-float16-128-128]
4.11s call     test/unit/language/test_core.py::test_convert2d[dst_layout0-interm_layout2-src_layout14-float16-128-128]
4.11s call     test/unit/language/test_core.py::test_convert2d[dst_layout5-interm_layout2-src_layout14-float16-128-128]
4.10s call     test/unit/language/test_core.py::test_convert2d[dst_layout1-interm_layout2-src_layout14-float16-128-128]
3.51s call     test/unit/language/test_core.py::test_convert2d[dst_layout11-None-src_layout7-float16-128-128]
3.44s call     test/unit/language/test_core.py::test_convert2d[dst_layout10-None-src_layout14-float16-128-128]
3.24s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout3-src_layout0-float16-128-128]
3.14s call     test/unit/language/test_core.py::test_scan_layouts[True-1-src_layout10-64-32]
3.13s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout2-src_layout0-float16-128-128]
3.12s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout4-src_layout0-float16-128-128]
3.09s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout1-src_layout5-float16-128-128]
3.04s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout1-src_layout2-float16-128-128]
3.01s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout1-src_layout1-float16-128-128]
3.01s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout1-src_layout0-float16-128-128]
2.98s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout4-src_layout1-float16-128-128]
2.98s call     test/unit/language/test_core.py::test_convert2d[dst_layout4-None-src_layout15-float16-128-128]
2.84s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout3-src_layout1-float16-128-128]
2.83s call     test/unit/language/test_core.py::test_convert2d[dst_layout13-None-src_layout7-float16-128-128]
2.82s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout4-src_layout5-float16-128-128]
2.74s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout2-src_layout1-float16-128-128]
2.72s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout4-src_layout2-float16-128-128]
2.71s call     test/unit/language/test_core.py::test_convert2d[dst_layout9-None-src_layout15-float16-128-128]
2.69s call     test/unit/language/test_core.py::test_convert2d[dst_layout13-None-src_layout15-float16-128-128]
2.66s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout3-src_layout2-float16-128-128]
2.66s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout3-src_layout5-float16-128-128]
2.65s call     test/unit/language/test_core.py::test_gather_warp_shuffle[src_shape1-indices_shape1-0-linear<{register = [[0, 2], [32, 0], [2, 0], [0, 16], [0, 32], [64, 0]], lane = [[0, 8], [8, 0], [1, 0], [4, 0], [16, 0]], warp = [[0, 1], [0, 4]], block = []}>-linear<{register = [[0, 2], [32, 0], [0, 32], [2, 0], [0, 16], [64, 0], [128, 0]], lane = [[0, 8], [8, 0], [1, 0], [4, 0], [16, 0]], warp = [[0, 1], [0, 4]], block = []}>]
2.56s call     test/unit/language/test_core.py::test_convert_mma2mma[mma_pair6-float16-256-256]
2.55s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout2-src_layout5-float16-128-128]
2.53s call     test/unit/language/test_core.py::test_convert2d[dst_layout14-interm_layout2-src_layout2-float16-128-128]
2.53s call     test/unit/language/test_core.py::test_convert2d[dst_layout10-None-src_layout15-float16-128-128]
2.51s call     test/unit/language/test_core.py::test_convert_mma2mma[mma_pair2-float16-256-256]

[alexbaden]

Two things of note:

1. MXFP is not supported on H100. Presumably this is due to a lack of hardware support? I assume we are emulating MXFP now - is this practice worthwhile given the performance?
2. The H100 heavy hitters primarily feature layout conversions and reductions whereas ours are reductions and scaled_dot. I compared H100 and PVC 1100 scaled dot:

H100

============================== 10 slowest durations ===============================
1.73s call     test/unit/language/test_core.py::test_scaled_dot[32-32-128-True-True-True-e2m1-e4m3-4-16-1]
1.73s call     test/unit/language/test_core.py::test_scaled_dot[32-32-64-True-True-False-e2m1-e4m3-4-16-1]
1.73s call     test/unit/language/test_core.py::test_scaled_dot[32-64-64-False-False-False-e4m3-bf16-4-16-1]
1.72s call     test/unit/language/test_core.py::test_scaled_dot[32-64-128-True-False-True-e2m1-e4m3-4-16-1]
1.70s call     test/unit/language/test_core.py::test_scaled_dot[32-64-128-False-False-True-e4m3-bf16-4-16-1]
1.70s call     test/unit/language/test_core.py::test_scaled_dot[32-32-64-False-True-False-e4m3-bf16-4-16-1]
1.70s call     test/unit/language/test_core.py::test_scaled_dot[32-64-64-True-False-False-e2m1-e4m3-4-16-1]
1.70s call     test/unit/language/test_core.py::test_scaled_dot[32-32-128-False-True-True-e4m3-bf16-4-16-1]
0.21s call     test/unit/language/test_core.py::test_scaled_dot[128-32-128-True-True-True-e4m3-e5m2-4-16-1]
0.20s call     test/unit/language/test_core.py::test_scaled_dot[128-32-64-False-False-False-e5m2-e5m2-4-16-1]

============================ 1728 passed in 12.93s =============================

PVC 1100

============================== 10 slowest durations ===============================
16.96s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e2m1-e4m3-4-16-1]
16.54s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e2m1-e4m3-4-16-1]
15.84s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e5m2-e4m3-4-16-1]
15.66s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e4m3-e4m3-4-16-1]
15.60s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e4m3-e5m2-4-16-1]
15.58s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e2m1-e4m3-4-16-1]
15.28s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-True-True-e2m1-e5m2-4-16-1]
15.25s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-False-False-False-e2m1-e4m3-4-16-1]
14.70s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-False-e2m1-e5m2-4-16-1]
14.62s call     test/unit/language/test_core.py::test_scaled_dot[128-128-128-True-False-True-e2m1-e5m2-4-16-1]

======================= 1728 passed in 546.72s (0:09:06) =======================

is this delta expected to get better or is this due to a PVC hardware limitation?
If the delta is due to hardware limitations then maybe we could skip some shapes or datatypes on the regular PRs and only run them nightly? I suspect the gap is even worse with BMG, but it would be interested to test.

[whitneywhtsang]

FYI #3711 to investigate the scaled_dot performance.