Skip to content

Commit 64dabb2

Browse files
laithsakkapytorchmergebot
authored andcommitted
only fail regressions>10% on pr_time benchmarks (pytorch#158577)
Moving to a new framework, maintaitning the pr_time benchmark test right now is hard and often breaking. 1. only fail PRs >10% regressions. 2. post monitor with pr_time benchmarks dashboard (oncall), and update expected results (frequently or on big changes) (supposed to already be doing https://www.internalfb.com/unidash/dashboard/pt2_diff_time_metrics) 3. setting up some one detections detectors warnings that would be triggered at regressions and notify internally post land https://www.internalfb.com/monitoring/detector/1140915271179237 Pull Request resolved: pytorch#158577 Approved by: https://github.com/xmfan, https://github.com/janeyx99
1 parent ab55742 commit 64dabb2

File tree

1 file changed

+23
-23
lines changed

1 file changed

+23
-23
lines changed
Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,89 +1,89 @@
1-
add_loop_eager,compile_time_instruction_count,3017000000,0.015
1+
add_loop_eager,compile_time_instruction_count,3070000000,0.10
22

33

44

5-
add_loop_eager_dynamic,compile_time_instruction_count,4352000000,0.025
5+
add_loop_eager_dynamic,compile_time_instruction_count,4432000000,0.10
66

77

88

9-
add_loop_inductor,compile_time_instruction_count,29490000000,0.015
9+
add_loop_inductor,compile_time_instruction_count,30280000000,0.10
1010

1111

1212

13-
add_loop_inductor_dynamic_gpu,compile_time_instruction_count,38760000000,0.025
13+
add_loop_inductor_dynamic_gpu,compile_time_instruction_count,39910000000,0.10
1414

1515

1616

17-
add_loop_inductor_gpu,compile_time_instruction_count,26000000000,0.015
17+
add_loop_inductor_gpu,compile_time_instruction_count,26800000000,0.10
1818

1919

2020

21-
basic_modules_ListOfLinears_eager,compile_time_instruction_count,947600000,0.015
21+
basic_modules_ListOfLinears_eager,compile_time_instruction_count,969100000,0.10
2222

2323

2424

25-
basic_modules_ListOfLinears_inductor,compile_time_instruction_count,18490000000,0.015
25+
basic_modules_ListOfLinears_inductor,compile_time_instruction_count,18030000000,0.10
2626

2727

2828

29-
basic_modules_ListOfLinears_inductor_gpu_force_shape_pad,compile_time_instruction_count,17020000000,0.015
29+
basic_modules_ListOfLinears_inductor_gpu_force_shape_pad,compile_time_instruction_count,17020000000,0.10
3030

3131

3232

33-
basic_modules_ListOfLinears_inductor_gpu,compile_time_instruction_count,10297683772,0.2
33+
basic_modules_ListOfLinears_inductor_gpu,compile_time_instruction_count,11090000000,0.2
3434

3535

3636

37-
update_hint_regression,compile_time_instruction_count,1673000000,0.02
37+
update_hint_regression,compile_time_instruction_count,1719000000,0.10
3838

3939

4040

41-
sum_floordiv_regression,compile_time_instruction_count,986800000,0.015
41+
sum_floordiv_regression,compile_time_instruction_count,966100000,0.10
4242

4343

4444

45-
symint_sum,compile_time_instruction_count,3166000000,0.015
45+
symint_sum,compile_time_instruction_count,3237000000,0.10
4646

4747

4848

49-
symint_sum_loop,compile_time_instruction_count,4202000000,0.015
49+
symint_sum_loop,compile_time_instruction_count,4299000000,0.10
5050

5151

5252

53-
aotdispatcher_inference_nosubclass_cpu,compile_time_instruction_count,2103000000,0.015
53+
aotdispatcher_inference_nosubclass_cpu,compile_time_instruction_count,2151000000,0.10
5454

5555

5656

57-
aotdispatcher_inference_subclass_cpu,compile_time_instruction_count,6004000000,0.015
57+
aotdispatcher_inference_subclass_cpu,compile_time_instruction_count,6124000000,0.10
5858

5959

6060

61-
aotdispatcher_partitioner_cpu,compile_time_instruction_count,8783000000,0.015
61+
aotdispatcher_partitioner_cpu,compile_time_instruction_count,9005000000,0.10
6262

6363

6464

65-
aotdispatcher_partitioner_cpu2,compile_time_instruction_count,1940000000,0.015
65+
aotdispatcher_partitioner_cpu2,compile_time_instruction_count,1989000000,0.10
6666

6767

6868

69-
aotdispatcher_training_nosubclass_cpu,compile_time_instruction_count,3885000000,0.015
69+
aotdispatcher_training_nosubclass_cpu,compile_time_instruction_count,3959000000,0.10
7070

7171

7272

73-
aotdispatcher_training_subclass_cpu,compile_time_instruction_count,10470000000,0.015
73+
aotdispatcher_training_subclass_cpu,compile_time_instruction_count,10650000000,0.10
7474

7575

7676

77-
mm_loop_inductor_gpu,compile_time_instruction_count,4324000000,0.015
77+
mm_loop_inductor_gpu,compile_time_instruction_count,4461000000,0.10
7878

7979

8080

81-
mm_loop_inductor_dynamic_gpu,compile_time_instruction_count,8116000000,0.015
81+
mm_loop_inductor_dynamic_gpu,compile_time_instruction_count,8417000000,0.10
8282

8383

8484

85-
basic_NestedModule_eager,compile_time_instruction_count,8152524390,0.015
85+
basic_NestedModule_eager,compile_time_instruction_count,8348000000,0.10
8686

8787

8888

89-
basic_InlineMod_eager,compile_time_instruction_count,7255000000,0.015
89+
basic_InlineMod_eager,compile_time_instruction_count,7464000000,0.10

0 commit comments

Comments
 (0)