Add S/ES plot in Analysis and Allowance Config in test_compiler (#299)

* Update utils

* Update

* Update

* Add analysis_GMRS.py and relative JSON config

* distinguish eager and compiled exec error

* Update precision config format

* Update precision config format

* Update test compiler on torch

* handle fail_type

* Update tolerance configurations

* Update

* Add fake_perf_degrad

* Update

* Fix

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* process subgraph

* FIx

* Fix

* Fix correctness examination

* Add S and ES plot

* Update

* Optimize fpd

* Optimize fpd

* add stats log

* Update violin plot

* Update violin plot

* Update

* Update GMRs

* Update

* Optimize plot

* Update

* Update

* Update gamma cal

* Update

* Update

* Update

* Update Readme and regulate args.

* Optimize anatations

* Add Allowance Config

* Clean samples

* Process multi models using Config

* Update

* Update

* Modified param name in test compiler

* Modified param name in test compiler

* Update

* Update

* Update

* Update

* Update

* Update torch model allow list

Author: JewelRoam

README.md

@@ -99,7 +99,7 @@ python -m graph_net.torch.test_compiler \
   --trials /times/to/test/ \
   > $GRAPH_NET_BENCHMARK_PATH/log.log 2>&1
 
-# Note: if --compiler is omitted, PyTorch’s built-in compiler is used by default
+# Note: If --compiler is omitted, PyTorch’s built-in compiler is used by default.
 ```
 
 After executing, `graph_net.torch.test_compiler` will:
@@ -114,24 +114,31 @@ This step is to extract information (including failure) from logs in benchmark.
 All the information will be saved to multiple `model_compiler.json` files via:
 
 ```bash
-python -m graph_net.torch.log2json \
+python -m graph_net.log2json \
   --log-file $GRAPH_NET_BENCHMARK_PATH/log.log \
-  --output-dir $GRAPH_NET_BENCHMARK_PATH
+  --output-dir $GRAPH_NET_BENCHMARK_PATH/JSON_results/
 ```
 
 **Step 3: Analysis**
 
-After processing, we provide `graph_net/analysis.py` to generate [violin plot](https://en.m.wikipedia.org/wiki/Violin_plot) based on the JSON results.
+After processing, we provide `graph_net.violin_analysis` to generate [violin plot](https://en.m.wikipedia.org/wiki/Violin_plot) and `graph_net.S_analysis` to generate S and ES plot based on the JSON results.
 
 ```bash
-python -m graph_net.analysis \
-  --benchmark-path /path/to/read/JSON/result/file/ \
-  --output-dir /path/to/save/output/figures/
-```
+python -m graph_net.violin_analysis \
+  --benchmark-path $GRAPH_NET_BENCHMARK_PATH/JSON_results/ \
+  --output-dir $GRAPH_NET_BENCHMARK_PATH
 
-After executing, one summary plot of results on all compilers, as well as multiple sub-plots of results in categories (model tasks, Library...) on a single compiler will be exported. 
+python -m graph_net.S_analysis \
+  --benchmark-path $GRAPH_NET_BENCHMARK_PATH/JSON_results/ \
+  --output-dir $GRAPH_NET_BENCHMARK_PATH \
+  --negative-speedup-penalty penalty/power/for/negative/speedup \
+  --fpdb base/penalty/for/severe/errors
+
+# Note: If --negative-speedup-penalty is omitted, p=0 is used by default.
+# If --fpdb, b=0.1 is used by default.
+```
 
-The script is designed to process a file structure as `/benchmark_path/compiler_name/category_name/` (for example `/benchmark_logs/paddle/nlp/`), and items on x-axis are identified by name of the folders. So you can modify  `read_all_speedups` function to fit the benchmark settings on your demand.
+The scripts are designed to process a file structure as `/benchmark_path/category_name/`, and items on x-axis are identified by name of the sub-directories. After executing, several summary plots of result in categories (model tasks, libraries...) will be exported to `$GRAPH_NET_BENCHMARK_PATH`.
 
 ## 📌 Roadmap