Merge branch 'develop' of https://github.com/PaddlePaddle/GraphNet into describe

Author: shanjiang7

.github/actions/check-bypass/action.yml

@@ -1,5 +1,5 @@
 name: "Check bypass"
-description: "A custom action to encapsulate PFCCLab/ci-bypass"
+description: "A custom action to encapsulate GraphNet"
 inputs:
   github-token:
     description: "GitHub token"
@@ -18,7 +18,7 @@ runs:
     - id: check-bypass
       name: Check Bypass
       env:
-        CI_TEAM_MEMBERS: '["SigureMo", "risemeup1", "tianshuo78520a", "0x3878f", "swgu98", "luotao1", "XieYunshen"]'
+        CI_TEAM_MEMBERS: '["lixinqi", "Xreki"]'
       uses: PFCCLab/ci-bypass@v1
       with:
         github-token: ${{ inputs.github-token }}

README.md

@@ -63,6 +63,8 @@ model = graph_net.torch.extract(name="model_name")(model)
 #   $GRAPH_NET_EXTRACT_WORKSPACE/model_name
 ```
 
+For details, see docstring of `graph_net.torch.extract` defined in `graph_net/torch/extractor.py`
+
 **graph_net.torch.validate**
 ```
 # Verify that the extracted model meets requirements
@@ -85,17 +87,44 @@ We define two key metrics here: **rectified speedup** and **GraphNet Score**. Re
 
 **Demo: How to benchmark your compiler on the model:**
 
+1. Benchmark
+
+We use ```graph_net/benchmark_demo.sh``` to benchmark GraphNet computation graph samples:
+
+```
+bash graph_net/benchmark_demo.sh &
+```
+
+The script will run ```graph_net.torch.test_compiler``` with specific batch and log configurations.
+
+Or you can customize and use ```graph_net.torch.test_compiler``` yourself:
+
 ```
 python3 -m graph_net.torch.test_compiler \
   --model-path $GRAPH_NET_EXTRACT_WORKSPACE/model_name/ \
-  --compiler /path/to/custom/compiler 
+  --compiler /path/to/custom/compiler/ \
+  --output-dir /path/to/save/JSON/result/file/
 # Note: if --compiler is omitted, PyTorch’s built-in compiler is used by default
 ```
 
+2. Analysis
+
+After processing, we provide ```graph_net/analysis.py``` to generate [violin plot](https://en.m.wikipedia.org/wiki/Violin_plot) based on the JSON results.
+
+```
+python3 graph_net/analysis.py \
+  --benchmark-path /path/to/read/JSON/result/file/ \
+  --output-dir /path/to/save/output/figures/
+```
+
+After executing, one summary plot of results on all compilers (as shown below in "Evaluation Results Example"), as well as multiple sub-plots of results in categories (model tasks, Library...) on a single compiler. 
+
+The script is designed to process a file structure as ```/benchmark_path/compiler_name/category_name/``` (for example ```/benchmark_logs/paddle/nlp/```), and items on x-axis are identified by name of the folders. So you can modify  ```read_all_speedups``` function to fit the benchmark settings on your demand.
+
 ### Evaluation Results Example
 
 <div align="center">
-<img src="/pics/Eval_result.jpg" alt="Violin plots of rectified speedup distributions" width="65%">
+<img src="/pics/Eval_result.png" alt="Violin plots of rectified speedup distributions" width="65%">
 </div>
 
 

graph_net/benchmark_demo.sh

@@ -31,7 +31,7 @@ for package_path in "${samples_dir}"/*/; do
 
                 echo "[$(date)] FINISHED: ${package_name}/${model_name}"
             fi
-        } >> "$global_log" 2>&1 &
+        } >> "$global_log" 2>&1
     done
 done
 

graph_net/test/nlp_model_getter.py

@@ -0,0 +1,109 @@
+def get_auto_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
+
+    config = AutoConfig.from_pretrained(model_name)
+    model = AutoModelForCausalLM.from_config(config, dtype=dtype)
+    model = model.eval()
+
+    tokenizer = AutoTokenizer.from_pretrained(model_name)
+    tokenizer.pad_token = tokenizer.eos_token
+    inputs = tokenizer(
+        text, return_tensors="pd", padding=True, truncation=True, max_length=2048
+    )
+    return model, inputs
+
+
+def get_bert_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import BertModel, BertTokenizer
+
+    model = BertModel.from_pretrained(model_name)
+    model.eval()
+
+    tokenizer = BertTokenizer.from_pretrained(model_name)
+    inputs = tokenizer(text, return_tensors="pd")
+    return model, inputs
+
+
+def get_convbert_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import ConvBertModel as ModelClass
+    from paddlenlp.transformers import ConvBertTokenizer as TokenizerClass
+
+    model = ModelClass.from_pretrained(model_name)
+    model.eval()
+
+    tokenizer = TokenizerClass.from_pretrained(model_name)
+    inputs = tokenizer(text, return_tensors="pd")
+    return model, inputs
+
+
+def get_ernie_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import ErnieModel, ErnieTokenizer
+
+    model = ErnieModel.from_pretrained(model_name)
+    tokenizer = ErnieTokenizer.from_pretrained(model_name)
+    inputs = tokenizer(text, return_tensors="pd")
+    return model, inputs
+
+
+def get_ernie_m_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import ErnieMModel as ModelClass
+    from paddlenlp.transformers import ErnieMTokenizer as TokenizerClass
+
+    model = ModelClass.from_pretrained(model_name)
+    model.eval()
+
+    tokenizer = TokenizerClass.from_pretrained(model_name)
+    inputs = tokenizer(text, return_tensors="pd")
+    return model, inputs
+
+
+def get_gpt_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import GPTModel, GPTTokenizer
+
+    model = GPTModel.from_pretrained(model_name)
+    model.eval()
+
+    tokenizer = GPTTokenizer.from_pretrained(model_name)
+    inputs = tokenizer(text, return_tensors="pd")
+    inputs.pop("token_type_ids")
+    return model, inputs
+
+
+def get_nezha_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import NeZhaModel as ModelClass
+    from paddlenlp.transformers import NeZhaTokenizer as TokenizerClass
+
+    model = ModelClass.from_pretrained(model_name)
+    tokenizer = TokenizerClass.from_pretrained(model_name)
+    inputs = tokenizer(text, return_tensors="pd")
+    return model, inputs
+
+
+def get_ppminilm_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import PPMiniLMModel as ModelClass
+    from paddlenlp.transformers import PPMiniLMTokenizer as TokenizerClass
+
+    model = ModelClass.from_pretrained(model_name)
+    tokenizer = TokenizerClass.from_pretrained(model_name)
+    inputs = tokenizer(text, return_tensors="pd")
+    return model, inputs
+
+
+def get_reformer_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import RoFormerModel as ModelClass
+    from paddlenlp.transformers import RoFormerTokenizer as TokenizerClass
+
+    model = ModelClass.from_pretrained(model_name)
+    tokenizer = TokenizerClass.from_pretrained(model_name)
+    inputs = tokenizer(text, return_tensors="pd")
+    return model, inputs
+
+
+def get_skep_model_and_inputs(model_name, text, dtype):
+    from paddlenlp.transformers import SkepModel as ModelClass
+    from paddlenlp.transformers import SkepTokenizer as TokenizerClass
+
+    model = ModelClass.from_pretrained(model_name)
+    tokenizer = TokenizerClass.from_pretrained(model_name)
+    inputs = tokenizer(text, return_tensors="pd")
+    return model, inputs