[TF-TRT] v2.0.0 - XLA FP16 Fix && HF BERT/BART Re-Alignment

Author: DEKHTIARJonathan

tftrt/benchmarking-python/CHANGELOG.md

@@ -30,12 +30,12 @@ Description of the change
 ##  How To Update The Changelog for a New Release ?
 
  - **Major Version:** Major refactoring, or changes that make the data
-                  non-comparable with any previous release of the benchmark.
-                  Changes in shell scripts are to be expected.
+                   non-comparable with any previous release of the benchmark.
+                   Changes in shell scripts are to be expected.
 
  - **Minor Version:** Changes that add a new functionality though not modifying
-                  existing metrics or models scripts in a way that would make
-                  metrics not comparable between minor releases.
+                   existing metrics or models scripts in a way that would make
+                   metrics not comparable between minor releases.
 
  - **Patch Version:** Changes that are expected to have no change to the
                    operation of the benchmark nor the way metrics are
@@ -46,6 +46,16 @@ Description of the change
 
 <!-- YOU CAN EDIT FROM HERE -->
 
+## [2.0.0] - 2022.08.04 - @DEKHTIARJonathan
+
+- Fix for XLA FP16 actually not being applied due to `"min_graph_nodes": -1`
+missing. Therefore AMP skipped when XLA is able to compile the whole graph in
+virtually one node.
+
+- HF BERT & BART target changed:
+    - `TFBertForPreTraining` -> `TFBertModel`
+    - `TFBartForConditionalGeneration` -> `TFBartModel`
+
 ## [1.2.0] - 2022.07.31 - @DEKHTIARJonathan
 
 Setting up the benchmarking suite to allow remote upload and storage of the

tftrt/benchmarking-python/benchmark_info.py

@@ -10,7 +10,7 @@
 # The `__version__` number shall be updated everytime core benchmarking files
 # are updated.
 # Please update CHANGELOG.md with a description of what this version changed.
-__version__ = "1.2.0"
+__version__ = "2.0.0"
 
 
 def get_commit_id():

tftrt/benchmarking-python/benchmark_runner.py

@@ -243,15 +243,20 @@ def load_model_from_disk(
             signature_key=signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY,
             precision="FP32"
         ):
+
+            tf.config.optimizer.set_experimental_options({
+                "disable_model_pruning": False,
+                "debug_stripper": True,
+                "auto_mixed_precision": precision != "FP32",
+                "layout_optimizer": True,
+                "dependency_optimization": True,
+                "min_graph_nodes": -1  # do not skip small graphs
+            })
+
             saved_model_loaded = tf.saved_model.load(export_dir=path, tags=tags)
 
             graph_func = saved_model_loaded.signatures[signature_key]
 
-            if precision == "FP16":
-                tf.config.optimizer.set_experimental_options({
-                    "auto_mixed_precision": True
-                })
-
             # Known TF Issue: https://github.com/tensorflow/tensorflow/issues/37615#issuecomment-767804930
             # it looks like if the original trackable object is released by
             # the Python garbage collector once it goes out of scope, and

tftrt/benchmarking-python/huggingface/bert/base_run_inference.sh

@@ -62,7 +62,7 @@ MIN_SEGMENT_SIZE=5
 VOCAB_SIZE=-1
 MAX_WORKSPACE_SIZE=$((2 ** (32 + 1)))  # + 1 necessary compared to python
 MAX_SAMPLES=1
-OUTPUT_TENSORS_NAME="prediction_logits,seq_relationship_logits"
+OUTPUT_TENSORS_NAME="last_hidden_state,pooler_output"
 
 case ${MODEL_NAME} in
   "bert_base_uncased" | "bert_large_uncased")
@@ -76,7 +76,7 @@ case ${MODEL_NAME} in
   "bart_base" | "bart_large")
     VOCAB_SIZE=50265
     MIN_SEGMENT_SIZE=90
-    OUTPUT_TENSORS_NAME="encoder_last_hidden_state,logits"
+    OUTPUT_TENSORS_NAME="encoder_last_hidden_state,last_hidden_state"
     ;;
 esac
 

tftrt/benchmarking-python/huggingface/bert/download_all_models.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/bash
+
+BASE_DIR="/models/huggingface/transformers"
+
+GPT2_MODELS=(
+    "bert-base-uncased"
+    "bert-base-cased"
+    "bert-large-uncased"
+    "bert-large-cased"
+    "facebook/bart-base"
+    "facebook/bart-large"
+)
+
+
+for model_name in "${GPT2_MODELS[@]}"; do
+    echo "Processing: ${model_name} ..."
+    sleep 1
+
+    MODEL_DIR=${model_name#*/}  # Remove `facebook/`
+    MODEL_DIR=${MODEL_DIR//-/_}  # Replace "-" by "_"
+
+    MODEL_DIR="${BASE_DIR}/${MODEL_DIR}"
+    rm -rf ${MODEL_DIR} && mkdir -p ${MODEL_DIR}
+    echo "Model Dir: ${MODEL_DIR}"
+
+    set -x
+    python generate_saved_models.py --output_directory ${MODEL_DIR} --model_name ${model_name}
+    saved_model_cli show --dir "${MODEL_DIR}/pb_model" --all 2>&1 | tee ${MODEL_DIR}/pb_model/analysis.txt
+    set +x
+done