Export Flan-t5-xl to onnx

[kiranalii]

I'm trying to export flan-t5-xl to onnx format using the command

python convert_to_onnx.py -m 'google/flan-t5-xl' -o --output /dir --use_gpu -e
but It gives the following error

 onnxruntime/onnxruntime/python/tools/transformers/models/t5/t5_encoder_decoder_init.py", line 253, in verify_onnx
    ort_inputs = ort_session.get_inputs()
AttributeError: 'NoneType' object has no attribute 'get_inputs'

When i tried to debug it. Added print(ort_session) at line 230, in export_onnx_models

it says

  sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model from /dir/flan-tt-xl/google/flan-t5-xl_encoder_decoder_init_fp32.onnx failed:This is an invalid model. In Node, ("Attention_23", Attention, "com.microsoft", -1) : ("/t5_encoder/encoder/block.0/layer.0/layer_norm/Mul_1_output_0": tensor(float),"Attention_23_qkv_weight": tensor(float),"","encoder_attention_mask": tensor(int32),"","/t5_encoder/encoder/block.0/layer.0/SelfAttention/Unsqueeze_5_output_0": tensor(float),) -> ("/t5_encoder/encoder/block.0/layer.0/SelfAttention/Reshape_3_output_0",) , Error Node (Attention_23)'s input 2 is marked single but has an empty string in the graph

Is there anything I'm doing wrong? I successfully exported the model using optimum but the results are different from original model. So i thought to give it a try using onnxruntime directly

[Rakesh-dhilip]

from pathlib import Path
import transformers as t
from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForSeq2SeqLM

print out the version of the transformers library

print("transformers version:", t.version)

models = [
#"google/flan-t5-small",
#"google/flan-t5-base",
#"google/flan-t5-large",
"google/flan-t5-xl",
"google/flan-t5-xxl",
]

for model_id in models:
model_name = model_id.split("/")[1]
onnx_path = Path("onnx/" + model_name)

# load vanilla transformers and convert to onnx
model = ORTModelForSeq2SeqLM.from_pretrained(model_id, from_transformers=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# save onnx checkpoint and tokenizer
model.save_pretrained(onnx_path)
tokenizer.save_pretrained(onnx_path)

[Rakesh-dhilip]

Use optimum for onnx convertion