Help running onnxruntime_training_bert sample

[georgen117]

Is there any documentation on how to run the onnxruntime_training_bert C++ sample?

I would love if there were a readme showing how to run the training samples but I have not been able to locate one. I was able to figure out how to run the MNIST training sample but I don't know how to run the BERT training sample.

What I would like to know:

• Where do I go to download the training data?
• Is there a recommended BERT model to use for training? If so, where can I download it?
• What set of command-line options should I provided to launch the sample? (most options look like they have reasonable defaults but not all.)
• Does the sample require CUDA or can I run it using GPU only?

[SherlockNoMad]

Hi @georgen117, do you have to use C++ onnxruntime_training_bert example?
We have a python frontend for bert-training. That would be a better starting point for user. We recommend you starting with https://github.com/microsoft/onnxruntime-training-examples

Where do I go to download the training data?
The data is hosted internally, and lives in a special internal format.

Does the sample require CUDA or can I run it using GPU only?
It should also run with CPU, but it will be significantly slower.

[georgen117]

I am adding NLP ops for an execution provider. For inference the onnx_test_runner is perfect for validating the ops but validating the backward pass is much more difficult.

I have already started with the onnxruntime-training-examples but debugging issues is much more difficult when you have to jump between two languages.

[pranav-prakash]

@georgen117 Out of curiosity did you ever manage to create a C++ trainer for bert? I think the main hurdle would be writing the training data loader (in the case of bert I think this would involve input tokenization).

[georgen117]

I have not created a C++ trainer for bert yet. Right now I have been focusing on the inference operators that are not yet supported by our execution provider. Once I get to the backward ops I will revisit the bert training sample.