Annotated Transformer

My learning record of the Annotated Transformer.

• Wrap the original code into a package.
• Visualization of the model.
• Train the model from scratch.
• Train the model with PyTorch Lightning.

Reference:

• The Annotated Transformer
• How to calculate the BLEU score?
• Jupyter Notebook provided as part of the Tensor2Tensor repo
• the attention.py in the Tensor2Tensor repo

From Scratch

python train_scratch.py

This line will call the train_scratch.py script to train the model from scratch. All the configure is in the script.

Once you have trained the model, you can evaluate the model, such as BLEU score, etc.

python eval_scratch.py

This is useful for me to understand the model and the training process.

Use Lightning to train on Multiple GPUs

python train_lightning.py

The model configuration is in the config.py file. You can change the configuration in the file.