Skip to content

Commit 60f6d6c

Browse files
frederick0329tensorflower-gardener
frederick0329
authored and
tensorflower-gardener
committed
Internal change
PiperOrigin-RevId: 418817477
1 parent 5fea53a commit 60f6d6c

File tree

1 file changed

+48
-32
lines changed

1 file changed

+48
-32
lines changed

‎official/nlp/MODEL_GARDEN.md

Lines changed: 48 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -2,53 +2,69 @@
22

33
## Introduction
44

5-
This TF-NLP library provides a collection of scripts for the training and
6-
evaluation of transformer-based models, on various tasks such as sentence
5+
The TF-NLP library provides a collection of scripts for training and
6+
evaluating transformer-based models, on various tasks such as sentence
77
classification, question answering, and translation. Additionally, we provide
88
checkpoints of pretrained models which can be finetuned on downstream tasks.
99

1010
### How to Train Models
1111

12-
Model Garden can be easily installed using PIP
13-
(`pip install tf-models-nightly`). After installation, check out
12+
Model Garden can be easily installed with
13+
`pip install tf-models-nightly`. After installation, check out
1414
[this instruction](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md)
1515
on how to train models with this codebase.
1616

17-
## Available Tasks
1817

19-
There are two available model configs (we will add more) under
20-
`configs/experiments/`:
18+
By default, the experiment runs on GPUs. To run on TPUs, one should overwrite
19+
`runtime.distribution_strategy` and set the tpu address. See [RuntimeConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py) for details.
20+
21+
In general, the experiments can run with the folloing command by setting the
22+
corresponding `${TASK}`, `${TASK_CONFIG}`, `${MODEL_CONFIG}`.
23+
```
24+
EXPERIMENT=???
25+
TASK_CONFIG=???
26+
MODEL_CONFIG=???
27+
EXRTRA_PARAMS=???
28+
MODEL_DIR=??? # a-folder-to-hold-checkpoints-and-logs
29+
python3 train.py \
30+
--experiment=${EXPERIMENT} \
31+
--mode=train_and_eval \
32+
--model_dir=${MODEL_DIR} \
33+
--config_file=${TASK_CONFIG} \
34+
--config_file=${MODEL_CONFIG} \
35+
--params_override=${EXRTRA_PARAMS}
36+
```
37+
38+
* `EXPERIMENT` can be found under `configs/`
39+
* `TASK_CONFIG` can be found under `configs/experiments/`
40+
* `MODEL_CONFIG` can be found under `configs/models/`
41+
42+
#### Order of params override:
43+
1. `train.py` looks up the registered `ExperimentConfig` with `${EXPERIMENT}`
44+
2. Overrides params in `TaskConfig` in `${TASK_CONFIG}`
45+
3. Overrides params `model` in `TaskConfig` with `${MODEL_CONFIG}`
46+
4. Overrides any params in `ExperimentConfig` with `${EXTRA_PARAMS}`
47+
48+
Note that
49+
1. `${TASK_CONFIG}`, `${MODEL_CONFIG}`, `${EXTRA_PARAMS}` can be optional when EXPERIMENT default is enough.
50+
2. `${TASK_CONFIG}`, `${MODEL_CONFIG}`, `${EXTRA_PARAMS}` are only guaranteed to be compatible to it's `${EXPERIMENT}` that defines it.
51+
52+
## Experiments
53+
54+
| NAME | EXPERIMENT | TASK_CONFIG | MODEL_CONFIG | EXRTRA_PARAMS |
55+
| ----------------- | ------------------------ | ------- | -------- | ----------- |
56+
| BERT-base GLUE/MNLI-matched finetune | [bert/sentence_prediction](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | <details> <summary>data and bert-base hub init</summary>task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4 </details> |
57+
| BERT-base GLUE/MNLI-matched finetune | [bert/sentence_prediction](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | <details> <summary>data and bert-base ckpt init</summary>task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.init_checkpoint=gs://tf_model_garden/nlp/bert/uncased_L-12_H-768_A-12/bert_model.ckpt </details> |
58+
| BERT-base SQuAD v1.1 finetune | [bert/squad](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | <details> <summary>data and bert-base hub init</summary>task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4 </details> |
59+
|ALBERT-base SQuAD v1.1 finetune | [bert/squad](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | [albert_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/albert_base.yaml)| <details> <summary>data and albert-base hub init</summary>task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/albert_en_base/3 </details>|
60+
| Transformer-large WMT14/en-de scratch |[wmt_transformer/large](https://github.com/tensorflow/models/blob/master/official/nlp/configs/wmt_transformer_experiments.py)| | | <details> <summary>ende-32k sentencepiece</summary>task.sentencepiece_model_path='gs://tf_model_garden/nlp/transformer_wmt/ende_bpe_32k.model'</details> |
2161

22-
| Dataset | Task | Config | Example command |
23-
| ----------------- | ------------------------ | ------- | ---- |
24-
| GLUE/MNLI-matched | bert/sentence_prediction | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | <details> <summary>finetune BERT-base on this task</summary> PARAMS=runtime.distribution_strategy=mirrored<br/>PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/<br/>PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4<br/><br/>python3 train.py \\<br/> --experiment=bert/sentence_prediction \\<br/> --mode=train \\<br/> --model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\<br/> --config_file=configs/models/bert_en_uncased_base.yaml \\<br/> --config_file=configs/experiments/glue_mnli_matched.yaml \\<br/> --params_override=${PARAMS}</details> |
25-
| SQuAD v1.1 | bert/squad | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | <details> <summary>finetune BERT-base on this task</summary> PARAMS=runtime.distribution_strategy=mirrored<br/>PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/<br/>PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4<br/><br/>python3 train.py \\<br/> --experiment=bert/squad \\<br/> --mode=train \\<br/> --model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\<br/> --config_file=configs/models/bert_en_uncased_base.yaml \\<br/> --config_file=configs/experiments/squad_v1.yaml \\<br/> --params_override=${PARAMS}</details> |
26-
27-
One example on how to use the config file: if you want to work on the SQuAD
28-
question answering task, set
29-
`--config_file=configs/experiments/squad_v1.yaml` and
30-
`--experiment=bert/squad`
31-
as arguments to `train.py`.
32-
33-
## Available Model Configs
34-
35-
There are two available model configs (we will add more) under
36-
`configs/models/`:
37-
38-
| Model | Config | Pretrained checkpoint & Vocabulary | TF-HUB SavedModel | Example command |
39-
| ------------ | ------- | ---------------------------------- | ----------------- | --------------- |
40-
| BERT-base | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | [uncased_L-12_H-768_A-12](https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/uncased_L-12_H-768_A-12.tar.gz) | [uncased_L-12_H-768_A-12](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/) | <details> <summary>finetune on SQuAD v1.1</summary> PARAMS=runtime.distribution_strategy=mirrored<br/>PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/<br/>PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4<br/><br/>python3 train.py \\<br/> --experiment=bert/squad \\<br/> --mode=train \\<br/> --model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\<br/> --config_file=configs/models/bert_en_uncased_base.yaml \\<br/> --config_file=configs/experiments/squad_v1.yaml \\<br/> --params_override=${PARAMS}</details> |
41-
| ALBERT-base | [albert_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/albert_base.yaml) | [albert_en_base](https://storage.googleapis.com/tf_model_garden/nlp/albert/albert_base.tar.gz) | [albert_en_base](https://tfhub.dev/tensorflow/albert_en_base/3) | <details> <summary>finetune on SQuAD v1.1</summary> PARAMS=runtime.distribution_strategy=mirrored<br/>PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/<br/>PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/albert_en_base/3<br/><br/>python3 train.py \\<br/> --experiment=bert/squad \\<br/> --mode=train \\<br/> --model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\<br/> --config_file=configs/models/albert_base.yaml \\<br/> --config_file=configs/experiments/squad_v1.yaml \\<br/> --params_override=${PARAMS}</details> |
42-
43-
One example on how to use the config file: if you want to train an ALBERT-base
44-
model, set `--config_file=configs/models/albert_base.yaml` as an argument to
45-
`train.py`.
4662

4763
## Useful links
4864

4965
[How to Train Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md)
5066

51-
[List of Pretrained Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrained_models.md)
67+
[List of Pretrained Models for finetuning](https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrained_models.md)
5268

5369
[How to Publish Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/tfhub.md)
5470

0 commit comments

Comments
 (0)