|
| 1 | +# Model Garden overview |
| 2 | + |
| 3 | +The TensorFlow Model Garden provides implementations of many state-of-the-art |
| 4 | +machine learning (ML) models for vision and natural language processing (NLP), |
| 5 | +as well as workflow tools to let you quickly configure and run those models on |
| 6 | +standard datasets. Whether you are looking to benchmark performance for a |
| 7 | +well-known model, verify the results of recently released research, or extend |
| 8 | +existing models, the Model Garden can help you drive your ML research and |
| 9 | +applications forward. |
| 10 | + |
| 11 | +The Model Garden includes the following resources for machine learning |
| 12 | +developers: |
| 13 | + |
| 14 | +- [**Official models**](#official) for vision and NLP, maintained by Google |
| 15 | + engineers |
| 16 | +- [**Research models**](#research) published as part of ML research papers |
| 17 | +- [**Training experiment framework**](#training_framework) for fast, |
| 18 | + declarative training configuration of official models |
| 19 | +- [**Specialized ML operations**](#ops) for vision and natural language |
| 20 | + processing (NLP) |
| 21 | +- [**Model training loop**](#orbit) management with Orbit |
| 22 | + |
| 23 | +These resources are built to be used with the TensorFlow Core framework and |
| 24 | +integrate with your existing TensorFlow development projects. Model |
| 25 | +Garden resources are also provided under an [open |
| 26 | +source](https://github.com/tensorflow/models/blob/master/LICENSE) license, so |
| 27 | +you can freely extend and distribute the models and tools. |
| 28 | + |
| 29 | +Practical ML models are computationally intensive to train and run, and may |
| 30 | +require accelerators such as Graphical Processing Units (GPUs) and Tensor |
| 31 | +Processing Units (TPUs). Most of the models in Model Garden were trained on |
| 32 | +large datasets using TPUs. However, you can also train and run these models on |
| 33 | +GPU and CPU processors. |
| 34 | + |
| 35 | +## Model Garden models |
| 36 | + |
| 37 | +The machine learning models in the Model Garden include full code so you can |
| 38 | +test, train, or re-train them for research and experimentation. The Model Garden |
| 39 | +includes two primary categories of models: *official models* and *research |
| 40 | +models*. |
| 41 | + |
| 42 | +### Official models {:#official} |
| 43 | + |
| 44 | +The [Official Models](https://github.com/tensorflow/models/tree/master/official) |
| 45 | +repository is a collection of state-of-the-art models, with a focus on |
| 46 | +vision and natural language processing (NLP). |
| 47 | +These models are implemented using current TensorFlow 2.x high-level |
| 48 | +APIs. Model libraries in this repository are optimized for fast performance and |
| 49 | +actively maintained by Google engineers. The official models include additional |
| 50 | +metadata you can use to quickly configure experiments using the Model Garden |
| 51 | +[training experiment framework](#training_framework). |
| 52 | + |
| 53 | +### Research models {:#research} |
| 54 | + |
| 55 | +The [Research Models](https://github.com/tensorflow/models/tree/master/research) |
| 56 | +repository is a collection of models published as code resources for research |
| 57 | +papers. These models are implemented using both TensorFlow 1.x and 2.x. Model |
| 58 | +libraries in the research folder are supported by the code owners and the |
| 59 | +research community. |
| 60 | + |
| 61 | +## Training experiment framework {:#training_framework} |
| 62 | + |
| 63 | +The Model Garden training experiment framework lets you quickly assemble and run |
| 64 | +training experiments using its official models and standard datasets. The |
| 65 | +training framework uses additional metadata included with the Model Garden's |
| 66 | +official models to allow you to configure models quickly using a declarative |
| 67 | +programming model. You can define a training experiment using Python commands in |
| 68 | +the |
| 69 | +[TensorFlow Model library](https://www.tensorflow.org/api_docs/python/tfm/core) |
| 70 | +or configure training using a YAML configuration file, like this |
| 71 | +[example](https://github.com/tensorflow/models/blob/master/official/vision/configs/experiments/image_classification/imagenet_resnet50_tpu.yaml). |
| 72 | + |
| 73 | +The training framework uses |
| 74 | +[`tfm.core.base_trainer.ExperimentConfig`](https://www.tensorflow.org/api_docs/python/tfm/core/base_trainer/ExperimentConfig) |
| 75 | +as the configuration object, which contains the following top-level |
| 76 | +configuration objects: |
| 77 | + |
| 78 | +- [`runtime`](https://www.tensorflow.org/api_docs/python/tfm/core/base_task/RuntimeConfig): |
| 79 | + Defines the processing hardware, distribution strategy, and other |
| 80 | + performance optimizations |
| 81 | +- [`task`](https://www.tensorflow.org/api_docs/python/tfm/core/config_definitions/TaskConfig): |
| 82 | + Defines the model, training data, losses, and initialization |
| 83 | +- [`trainer`](https://www.tensorflow.org/api_docs/python/tfm/core/base_trainer/TrainerConfig): |
| 84 | + Defines the optimizer, training loops, evaluation loops, summaries, and |
| 85 | + checkpoints |
| 86 | + |
| 87 | +For a complete example using the Model Garden training experiment framework, see |
| 88 | +the [Image classification with Model Garden](vision/image_classification.ipynb) |
| 89 | +tutorial. For information on the training experiment framework, check out the |
| 90 | +[TensorFlow Models API documentation](https://tensorflow.org/api_docs/python/tfm/core). |
| 91 | +If you are looking for a solution to manage training loops for your model |
| 92 | +training experiments, check out [Orbit](#orbit). |
| 93 | + |
| 94 | +## Specialized ML operations {:#ops} |
| 95 | + |
| 96 | +The Model Garden contains many vision and NLP operations specifically designed |
| 97 | +to execute state-of-the-art models that run efficiently on GPUs and TPUs. Review |
| 98 | +the TensorFlow Models Vision library API docs for a list of specialized |
| 99 | +[vision operations](https://www.tensorflow.org/api_docs/python/tfm/vision). |
| 100 | +Review the TensorFlow Models NLP Library API docs for a list of |
| 101 | +[NLP operations](https://www.tensorflow.org/api_docs/python/tfm/nlp). These |
| 102 | +libraries also include additional utility functions used for vision and NLP data |
| 103 | +processing, training, and model execution. |
| 104 | + |
| 105 | +## Training loops with Orbit {:#orbit} |
| 106 | + |
| 107 | +There are two default options for training TensorFlow models: |
| 108 | + |
| 109 | +* Use the high-level Keras |
| 110 | +[Model.fit](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) |
| 111 | +function. If your model and training procedure fit the assumptions of Keras' |
| 112 | +`Model.fit` (incremental gradient descent on batches of data) method this can |
| 113 | +be very convenient. |
| 114 | +* Write a custom training loop |
| 115 | +[with keras](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch), |
| 116 | +or [without](https://www.tensorflow.org/guide/core/logistic_regression_core). |
| 117 | +You can write a custom training loop with low-level TensorFlow methods such as |
| 118 | +`tf.GradientTape` or `tf.function`. However, this approach requires a lot of |
| 119 | +boilerplate code, and doesn't do anything to simplify distributed training. |
| 120 | + |
| 121 | +Orbit tries to provide a third option in between these two extremes. |
| 122 | + |
| 123 | +Orbit is a flexible, lightweight library designed to make it easier to |
| 124 | +write custom training loops in TensorFlow 2.x, and works well with the Model |
| 125 | +Garden [training experiment framework](#training_framework). Orbit handles |
| 126 | +common model training tasks such as saving checkpoints, running model |
| 127 | +evaluations, and setting up summary writing. It seamlessly integrates with |
| 128 | +`tf.distribute` and supports running on different device types, including CPU, |
| 129 | +GPU, and TPU hardware. The Orbit tool is also [open |
| 130 | +source](https://github.com/tensorflow/models/blob/master/orbit/LICENSE), so you |
| 131 | +can extend and adapt to your model training needs. |
| 132 | + |
| 133 | +The Orbit guide is available [here](orbit/index.ipynb). |
| 134 | + |
| 135 | +Note: You can customize how the Keras API executes training. Mainly you must |
| 136 | +override the `Model.train_step` method or use `keras.callbacks` like |
| 137 | +`callbacks.ModelCheckpoint` or `callbacks.TensorBoard`. For more information |
| 138 | +about modifying the behavior of `train_step`, check out the |
| 139 | +[Customize what happens in Model.fit](https://www.tensorflow.org/guide/keras/customizing_what_happens_in_fit) |
| 140 | +page. |
0 commit comments