Notebook documentation for classification_with_model_garden

PiperOrigin-RevId: 447557463

Author: ktonthat

site/en/tutorials/_toc.yaml

@@ -109,6 +109,9 @@ toc:
     path: /tutorials/images/data_augmentation
   - title: "Image segmentation"
     path: /tutorials/images/segmentation
+  - title: "Image classification with Model Garden"
+    path: /tutorials/images/classification_with_model_garden
+    status: new
   - title: "Object detection with TF Hub"
     path: https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_object_detection.ipynb
     status: external

site/en/tutorials/images/models_vision.ipynb renamed to site/en/tutorials/images/classification_with_model_garden.ipynb

@@ -37,7 +37,7 @@
         "id": "qFdPvlXBOdUN"
       },
       "source": [
-        "# Use TensorFlow Models: Fine tune a ResNet"
+        "# Image classification with Model Garden"
       ]
     },
     {
@@ -48,16 +48,16 @@
       "source": [
         "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
         "  <td>\n",
-        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/images/models_vision\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
+        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/images/classification_with_model_garden\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/images/models_vision.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
+        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/images/classification_with_model_garden.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/images/models_vision.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View on GitHub</a>\n",
+        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/images/classification_with_model_garden.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View on GitHub</a>\n",
         "  </td>\n",
         "  <td>\n",
-        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/images/models_vision.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
+        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/images/classification_with_model_garden.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
         "  </td>\n",
         "</table>"
       ]
@@ -68,7 +68,16 @@
         "id": "Ta_nFXaVAqLD"
       },
       "source": [
-        "This tutorial uses the TensorFlow Models package to fine-tune a ResNet."
+        "This tutorial fine-tunes a Residual Network (ResNet) from the TensorFlow [Model Garden](https://github.com/tensorflow/models) package (`tensorflow-models`) to classify images in the [CIFAR](https://www.cs.toronto.edu/~kriz/cifar.html) dataset.\n",
+        "\n",
+        "Model Garden contains a collection of state-of-the-art vision models, implemented with TensorFlow's high-level APIs. The implementations demonstrate the best practices for modeling, letting users to take full advantage of TensorFlow for their research and product development.\n",
+        "\n",
+        "This tutorial uses a [ResNet](https://arxiv.org/pdf/1512.03385.pdf) model, a state-of-the-art image classifier. This tutorial uses the ResNet-18 model, a convolutional neural network with 18 layers.\n",
+        "\n",
+        "This tutorial demonstrates how to:\n",
+        "1. Use models from the TensorFlow Models package.\n",
+        "2. Fine-tune a pre-built ResNet for image classification.\n",
+        "3. Export the tuned ResNet model."
       ]
     },
     {
@@ -79,7 +88,7 @@
       "source": [
         "## Setup\n",
         "\n",
-        "Install and import the necessary modules"
+        "Install and import the necessary modules. This tutorial uses the `tf-models-nightly` version of Model Garden."
       ]
     },
     {
@@ -94,6 +103,15 @@
         "!pip install -q tf-models-nightly"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "CKYMTPjOE400"
+      },
+      "source": [
+        "Import TensorFlow, TensorFlow Datasets, and a few helper libraries."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -102,7 +120,6 @@
       },
       "outputs": [],
       "source": [
-        "# Import helper libraries\n",
         "import pprint\n",
         "import tempfile\n",
         "\n",
@@ -113,6 +130,15 @@
         "import tensorflow_datasets as tfds"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AVTs0jDd1b24"
+      },
+      "source": [
+        "The `tensorflow_models` package contains the ResNet vision model, and the `official.vision.serving` model contains the function to save and export the tuned model."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -124,7 +150,7 @@
         "import tensorflow_models as tfm\n",
         "\n",
         "# Not in the tfm public API for v2.9. Will be available as `vision.serving` in v2.10\n",
-        "from official.vision.serving import export_saved_model_lib  "
+        "from official.vision.serving import export_saved_model_lib"
       ]
     },
     {
@@ -133,7 +159,7 @@
         "id": "aKv3wdqkQ8FU"
       },
       "source": [
-        "## Cifar-10 with ResNet-18 Backbone"
+        "## Configure the ResNet-18 model for the Cifar-10 dataset"
       ]
     },
     {
@@ -142,7 +168,11 @@
         "id": "5iN8mHEJjKYE"
       },
       "source": [
-        "Base the experiment on `\"resnet_imagenet\"` configuration (defined by `tfm.vision.configs.image_classification.image_classification_imagenet`)."
+        "The CIFAR10 dataset contains 60,000 color images in mutually exclusive 10 classes, with 6,000 images in each class.\n",
+        "\n",
+        "In Model Garden, the collections of parameters that define a model are called *configs*. Model Garden can create a config based on a known set of parameters via a [factory](https://en.wikipedia.org/wiki/Factory_method_pattern).\n",
+        "\n",
+        "Use the `resnet_imagenet` factory configuration, as defined by `tfm.vision.configs.image_classification.image_classification_imagenet`. The configuration is set up to train ResNet to converge on [ImageNet](https://www.image-net.org/)."
       ]
     },
     {
@@ -165,7 +195,7 @@
         "id": "U6PVwXA-j3E7"
       },
       "source": [
-        "Next adjust the configuration so that it works with `cifar10`."
+        "Adjust the model and dataset configurations so that it works with Cifar-10 (`cifar10`)."
       ]
     },
     {
@@ -176,12 +206,12 @@
       },
       "outputs": [],
       "source": [
-        "# Change model\n",
+        "# Configure model\n",
         "exp_config.task.model.num_classes = 10\n",
         "exp_config.task.model.input_size = list(ds_info.features[\"image\"].shape)\n",
         "exp_config.task.model.backbone.resnet.model_id = 18\n",
         "\n",
-        "# Change train, eval data\n",
+        "# Configure training and testing data\n",
         "batch_size = 128\n",
         "\n",
         "exp_config.task.train_data.input_path = ''\n",
@@ -201,7 +231,7 @@
         "id": "DE3ggKzzTD56"
       },
       "source": [
-        "Adjust the trainer configuration:"
+        "Adjust the trainer configuration."
       ]
     },
     {
@@ -212,8 +242,24 @@
       },
       "outputs": [],
       "source": [
-        "# Change trainer config\n",
-        "train_steps = 5000\n",
+        "logical_device_names = [logical_device.name for logical_device in tf.config.list_logical_devices()]\n",
+        "\n",
+        "if 'GPU' in ''.join(logical_device_names):\n",
+        "  print('This may be broken in Colab.')\n",
+        "  device = 'GPU'\n",
+        "elif 'TPU' in ''.join(logical_device_names):\n",
+        "  print('This may be broken in Colab.')\n",
+        "  device = 'TPU'\n",
+        "else:\n",
+        "  print('This is slow, and doesn\\'t train to convergence.')\n",
+        "  device = 'CPU'\n",
+        "\n",
+        "if device=='CPU':\n",
+        "  train_steps = 20\n",
+        "  exp_config.trainer.steps_per_loop = 5\n",
+        "else:\n",
+        "  train_steps=5000\n",
+        "  exp_config.trainer.steps_per_loop = 100\n",
         "\n",
         "exp_config.trainer.steps_per_loop = 100\n",
         "exp_config.trainer.summary_interval = 100\n",
@@ -233,7 +279,7 @@
         "id": "5mTcDnBiTOYD"
       },
       "source": [
-        "And set the runtime configuration."
+        "Print the modified configuration."
       ]
     },
     {
@@ -255,7 +301,7 @@
         "id": "w7_X0UHaRF2m"
       },
       "source": [
-        "Set up the distribution strategy:"
+        "Set up the distribution strategy."
       ]
     },
     {
@@ -288,7 +334,9 @@
         "id": "W4k5YH5pTjaK"
       },
       "source": [
-        "Create the `Task` object (ref: `tfm.core.base_task.Task`) form the `config_definitions.TaskConfig`:"
+        "Create the `Task` object (`tfm.core.base_task.Task`) from the `config_definitions.TaskConfig`.\n",
+        "\n",
+        "The `Task` object has all the methods necessary for building the dataset, building the model, and running training & evaluation. These methods are driven by `tfm.core.train_lib.run_experiment`."
       ]
     },
     {
@@ -326,7 +374,7 @@
         "id": "yrwxnGDaRU0U"
       },
       "source": [
-        "## Visualize Training Dataloader"
+        "## Visualize the training data"
       ]
     },
     {
@@ -335,8 +383,8 @@
         "id": "683c255c6c52"
       },
       "source": [
-        "The data-loader applies a z-score normalization using \n",
-        "`preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`, so the images returned by the dataset can't be directly displayed by standard tools, so rescale the minimum to 0.0 and the maximum to 1.0: "
+        "The dataloader applies a z-score normalization using \n",
+        "`preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`, so the images returned by the dataset can't be directly displayed by standard tools. The visualization code needs to rescale the data into the [0,1] range."
       ]
     },
     {
@@ -356,7 +404,7 @@
         "id": "7a8582ebde7b"
       },
       "source": [
-        "You can use the `tfds.core.DatasetInfo` (`ds_info` from earlier) to lookup the text descriptions of each class ID. "
+        "Use `ds_info` (which is an instance of `tfds.core.DatasetInfo`) to lookup the text descriptions of each class ID."
       ]
     },
     {
@@ -377,7 +425,7 @@
         "id": "8c652a6fdbcf"
       },
       "source": [
-        "Use these to disualize a batch of the data:"
+        "Visualize a batch of the data."
       ]
     },
     {
@@ -427,7 +475,16 @@
         "id": "v_A9VnL2RbXP"
       },
       "source": [
-        "## Visualize Evaluation Dataloader"
+        "## Visualize the testing data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "AXovuumW_I2z"
+      },
+      "source": [
+        "Visualize a batch of images from the validation dataset."
       ]
     },
     {
@@ -449,7 +506,7 @@
         "id": "ihKJt2FHRi2N"
       },
       "source": [
-        "## Train and Evaluate"
+        "## Train and evaluate"
       ]
     },
     {
@@ -480,6 +537,15 @@
         "tf.keras.utils.plot_model(model, show_shapes=True)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "L7nVfxlBA8Gb"
+      },
+      "source": [
+        "Print the `accuracy`, `top_5_accuracy`, and `validation_loss` evaluation metrics."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -492,6 +558,33 @@
         "    print(f'{key:20}: {value.numpy():.3f}')"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TDys5bZ1zsml"
+      },
+      "source": [
+        "Run a batch of the processed training data through the model, and view the results"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "GhI7zR-Uz1JT"
+      },
+      "outputs": [],
+      "source": [
+        "for images, labels in task.build_inputs(exp_config.task.train_data).take(1):\n",
+        "  predictions = model.predict(images)\n",
+        "  predictions = tf.argmax(predictions, axis=-1)\n",
+        "\n",
+        "show_batch(images, labels, tf.cast(predictions, tf.int32))\n",
+        "\n",
+        "if device=='CPU':\n",
+        "  plt.title('The model was only trained for a few steps, so it is not expected to do well.')"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -507,7 +600,7 @@
         "id": "9669d08c91af"
       },
       "source": [
-        "The `keras.Model` object returned by `train_lib.run_experiment` expects the data to be normalized by the dataset loader using the same mean and variance statiscics in `preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`. This export function handles those details so you can pass `tf.uint8` images and get correct result.\n"
+        "The `keras.Model` object returned by `train_lib.run_experiment` expects the data to be normalized by the dataset loader using the same mean and variance statiscics in `preprocess_ops.normalize_image(image, offset=MEAN_RGB, scale=STDDEV_RGB)`. This export function handles those details, so you can pass `tf.uint8` images and get the correct results.\n"
       ]
     },
     {
@@ -534,7 +627,7 @@
         "id": "vVr6DxNqTyLZ"
       },
       "source": [
-        "Test the exported model"
+        "Test the exported model."
       ]
     },
     {
@@ -556,7 +649,7 @@
         "id": "GiOp2WVIUNUZ"
       },
       "source": [
-        "Visualize the predictions"
+        "Visualize the predictions."
       ]
     },
     {
@@ -573,7 +666,10 @@
         "  for image in data['image']:\n",
         "    index = tf.argmax(model_fn(image[tf.newaxis, ...])['logits'], axis=1)[0]\n",
         "    predictions.append(index)\n",
-        "  show_batch(data['image'], data['label'], predictions)"
+        "  show_batch(data['image'], data['label'], predictions)\n",
+        "\n",
+        "  if device=='CPU':\n",
+        "    plt.title('The model was only trained for a few steps, it is not expected to do well.')"
       ]
     }
   ],