Lint Parameter server strategy guide

PiperOrigin-RevId: 436863751

Author: 8bitmp3

site/en/tutorials/distribute/parameter_server_training.ipynb

@@ -108,7 +108,7 @@
         "- Multiple _worker_ jobs (job name `worker`); and\n",
         "- Multiple _parameter server_ jobs (job name `ps`)\n",
         "\n",
-        "While the _coordinator_ creates resources, dispatches training tasks, writes checkpoints, and deals with task failures, _workers_ and _parameter servers_ run `tf.distribute.Server` that listen for requests from the coordinator."
+        "The _coordinator_ creates resources, dispatches training tasks, writes checkpoints, and deals with task failures. The _workers_ and _parameter servers_ run `tf.distribute.Server` instances that listen for requests from the coordinator."
       ]
     },
     {
@@ -348,15 +348,15 @@
       },
       "source": [
         "When a `variable_partitioner` is passed in and if you create a variable directly\n",
-        "under `strategy.scope()`, it will become a container type with a `variables`\n",
-        "property which provides access to the list of shards. In most cases, this\n",
+        "under `Strategy.scope`, it will become a container type with a `variables`\n",
+        "property, which provides access to the list of shards. In most cases, this\n",
         "container will be automatically converted to a Tensor by concatenating all the\n",
         "shards. As a result, it can be used as a normal variable. On the other hand,\n",
         "some TensorFlow methods such as `tf.nn.embedding_lookup` provide efficient\n",
         "implementation for this container type and in these methods automatic\n",
         "concatenation will be avoided.\n",
         "\n",
-        "Please see the API docs of `tf.distribute.experimental.ParameterServerStrategy` for more details."
+        "Refer to the API docs of `tf.distribute.experimental.ParameterServerStrategy` for more details."
       ]
     },
     {
@@ -384,7 +384,7 @@
         "\n",
         "Note that it is recommended to shuffle and repeat the data with parameter server training, and specify `steps_per_epoch` in `fit` call so the library knows the epoch boundaries.\n",
         "\n",
-        "Please see the [Distributed input](https://www.tensorflow.org/tutorials/distribute/input#usage_2) tutorial for more information about the `InputContext` argument."
+        "Refer to the [Distributed input](https://www.tensorflow.org/tutorials/distribute/input#usage_2) tutorial for more information about the `InputContext` argument."
       ]
     },
     {
@@ -420,8 +420,7 @@
         "id": "v_jhF70K7zON"
       },
       "source": [
-        "The code in `dataset_fn` will be invoked on the input device, which is usually the CPU, on each of the worker machines.\n",
-        "\n"
+        "The code in `dataset_fn` will be invoked on the input device, which is usually the CPU, on each of the worker machines.\n"
       ]
     },
     {
@@ -463,7 +462,7 @@
         "\n",
         "- `ModelCheckpoint`: to save the model weights.\n",
         "- `BackupAndRestore`: to make sure the training progress is automatically backed up, and recovered if the cluster experiences unavailability (such as abort or preemption); or\n",
-        "- `TensorBoard`: to save the progress reports into summary files, which get visualized in TensorBoard tool.\n",
+        "- `TensorBoard`: to save the progress reports into summary files, which can be visualized in the TensorBoard tool.\n",
         "\n",
         "Note: Due to performance consideration, custom callbacks cannot have batch level callbacks overridden when used with `ParameterServerStrategy`. Please modify your custom callbacks to make them epoch level calls, and adjust `steps_per_epoch` to a suitable value. In addition, `steps_per_epoch` is a required argument for `Model.fit` when used with `ParameterServerStrategy`."
       ]
@@ -530,11 +529,11 @@
         "\n",
         "First, write a function that creates a dataset.\n",
         "\n",
-        "If you would like to preprocess the data with [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) or [Tensorflow Transform layers](https://www.tensorflow.org/tfx/tutorials/transform/simple), create these layers **outside the `dataset_fn`** and **under `strategy.scope()`** like you would do for any other Keras layers. This is because the `dataset_fn` will be wrapped into a `tf.function` and then executed on each worker to generate the data pipeline. \n",
+        "If you would like to preprocess the data with [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) or [Tensorflow Transform layers](https://www.tensorflow.org/tfx/tutorials/transform/simple), create these layers **outside the `dataset_fn`** and **under `Strategy.scope`** like you would do for any other Keras layers. This is because the `dataset_fn` will be wrapped into a `tf.function` and then executed on each worker to generate the data pipeline.\n",
         "\n",
-        "If you don't follow the above procedure, creating the layers might create Tensorflow states which will be lifted out of the `tf.function` to the coordinator, and thus accessing them on workers would incur repetitive RPC calls between coordinator and workers and causes significant slowdown. \n",
+        "If you don't follow the above procedure, creating the layers might create Tensorflow states which will be lifted out of the `tf.function` to the coordinator. Thus, accessing them on workers would incur repetitive RPC calls between coordinator and workers, and cause significant slowdown.\n",
         "\n",
-        "Placing the layers under `strategy.scope()` will instead create them on all workers. Then, you will apply the transformation inside the `dataset_fn` via `tf.data.Dataset.map`. Please refer to [this tutorial](https://www.tensorflow.org/tutorials/distribute/input#data_preprocessing) for more information on data preprocessing with distributed input."
+        "Placing the layers under `Strategy.scope` will instead create them on all workers. Then, you will apply the transformation inside the `dataset_fn` via `tf.data.Dataset.map`. Refer to _Data preprocessing_ in the [Distributed input](https://www.tensorflow.org/tutorials/distribute/input) tutorial for more information on data preprocessing with distributed input."
       ]
     },
     {
@@ -644,7 +643,7 @@
       "source": [
         "### Build the model\n",
         "\n",
-        "Next, create the model and other objects. Make sure to create all variables under `strategy.scope`."
+        "Next, create the model and other objects. Make sure to create all variables under `Strategy.scope`."
       ]
     },
     {
@@ -655,7 +654,7 @@
       },
       "outputs": [],
       "source": [
-        "# These variables created under the `strategy.scope` will be placed on parameter\n",
+        "# These variables created under the `Strategy.scope` will be placed on parameter\n",
         "# servers in a round-robin fashion.\n",
         "with strategy.scope():\n",
         "  # Create the model. The input needs to be compatible with Keras processing layers.\n",
@@ -875,9 +874,9 @@
       "source": [
         "### More about dataset creation\n",
         "\n",
-        "The dataset in the above code is created using the `ClusterCoordinator.create_per_worker_dataset` API). It creates one dataset per worker and returns a container object. You can call the `iter` method on it to create a per-worker iterator. The per-worker iterator contains one iterator per worker and the corresponding slice of a worker will be substituted in the input argument of the function passed to the `ClusterCoordinator.schedule` method before the function is executed on a particular worker.\n",
+        "The dataset in the above code is created using the `ClusterCoordinator.create_per_worker_dataset` API. It creates one dataset per worker and returns a container object. You can call the `iter` method on it to create a per-worker iterator. The per-worker iterator contains one iterator per worker and the corresponding slice of a worker will be substituted in the input argument of the function passed to the `ClusterCoordinator.schedule` method before the function is executed on a particular worker.\n",
         "\n",
-        "Currently, the `ClusterCoordinator.schedule` method assumes workers are equivalent and thus assumes the datasets on different workers are the same except they may be shuffled differently if they contain a `Dataset.shuffle` operation. Because of this, it is also recommended that the datasets to be repeated indefinitely and you schedule a finite number of steps instead of relying on the `OutOfRangeError` from a dataset.\n",
+        "The `ClusterCoordinator.schedule` method assumes workers are equivalent and thus assumes the datasets on different workers are the same (except that they may be shuffled differently). Because of this, it is also recommended to repeat datasets, and schedule a finite number of steps instead of relying on the `OutOfRangeError` from a dataset.\n",
         "\n",
         "Another important note is that `tf.data` datasets don’t support implicit serialization and deserialization across task boundaries. So it is important to create the whole dataset inside the function passed to `ClusterCoordinator.create_per_worker_dataset`."
       ]
@@ -1227,9 +1226,9 @@
         "2. Avoid creating a hotspot variable that is required by all parameter servers in a single step if possible. For example, use a constant learning rate or subclass `tf.keras.optimizers.schedules.LearningRateSchedule` in optimizers since the default behavior is that the learning rate will become a variable placed on a particular parameter server and requested by all other parameter servers in each step.\n",
         "3. Shuffle your large vocabularies before passing them to Keras preprocessing layers.\n",
         "\n",
-        "Another possible reason for performance issues is the coordinator. The implementation of `schedule`/`join` is Python-based and thus may have threading overhead. Also, the latency between the coordinator and the workers can be large. If this is the case,\n",
+        "Another possible reason for performance issues is the coordinator. The implementation of `schedule`/`join` is Python-based and thus may have threading overhead. Also, the latency between the coordinator and the workers can be large. If this is the case:\n",
         "\n",
-        "- For `Model.fit`, you can set `steps_per_execution` argument provided at `Model.compile` to a value larger than 1.\n",
+        "- For `Model.fit`, you can set the `steps_per_execution` argument provided at `Model.compile` to a value larger than 1.\n",
         "\n",
         "- For a custom training loop, you can pack multiple steps into a single `tf.function`:\n",
         "\n",
@@ -1271,9 +1270,7 @@
         "- It is not supported to load a saved_model via `tf.saved_model.load` containing sharded variables. Note loading such a saved_model using TensorFlow Serving is expected to work.\n",
         "- It is not supported to load a checkpoint containing sharded optimizer slot variables into a different number of shards.\n",
         "- It is not supported to recover from parameter server failure without restarting the coordinator task.\n",
-        "- Creation of `tf.lookup.StaticHashTable`, commonly employed by some Keras preprocessing layers, such as `tf.keras.layers.IntegerLookup`, `tf.keras.layers.StringLookup`, and `tf.keras.layers.TextVectorization`, should be placed under `strategy.scope()`. Otherwise, resources will be placed on the coordinator, and lookup RPCs from workers to the coordinator incur performance implications. \n",
-        "\n",
-        "\n"
+        "- Creation of `tf.lookup.StaticHashTable`, commonly employed by some Keras preprocessing layers, such as `tf.keras.layers.IntegerLookup`, `tf.keras.layers.StringLookup`, and `tf.keras.layers.TextVectorization`, should be placed under `Strategy.scope`. Otherwise, resources will be placed on the coordinator, and lookup RPCs from workers to the coordinator incur performance implications.\n"
       ]
     },
     {
@@ -1286,9 +1283,8 @@
         "\n",
         "- `steps_per_epoch` argument is required in `Model.fit`. You can select a value that provides appropriate intervals in an epoch.\n",
         "- `ParameterServerStrategy` does not have support for custom callbacks that have batch-level calls for performance reasons. You should convert those calls into epoch-level calls with suitably picked `steps_per_epoch`, so that they are called every `steps_per_epoch` number of steps. Built-in callbacks are not affected: their batch-level calls have been modified to be performant. Supporting batch-level calls for `ParameterServerStrategy` is being planned.\n",
-        "- For the same reason, unlike other strategies, progress bar and metrics are logged only at epoch boundaries.\n",
-        "- `run_eagerly` is not supported.\n",
-        "\n"
+        "- For the same reason, unlike other strategies, progress bars and metrics are logged only at epoch boundaries.\n",
+        "- `run_eagerly` is not supported.\n"
       ]
     },
     {
@@ -1310,7 +1306,6 @@
     "colab": {
       "collapsed_sections": [],
       "name": "parameter_server_training.ipynb",
-      "provenance": [],
       "toc_visible": true
     },
     "kernelspec": {