Merge pull request #806 from MohamedNourArm:toupstream/update_cluster_guide

PiperOrigin-RevId: 393948588

Author: tensorflower-gardener

tensorflow_model_optimization/g3doc/guide/clustering/clustering_comprehensive_guide.ipynb

@@ -214,7 +214,7 @@
         "\n",
         "clustering_params = {\n",
         "  'number_of_clusters': 3,\n",
-        "  'cluster_centroids_init': CentroidInitialization.DENSITY_BASED\n",
+        "  'cluster_centroids_init': CentroidInitialization.KMEANS_PLUS_PLUS\n",
         "}\n",
         "\n",
         "model = setup_model()\n",
@@ -279,6 +279,61 @@
         "clustered_model.summary()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WcFrw1dHmxTr"
+      },
+      "source": [
+        "### Cluster custom Keras layer or specify which weights of layer to cluster\n",
+        "\n",
+        "`tfmot.clustering.keras.ClusterableLayer` serves two use cases:\n",
+        "1. Cluster any layer that is not supported natively, including a custom Keras layer.\n",
+        "2. Specify which weights of a supported layer are to be clustered.\n",
+        "\n",
+        "For an example, the API defaults to only clustering the kernel of the\n",
+        "`Dense` layer. The example below shows how to modify it to also cluster the bias. Note that when deriving from the keras layer, you need to override the function `get_clusterable_weights`, where you specify the name of the trainable variable to be clustered and the trainable variable itself. For example, if you return an empty list [], then no weights will be clusterable.\n",
+        "\n",
+        "**Common mistake:** clustering the bias usually harms model accuracy too much."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "73iboQ7MmxTs"
+      },
+      "outputs": [],
+      "source": [
+        "class MyDenseLayer(tf.keras.layers.Dense, tfmot.clustering.keras.ClusterableLayer):\n",
+        "\n",
+        "  def get_clusterable_weights(self):\n",
+        "   # Cluster kernel and bias. This is just an example, clustering\n",
+        "   # bias usually hurts model accuracy.\n",
+        "   return [('kernel', self.kernel), ('bias', self.bias)]\n",
+        "\n",
+        "# Use `cluster_weights` to make the `MyDenseLayer` layer train with clustering as usual.\n",
+        "model_for_clustering = tf.keras.Sequential([\n",
+        "  tfmot.clustering.keras.cluster_weights(MyDenseLayer(20, input_shape=[input_dim]), **clustering_params),\n",
+        "  tf.keras.layers.Flatten()\n",
+        "])\n",
+        "\n",
+        "model_for_clustering.summary()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "SYlWPXEWmxTs"
+      },
+      "source": [
+        "You may also use `tfmot.clustering.keras.ClusterableLayer` to cluster a keras custom layer. To do this, you extend `tf.keras.Layer` as usual and implement the `__init__`, `call`, and `build` functions, but you also need to extend the `clusterable_layer.ClusterableLayer` class and implement:\n",
+        "1. `get_clusterable_weights`, where you specify the weight kernel to be clustered, as shown above.\n",
+        "2. `get_clusterable_algorithm`, where you specify the clustering algorithm for the weight tensor. This is because you need to specify how the custom layer weights are shaped for clustering. The returned clustering algorithm class should be derived from the `clustering_algorithm.ClusteringAlgorithm` class and the function `get_pulling_indices` should be overwritten. An example of this function, which supports weights of ranks 1D, 2D, and 3D, can be found [here]( https://github.com/tensorflow/model-optimization/blob/18e87d262e536c9a742aef700880e71b47a7f768/tensorflow_model_optimization/python/core/clustering/keras/clustering_algorithm.py#L62).\n",
+        "\n",
+        "An example of this use case can be found [here](https://github.com/tensorflow/model-optimization/blob/master/tensorflow_model_optimization/python/core/clustering/keras/mnist_clusterable_layer_test.py)."
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -338,7 +393,7 @@
       "source": [
         "For your specific use case, there are tips you can consider:\n",
         "\n",
-        "* Centroid initialization plays a key role in the final optimized model accuracy. In general, linear initialization outperforms density and random initialization since it does not tend to miss large weights. However, density initialization has been observed to give better accuracy for the case of using very few clusters on weights with bimodal distributions.\n",
+        "* Centroid initialization plays a key role in the final optimized model accuracy. In general, kmeans++ initialization outperforms linear, density and random initialization. When not using kmeans++, linear initialization tends to outperform density and random initialization, since it does not tend to miss large weights. However, density initialization has been observed to give better accuracy for the case of using very few clusters on weights with bimodal distributions.\n",
         "\n",
         "* Set a learning rate that is lower than the one used in training when fine-tuning the clustered model.\n",
         "\n",