add new doc for structured parameter

Signed-off-by: reiase <[email protected]>

Author: reiase

docs/structured_parameter.md

@@ -0,0 +1,185 @@
+Structured Parameter
+====================
+
+The main idea of `HyperParameter` is to organize the parameters and parameter accesses into a tree structure, by which we can refer to each parameter with a unique name and identify each access to the parameters. Then we can modify each parameter and control the access to the parameters.
+
+Nested Parameters and Parameter Path
+------------------------------------
+
+The parameters are stored in a nested dict, a very common solution for python programs, and can be easily serialized into JSON or YAML format. For example:
+
+```python
+cfg = {
+    "param1": 1, # unique name: param1
+    "obj2": {
+        "param3": "value4", # unique name: obj2.param3
+        "param5": 6,        # unique name: obj2.param5
+    },
+}
+```
+
+We can directly refer to the value `value4` by `cfg["obj2"]["param3"]`. But we also need to check whether the parameter is missing from the cfg, and handle the default value.
+
+`HyperParameter` offers a tiny DSL to access the parameters. The DSL syntax is very similar to `jsonpath`, but compatible with python syntax so that the interpreter and code editor can check for syntax errors. (I found this feature saves me a lot of time.). The first thing to use the DSL is converting the cfg into `HyperParameter`, and then we can use the DSL:
+
+```python
+# convert a nested dict into HyperParameter
+hp = HyperParameter(**cfg) 
+
+# or create the HyperParameter object from scratch
+hp = HyperParameter(param1=1, obj2={"param3": "value4"})
+
+# the DSL for access the parameter
+param = hp().obj2.param3(default="undefined")
+```
+
+`hp().obj2.param3(default="undefined")` is the inline DSL for reading parameter from `HyperParameter` object. It looks like a `jsonpath` expression but has support for default values.
+
+Best Practice for Structure Parameters with Parameter Path
+-----------------------------------------------
+
+### A Quick Example of Recommendation Model
+
+Suppose we are building a wide&deep model with `keras`.
+
+```python
+class WideAndDeepModel(keras.Model):
+    def __init__(self, units=30, activation="relu", **kwargs):
+        super().__init__(**kwargs)
+        self.hidden1 = keras.layers.Dense(units, activation=activation)
+        self.hidden2 = keras.layers.Dense(units, activation=activation)
+        self.main_output = keras.layers.Dense(1)
+        self.aux_output = keras.layers.Dense(1)
+        
+    def call(self, inputs):
+        input_A , input_B = inputs
+        hidden1 = self.hidden1(input_B)
+        hidden2 = self.hidden2(hidden1)
+        concat = keras.layers.concatenate([input_A, hidden2])
+        main_output = self.main_output()(concat)
+        aux_output = self.aux_output()(hidden2)
+        return main_outputi, aux_output
+```
+
+The model is straightforward and does not support many parameters. If we want to add batch normalization, dropout, and leaky-relu tricks to the model, we have to modify the code as follows:
+
+```python
+class WideAndDeepModel(keras.Model):
+    def __init__(self, 
+        units=[30, 30, 30],
+        activation="relu",
+        use_bn=False,
+        bn_momentum=0.99,
+        bn_epsilon=0.001,
+        bn_center=True,
+        bn_scale=True,
+        bn_beta_initializer="zeros",
+        bn_gamma_initializer="ones",
+        bn_moving_mean_initializer="zeros",
+        bn_moving_variance_initializer="ones",
+        use_dropout=False,
+        ...):
+
+        ...
+        self.bn1 = keras.layers.BatchNormalization(
+            momentum=bn_momentum,
+            epsilon=bn_epsilon,
+            center=bn_center,
+            scale=bn_scale,
+            beta_initializer=bn_beta_initializer,
+            gamma_initializer=bn_gamma_initializer,
+            ...,
+        )
+```
+
+The code becomes too complicated, having dozens of parameters to handle, most of which are not used.
+
+### A Fast Trial of Structured Parameter
+
+We can simplify the code with `auto_param`, which automatically converts the parameters into a parameter tree. And then, we can specify the parameters by name:
+
+```python
+# add parameter support for custom functions with a decorator
+@auto_param("myns.rec.rank.dropout")
+class dropout:
+    def __init__(self, ratio=0.5):
+        ...
+
+# add parameter support for library functions 
+wrapped_bn = auto_param("myns.rec.rank.bn")(keras.layers.BatchNormalization)
+```
+
+`myns.rec.rank` is the namespace for my project, and `myns.rec.rank.dropout` refers to the function defined in our code. We can refer to the keyword arguments (e.g. `ratio=0.5`) with the path `hp().myns.rec.rank.dropout`. 
+
+After making the building block configurable, we can simplify the model:
+```python
+class WideAndDeepModel(keras.Model):
+    def __init__(self, 
+        units=[30, 30, 30],
+        activation="relu",
+        ...):
+
+        ...
+        self.bn1 = wrapped_bn()
+        self.dropout1 = dropout()
+```
+And we can change the parameters of the `BN` layers with `param_scope`:
+
+```python
+with param_scope(**{
+    "myns.rec.rank.dropout.ratio": 0.6,
+    "myns.rec.rank.bn.center": False,
+    ...
+}):
+    model = WideAndDeepModel()
+```
+
+Or read the parameters from a JSON file:
+
+```python
+with open("model.cfg.json") as f:
+    cfg = json.load(f)
+with param_scope(**cfg):
+    model = WideAndDeepModel() 
+```
+
+### Fine-grained Control of Structured Parameters
+
+In the last section, we have introduced how to structure the parameters with `auto_param` and modify them with `param_scope` by their path.
+However, we may also need to access the same parameter in different places in our code, e.g., different layers in a DNN model.
+
+In such situation, we can break our code into named scopes. And then, we can identify each access to the parameters and set a value for each access.
+
+To add named scopes to our code, we can use `param_scope`:
+
+```python
+class WideAndDeepModel(keras.Model):
+    def __init__(self, 
+        units=[30, 30, 30],
+        activation="relu",
+        ...):
+
+        ...
+        with param_scope["layer1"]():
+            self.bn1 = wrapped_bn()
+            self.dropout1 = dropout()
+        with param_scope["layer2"]():
+            self.bn2 = wrapped_bn()
+            self.dropout2 = dropout()
+        ...
+
+with param_scope["wdmodel"]():
+    model = WideAndDeepModel()
+```
+
+`param_scope["layer1"]` creates a named scope called `layer1`. Since the scope is created inside another named scope `param_scope["wdmodel"]`, its full path should be `wdmodel.layer1`. We can specify different values of a parameter according to its path. For example:
+
+```python
+with param_scope["wdmodel"](**{
+    "myns.rec.rank.dropout.ratio@wdmodel#layer1": 0.6,
+    "myns.rec.rank.dropout.ratio@wdmodel#layer2": 0.7,
+}):
+    model = WideAndDeepModel()
+```
+
+With the above code, we get a drop ratio of 0.6 for `layer1` and 0.7 for `layer2`.

mkdocs.yml

@@ -24,11 +24,18 @@ plugins:
 nav:
   - Home: index.md
   - Quick Start: quick_start.md
+  - Best Practice: structured_parameter.md
   - Reference: reference.md
 
 watch:
   - hyperparameter
 
+markdown_extensions:
+  - pymdownx.highlight:
+      anchor_linenums: true
+      use_pygments: true
+  - pymdownx.superfences 
+
 extra:
   alternate:
     - name: English