parent 7d5ba61

author Reiase <[email protected]> 1641453952 +0800
committer Reiase <[email protected]> 1642438724 +0800

init commit

Author: reiase

README.md

@@ -1,2 +1,89 @@
-# hyperparameter
-a library for hyper parameter
+**H**_yper_**P**_arameter_
+===========================
+
+A hyper-parameter library for researchers, data scientists and machine learning engineers.
+
+- [**H**_yper_**P**_arameter_](#hyperparameter)
+- [Quick Start](#quick-start)
+  - [Object-Style API:](#object-style-api)
+  - [Scoped Parameter](#scoped-parameter)
+- [Examples](#examples)
+  - [parameter tunning for researchers](#parameter-tunning-for-researchers)
+  - [experiment tracing for data scientists](#experiment-tracing-for-data-scientists)
+  - [design-pattern for system engineers](#design-pattern-for-system-engineers)
+
+Quick Start
+============
+
+## Object-Style API:
+
+```python
+from hyperparameter import HyperParameter
+
+params = HyperParameter(a=1, b={'c': 2})
+params.a == 1   # True
+params.b.c == 2 # True (nested parameter)
+```
+
+or becomes powerful with `params()`:
+
+```python
+params().a.b.c.getOrElse(3) # 3 (default value)
+params().a.b.c(3)           # 3 (shortcut for default value)
+
+params().a.b.c = 4          # set value to param `a.b.c`
+params().a.b.c(3)           # 4 (default value is ignored) 
+```
+
+## Scoped Parameter
+
+```python
+from hyperparameter import param_scope
+
+# scoped parameter
+with param_scope(a=1) as hp: 
+    hp.a == 1 # True
+```
+or becomes powerful with `nested scope`:
+``` python
+with param_scope(a=1) as hp: 
+    with param_scope(a=2) as hp: 
+        hp.a == 2 # True, a=2 for inner scope
+    hp.a == 1     # True, a=1 for outer scope
+```
+
+even more powerful when using `param_scope` in function:
+
+```python
+#change function behavior with scoped parameter:
+def foo(arg):
+    # receive parameter using param_scope
+    with param_scope() as hp: 
+        if (hp().param1.getOrElse(1) == 1):
+            return 1
+        else:
+            return 2
+        ...
+
+# call function with default parameter
+foo() # 1
+
+# passing parameter using param_scope
+with param_scope(param1=2): 
+    foo() # 2
+```
+
+Examples
+========
+
+## [parameter tunning for researchers](examples/sparse_lr/README.md)
+
+This example shows how to use hyperparameter in your research projects, and make your experiments reproducible.
+
+## experiment tracing for data scientists
+
+Todo.
+
+## design-pattern for system engineers
+
+Todo.

README_CN.md

@@ -0,0 +1,19 @@
+Overview
+========
+
+hyperparameter 是一个参数管理工具库，用于解决如下问题：
+1. 应用级别/库级别参数配置与管理；
+2. 机器学习模型超参配置和调优；
+3. AutoML参数寻优；
+
+Quick Start
+===========
+
+``` python
+def foo():
+    with param_scope() as hp:
+        return hp.a
+    
+with param_scope(a=1, b=2) as hp:
+    hp() # 1
+```

docs/_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-slate

docs/index.md

@@ -0,0 +1,61 @@
+**H**_yper_**P**_arameter_
+===========================
+A hyper-parameter library for researchers, data scientists and machine learning engineers.
+
+## Object-Style API:
+
+```python
+from hyperparameter import HyperParameter
+
+params = HyperParameter(a=1, b={'c': 2})
+params.a == 1   # True
+params.b.c == 2 # True (nested parameter)
+```
+
+or becomes powerful with `params()`:
+
+```python
+params().a.b.c.getOrElse(3) # 3 (default value)
+params().a.b.c(3) # 3 (shortcut for default value)
+
+params().a.b.c = 4 # set value to param `a.b.c`
+params().a.b.c(3) # 4 (default value is ignored) 
+```
+
+## Scoped Parameter
+
+```python
+from hyperparameter import param_scope
+
+# scoped parameter
+with param_scope(a=1) as hp: 
+    hp.a == 1 # True
+```
+or becomes powerful with `nested scope`:
+``` python
+with param_scope(a=1) as hp: 
+    with param_scope(a=2) as hp: 
+        hp.a == 2 # True
+    hp.a == 1 # True
+```
+
+even more powerful when using `param_scope` in function:
+
+```python
+#change function behavior with scoped parameter:
+def foo(arg):
+    # receive parameter using param_scope
+    with param_scope() as hp: 
+        if (hp().param1.getOrElse(1) == 1):
+            return 1
+        else:
+            return 2
+        ...
+
+# call function with default parameter
+foo() # 1
+
+# passing parameter using param_scope
+with param_scope(param1=2): 
+    foo() # 2
+```

examples/sparse_lr/README.md

@@ -0,0 +1,48 @@
+Sparse LR Examples
+==================
+
+This example shows l1 penalty and sparsity in logistic regression, based on `scikit-learn` example from https://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_l1_l2_sparsity.html#sphx-glr-auto-examples-linear-model-plot-logistic-l1-l2-sparsity-py.
+
+`sparse_lr_train` (from `model.py`) classifies 8x8 images of digits into two classes: 0-4 against 5-9, 
+and visualize the coefficients of the model for different penalty methods(l1 or l2) and C.
+
+We use the `let` decorator to declare hyper-parameters for our algorithm:
+``` python
+@let(learning_rate=0.01, penalty='l1', C=0.01, tol=0.01)
+def sparse_lr_train(X, y):
+    C = local_param('C')
+    penalty = local_param('penalty')
+    tol = local_param('tol')
+    print({'C': C, 'penalty': penalty, 'tol': tol})
+    ...
+```
+
+Four hyper-parameter are defined for function `sparse_lr_train`: `learning_rate`, `penalty`, `C` and `tol`. 
+There are two ways to control the hyper-parameters:
+1. parameter scope (see detail in `example_1.py`):
+
+``` python 
+with param_scope('model.sparse_lr_train.C=0.1'):
+    sparse_lr_train(X, y)
+```
+
+2. command line arguments (see detail in `example_2.py`):
+
+``` python
+def run(args):
+    # run the lr model with parameter from cmdline
+    with param_scope(*args.define):  # set parameters according to cmd line
+        sparse_lr_train(X, y)
+        ...
+
+
+if __name__ == '__main__':
+    # create cmd line arguments parser
+    import argparse
+    parser = argparse.ArgumentParser('example')
+    parser.add_argument('-D', '--define', nargs='*', default=[])
+    args = parser.parse_args()
+
+    run(args)
+
+```

examples/sparse_lr/example_1.py

@@ -0,0 +1,40 @@
+from hyperparameter.hp import param_scope, Tracker
+from model import sparse_lr_train
+import numpy as np
+
+from sklearn import datasets
+from sklearn.preprocessing import StandardScaler
+import matplotlib.pyplot as plt
+
+X, y = datasets.load_digits(return_X_y=True)
+
+X = StandardScaler().fit_transform(X)
+
+# classify small against large digits
+y = (y > 4).astype(int)
+
+print('parameter list: \n  {}'.format('\n  '.join(Tracker.all())))
+
+# run the lr model with default parameters
+coef = sparse_lr_train(X, y)
+
+plt.imshow(
+    np.abs(coef.reshape(8, 8)),
+    interpolation="nearest",
+    cmap="binary",
+    vmax=1,
+    vmin=0,
+)
+plt.show()
+
+# run the lr model with another parameter
+with param_scope('model.sparse_lr_train.C=0.1'):
+    coef = sparse_lr_train(X, y)
+    plt.imshow(
+        np.abs(coef.reshape(8, 8)),
+        interpolation="nearest",
+        cmap="binary",
+        vmax=1,
+        vmin=0,
+    )
+    plt.show()

examples/sparse_lr/example_2.py

@@ -0,0 +1,46 @@
+from hyperparameter.hp import param_scope, Tracker
+from model import sparse_lr_train
+import numpy as np
+
+from sklearn import datasets
+from sklearn.preprocessing import StandardScaler
+import matplotlib.pyplot as plt
+
+X, y = datasets.load_digits(return_X_y=True)
+
+X = StandardScaler().fit_transform(X)
+
+# classify small against large digits
+y = (y > 4).astype(int)
+
+
+def run(args):
+    # run the lr model with parameter from cmdline
+    with param_scope(*args.define):  # set parameters according to cmd line
+        coef = sparse_lr_train(X, y)
+        plt.imshow(
+            np.abs(coef.reshape(8, 8)),
+            interpolation="nearest",
+            cmap="binary",
+            vmax=1,
+            vmin=0,
+        )
+        plt.show()
+
+
+if __name__ == '__main__':
+    # create cmd line arguments parser
+    import argparse
+    parser = argparse.ArgumentParser('example')
+    parser.add_argument(
+        '-D',
+        '--define',
+        nargs='*',
+        default=[],
+        help=
+        'define a parameter `param_name=param_value`, supported parameter list: \n\n  {}'
+        .format('\n  '.join(Tracker.all())))
+    args = parser.parse_args()
+    run(args)
+
+    print(Tracker.report())

examples/sparse_lr/model.py

@@ -0,0 +1,18 @@
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+
+from hyperparameter import let, local_param, param_scope
+
+
+@let(learning_rate=0.01, penalty='l1', C=0.01, tol=0.01)
+def sparse_lr_train(X, y):
+    C = local_param('C')
+    penalty = local_param('penalty')
+    tol = local_param('tol')
+    LR = LogisticRegression(C=C, penalty=penalty, tol=tol, solver='saga')
+
+    LR.fit(X, y)
+    coef =  LR.coef_.ravel()
+    sparsity_LR = np.mean(coef == 0) * 100
+    print({'C': C, 'penalty': penalty, 'tol': tol, 'sparsity': '%.2f%%'%(sparsity_LR)})
+    return coef

hyperparameter/__init__.py

@@ -0,0 +1 @@
+from .hp import *