ENH: Model-based optimization and randomized grid search (#154)

* initial commit

* initial commit

* added optimize_skopt to backtesting

* _optomize_skopt refactor

* made dimensions dynamic

* added unit test-apply PR comments

* added heatmap to skopt and update unit tests

* removed eggs folder

* remove egg folder

* fixed gitignore

* add scikit-optimize dependancy for test

* comment out pickle TRUE

* fixed flake8 errors

* added skopt to Parameter Heatmap notebook

* Revert unwanted changes

* Fixup .gitignore

* Reword docstring

* Refactor Backtest.optimize() code

* make Backtest.optimize() arguments kw-only
* add random_state for reproducible results
* ensure function arguments consistency
* ensure all kwargs have values
* make scikit-optimize package optional
* cast timedelta/datetime dimensions to int
* cache objective_function evaluations (avoid warning)
* ensure param combo matches constraint=
* adjust skopt.forest_minimize() params
* return ordering: stats, heatmap, optimize_result
* clean heatmap and optimize_result

* Make max_tries for method=grid be randomized search

* Update example notebook

* doc/build.sh: unescape URLs

* mypy happy

* minor restyle

* fix typo

* Add changelog entry

Co-authored-by: Tony Freeman <[email protected]>
Co-authored-by: Kernc <[email protected]>

Author: 3 people

.gitignore

@@ -15,5 +15,9 @@ htmlcov/*
 doc/build/*
 
 .idea/*
+.vscode/
+
 **/.ipynb_checkpoints
 *~*
+
+.venv/

CHANGELOG.md

@@ -6,6 +6,8 @@ These were the major changes contributing to each release:
 
 ### 0.x.x
 
+* Faster [model-based optimization](https://kernc.github.io/backtesting.py/doc/examples/Parameter%20Heatmap%20&%20Optimization.html#Model-based%20optimization) using scikit-optimize (#154)
+* Optionally faster [optimization](https://kernc.github.io/backtesting.py/doc/backtesting/backtesting.html#backtesting.backtesting.Backtest.optimize) by randomized grid search (#154)
 * _Annualized_ Return/Volatility/Sharpe/Sortino/Calmar stats (#156)
 * Auto close open trades on backtest finish
 * Add `Backtest.plot(plot_return=)`, akin to `plot_equity=`

backtesting/__init__.py

@@ -7,7 +7,7 @@
 
 * [Library of Utilities and Composable Base Strategies](../examples/Strategies Library.html)
 * [Multiple Time Frames](../examples/Multiple Time Frames.html)
-* [Parameter Heatmap](../examples/Parameter Heatmap.html)
+* [**Parameter Heatmap & Optimization**](../examples/Parameter Heatmap & Optimization.html)
 * [Trading with Machine Learning](../examples/Trading with Machine Learning.html)
 
 These tutorials are also available as live Jupyter notebooks:

backtesting/backtesting.py

@@ -513,6 +513,7 @@ def test_optimize(self):
         self.assertRaises(TypeError, bt.optimize, maximize=15, **OPT_PARAMS)
         self.assertRaises(TypeError, bt.optimize, constraint=15, **OPT_PARAMS)
         self.assertRaises(ValueError, bt.optimize, constraint=lambda d: False, **OPT_PARAMS)
+        self.assertRaises(ValueError, bt.optimize, return_optimization=True, **OPT_PARAMS)
 
         res = bt.optimize(**OPT_PARAMS)
         self.assertIsInstance(res, pd.Series)
@@ -531,6 +532,40 @@ def test_optimize(self):
         with _tempfile() as f:
             bt.plot(filename=f, open_browser=False)
 
+    def test_method_skopt(self):
+        bt = Backtest(GOOG.iloc[:100], SmaCross)
+        res, heatmap, skopt_results = bt.optimize(
+            fast=range(2, 20), slow=np.arange(2, 20, dtype=object),
+            constraint=lambda p: p.fast < p.slow,
+            max_tries=30,
+            method='skopt',
+            return_optimization=True,
+            return_heatmap=True,
+            random_state=2)
+        self.assertIsInstance(res, pd.Series)
+        self.assertIsInstance(heatmap, pd.Series)
+        self.assertGreater(heatmap.max(), 1.1)
+        self.assertGreater(heatmap.min(), -2)
+        self.assertEqual(-skopt_results.fun, heatmap.max())
+        self.assertEqual(heatmap.index.tolist(), heatmap.dropna().index.unique().tolist())
+
+    def test_max_tries(self):
+        bt = Backtest(GOOG.iloc[:100], SmaCross)
+        OPT_PARAMS = dict(fast=range(2, 10, 2), slow=[2, 5, 7, 9])
+        for method, max_tries, random_state in (('grid', 5, 2),
+                                                ('grid', .3, 2),
+                                                ('skopt', 7, 0),
+                                                ('skopt', .45, 0)):
+            with self.subTest(method=method,
+                              max_tries=max_tries,
+                              random_state=random_state):
+                _, heatmap = bt.optimize(max_tries=max_tries,
+                                         method=method,
+                                         random_state=random_state,
+                                         return_heatmap=True,
+                                         **OPT_PARAMS)
+                self.assertEqual(len(heatmap), 6)
+
     def test_nowrite_df(self):
         # Test we don't write into passed data df by default.
         # Important for copy-on-write in Backtest.optimize()

backtesting/test/_test.py

@@ -90,6 +90,7 @@ for line in sys.stdin.readlines():
     grep -v $'\t''$' |
     while read -r line; do
         while IFS=$'\t' read -r file url; do
+            url=$(python -c 'import html, sys; print(html.unescape(sys.argv[-1]))' "$url")
             [ -f "$url" ] ||
                 curl --silent --fail --retry 2 --user-agent 'Mozilla/5.0 Firefox 61' "$url" >/dev/null 2>&1 ||
                 die "broken link in $file:  $url"