Fixes several bugs with example scripts and notebooks.

Adds random sampling to acq maximization. This allows for both brute force and clever optimizations to work together, and is more robust in situations where the acquisition function is very flat with a hard to find maximum.

Author: fmfn

README.md

@@ -1,31 +1,51 @@
 # Bayesian Optimization
 
-Pure Python implementation of bayesian global optimization with gaussian processes.
+Pure Python implementation of bayesian global optimization with gaussian
+processes.
 
     pip install git+https://github.com/fmfn/BayesianOptimization.git
 
-This is a constrained global optimization package built upon bayesian inference and gaussian process, that attempts to find the maximum value of an unknown function in as few iterations as possible. This technique is particularly suited for optimization of high cost functions, situations where the balance between exploration and exploitation is important.
+This is a constrained global optimization package built upon bayesian inference
+and gaussian process, that attempts to find the maximum value of an unknown
+function in as few iterations as possible. This technique is particularly
+suited for optimization of high cost functions, situations where the balance
+between exploration and exploitation is important.
+
+To get a grip of how this method and package works in the [examples](https://github.com/fmfn/BayesianOptimization/tree/master/examples)
+folder you can:
+- Checkout this [notebook](https://github.com/fmfn/BayesianOptimization/blob/master/examples/visualization.ipynb)
+with a step by step visualization of how this method works.
+- Go over this [script](https://github.com/fmfn/BayesianOptimization/blob/master/examples/usage.py)
+to become familiar with this packages basic functionalities.
+- Explore this [notebook](https://github.com/fmfn/BayesianOptimization/blob/master/examples/exploitation%20vs%20exploration.ipynb)
+exemplifying the balance between exploration and exploitation and how to
+control it.
+- Checkout these scripts ([sklearn](https://github.com/fmfn/BayesianOptimization/blob/master/examples/sklearn_example.py),
+[xgboost](https://github.com/fmfn/BayesianOptimization/blob/master/examples/xgboost_example.py))
+for examples of how to use this package to tune parameters of ML estimators
+using cross validation and bayesian optimization
 
-Checkout this [notebook](https://github.com/fmfn/BayesianOptimization/blob/master/examples/visualization.ipynb) with a step by step visualization of how this method works.
 
 ![BayesianOptimization in action](https://github.com/fmfn/BayesianOptimization/blob/master/examples/bo_example.png)
 
-Checkout the [examples](https://github.com/fmfn/BayesianOptimization/tree/master/examples) folder for more scripts with examples of how to use this package.
-
 ![BayesianOptimization in action](https://github.com/fmfn/BayesianOptimization/blob/master/examples/bayesian_optimization.gif)
 
+
+This project is under active development, if you find a bug, or anything that
+needs correction, please let me know.
+
 Installation
 ============
 
 ### Installation
 
-BayesianOptimization is not currently available on the PyPi's reporitories, 
+BayesianOptimization is not currently available on the PyPi's reporitories,
 however you can install it via `pip`:
 
     pip install git+https://github.com/fmfn/BayesianOptimization.git
 
-If you prefer, you can clone it and run the setup.py file. Use the following commands to get a 
-copy from Github and install all dependencies:
+If you prefer, you can clone it and run the setup.py file. Use the following
+commands to get a copy from Github and install all dependencies:
 
     git clone https://github.com/fmfn/BayesianOptimization.git
     cd BayesianOptimization
@@ -36,8 +56,6 @@ copy from Github and install all dependencies:
 * Scipy
 * Scikit-learn
 
-Disclaimer: This project is under active development, if you find a bug, or anything that needs correction, please let me know.
-
 ### References:
 * http://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf
 * http://arxiv.org/pdf/1012.2599v1.pdf

bayes_opt/bayesian_optimization.py

@@ -5,7 +5,6 @@
 from sklearn.gaussian_process import GaussianProcessRegressor
 from sklearn.gaussian_process.kernels import Matern
 from .helpers import UtilityFunction, unique_rows, PrintLog, acq_max
-__author__ = 'fmfn'
 
 
 class BayesianOptimization(object):
@@ -88,7 +87,8 @@ def init(self, init_points):
         """
 
         # Generate random points
-        l = [np.random.uniform(x[0], x[1], size=init_points) for x in self.bounds]
+        l = [np.random.uniform(x[0], x[1], size=init_points)
+             for x in self.bounds]
 
         # Concatenate new random points to possible existing
         # points from self.explore method.
@@ -172,7 +172,9 @@ def initialize_df(self, points_df):
         Method to introduce point for which the target function
         value is known from pandas dataframe file
 
-        :param points_df: pandas dataframe with columns (target, {list of columns matching self.keys})
+        :param points_df:
+            pandas dataframe with columns (target, {list of columns matching
+            self.keys})
 
         ex:
               target        alpha      colsample_bytree        gamma
@@ -333,7 +335,8 @@ def points_to_csv(self, file_name):
         After training all points for which we know target variable
         (both from initialization and optimization) are saved
 
-        :param file_name: name of the file where points will be saved in the csv format
+        :param file_name: name of the file where points will be saved in the csv
+            format
 
         :return: None
         """

bayes_opt/helpers.py

@@ -8,8 +8,11 @@
 
 def acq_max(ac, gp, y_max, bounds):
     """
-    A function to find the maximum of the acquisition function using
-    the 'L-BFGS-B' method.
+    A function to find the maximum of the acquisition function
+
+    It uses a combination of random sampling (cheap) and the 'L-BFGS-B'
+    optimization method. First by sampling 1e5 points at random, and then
+    running L-BFGS-B from 250 random starting points.
 
     Parameters
     ----------
@@ -31,24 +34,27 @@ def acq_max(ac, gp, y_max, bounds):
     :return: x_max, The arg max of the acquisition function.
     """
 
-    # Start with the lower bound as the argmax
-    x_max = bounds[:, 0]
-    max_acq = None
-
+    # Warm up with random points
     x_tries = np.random.uniform(bounds[:, 0], bounds[:, 1],
-                                size=(100, bounds.shape[0]))
-
-    for x_try in x_tries:
+                                 size=(100000, bounds.shape[0]))
+    ys = ac(x_tries, gp=gp, y_max=y_max)
+    x_max = x_tries[ys.argmax()]
+    max_acq = ys.max()
+
+    # Explore the parameter space more throughly
+    x_seeds = np.random.uniform(bounds[:, 0], bounds[:, 1],
+                                size=(250, bounds.shape[0]))
+    for x_try in x_seeds:
         # Find the minimum of minus the acquisition function
         res = minimize(lambda x: -ac(x.reshape(1, -1), gp=gp, y_max=y_max),
                        x_try.reshape(1, -1),
                        bounds=bounds,
                        method="L-BFGS-B")
 
         # Store it if better than previous minimum(maximum).
-        if max_acq is None or -res.fun >= max_acq:
+        if max_acq is None or -res.fun[0] >= max_acq:
             x_max = res.x
-            max_acq = -res.fun
+            max_acq = -res.fun[0]
 
     # Clip output to make sure it lies within the bounds. Due to floating
     # point technicalities this is not always the case.
@@ -165,7 +171,8 @@ def print_header(self, initialization=True):
             print("{}Bayesian Optimization{}".format(BColours.RED,
                                                      BColours.ENDC))
 
-        print(BColours.BLUE + "-" * (29 + sum([s + 5 for s in self.sizes])) + BColours.ENDC)
+        print(BColours.BLUE + "-" * (29 + sum([s + 5 for s in self.sizes])) +
+            BColours.ENDC)
 
         print("{0:>{1}}".format("Step", 5), end=" | ")
         print("{0:>{1}}".format("Time", 6), end=" | ")
@@ -193,10 +200,12 @@ def print_step(self, x, y, warning=False):
                   end=" | ")
 
             for index in self.sorti:
-                print("{0}{2: >{3}.{4}f}{1}".format(BColours.GREEN, BColours.ENDC,
-                                                    x[index],
-                                                    self.sizes[index] + 2,
-                                                    min(self.sizes[index] - 3, 6 - 2)),
+                print("{0}{2: >{3}.{4}f}{1}".format(
+                            BColours.GREEN, BColours.ENDC,
+                            x[index],
+                            self.sizes[index] + 2,
+                            min(self.sizes[index] - 3, 6 - 2)
+                        ),
                       end=" | ")
         else:
             print("{: >10.5f}".format(y), end=" | ")