Update docs and examples

Author: mjhough

docs/build/doctrees/environment.pickle

@@ -49,3 +49,4 @@ Version 1.2.3 (1 Jun 2021)
 Version 1.3.0 (16 Oct 2021)
 ---------------------------
 * Handle finitely many arbitrary convex constraints in addition to simple bound constraints.
+* Only new functionality is added, so there is no change to the solver for unconstrained/bound-constrained problems.

docs/build/doctrees/history.doctree

@@ -11,16 +11,17 @@ DFO-LS: Derivative-Free Optimizer for Least-Squares Minimization
 
 **Author:** `Lindon Roberts <[email protected]>`_
 
-DFO-LS is a flexible package for finding local solutions to nonlinear least-squares minimization problems (with optional convex constraints), without requiring any derivatives of the objective. DFO-LS stands for Derivative-Free Optimizer for Least-Squares.
+DFO-LS is a flexible package for finding local solutions to nonlinear least-squares minimization problems (with optional constraints), without requiring any derivatives of the objective. DFO-LS stands for Derivative-Free Optimizer for Least-Squares.
 
 That is, DFO-LS solves
 
 .. math::
 
    \min_{x\in\mathbb{R}^n}  &\quad  f(x) := \sum_{i=1}^{m}r_{i}(x)^2 \\
-   \text{s.t.} &\quad x \in C
+   \text{s.t.} &\quad x \in C\\
+               &\quad  a \leq x \leq b
 
-The constraint set :math:`C` is assumed to be non-empty, closed and convex. Moreover, the constraints are non-relaxable (i.e. DFO-LS will never ask to evaluate a point that is not feasible).
+The constraint set :math:`C` is the intersection of multiple convex sets provided as input by the user. All constraints are non-relaxable (i.e. DFO-LS will never ask to evaluate a point that is not feasible).
 
 Full details of the DFO-LS algorithm are given in our paper: C. Cartis, J. Fiala, B. Marteau and L. Roberts, `Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers <https://doi.org/10.1145/3338517>`_, *ACM Transactions on Mathematical Software*, 45:3 (2019), pp. 32:1-32:41 [`preprint <https://arxiv.org/abs/1804.00154>`_] . DFO-LS is a more flexible version of `DFO-GN <https://github.com/numericalalgorithmsgroup/dfogn>`_.
 

docs/build/doctrees/index.doctree

@@ -8,10 +8,11 @@ DFO-LS is designed to solve the nonlinear least-squares minimization problem (wi
 .. math::
 
    \min_{x\in\mathbb{R}^n}  &\quad  f(x) := \sum_{i=1}^{m}r_{i}(x)^2 \\
-   \text{s.t.} &\quad x \in C
+   \text{s.t.} &\quad x \in C\\
+               &\quad  a \leq x \leq b
 
 We call :math:`f(x)` the objective function and :math:`r_i(x)` the residual functions (or simply residuals).
-Here :math:`C` is a non-empty and closed convex set.
+:math:`C` is the intersection of multiple convex sets given as input by the user.
 
 DFO-LS is a *derivative-free* optimization algorithm, which means it does not require the user to provide the derivatives of :math:`f(x)` or :math:`r_i(x)`, nor does it attempt to estimate them internally (by using finite differencing, for instance). 
 

docs/build/doctrees/info.doctree

@@ -144,7 +144,7 @@ Note that DFO-LS is a randomized algorithm: in its first phase, it builds an int
 
 This and all following problems can be found in the `examples <https://github.com/numericalalgorithmsgroup/dfols/tree/master/examples>`_ directory on the DFO-LS Github page.
 
-Adding Constraints and More Output
+Adding Bounds and More Output
 -----------------------------
 We can extend the above script to add constraints. To add bound constraints alone, we can add the lines
 
@@ -181,41 +181,6 @@ However, we also get a warning that our starting point was outside of the bounds
 
 DFO-LS automatically fixes this, and moves :math:`x_0` to a point within the bounds, in this case :math:`x_0=(-1.2,0.85)`.
 
-If we want more complex constraints, we can instead write something like the following:
-
-  .. code-block:: python
-  
-      # Define the projection functions
-      def pball(x):
-          c = np.array([0.7,1.5]) # ball centre
-          r = 0.4 # ball radius
-          return c + (r/np.max([np.linalg.norm(x-c),r]))*(x-c)
-
-      def pbox(x):
-          l = np.array([-2, 1.1]) # lower bound
-          u = np.array([0.9, 3]) # upper bound
-          return np.minimum(np.maximum(x,l), u)
-
-      # Call DFO-LS (with box and ball constraints)
-      soln = dfols.solve(rosenbrock, x0, projections=[pball,pbox])
-
-DFO-LS correctly finds the solution to this constrained problem too. Note that we get a warning because the step computed in the trust region subproblem
-gave an increase in the model. This is common in the case where multiple constraints are active at the optimal point.
-
-  .. code-block:: none
-
-      ****** DFO-LS Results ******
-      Solution xmin = [0.9        1.15359245]
-      Residual vector = [3.43592448 0.1       ]
-      Objective value f(xmin) = 11.81557703
-      Needed 10 objective evaluations (at 10 points)
-      Approximate Jacobian = [[-1.79826221e+01  1.00004412e+01]
-       [-1.00000000e+00 -1.81976605e-15]]
-      Exit flag = 5
-      Warning (trust region increase): Either multiple constraints are active or trust region step gave model increase
-      ****************************
-
-
 We can also get DFO-LS to print out more detailed information about its progress using the `logging <https://docs.python.org/3/library/logging.html>`_ module. To do this, we need to add the following lines:
 
   .. code-block:: python
@@ -259,6 +224,95 @@ An alternative option available is to get DFO-LS to print to terminal progress i
         1    55    1.00e-02  2.00e-01  1.50e-08  1.00e-08   56   
         1    56    1.00e-02  2.00e-01  1.50e-08  1.00e-08   57
 
+Handling Arbitrary Convex Constraints
+-----------------------------
+DFO-LS can also handle more general constraints where they can be written as the intersection of finitely many convex sets. For example, the below code
+minimizes the Rosenbrock function subject to a constraint set given by the intersection of two convex sets. Note the intersection of the user-provided convex
+sets must be non-empty.
+
+  .. code-block:: python
+  
+      '''
+      DFO-LS example: minimize the Rosenbrock function with arbitrary convex constraints
+
+      This example defines two functions pball(x) and pbox(x) that project onto ball and
+      box constraint sets respectively. It then passes both these functions to the DFO-LS
+      solver so that it can find a constrained minimizer to the Rosenbrock function.
+      Such a minimizer must lie in the intersection of constraint sets corresponding to
+      projection functions pball(x) and pbox(x). The description of the problem is as follows:
+
+          min rosenbrock(x)
+          s.t.
+              -2 <= x[0] <= 1.1,
+              1.1 <= x[1] <= 3,
+              norm(x-c) <= 0.4
+
+      where c = [0.7, 1.5] is the centre of the ball.
+      '''
+      from __future__ import print_function
+      import numpy as np
+      import dfols
+
+      # Define the objective function
+      def rosenbrock(x):
+          return np.array([10.0 * (x[1] - x[0] ** 2), 1.0 - x[0]])
+
+      # Define the starting point
+      x0 = np.array([-1.2, 1])
+
+      '''
+      Define ball projection function
+      Projects the input x onto a ball with
+      centre point (0.7,1.5) and radius 0.4.
+      '''
+      def pball(x):
+          c = np.array([0.7,1.5]) # ball centre
+          r = 0.4 # ball radius
+          return c + (r/np.max([np.linalg.norm(x-c),r]))*(x-c)
+
+      '''
+      Define box projection function
+      Projects the input x onto a box
+      such that -2 <= x[0] <= 0.9 and
+      1.1 <= x[1] <= 3.
+
+      Note: One could equivalently add bound
+      constraints as a separate input to the solver
+      instead.
+      '''
+      def pbox(x):
+          l = np.array([-2, 1.1]) # lower bound
+          u = np.array([0.9, 3]) # upper bound
+          return np.minimum(np.maximum(x,l), u)
+
+      # For optional extra output details
+      import logging
+      logging.basicConfig(level=logging.DEBUG, format='%(message)s')
+
+      # Call DFO-LS
+      soln = dfols.solve(rosenbrock, x0, projections=[pball,pbox])
+
+      # Display output
+      print(soln)
+
+Note that for bound constraints one can choose to either implement them by defining a projection function as above, or by passing the bounds as input like in the example from the section on adding bound constraints.
+
+DFO-LS correctly finds the solution to this constrained problem too. Note that we get a warning because the step computed in the trust region subproblem
+gave an increase in the model. This is common in the case where multiple constraints are active at the optimal point.
+
+  .. code-block:: none
+
+      ****** DFO-LS Results ******
+      Solution xmin = [0.9        1.15359245]
+      Residual vector = [3.43592448 0.1       ]
+      Objective value f(xmin) = 11.81557703
+      Needed 10 objective evaluations (at 10 points)
+      Approximate Jacobian = [[-1.79826221e+01  1.00004412e+01]
+       [-1.00000000e+00 -1.81976605e-15]]
+      Exit flag = 5
+      Warning (trust region increase): Either multiple constraints are active or trust region step gave model increase
+      ****************************
+
 Example: Noisy Objective Evaluation
 -----------------------------------
 As described in :doc:`info`, derivative-free algorithms such as DFO-LS are particularly useful when :code:`objfun` has noise. Let's modify the previous example to include random noise in our objective evaluation, and compare it to a derivative-based solver:

docs/build/doctrees/userguide.doctree

@@ -1,7 +1,7 @@
 pre { line-height: 125%; }
-td.linenos pre { color: #000000; background-color: #f0f0f0; padding-left: 5px; padding-right: 5px; }
-span.linenos { color: #000000; background-color: #f0f0f0; padding-left: 5px; padding-right: 5px; }
-td.linenos pre.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
 span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
 .highlight .hll { background-color: #ffffcc }
 .highlight { background: #eeffcc; }