Deprecate the Lambda() and Sigma() functions for AugLagrangian (#439)

* Deprecate the Lambda() and Sigma() functions for AugLagrangian.

* Add utility struct to differentiate between matrices and callbacks.

* Update HISTORY.

* Fix overload issues with LRSDP and new AugLagrangian types.

* Fix some documentation to kick off another build.  (I think there was an issue with dealgood.)

* Remove spaces to kick the build off again.

Author: rcurtin

CONTRIBUTING.md

@@ -8,9 +8,10 @@ you have an improvement you would like to see, we would love to include it!
 
 The ensmallen maintainer community overlaps heavily with the
 [mlpack](https://github.com/mlpack/mlpack) community, so development discussions
-can happen either here on Github, on the [mlpack mailing
-list](http://lists.mlpack.org/mailman/listinfo/mlpack), or in the #mlpack
-IRC channel on irc.freenode.net.
+can happen either here on Github or in the `#mlpack:matrix.org` channel on
+[Matrix](https://www.matrix.org/).  See
+[here](https://www.mlpack.org/doc/developer/community.html) for more
+information.
 
 Once a pull request is submitted, it must be reviewed and approved before a
 merge, to ensure that:

HISTORY.md

@@ -41,12 +41,24 @@
     ActiveCMAES<FullSelection, BoundaryBoxConstraint> opt(lambda,
         BoundaryBoxConstraint(lowerBound, upperBound), ...);
     ```
-
  * Add proximal gradient optimizers for L1-constrained and other related
    problems: `FBS`, `FISTA`, and `FASTA`
    ([#427](https://github.com/mlpack/ensmallen/pull/427)).  See the
    documentation for more details.
 
+ * The `Lambda()` and `Sigma()` functions of the `AugLagrangian` optimizer,
+   which could be used to retrieve the Lagrange multipliers and penalty
+   parameter after optimization, are now deprecated
+   ([#439](https://github.com/mlpack/ensmallen/pull/439)).  Instead, pass a
+   vector and a double to the `Optimize()` function directly:
+
+    ```c++
+    augLag.Optimize(function, coordinates, lambda, sigma)
+    ```
+
+   and these will be filled with the final Lagrange multiplier estimates and
+   penalty parameters.
+
 ### ensmallen 2.22.2: "E-Bike Excitement"
 ###### 2025-04-30
  * Fix include statement in `tests/de_test.cpp`

README.md

@@ -22,12 +22,12 @@ Documentation and downloads: https://ensmallen.org
 
 ### Installation
 
-ensmallen can be installed in several ways: either manually or via cmake, 
+ensmallen can be installed in several ways: either manually or via cmake,
 with or without root access.
 
-The cmake based installation will check the requirements 
-and optionally build the tests. If cmake 3.3 (or a later version) 
-is not already available on your system, it can be obtained 
+The cmake based installation will check the requirements
+and optionally build the tests. If cmake 3.3 (or a later version)
+is not already available on your system, it can be obtained
 from [cmake.org](https://cmake.org).
 
 Example cmake based installation with root access:
@@ -39,7 +39,7 @@ cmake ..
 sudo make install
 ```
 
-Example cmake based installation without root access, 
+Example cmake based installation without root access,
 installing into `/home/blah/` (adapt as required):
 
 ```
@@ -49,7 +49,7 @@ cmake .. -DCMAKE_INSTALL_PREFIX:PATH=/home/blah/
 make install
 ```
 
-The above will create a directory named `/home/blah/include/` 
+The above will create a directory named `/home/blah/include/`
 and place all ensmallen headers there.
 
 To optionally build and run the tests
@@ -61,10 +61,10 @@ make ensmallen_tests
 ./ensmallen_tests --durations yes
 ```
 
-Manual installation involves simply copying the `include/ensmallen.hpp` header 
-***and*** the associated `include/ensmallen_bits` directory to a location 
+Manual installation involves simply copying the `include/ensmallen.hpp` header
+***and*** the associated `include/ensmallen_bits` directory to a location
 such as `/usr/include/` which is searched by your C++ compiler.
-If you can't use `sudo` or don't have write access to `/usr/include/`, 
+If you can't use `sudo` or don't have write access to `/usr/include/`,
 use a directory within your own home directory (eg. `/home/blah/include/`).
 
 
@@ -73,19 +73,19 @@ use a directory within your own home directory (eg. `/home/blah/include/`).
 If you have installed ensmallen in a standard location such as `/usr/include/`:
 
     g++ prog.cpp -o prog -O2 -larmadillo
-    
-If you have installed ensmallen in a non-standard location, 
-such as `/home/blah/include/`, you will need to make sure 
-that your C++ compiler searches `/home/blah/include/` 
-by explicitly specifying the directory as an argument/option. 
+
+If you have installed ensmallen in a non-standard location,
+such as `/home/blah/include/`, you will need to make sure
+that your C++ compiler searches `/home/blah/include/`
+by explicitly specifying the directory as an argument/option.
 For example, using the `-I` switch in gcc and clang:
 
     g++ prog.cpp -o prog -O2 -I /home/blah/include/ -larmadillo
 
 
 ### Example Optimization
 
-See [`example.cpp`](example.cpp) for example usage of the L-BFGS optimizer 
+See [`example.cpp`](example.cpp) for example usage of the L-BFGS optimizer
 in a linear regression setting.
 
 
@@ -103,8 +103,8 @@ Please cite the following paper if you use ensmallen in your research and/or
 software. Citations are useful for the continued development and maintenance of
 the library.
 
-* Ryan R. Curtin, Marcus Edel, Rahul Ganesh Prabhu, Suryoday Basak, Zhihao Lou, Conrad Sanderson.  
-  [The ensmallen library for flexible numerical optimization](https://jmlr.org/papers/volume22/20-416/20-416.pdf).  
+* Ryan R. Curtin, Marcus Edel, Rahul Ganesh Prabhu, Suryoday Basak, Zhihao Lou, Conrad Sanderson.
+  [The ensmallen library for flexible numerical optimization](https://jmlr.org/papers/volume22/20-416/20-416.pdf).
   Journal of Machine Learning Research, Vol. 22, No. 166, 2021.
 
 ```

doc/optimizers.md

@@ -808,44 +808,20 @@ corresponding vector type (e.g. `arma::vec` or `coot::fvec`).
 The attributes of the optimizer may also be modified via the member methods
 `MaxIterations()`, `PenaltyThresholdFactor()`, `SigmaUpdateFactor()` and `LBFGS()`.
 
-<details open>
-<summary>Click to collapse/expand example code.
-</summary>
+The `AugLagrangian` optimizer also allows manually specifying the initial
+Lagrange multipliers (`lambda`) and penalty parameter (`sigma`) directly in the
+call to `Optimize()`.  For this, the following version of `Optimize()` should be
+used:
 
-```c++
-/**
- * Optimize the function.  The value '1' is used for the initial value of each
- * Lagrange multiplier.  To set the Lagrange multipliers yourself, use the
- * other overload of Optimize().
- *
- * @tparam LagrangianFunctionType Function which can be optimized by this
- *     class.
- * @param function The function to optimize.
- * @param coordinates Output matrix to store the optimized coordinates in.
- */
-template<typename LagrangianFunctionType>
-bool Optimize(LagrangianFunctionType& function,
-              arma::mat& coordinates);
+ * `opt.Optimize(`_`function, coordinates, lambda, sigma, callbacks...`_`)`
 
-/**
- * Optimize the function, giving initial estimates for the Lagrange
- * multipliers.  The vector of Lagrange multipliers will be modified to
- * contain the Lagrange multipliers of the final solution (if one is found).
- *
- * @tparam LagrangianFunctionType Function which can be optimized by this
- *      class.
- * @param function The function to optimize.
- * @param coordinates Output matrix to store the optimized coordinates in.
- * @param initLambda Vector of initial Lagrange multipliers.  Should have
- *     length equal to the number of constraints.
- * @param initSigma Initial penalty parameter.
- */
-template<typename LagrangianFunctionType>
-bool Optimize(LagrangianFunctionType& function,
-              arma::mat& coordinates,
-              const arma::vec& initLambda,
-              const double initSigma);
-```
+In that call, `lambda` should be a column vector of the same type as
+`coordinates`, and `sigma` is a `double`.  `lambda` and `sigma` will be
+overwritten with the final values of the Lagrange multipliers and penalty
+parameters.
+
+If `lambda` and `sigma` are not specified, then 0 is used as the initial value
+for all Lagrange multipliers and 10 is used as the initial penalty parameter.
 
 </details>
 
@@ -2400,6 +2376,20 @@ The attributes of the LRSDP optimizer may only be accessed via member methods.
 | `size_t` | **`MaxIterations()`** | Maximum number of iterations before termination. | `1000` |
 | `AugLagrangian` | **`AugLag()`** | The internally-held Augmented Lagrangian optimizer. | **n/a** |
 
+Because `LRSDP` uses the [`AugLagrangian`](#auglagrangian) optimizer internally,
+an additional overload of `Optimize()` is supplied to allow specifying the
+initial Lagrange multiplier estimates and penalty parameter:
+
+ * `lrsdp.Optimize(`_`coordinates, lambda, sigma, callbacks...`_`)`
+
+In that call, `lambda` should be a column vector of the same type as
+`coordinates`, and `sigma` is a `double`.  `lambda` and `sigma` will be
+overwritten with the final values of the Lagrange multipliers and penalty
+parameters.
+
+If `lambda` and `sigma` are not specified, then 0 is used as the initial value
+for all Lagrange multipliers and 10 is used as the initial penalty parameter.
+
 #### See also:
 
  * [A Nonlinear Programming Algorithm for Solving Semidefinite Programs via Low-rank Factorization](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.682.1520&rep=rep1&type=pdf)

include/ensmallen.hpp

@@ -81,6 +81,7 @@
 #include "ensmallen_bits/utility/proxies.hpp"
 #include "ensmallen_bits/utility/function_traits.hpp"
 #include "ensmallen_bits/utility/using.hpp"
+#include "ensmallen_bits/utility/detect_callbacks.hpp"
 #include "ensmallen_bits/utility/indicators/epsilon.hpp"
 #include "ensmallen_bits/utility/indicators/igd.hpp"
 #include "ensmallen_bits/utility/indicators/igd_plus.hpp"

include/ensmallen_bits/aug_lagrangian/aug_lagrangian.hpp

@@ -30,7 +30,7 @@ namespace ens {
  * documentation on function types included with this distribution or on the
  * ensmallen website.
  */
-template<typename VecType = arma::vec>
+template<typename VecType = arma::vec> // TODO: remove for ensmallen 4.x
 class AugLagrangianType
 {
  public:
@@ -50,7 +50,7 @@ class AugLagrangianType
                     const L_BFGS& lbfgs = L_BFGS());
 
   /**
-   * Optimize the function.  The value '1' is used for the initial value of each
+   * Optimize the function.  The value '0' is used for the initial value of each
    * Lagrange multiplier.  To set the Lagrange multipliers yourself, use the
    * other overload of Optimize().
    *
@@ -67,7 +67,8 @@ class AugLagrangianType
            typename MatType,
            typename GradType,
            typename... CallbackTypes>
-  typename std::enable_if<IsMatrixType<GradType>::value, bool>::type
+  typename std::enable_if<IsMatrixType<GradType>::value &&
+                          IsAllNonMatrix<CallbackTypes...>::value, bool>::type
   Optimize(LagrangianFunctionType& function,
            MatType& coordinates,
            CallbackTypes&&... callbacks);
@@ -76,9 +77,10 @@ class AugLagrangianType
   template<typename LagrangianFunctionType,
            typename MatType,
            typename... CallbackTypes>
-  bool Optimize(LagrangianFunctionType& function,
-                MatType& coordinates,
-                CallbackTypes&&... callbacks)
+  typename std::enable_if<IsAllNonMatrix<CallbackTypes...>::value, bool>::type
+  Optimize(LagrangianFunctionType& function,
+           MatType& coordinates,
+           CallbackTypes&&... callbacks)
   {
     return Optimize<LagrangianFunctionType, MatType, MatType,
         CallbackTypes...>(function, coordinates,
@@ -97,26 +99,50 @@ class AugLagrangianType
    * @tparam CallbackTypes Types of callback functions.
    * @param function The function to optimize.
    * @param coordinates Output matrix to store the optimized coordinates in.
-   * @param initLambda Vector of initial Lagrange multipliers.  Should have
-   *     length equal to the number of constraints.
-   * @param initSigma Initial penalty parameter.
+   * @param lambda Vector containing initial Lagrange multipliers.  Should have
+   *     length equal to the number of constraints.  This will be overwritten
+   *     with the Lagrange multipliers that are found during optimization.
+   * @param sigma Initial penalty parameter.  This will be overwritten with the
+   *     final penalty value used during optimization.
    * @param callbacks Callback functions.
    */
   template<typename LagrangianFunctionType,
            typename MatType,
+           typename InVecType,
            typename GradType,
            typename... CallbackTypes>
+  [[deprecated("use Optimize() with non-const lambda/sigma instead")]]
   typename std::enable_if<IsMatrixType<GradType>::value, bool>::type
   Optimize(LagrangianFunctionType& function,
            MatType& coordinates,
-           const VecType& initLambda,
+           const InVecType& initLambda,
            const double initSigma,
+           CallbackTypes&&... callbacks)
+  {
+    deprecatedLambda = initLambda;
+    deprecatedSigma = initSigma;
+    const bool result = Optimize(function, coordinates, this->deprecatedLambda,
+        this->deprecatedSigma,
+        std::forward<CallbackTypes>(callbacks)...);
+  }
+
+  template<typename LagrangianFunctionType,
+           typename MatType,
+           typename InVecType,
+           typename GradType,
+           typename... CallbackTypes>
+  typename std::enable_if<IsMatrixType<GradType>::value, bool>::type
+  Optimize(LagrangianFunctionType& function,
+           MatType& coordinates,
+           InVecType& lambda,
+           double& sigma,
            CallbackTypes&&... callbacks);
 
   //! Forward the MatType as GradType.
   template<typename LagrangianFunctionType,
            typename MatType,
            typename... CallbackTypes>
+  [[deprecated("use Optimize() with non-const lambda/sigma instead")]]
   bool Optimize(LagrangianFunctionType& function,
                 MatType& coordinates,
                 const VecType& initLambda,
@@ -128,20 +154,39 @@ class AugLagrangianType
         std::forward<CallbackTypes>(callbacks)...);
   }
 
+  template<typename LagrangianFunctionType,
+           typename MatType,
+           typename InVecType,
+           typename... CallbackTypes>
+  bool Optimize(LagrangianFunctionType& function,
+                MatType& coordinates,
+                InVecType& lambda,
+                double& sigma,
+                CallbackTypes&&... callbacks)
+  {
+    return Optimize<LagrangianFunctionType, MatType, InVecType, MatType,
+        CallbackTypes...>(function, coordinates, lambda, sigma,
+        std::forward<CallbackTypes>(callbacks)...);
+  }
+
   //! Get the L-BFGS object used for the actual optimization.
   const L_BFGS& LBFGS() const { return lbfgs; }
   //! Modify the L-BFGS object used for the actual optimization.
   L_BFGS& LBFGS() { return lbfgs; }
 
   //! Get the Lagrange multipliers.
-  const VecType& Lambda() const { return lambda; }
+  [[deprecated("use Optimize() with lambda/sigma parameters instead")]]
+  const VecType& Lambda() const { return deprecatedLambda; }
   //! Modify the Lagrange multipliers (i.e. set them before optimization).
-  VecType& Lambda() { return lambda; }
+  [[deprecated("use Optimize() with lambda/sigma parameters instead")]]
+  VecType& Lambda() { return deprecatedLambda; }
 
   //! Get the penalty parameter.
-  double Sigma() const { return sigma; }
+  [[deprecated("use Optimize() with lambda/sigma parameters instead")]]
+  double Sigma() const { return deprecatedSigma; }
   //! Modify the penalty parameter.
-  double& Sigma() { return sigma; }
+  [[deprecated("use Optimize() with lambda/sigma parameters instead")]]
+  double& Sigma() { return deprecatedSigma; }
 
   //! Get the maximum iterations
   size_t MaxIterations() const { return maxIterations; }
@@ -174,35 +219,37 @@ class AugLagrangianType
   //! Controls early termination of the optimization process.
   bool terminate;
 
+  // NOTE: these will be removed in ensmallen 4.x!
   //! Lagrange multipliers.
-  VecType lambda;
-
+  VecType deprecatedLambda;
   //! Penalty parameter.
-  double sigma;
+  double deprecatedSigma;
 
   /**
    * Internal optimization function: given an initialized AugLagrangianFunction,
    * perform the optimization itself.
    */
   template<typename LagrangianFunctionType,
            typename MatType,
+           typename InVecType,
            typename GradType,
            typename... CallbackTypes>
   typename std::enable_if<IsMatrixType<GradType>::value, bool>::type
-  Optimize(AugLagrangianFunction<LagrangianFunctionType, VecType>& augfunc,
+  Optimize(AugLagrangianFunction<LagrangianFunctionType, InVecType>& augfunc,
            MatType& coordinates,
            CallbackTypes&&... callbacks);
 
   //! Forward the MatType as GradType.
   template<typename LagrangianFunctionType,
            typename MatType,
+           typename InVecType,
            typename... CallbackTypes>
   bool Optimize(
-      AugLagrangianFunction<LagrangianFunctionType, VecType>& function,
+      AugLagrangianFunction<LagrangianFunctionType, InVecType>& function,
       MatType& coordinates,
       CallbackTypes&&... callbacks)
   {
-    return Optimize<LagrangianFunctionType, MatType, MatType,
+    return Optimize<LagrangianFunctionType, MatType, InVecType, MatType,
         CallbackTypes...>(function, coordinates,
         std::forward<CallbackTypes>(callbacks)...);
   }