add least squares example

Author: MaxenceGollier

docs/Project.toml

@@ -2,8 +2,10 @@
 ADNLPModels = "54578032-b7ea-4c30-94aa-7cbd1cce6c9a"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 DocumenterCitations = "daee34ce-89f3-4625-b898-19384cb65244"
+LLSModels = "39f5bc3e-5160-4bf8-ac48-504fd2534d24"
 NLPModelsModifiers = "e01155f1-5c6f-4375-a9d8-616dd036575f"
 ProximalOperators = "a725b495-10eb-56fe-b38b-717eba820537"
+Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 RegularizedProblems = "ea076b23-609f-44d2-bb12-a4ae45328278"
 
 [compat]

docs/src/examples/ls.md

@@ -1 +1,85 @@
-# A regularized least-square problem
+# A regularized least-square problem
+
+In this tutorial, we will show how to model and solve the nonconvex nonsmooth least-square problem
+```math
+  \min_{x \in \mathbb{R}^n} \frac{1}{2} \|Ax - b\|_2^2 + \lambda \|x\|_0.
+```
+
+## Modelling the problem
+We first formulate the objective function as the sum of a smooth function $f$ and a nonsmooth regularizer $h$:
+```math
+  \frac{1}{2} \|Ax - b\|_2^2 + \lambda \|x\|_0 = f(x) + h(x),
+```
+where 
+```math
+\begin{align*}
+f(x) &:= \frac{1}{2} \|Ax - b\|_2^2,\\
+h(x) &:= \lambda\|x\|_0.
+\end{align*}
+```
+
+To model $f$, we are going to use [LLSModels.jl](https://github.com/JuliaSmoothOptimizers/LLSModels.jl).
+For the nonsmooth regularizer, we observe that $h$ is actually readily available in [ProximalOperators.jl](https://github.com/JuliaFirstOrder/ProximalOperators.jl), you can refer to [this section](@ref regularizers) for a list of readily available regularizers.
+We then wrap the smooth function and the regularizer in a `RegularizedNLPModel`.
+
+```@example
+using LLSModels
+using ProximalOperators
+using Random
+using RegularizedProblems
+
+Random.seed!(0)
+
+# Generate A, b
+m, n = 5, 10
+A = randn((m, n))
+b = randn(m)
+
+# Choose a starting point for the optimization process
+x0 = randn(n)
+
+# Get an NLSModel corresponding to the smooth function f
+f_model = LLSModel(A, b, x0 = x0, name = "NLS model of f") 
+
+# Get the regularizer from ProximalOperators
+λ  = 1.0   
+h = NormL0(λ)
+
+# Wrap into a RegularizedNLPModel
+regularized_pb = RegularizedNLPModel(f_model, h)
+```
+
+## Solving the problem
+We can now choose one of the algorithms presented [here](@ref algorithms) to solve the problem we defined above.
+In the case of least-squares, it is usually more appropriate to choose LM or LMTR.
+```@example
+using LLSModels
+using ProximalOperators
+using Random
+using RegularizedProblems
+
+Random.seed!(0)
+
+m, n = 5, 10
+λ  = 0.1
+A = randn((m, n))
+b = randn(m)
+
+x0 = 10*randn(n)
+
+f_model = LLSModel(A, b, x0 = x0, name = "NLS model of f") 
+h = NormL0(λ)
+regularized_pb = RegularizedNLPModel(f_model, h)
+
+using RegularizedOptimization
+
+# LM is a quadratic regularization method, we specify the verbosity and the tolerance of the solver
+out = LM(regularized_pb, verbose = 1, atol = 1e-3)
+println("LM converged after $(out.iter) iterations.")
+println("--------------------------------------------------------------------------------------")
+
+# We can choose LMTR instead which is a trust-region method
+out = LMTR(regularized_pb, verbose = 1, atol = 1e-3)
+println("LMTR converged after $(out.iter) iterations.")
+
+```