Add underdetermined Levenberg-Marquardt support (minimum-norm mode)

[ChrisRackauckas-Claude]

PR: Add underdetermined Levenberg-Marquardt support (minimum-norm mode)

Summary

This PR implements support for underdetermined nonlinear least-squares problems (n > m, more unknowns than equations) in the Levenberg-Marquardt algorithm, addressing issue #851.

Background

The current L-M implementation in DampedNewtonDescent assumes overdetermined systems (m ≥ n) and forms JᵀJ + λD. For underdetermined systems (n > m), this is inefficient because:

• JᵀJ is n×n (large)
• We want the minimum-norm solution among infinitely many solutions

Algorithm

For underdetermined systems, instead of solving (JᵀJ + λD)δ = -Jᵀf, we solve:

1. (JJᵀ + λI) z = -f(x) — This is an m×m system (smaller when n > m)
2. δ = Jᵀz — This gives the minimum-norm step

This approach:

• Keeps the linear system small (m×m instead of n×n)
• Finds the minimum-norm step that solves the linearized equations
• Is numerically well-conditioned with appropriate λ

Changes

lib/NonlinearSolveBase/src/descent/damped_newton.jl

• New mode: Added :minimum_norm to DampedNewtonDescentCache
• New kwarg: min_norm_mode in DampedNewtonDescent:
  • :auto (default): Auto-detect and use minimum-norm for underdetermined systems
  • :minimum_norm: Force minimum-norm mode
  • :disabled: Never use minimum-norm mode
• Auto-detection: Checks length(fu) < length(u) to detect underdetermined systems
• New caches: JJᵀ_cache, z_cache, J_raw for minimum_norm computations
• Helper functions: dampen_jacobian_minimum_norm!! and _extract_scalar_damping

lib/NonlinearSolveFirstOrder/test/underdetermined_tests.jl

New test file with comprehensive tests:

• Linear underdetermined systems
• Nonlinear underdetermined systems
• In-place formulation
• Overdetermined regression tests (ensure we didn't break anything)
• Large underdetermined systems (50 unknowns, 10 equations)

Usage

using NonlinearSolve

# Underdetermined: 2 equations, 4 unknowns
function f(u, p)
    return [
        u[1] + u[2] - 1.0,
        u[3] + u[4] - 2.0,
    ]
end

u0 = zeros(4)
prob = NonlinearLeastSquaresProblem(f, u0)
sol = solve(prob, LevenbergMarquardt())
# Automatically uses minimum-norm mode

References

• Issue: Levenberg–Marquardt variant for underdetermined least-square problems? #851
• Discussion on Discourse
• Chen et al. (2026), appendix B - Optica Express 34(4):5729
• Reference implementation: LeastSquaresOptim fork

Checklist

• Implementation in damped_newton.jl
• Tests for underdetermined systems
• Documentation updated in docstring
• CI passes (needs testing)