Multivariate regression p = argmin |y-f(p; x)|

[grantyuc]

I notice that there might be some inconsistency in namespace MathNet.Numerics.Optimization.

The method ObjectiveFunction.NonlinearModel accepts multi-dimensional model f(p; x), especially **x** a vector datum. However, it only take one-dimensional ObservedX as input; this means I cannot create a model with N vectors of independent data **x**_1, **x**_2, .., **x**_N but only one-dimensional data.

That all the four models in MathNet.Numerics.UnitTests.OptimizationTests.NonLinearCurveFittingTests only accept x in scalar form also approves my guess.

Is there any support for multivariate regression in MathNet.Numerics?

Updated:
After carefully reading, I think that the NonlinearModel f(p; x) is actually accepts scalar x; it accepts a collection of data **x**=(x_i) but calculates all scalar x_i at once.

[diluculo]

@willy1222, yes, 'NonlinearModel' is not designed for multivariate models. The expression f(p; x) [or f(x, p)] should be read as a model function with a single independent variable x and parameters p.

However, in the Levenberg-Marquardt minimizer, x values are considered only in user function. So, as a workaround, you can define your model with multiple independent variables. Hopefully someday 'NonlinearModel' will be extended to multivariate models.

For example:

private Vector<double> area = Vector<double>.Build.DenseOfArray(new double[] { 2600, 3000, 3200, 3600, 4000, 4100 });
private Vector<double> bedrooms = Vector<double>.Build.DenseOfArray(new double[] { 3, 4, 5, 3, 5, 6 });
private Vector<double> age = Vector<double>.Build.DenseOfArray(new double[] { 20, 15, 18, 30, 8, 8 });
private Vector<double> price = Vector<double>.Build.DenseOfArray(new double[] { 550, 565, 610, 595, 760, 810 });
private Vector<double> initialGuess = Vector<double>.Build.DenseOfArray(new double[] { 1.0, 1.0, 1.0, 1.0 });

// define model: price[i] = p[0] + p[1] * area[i] + p[2] * bedrooms[i] + p[3] * age[i]
private Vector<double> MultivariateModel(Vector<double> p, Vector<double> x)
{
    var y = CreateVector.Dense<double>(x.Count);
    for (var i = 0; i < x.Count; i++)
    {
        y[i] = p[0] + p[1] * area[i] + p[2] * bedrooms[i] + p[3] * age[i];
    }
    return y;
}

public NonlinearMinimizationResult SolveMultivariateProblem()
{
    var obj = ObjectiveFunction.NonlinearModel(MultivariateModel, area, price);
    var solver = new LevenbergMarquardtMinimizer(maximumIterations: 10000);
    var result = solver.FindMinimum(obj, initialGuess);
    return result;

    // result.MinimizingPoint
    // [0] 264.78007
    // [1]   0.12844
    // [2]   5.91352
    // [3]  -4.90255

    // result.Minimizedvalues (prices)
    // [0] 518.40104
    // [1] 600.20134
    // [2] 617.09425
    // [3] 597.81071
    // [4] 768.86781
    // [5] 787.62484
}

[diluculo]

Just now, I found #646 already addressed this.