Is there any actual need for array input in `SymbolicNeuralNetwork`-created neural networks?

[TorkelE]

When you create a symbolic neural network, e.g.

sym_nn, θ = SymbolicNeuralNetwork(; nn_p_name = :θ, chain = nn_arch, n_input = 1, n_output = 1)

in a model you have to give inputs as:

eqs_ude = [
    D(X) ~ sym_nn([Y], θ)[1] - d*X,
    D(Y) ~ X - d*Y
]

i.e. sym_nn([Y], θ)[1] . For two inputs it would be sym_nn([X,Y], θ)[1] .

While Lux uses array, there does not seem to be any reason for the array form here? I.e. a more natural syntax would have for this to be already considered in the symbolic neural network, i.e. be sym_nn(Y, θ)[1] ? Potentially, even the θ could be baked into the function as well, so just be sym_nn(Y), but there might be more to consider here. For the array, however, it seems like a straight-up syntax improvement?

[ChrisRackauckas]

If y is a symbolic array it's not needed. That is just to array the scalers

[TorkelE]

Good point, should have clarified. Was meaning cases where variables are scalar non-vectors.
(I made the, potentially erroneous assumption, that scalar variables were more common, and expanding the vector ones would be more intuitive)

[ChrisRackauckas]

If it's a vector variable then sym_nn(y, theta) should be fine

[TorkelE]

I was primarily thinking of cases like

using ModelingToolkit, Lux, ModelingToolkitNeuralNets
using ModelingToolkit: t_nounits as t, D_nounits as D
@variables X(t) Y(t)
@parameters d

chain = Lux.Chain(
    Lux.Dense(1 => 3, Lux.softplus, use_bias = false),
    Lux.Dense(3 => 3, Lux.softplus, use_bias = false),
    Lux.Dense(3 => 1, Lux.softplus, use_bias = false)
)

sym_nn, θ = SymbolicNeuralNetwork(; nn_p_name = :θ, chain, n_input = 1, n_output = 1)
eqs_ude = [
    D(X) ~ sym_nn([X], θ)[1] - d*X,
    D(Y) ~ X - d*Y
]

where the variables are non-vectors, thinking that thosem ight be more common?

Admitedly, supporting both these cases directly might not be possible in one function? In which case, making it work nicer for scalar variables might force sym_nn(x..., θ)[1] for the case where X is a vector. My hunch was that scalar variables might be more common for UDEs, so making that the default case would be nicer, but I might very well be wrong.

[ChrisRackauckas]

You're specifically talking about the case where you have one scalar? That's really an edge case, it's not common to have a neural network with 1 input 😅. If you do have that, surely a different universal approximator is more efficient? But yeah it could get specialized, I don't see it as the highest priority but it might be doable with just one dispatch. But the bigger thing is that a neural network is probably not the best approach if that's what you're doing.

[TorkelE]

So, I kind of agree on that neural networks probably are not the go-to one here, however, they still work and the support for working with them (e.g. using this package) just make them by far the most convenient choice (unless there are not some to me unknown packages/functions lying around!)

[TorkelE]

(on that topic, however, it would have been cool to have a general interface where SymbolicNeuralNetwork was SymbolicUniversalApproximator, and then one not only had the current options, but could also designate approximator type, but use the same interface. That would however, likely be a decent amount of work)

[ChrisRackauckas]

It probably just needs a ::Number dispatch that adds the array. It shouldn't be done on the Lux type because that's piracy though. Probably just here:

https://github.com/SciML/ModelingToolkitNeuralNets.jl/blob/main/src/ModelingToolkitNeuralNets.jl#L120-L122

function (wrapper::StatelessApplyWrapper)(input::AbstractArray, nn_p::AbstractArray)
StatelessApplyWrapper([input], nn_p)
end

[ChrisRackauckas]

(on that topic, however, it would have been cool to have a general interface where SymbolicNeuralNetwork was SymbolicUniversalApproximator, and then one not only had the current options, but could also designate approximator type, but use the same interface. That would however, likely be a decent amount of work)

https://github.com/ACEsuit/Polynomials4ML.jl?tab=readme-ov-file looks reasonable.

[TorkelE]

Yeah, that link definitely looks promising