Multi target prediction

[MichaelMedek]

Is there a way to predict multiple targets at once? Currently X is a DataFrame and y is a Series. Sklearn also supports y as DataFrame with e.g. 2 columns for some models. Especially MLPs could scale well if they are trained with 2 targets simultaneously, since they might depend on each other and it would make things faster if one is interested in more targets at once.

[dholzmueller]

Thanks for the question. I didn't try passing y as a dataframe yet, but passing it as a numpy array of shape (n_samples, n_outputs) should work, I think? I can try to allow dataframes in future versions.

[MichaelMedek]

Thanks for your answer! But does it also work with original non numeric DataFrames? I am using your code with a DataFrame with a mixture of categorical and numeric dtype DataFrames.

[dholzmueller]

Mixed classification and regression is currently not supported, neither is multi-label classification (and I don't intend on adding it, as it would complicate things a bit, unfortunately...). My comment was solely for multi-output regression.

[ogencoglu]

+1 for multi-task learning (regression + classification simultaneously). Most real world problems are not pure classification and pure regression to my experience.

[dholzmueller]

Interesting, what metric would you optimize for in this case? Would one need to balance the classification and regression metrics somehow?

By the way, in principle you can do multi-task learning with RealMLP in pytabkit by one-hot encoding classification labels and performing multi-output regression with MSE loss. You might need to post-process the classification outputs to make sure that they're probabilities, though.

[ogencoglu]

Whatever loss the model is minimizing. Let's say I have a dataset where for a subset of the data I have numerical targets and for the other part I do not have numerical targets but I know they should be at least a value (this is typical for example if your measurement device fails above a value your get some error code). I want to minimize a regression loss for the first subset and a classification loss for the second.

One can do this (I have done before) for example in lightgbm by simply writing the gradients and hessians and optimizing for both simultaneously in a single model. So it is not multi-output but multi-task learning. In my opinion scikit-learn is unusable only for this very reason that it does not allow custom loss function definitions which is almost always needed in real life.

[dholzmueller]

Interesting! Yeah I think there is much potential to popularize NNs on tabular data by having some proper software development team giving them all these features (custom loss functions, sample weights, etc.). Unfortunately I'm not in a position to do all of this at the moment.