Out-of-Sample Predictions

[louismagowan]

Does anyone know how to get out-of-sample or test-set predictions with a fitted DelayedSaturatedMMM model?

When I try to predict posterior samples using test data I receive dimension errors (since X_test has fewer rows than X_train)

e.g. this error ValueError: The 'channel_data' variable already had 163 coord values defined for its date dimension. With the new values this dimension changes to length 29, so new coord values for the date dimension are required.

It seems like the set_data() method might be missing from the MMM Class?

[williambdean]

Hello @louismagowan , have you come across the methods that are inherited from the ModelBuilder?
predict_posterior might be of interest to you. It will take advantage of the set_data functionality.
They are in the docs as well, for instance: https://www.pymc-marketing.io/en/stable/api/generated/pymc_marketing.mmm.delayed_saturated_mmm.DelayedSaturatedMMM.predict_posterior.html#pymc_marketing.mmm.delayed_saturated_mmm.DelayedSaturatedMMM.predict_posterior

[kmostdiek-k]

Sorry for the delayed response! I'm then getting a NaN error (see screenshot) but I've confirmed that I do not have any NaNs in either the X's or the Y's.

[williambdean]

Interesting. Can you check for nan values in mmm.preprocessed_data["X"] rather than X and y
This seems related to the #402

On top of that, there is an issue to correctly handle new data at the moment. I linked it above

[kmostdiek-k]

Yeah, looks like there are no NaN there either. Both of the following, result in False

mmm.preprocessed_data["X"].isnull().any().any()
np.isnan(np.min(mmm.preprocessed_data["y"]))

And yeah, I'm fine with waiting until either issue 400 or 396 are resolved. I just figured I'd help with the discussion here.

[williambdean]

There is another PR up to support the new data use case #444
Hopefully will be merge sometime soon

[williambdean]

For anyone in the thread, a reworked PR (#482) is in the final stages of review

[juanitorduz]

@wd60622 did a fantastic job! #482 . We will provide and example soon. Stay tuned!

[williambdean]

Example notebook up!
https://www.pymc-marketing.io/en/stable/notebooks/mmm/mmm_lift_test.html