[FEAT] Add example of modifying the default configure_optimizers() behavior (use of ReduceLROnPlateau scheduler)

[JQGoh]

Rationale

• Users have asked about whether we can modify the default learning rate scheduler behaviors. See How to Use Different Optimizers with NeuralForecast Models #852 (comment) and https://nixtlacommunity.slack.com/archives/C031M8RLC66/p1713516602654109
• This provides an example of how to achieve a full control of configure_optimizers() behavior and how we can use ReduceLROnPlateau scheduler during the optimization
• Instead of adding arguments to all the models, passing from NeuralForecast class, we introduce a set_configure_optimizers function for the BaseModel class such that individual model can overwrite the default configure_optimizers() behavior
• Note that this requires users to specify both optimizer and scheduler to have an effective optimization configuration, otherwise it shall fallback to the default option.

This shall deprecate the earlier work introduced in: #998

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[jmoralez]

Hey @JQGoh, thanks a lot for trying this approach! Although I think it's fine to add more arguments for now, we can give this a shot when we work towards v2.0.

[JQGoh]

Hey @JQGoh, thanks a lot for trying this approach! Although I think it's fine to add more arguments for now, we can give this a shot when we work towards v2.0.

In that case, I shall put this on hold then.

[JQGoh]

@jmoralez @elephaint @marcopeix
Please review. After the first round of review, I can improve it with raising a warning when set_configure_optimizers will suggest the fallback option.

[JQGoh]

This breaking change shall fix #1096

[JQGoh]

@jmoralez I have modified it based on your suggestion.

[JQGoh]

@jmoralez

Thanks for your tips on using a subclass approach to customize the default optimizer behavior. Moving forward I can see a few practical ways to support the NeuralForecast users regarding this.

Options

1. We do not deprecate the already introduced parameters optimizer, optimizer_kwargs, lr_scheduler, lr_scheduler_kwargs. However, if users want to use ReduceLROnPlateau which requires customization of the monitor parameter within the custom_optimizer, we can educate NeuralForecast users by demonstrating an example in the documentation. See an example of how to customize the optimizer via subclass https://gist.github.com/JQGoh/1f9fe1613bdc6380b4c68afddec00b91.

2. We deprecate the already introduced parameters optimizer, optimizer_kwargs, lr_scheduler, lr_scheduler_kwargs. However, we introduce a new parameter config_optimizers which can allow the users to specify the desired function customizing the optimization behavior, see this commit
   3bab7c0

3. We deprecate the already introduced parameters optimizer, optimizer_kwargs, lr_scheduler, lr_scheduler_kwargs and we no longer introduce any new parameter. But this requires the users to customize the optimizer behavior via subclass approach https://gist.github.com/JQGoh/1f9fe1613bdc6380b4c68afddec00b91

Options 2 & 3 shall introduce a breaking change and require us to update the NeuralForecast to version 3.0.0. Personally, I prefer the option 1 but I would like to hear more the feedback from the Nixtla's team regarding this.

[jmoralez]

Thanks. Option 1 sounds good.

[JQGoh]

@jmoralez please review. If you agree that this shall resolve #1096, please help to close that

[JQGoh]

@elephaint @jmoralez Mind helping to review this? Providing an example of how to address #1096

[jmoralez]

Thanks!