scikit-learn stability and failures of extension contract at minor upgrades

[fkiraly]

FYI, breakage of the extension contract at scikit-learn minor version upgrades is something I have observed in multiple packages by now.

I have opened an issue in scikit-learn here, please feel free to comment!
scikit-learn/scikit-learn#31728

FYI @solegalli