2024/02/17/sklearn-pipeline-pt2

[giscus[bot]]

2024/02/17/sklearn-pipeline-pt2

https://it176131.github.io/2024/02/17/sklearn-pipeline-pt2.html

[Mrwent21]

It’s not where you’re at, it’s where you’re going. Good for you for being vulnerable and trying to grow.

[BenKite]

So, do you think this shows a need in scikit-learn? You showed two ways to avoid retraining expensive steps, but those seem difficult for less experienced data scientists. I’d love to have Pipeline cache things for me by default…

[it176131]

Hi Ben, sorry for the delay.

So, do you think this shows a need in scikit-learn?

A need? I'm not sure. One could say the docs for taking advantage of some of these lesser-known—and useful—features are hard to find even for a seasoned user. And while tutorials like this and the resources linked throughout are helpful, maybe a cookbook of scikit-learn recipes would be useful 🤷. Though it would probably be better on the official website so that it could be updated as new versions are released.

You showed two ways to avoid retraining expensive steps, but those seem difficult for less experienced data scientists.

I think almost everything could be difficult in the beginning, even for an experienced data scientist. The data science space is always changing, and exploring new/old/different techniques and processes are part of the journey to becoming a better data scientist. Personally, I like to look at difficult or hard-to-understand things as opportunities to learn and simplify for the next person.

I’d love to have Pipeline cache things for me by default…

There has been much debate around caching amongst the scikit-learn contributors. And I would hesitate to treat caching as a silver bullet. Caching may save you time in training, but each cached item takes up valuable memory space. This may eventually lead to failed training as there's no memory left. This isn't to say that you couldn't patch the Pipeline to always use the memory parameter by default, but there's no such thing as free lunch.