Datasets and I/O in the GenAI era

[astrojuanlu]

Usually we tell users that datasets do all the I/O so that nodes don't have to. However, passing an object that's essentially a wrapper around a REST API essentially breaks that rule. Lastly, the fact that these APIs aren't versioned and therefore might return completely different things at different times is a threat to Kedro promise of reproducibility.

On the flip side, we already support our Hugging Face datasets, and I see how a HF model that spits text and an OpenAI wrapper that spits text have the same interface, but work in completely different ways, with I/O happening in different moments. I don't have a good solution for this - it somehow bends Kedro conventions and forces us to think what is the responsibility of the dataset and what should be written in the node.

Originally posted by @astrojuanlu in kedro-org/kedro-plugins#434 (review)

[astrojuanlu]

Closing this in favour of the more active #3979.