Kubeflow run fails when downloading models with "too many requests"

[ceberam]

With a high number of documents and low batch size resulting into a potential high number of parallel pods, models are downloaded concurrently and it creates a too many requests error, which results in failures.

As pointed out by @SantanaTiago

Regardless if batch size fix this issue or not, I think some kind of retry mechanism to get models (init Docling converter) should be implemented