Wise to call outside API on custom node?

[rezatakhshid]

Hi,
We want to host our models on Hugging Face. Anyone can think of a reason that it's a bad idea to call the HF inference api inside a custom node and put that in the pipeline as reader?

I did see this Outsourcing Reader but it didn't answer my question.

I would appreciate some insight

[julian-risch]

Hi @rezatakhshid that's a great question! Your idea should in general work. You could implement a custom node that communicates with an API, yes. Our documentation page about custom nodes will be helpful for that.
The downsides that I can think of are first of all latency and also some limitations on the responses you get back from the API compared to the answers you get from FARMReader. For example, with a FARMReader you can have calibrated confidence scores and you can use the evaluation functionalities of Haystack. That won't be possible otherwise. And while a custom node that communicates with an external API gives you the advantage that you would not need to host a reader model yourself, you would still need to host all other parts of the pipeline. In case you are considering to host and manage not only nodes but entire pipelines instead, you could maybe have a look at deepset Cloud as another option.

[rezatakhshid]

@julian-risch Thanks. More than what I was hoping to learn. Awesome