How to use Hugging Face inference endpoint with Langchain?

[Abe410]

Hi

I have used the HuggingFacePipeline with different models such as flan-t5 and stablelm-7b etc., and it works with local inference.

I tried using the HuggingFaceHub as well, but it constantly gives a time out error for basically every model.

Now I have created an inference endpoint on HF, but how do I use that with langchain? The HuggingFaceHub class only accepts a text parameter which is the repo_id or model name, but the inference endpoint gives me a URL only.

I can get individual text samples by a simple API request, but how do I integrate this with langchain?

[Galeos93]

It seems this functionality was added here. Here is the class that allows the integration. I have not tried it yet, but you can give it a try. Unfortunately, this integration is not in the official documentation.

[Abe410]

Yupp, I found the integration after browsing through the actual code.

Would help if it is included in the official documentation as well!

[unnati-rane]

can you please provide the code that you find useful. i am not able to find the actual code

[levalencia]

Looks like its very new functionality! some docs are needed :)

[levalencia]

Hi

I have used the HuggingFacePipeline with different models such as flan-t5 and stablelm-7b etc., and it works with local inference.

I tried using the HuggingFaceHub as well, but it constantly gives a time out error for basically every model.

Now I have created an inference endpoint on HF, but how do I use that with langchain? The HuggingFaceHub class only accepts a text parameter which is the repo_id or model name, but the inference endpoint gives me a URL only.

I can get individual text samples by a simple API request, but how do I integrate this with langchain?

How to get the URL: https://huggingface.co/docs/inference-endpoints/guides/test_endpoint

You have to deploy the model you want as an inference point, basically you have to pay in HF for inference compute time

[syeminpark]

the provided class
https://github.com/hwchase17/langchain/blob/370becdfc2dea35eab6b56244872001116d24f0b/langchain/llms/huggingface_endpoint.py#L15
has a bug.
it should be

if self.task == "text-generation":
# Text generation return includes the starter text.
text = generated_text[0]["generated_text"]

not
text = generated_text[0]["generated_text"][len(prompt) :]

the current class will likely just return a 0.

[uncensorie]

@syeminpark did you test it? It seems to work fine