docs/integrations/chat/huggingface/

[giscus[bot]]

docs/integrations/chat/huggingface/

This will help you getting started with langchainhuggingface chat models. For detailed documentation of all ChatHuggingFace features and configurations head to the API reference. For a list of models supported by Hugging Face check out this page.

https://python.langchain.com/docs/integrations/chat/huggingface/

[sorooshsorkhani]

I have an issue with the output of the chat_model. I see a lot of tokens like <|eot_id|>``<|start_header_id|>``user<|end_header_id|>.
I use the code provided on this page. I had to modify it a little bit to avoid some errors.

Below is my code:

import transformers, torch
from langchain_huggingface import ChatHuggingFace, HuggingFacePipeline

model_id = "meta-llama/Llama-3.2-1B-Instruct"
pipe = transformers.pipeline(
    "text-generation",
    model=model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    max_new_tokens=2048,
    do_sample=False,
    repetition_penalty=1.03
)
pipe.tokenizer.pad_token_id = pipe.model.config.eos_token_id

model_kwargs = {'temperature':0.3}
llm = HuggingFacePipeline(pipeline = pipe, model_kwargs = model_kwargs)

chat_model = ChatHuggingFace(llm=llm)

from langchain_core.messages import (
    HumanMessage,
    SystemMessage,
)

messages = [
    SystemMessage(content="You're a helpful assistant"),
    HumanMessage(
        content="What happens when an unstoppable force meets an immovable object?"
    ),
]

ai_msg = chat_model.invoke(messages)
print(ai_msg.content)

And this is the output:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 14 Nov 2024

You're a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

What happens when an unstoppable force meets an immovable object?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

In the context of physics, when an unstoppable force meets an immovable object, the outcome is often referred to as a "static frictional lock" or a "static frictional resistance." This occurs when two objects are in contact and cannot move relative to each other due to their immense weight or inertia.

In this scenario, the force of static friction (also known as "inertial force") acts to resist the motion of one object against the other. The force of static friction is proportional to the normal force (the force exerted by the surface on the objects) and the coefficient of static friction between them.

When an unstoppable force meets an immovable object, the following events can occur:

1. **The force of static friction increases**: As the unstoppable force tries to overcome the static friction, it generates a greater force to resist the motion. This increased force causes the object to slow down or even come to a complete stop.
2. **The object may not move**: If the force of static friction is strong enough, it can overcome the unstoppable force and prevent the object from moving. In this case, the object will remain stationary.
3. **The unstoppable force may cause the object to deform**: If the force of static friction is significant, it can cause the object to deform or change shape, such as bending or crushing, due to the immense pressure exerted by the unstoppable force.

Examples of unstoppable forces include:

* Gravity (the force of attraction between masses)
* Friction (the force that opposes motion between two surfaces in contact)
* Electromagnetic forces (such as repulsion or attraction between charged particles)

Examples of immovable objects include:

* A heavy, solid object on a frictionless surface
* A massive, rigid structure (like a mountain or a skyscraper)
* A planet or moon orbiting a star

Keep in mind that the concept of unstoppable force and immovable object is a bit of a simplification, as in reality, there are many exceptions and nuances to consider. However, this basic principle provides a useful starting point for understanding the behavior of these fundamental forces and objects.

Is this what I'm supposed to have?
I know I can have a better output with post processing the message. But, is there anything I should've done differently to prevent this?

[shan-max]

buy a car

[shan-max]

build a home.

[A96M]

add in the pipeline_kwargs { "return_full_text": False}. By the way, I would take into consideration buying a car

[vishnun0027]

Here is how can instantiate a ChatHuggingFace model using both HuggingFaceEndpoint and HuggingFacePipeline. Since HuggingFacePipeline does support function calling/tool calling ?

[B-Ismail]

i'm wondering if you fond a good model from huggingface that executes tools with HuggingFacePipeline or no ?

[A96M]

https://huggingface.co/vicgalle/Configurable-Llama-3.1-8B-Instruct try this.

[B-Ismail]

does HuggingFacePipeline still not support tool calling ? only the inference one

[hakan-erdem]

I also worked on this for a while, but didn't carry on, because it didn't work with Llama. However, I was doing the exact same thing with the tutorials on the web. To my understanding, in order to call tools you need more powerful LLMs like GPT, open source just can not do it.

I might be wrong though.

[B-Ismail]

@alxtrtw However if you have time and ressources you can try with https://python.langchain.com/docs/integrations/chat/vllm/ and choose a model capable of calling tools like Salesforce/Llama-xLAM-2-8b-fc-r

[B-Ismail]

@hakan-erdem plenty of open source models capable here : https://gorilla.cs.berkeley.edu/leaderboard.html

[Arya920]

hey @B-Ismail I tried to work this with Gemma3-1B, faced the same issue, now I am trying to write this function calling for "ChatHuggingFace", anyone wants to help ?

[B-Ismail]

@Arya920 well Gemma3 dones't have tool calling capabilities search other models

[TomH-Cott]

Why are there two "Model features" sections that show different values for the same class/package? Am I missing something?

[hakan-erdem]

Apparently two days ago, langchain_huggingface got a big update, from 0.1.2 release way up to 0.2.0 . Well yesterday my code suddenly broke and i was wondering why. If anyone have come up with the following issue:

OSError: None is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'

try downgrading to the 0.1.2 version, at least that worked for me.

[alxtrtw]

forgot to paste the import, of course it's this:

from transformers import AutoTokenizer

[hakan-erdem]

That's interesting, the following was my implementation and it does not look that different, no? However, I struggled to run this code in 0.2.0

reasoning_model_id = "meta-llama/Llama-3.1-8B-Instruct"

reasoning_tokenizer = AutoTokenizer.from_pretrained(reasoning_model_id)
reasoning_tokenizer.pad_token = reasoning_tokenizer.eos_token  # Set pad_token explicitly

reasoning_model = AutoModelForCausalLM.from_pretrained(
    reasoning_model_id,
    trust_remote_code=True,
    torch_dtype=torch.float16,  # Use float16 to save VRAM
    device_map={"", 0},
    token=HUGGINGFACEHUB_API_TOKEN
)

reasoning_pipeline = pipeline(
    "text-generation",
    model=reasoning_model,
    tokenizer=reasoning_tokenizer,
    max_new_tokens=1024,
    do_sample=True,
    temperature=0.3,
    repetition_penalty=1.03,
)

reasoning_llm = ChatHuggingFace(llm=HuggingFacePipeline(pipeline=reasoning_pipeline))

[alxtrtw]

There is a difference - it seems like it needs to be passed not only in the pipeline, bu also in ChatHuggingFace() at the end again. Only after doing that it worked for me

chat = ChatHuggingFace(llm=pipe, tokenizer=tokenizer)

[hakan-erdem]

Oh yes, now I see it! Thanks @alxtrtw I didn't know ChatHuggingFace had a tokenizer parameter.

[Arya920]

I am trying to use this something like this ~

from transformers import AutoModelForCausalLM, AutoTokenizer
from langchain_huggingface import HuggingFacePipeline, ChatHuggingFace
import torch
from transformers import pipeline

def load_LLM(model_id):

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to('cpu')
model.eval()

pipe = HuggingFacePipeline(pipeline=pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=32))
llm = ChatHuggingFace(llm= pipe, tokenizer= tokenizer)
return llm

llm = load_LLM("model/gemma3-1b-it")
"""
Example
"""
print(llm.invoke('what is GEN AI'))

getting answer like
Device set to use cpu
content="<start_of_turn>user\nwhat is GEN AI<end_of_turn>\n<start_of_turn>model\nOkay, let's break down what Generative AI is. It's a really hot and rapidly evolving field, so here's a comprehensive explanation:" additional_kwargs={} response_metadata={} id='run--02bca282-ad00-45c2-93e6-1be1bf0b537a-0'