How to implement bind_tools to custom LLM from huggingface pipeline(Llama-3) for a custom agent

[talhaty]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

name = "meta-llama/Meta-Llama-3-8B-Instruct"

auth_token = ""

tokenizer = AutoTokenizer.from_pretrained(name,use_auth_token=auth_token)

bnb_config = BitsAndBytesConfig(
    load_in_8bit=True,
)

model_config = AutoConfig.from_pretrained(
    name,
    use_auth_token=auth_token,
    tempreature=0.1,
)

model = AutoModelForCausalLM.from_pretrained(
    name,
    trust_remote_code=True,
    config=model_config,
    quantization_config=bnb_config,
    device_map='auto',
    use_auth_token=auth_token,
)

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=4096, device_map="auto", streamer = streamer)
llm = HuggingFacePipeline(pipeline=pipe)

@tool
def some_custom_tool(input_string: str) -> str:
    """Executes some work and returns a success message if successfull else it return the error message"""
    return "SUCCESS"


tools = [some_custom_tool]
prompt = ChatPromptTemplate.from_messages(
  [
      (
          "system",
          f"""
              You are an Assistant......
          """,
      ),
      ("user", "{input}"),
      MessagesPlaceholder(variable_name="agent_scratchpad"),
  ]
)
            
llm_with_tools = llm.bind_tools(tools)

           

agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_to_openai_tool_messages(
            x["intermediate_steps"]
        ),
    }
    | prompt
    | llm
    | JsonOutputParser()
)
            

agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, return_intermediate_steps= True)

Description

I am trying to bind a custom tool with the LLM just like ChatOpenAI but i am getting the following error.
bind_tools does exist in HuggingFacePipeline

AttributeError: 'HuggingFacePipeline' object has no attribute 'bind_tools'

System Info

langchain==0.2.6
langchain-community==0.2.6
langchain-core==0.2.11
langchain-openai==0.1.14
langchain-text-splitters==0.2.2
Python 3.10.13
I am doing this on Kaggle GPU t4x2

[dosubot[bot]]

Hey there, @talhaty! I'm here to help you out with your question. Let's get that bug sorted!

To bind custom tools to the HuggingFacePipeline LLM, you can use the bind_tools method provided in the huggingface_pipeline.py file. Here is an example of how to do it:

from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline
from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM, pipeline
from transformers import BitsAndBytesConfig, TextStreamer
from langchain_core.tools import tool
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.agents import AgentExecutor, JsonOutputParser

# Define your custom tool
@tool
def some_custom_tool(input_string: str) -> str:
    """Executes some work and returns a success message if successful else it returns the error message"""
    return "SUCCESS"

# Load your model and tokenizer
name = "meta-llama/Meta-Llama-3-8B-Instruct"
auth_token = ""

tokenizer = AutoTokenizer.from_pretrained(name, use_auth_token=auth_token)

bnb_config = BitsAndBytesConfig(load_in_8bit=True)

model_config = AutoConfig.from_pretrained(name, use_auth_token=auth_token, temperature=0.1)

model = AutoModelForCausalLM.from_pretrained(
    name,
    trust_remote_code=True,
    config=model_config,
    quantization_config=bnb_config,
    device_map='auto',
    use_auth_token=auth_token,
)

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=4096, device_map="auto", streamer=streamer)
llm = HuggingFacePipeline(pipeline=pipe)

# Bind the custom tool to the LLM
tools = [some_custom_tool]
llm_with_tools = llm.bind_tools(tools)

# Define the prompt
prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            """
            You are an Assistant......
            """,
        ),
        ("user", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

# Define the agent
agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_to_openai_tool_messages(
            x["intermediate_steps"]
        ),
    }
    | prompt
    | llm_with_tools
    | JsonOutputParser()
)

# Create the agent executor
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, return_intermediate_steps=True)

# Example usage
response = agent_executor.invoke({"input": "What is the weather today?", "chat_history": []})
print(response)

This code snippet demonstrates how to define a custom tool (some_custom_tool), bind it to the HuggingFacePipeline LLM using the bind_tools method, and then invoke the model with a query that utilizes this tool. The HuggingFacePipeline class supports various tasks such as text-generation, text2text-generation, summarization, and translation, making it versatile for different use cases ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[cordycepssystems]

I think the problem here is that bind_tools is not actually defined on HuggingFacePipeline ? @dosu

[mli-tian]

I also encountered this problem, have you solved it?

[MinKhantYCC]

I think we can use ChatHuggingFace(llm=hf_pipe, **kwargs).
It has a .bind_tools function. But I am sure it has the same functionality as OpenAPI.

[boerste]

From my understanding, the ChatHuggingFace with HuggingFacePipeline as input does not support tool calling even though invoking .bind_tools on ChatHuggingFace causes no immediate issues. See: Issue 22379 and Issue 24430.