Support for conversational retrieval agents with llama v2 locally hosted model

[jjeng0]

I would like to use the llama v2 chat models locally (in my case llama-2-13b-chat.ggmlv3.q6_K.bin, through llama cpp server) with the conversational retrieval agent framework that was recently released. Are there any ways to get it working (or plans to support the open source models in the near future)?

The conversational retrieval agent seems to only currently work with ChatOpenAI models as it errors out: ""Only supported with ChatOpenAI models." I am very new to the langchain framework, so apologies if I'm missing something obvious. Thanks!

[dosubot[bot]]

Answer generated by a 🤖

Answer

Hello,

Thank you for your interest in the LangChain framework. Based on the information provided, it seems you're trying to use the llama v2 chat models with the conversational retrieval agent framework.

From the context provided, it's clear that LangChain does support the integration of llama chat models as shown by the import statement from langchain_experimental.llms.llamaapi import ChatLlamaAPI. However, it's not explicitly mentioned if it supports the v2 version of the llama chat models.

The error message you're encountering suggests that the conversational retrieval agent framework currently only supports ChatOpenAI models. This might be the reason why you're unable to use the llama v2 chat models.

There was a similar issue titled Can I use vectorstore with LLMChain? which might provide some insights. However, it doesn't directly address your issue.

Unfortunately, without more information, it's hard to provide a definitive answer. It would be helpful if you could provide more details about the error message you're encountering and the steps you've taken so far. This will help us understand the issue better and provide a more accurate solution.

Best regards,
Dosu

Sources

• libs/experimental/langchain_experimental/llms/init.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[manmax31]

I am having a similar issue. I am trying with Qwen-7b-chat. When I try to run the following code, I get: ValueError: Only supported with ChatOpenAI models.

from langchain.docstore.document import Document
from langchain.embeddings import HuggingFaceBgeEmbeddingsa
from langchain.llms import HuggingFacePipeline, LlamaCpp
from langchain.text_splitter import Language, RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma, FAISS
from langchain.document_loaders import TextLoader
from langchain.agents.agent_toolkits import create_retriever_tool
from langchain.agents.agent_toolkits import create_conversational_retrieval_agent
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    GenerationConfig,
    LlamaForCausalLM,
    LlamaTokenizer,
    pipeline,
    BitsAndBytesConfig,
)
from langchain.memory.vectorstore import VectorStoreRetrieverMemory
from transformers.generation import GenerationConfig
import torch


# Documents
loader = TextLoader("state_of_the_union.txt")
documents = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1100, chunk_overlap=200)
texts = text_splitter.split_documents(documents)

# Embeddings
embeddings = HuggingFaceBgeEmbeddings(
    model_name="BAAI/bge-large-en",
    model_kwargs={"device": "cuda:2"},
    encode_kwargs={"normalize_embeddings": True},
)
db = FAISS.from_documents(texts, embeddings)

# Tool
retriever = db.as_retriever()
tool = create_retriever_tool(
    retriever,
    "search_state_of_union",
    "Searches and returns documents regarding the state-of-the-union.",
)
tools = [tool]

# LLM
model_id = "Qwen/Qwen-7B-Chat"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True, 
    bnb_4bit_quant_type="nf4", 
    bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="cuda:2",
    quantization_config=quantization_config,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
).eval()
generation_config = GenerationConfig.from_pretrained(model_id)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_length=2048,
    temperature=0.01,
    top_p=0.95,
    repetition_penalty=1.15,
    generation_config=generation_config,
)
llm = HuggingFacePipeline(pipeline=pipe)

# Agent
agent_executor = create_conversational_retrieval_agent(llm, tools, verbose=True)

[jntv]

did you get a solution for this

[manmax31]

I have not looked at it since then