Overriding BaseChatModel with HuggingFace chat models ?

[RamishSiddiqui]

Has anybody tried to override BaseChatModel class for HuggingFace models like llama-2-7b-chat or ggml-gpt4all-j-v1.3-groovy ?
I am trying to override this class so i don't have to make my data or documents public.

[dosubot[bot]]

🤖

Yes, it is possible to override the BaseChatModel class for HuggingFace models like llama-2-7b-chat or ggml-gpt4all-j-v1.3-groovy. The BaseChatModel class in LangChain is designed to be extended by different models, each potentially having its own unique implementation of the abstract methods present in the BaseChatModel class.

Here are the steps to override the BaseChatModel class for HuggingFace models:

1. Import the necessary modules and classes.

from typing import List, Optional, Any
from libs.langchain.langchain.chat_models.base import BaseChatModel, BaseMessage, CallbackManagerForLLMRun, AsyncCallbackManagerForLLMRun, ChatResult, ChatGeneration
from libs.langchain.langchain.message import AIMessage
from functools import partial
import asyncio
from libs.langchain.langchain.llms.huggingface_hub import HuggingFaceHub

2. Define your new class that inherits from BaseChatModel.

class MyHuggingFaceModel(BaseChatModel):

3. Override the _generate method. This method is responsible for generating the chat response. You need to implement the logic for generating the response using the HuggingFace model.

def _generate(
    self,
    messages: List[BaseMessage],
    stop: Optional[List[str]] = None,
    run_manager: Optional[CallbackManagerForLLMRun] = None,
    **kwargs: Any,
) -> ChatResult:
    # Implement the logic for generating the response using the HuggingFace model
    output_str = self._call(messages, stop=stop, run_manager=run_manager, **kwargs)
    message = AIMessage(content=output_str)
    generation = ChatGeneration(message=message)
    return ChatResult(generations=[generation])

4. Override the _call method. This method is a simpler interface for generating the chat response. You need to implement the logic for generating the response using the HuggingFace model.

def _call(
    self,
    messages: List[BaseMessage],
    stop: Optional[List[str]] = None,
    run_manager: Optional[CallbackManagerForLLMRun] = None,
    **kwargs: Any,
) -> str:
    # Implement the logic for generating the response using the HuggingFace model
    hf_hub = HuggingFaceHub(
        repo_id="<your_repo_id>",
        huggingfacehub_api_token="<your_api_token>"
    )
    prompt = " ".join([message.content for message in messages])
    return hf_hub._call(prompt, stop=stop, **kwargs)

5. Override the _agenerate method. This method is the asynchronous version of the _generate method. You need to implement the logic for generating the response using the HuggingFace model.

async def _agenerate(
    self,
    messages: List[BaseMessage],
    stop: Optional[List[str]] = None,
    run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,
    **kwargs: Any,
) -> ChatResult:
    func = partial(
        self._generate, messages, stop=stop, run_manager=run_manager, **kwargs
    )
    return await asyncio.get_event_loop().run_in_executor(None, func)

Please replace the comments with the actual logic for generating the response using the HuggingFace model. Also, replace <your_repo_id> and <your_api_token> with your actual HuggingFace repository ID and API token.

You can find more details in the LangChain source code:

• libs/langchain/langchain/chat_models/base.py
• libs/langchain/langchain/llms/huggingface_endpoint.py
• libs/langchain/langchain/llms/huggingface_hub.py

I hope this helps! If you have any further questions, feel free to ask.

Sources

• libs/langchain/langchain/chat_models/base.py
• libs/langchain/langchain/llms/huggingface_endpoint.py
• libs/langchain/langchain/llms/huggingface_hub.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.