Unexpected Output Format in ask_question Method Using Langchain v0.2

[eeleedev]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from abc import ABC

import logging
import re

from typing import List
from core.reader.simple_reader import SimpleDirectoryReader
from core.reader.schema.document import Document
from core.reader.token_func import group_split
from core.config.settings import settings
from core.manager.load_models import (
    load_quantized_model_gguf_ggml,
    load_quantized_model_qptq,
    load_full_model,
)
from core.manager.prompt.prompt_template import PromptTemplate as PT
from core.manager.prompt_template_utils import get_prompt_template
from core.manager.callback.streaming_callback import StreamingCallbackHandler

from chromadb.config import Settings as ChromaSettings

import click
import torch
import json
import os
import sys
from huggingface_hub import hf_hub_download
from langchain_core.prompts import PromptTemplate, ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain.chains import RetrievalQA, LLMChain, VectorDBQA
from langchain_community.llms import HuggingFacePipeline, LlamaCpp
from langchain_community.vectorstores import Chroma
from langchain_core.documents import Document as LCDocument
from langchain_community.embeddings import HuggingFaceInstructEmbeddings
from langchain_core.callbacks import CallbackManager, StreamingStdOutCallbackHandler  # for streaming response
from langchain_openai import OpenAIEmbeddings
import transformers


class AppManager(ABC):

    def __init__(self, use_history=False,promptTemplate_type="llama"):
        self.device_type = settings.DEVICE_TYPE

        # load the llm pipeline
        llm = self._load_local_llm(model_id=settings.LLM_ID, model_basename=settings.LLM_BASENAME) 

        if settings.DEBUG:
            print(llm)

        # get the prompt template and memory if set by the user.
        self.llm = llm
        self.prompt = PromptTemplate.from_template("{prompt}")
        
        # Use the HuggingFace token from the environment variable
        hf_token = os.getenv("HUGGINGFACE_TOKEN")

        # Initialize the HuggingFaceInstructEmbeddings with the token
        self.embedding = HuggingFaceInstructEmbeddings(
            model_name=settings.EMBEDDING_MODEL_NAME,
            model_kwargs={"device": self.device_type}
        )


        # The Chroma class is used to load a vector store containing pre-computed embeddings.
        self.vector_store_settings = ChromaSettings(
            is_persistent=True,
            persist_directory=settings.VECTOR_STORE_PATH,
            anonymized_telemetry=False)

        # load the vectorstore
        db = Chroma(
            persist_directory=settings.VECTOR_STORE_PATH,
            embedding_function=self.embedding,
            client_settings=self.vector_store_settings,
        )

        retriever = db.as_retriever(search_type="similarity")
        prompt, memory = get_prompt_template(promptTemplate_type=promptTemplate_type, history=use_history)

        if use_history:
            self.qa = RetrievalQA.from_chain_type(
                llm=self.llm,
                chain_type="stuff",  # try other chains types as well. refine, map_reduce, map_rerank
                retriever=retriever,
                return_source_documents=False,  # verbose=True,
                callbacks=callback_manager,
                chain_type_kwargs={
                    "prompt": self.prompt,
                    "memory": memory
                },
            )
        else:
            self.qa = RetrievalQA.from_chain_type(
                llm=self.llm,
                chain_type="stuff",  # try other chains types as well. refine, map_reduce, map_rerank
                retriever=retriever,
                return_source_documents=False,  # verbose=True,
                callbacks=callback_manager,
                chain_type_kwargs={
                    "prompt": prompt,
                    #"verbose": True,
                },
            )

       ....

       def ask_question(self, question, show_sources=False):
        
          # Get the answer from the chain
          res = self.qa(question)
          answer = res["result"]
          print("answer:", answer)
      
          # Print the structure of res
          print("Response structure:", res)

          # Log the structure of res
          logging.info(f"Response structure: {res}")

          return answer

Description

I am using Langchain v0.2 for a project involving a question-answering application. The ask_question method is expected to return only the model's answer to a given question. However, the current output includes additional information such as system instructions (SYS) and context metadata. I want to modify the output to return only the model's response without the extraneous information. Here is an example of the current output:

[INST]<<SYS>> You are a helpful assistant, you will use the provided context to answer user questions. ... Do not use any other information for answering user. Provide a detailed answer to the question. <</SYS>>

Context: | title | text | source |

| --- | --- | --- | | Paget's Disease of Bone | Diagnosis iew Title Prevention of Surgical Site Infections After Major Extremity Trauma Authoring Organization American Academy of Orthopaedic.... 
```User: Tell me about clavicle fracture ...``` 
answer: LLM answer...

I need guidance on how to extract only the model's response (LLM answer...) from this output format.

System Info

System Information

OS: Linux
OS Version: #182-Ubuntu SMP Mon Oct 2 19:43:28 UTC 2023
Python Version: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0]

Package Information

langchain_core: 0.2.22
langchain: 0.2.10
langchain_community: 0.2.9
langsmith: 0.1.93
langchain_cli: 0.0.25
langchain_openai: 0.1.17
langchain_text_splitters: 0.2.2
langserve: 0.2.2

[dosubot[bot]]

Hello, @eeleedev! I'm here to assist you with any bugs, questions, or contributions. Let's tackle this together while we wait for a human maintainer.

To modify the ask_question method in your Langchain v0.2 project to return only the model's response without the extraneous information, you can use an OutputParser to extract and return only the relevant part of the response. Here is how you can do it:

from langchain_core.output_parsers import StrOutputParser

def ask_question(self, question, show_sources=False):
    """Ask a question and return only the model's response."""
    try:
        # Get the answer from the chain
        res = self.qa(question)
        response = res["result"]
        
        # Use StrOutputParser to extract and return only the model's response
        parser = StrOutputParser()
        answer = parser.parse(response)
        
        print("answer:", answer)
        return answer
    except Exception as e:
        logging.error(f"Error in ask_question: {e}")
        raise e

This modification ensures that the ask_question method uses StrOutputParser to return only the model's response content ^[1].

To continue talking to Dosu, mention @dosu.