How do I override PromtTemplate for ConversationalRetrievalChain.from_llm method?

[hodgesz]

I am trying to create a support chatbot with our indexed knowledge base as embeddings in Pinecone. When I don't try to override the prompt template, it is functional, but I want to better control its responses, so I want to override it.

template = """You are a helpful support chatbot having a conversation with a human.
        Follow exactly these 3 steps:
        1. Read the context below
        2. Answer the question using only the context information
        3. Show the source for your answers
        
        Context : {context}
        User Question: {question}

        If you don't know the answer, just say you don't know. Do NOT try to make up an answer. If the question is 
        not related to the context, politely respond that you are tuned to only answer questions that are related to 
        the context. Use as much detail as possible when responding."""

    prompt = PromptTemplate(
        input_variables=["question", "context"], template=template
    )

    memory = ConversationBufferMemory(memory_key="chat_history", input_key="question", return_messages=True)
    qa = ConversationalRetrievalChain.from_llm(
        llm=llm,
        chain_type="stuff",
        retriever=doc_db.as_retriever(),
        memory=memory,
        verbose=True,
        condense_question_prompt=prompt,
        max_tokens_limit=4097
    )

    qa.run(query)

The first time I run the code, it correctly returns a summarized result from indexed knowledge base documents. However, on the 2nd run, it fails because context input key is null. I was assuming the Pinecone retriever was setting the context variable from the most similar documents to the passed in query or question from the user. That appears to happen the first time, but not the 2nd time. Any ideas?

The following is the error and the output from the first successful run.

Please ask a question: When was Userpilot founded?

Traceback (most recent call last):
  File "/Users/hodgesz/PycharmProjects/ai/generative_ai/chatbot_old.py", line 181, in <module>
    main()
  File "/Users/hodgesz/PycharmProjects/ai/generative_ai/chatbot_old.py", line 175, in main
    answer = query_llm(qa, question)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hodgesz/PycharmProjects/ai/generative_ai/chatbot_old.py", line 156, in query_llm
    return qa.run(query)
           ^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/langchain_dev/lib/python3.11/site-packages/langchain/chains/base.py", line 236, in run
    return self(args[0], callbacks=callbacks)[self.output_keys[0]]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/langchain_dev/lib/python3.11/site-packages/langchain/chains/base.py", line 140, in __call__
    raise e
  File "/opt/homebrew/Caskroom/miniforge/base/envs/langchain_dev/lib/python3.11/site-packages/langchain/chains/base.py", line 134, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/langchain_dev/lib/python3.11/site-packages/langchain/chains/conversational_retrieval/base.py", line 101, in _call
    new_question = self.question_generator.run(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/langchain_dev/lib/python3.11/site-packages/langchain/chains/base.py", line 239, in run
    return self(kwargs, callbacks=callbacks)[self.output_keys[0]]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/langchain_dev/lib/python3.11/site-packages/langchain/chains/base.py", line 123, in __call__
    inputs = self.prep_inputs(inputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/langchain_dev/lib/python3.11/site-packages/langchain/chains/base.py", line 216, in prep_inputs
    self._validate_inputs(inputs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/langchain_dev/lib/python3.11/site-packages/langchain/chains/base.py", line 83, in _validate_inputs
    raise ValueError(f"Missing some input keys: {missing_keys}")
ValueError: Missing some input keys: {'context'}

Please ask a question: What is Userpilot used for?

> Entering new StuffDocumentsChain chain...


> Entering new LLMChain chain...
Prompt after formatting:
System: Use the following pieces of context to answer the users question. 
If you don't know the answer, just say that you don't know, don't try to make up an answer.
----------------
Q&A Knowledge Base
1
Q&A Knowledge Base
1. Does Userpilot work on mobile applications?
No, Userpilot only supports web applications on all screen sizes. W e are looking to add support to native mobile platforms in the  
future.
2. Does Userpilot support right to left languages?
Yes, Userpilot supports right to left languages, you can create content in any language!
3. Is Userpilot pricing per application or domain?
Userpilot’ s pricing is based on monthly active users (MAU) which is the number of users that login in a web application (end users).  
Accordingly , It can be installed on as many URLs, domains, web applications, and domains as you want.
4. Can I trigger a flow on a dynamic URL?
Yes, you can replace the dynamic part of a URL with an Asterisk (*).  
This is explained in more detail here
5. How can I add videos or images to my flow?
You can easily embed a video in a flow by clicking on the plus icon in a UI module and selecting 'V ideo’ or “Image”. However , if your  
video is hosted on Loom, then you can directly embed it by including the URL, the steps are mentioned in  this document
6. How does Userpilot count monthly active users(MAU)?
Users who access pages, where the Userpilot script is installed, will be counted as monthly active users – Userpilot tracks the  
numbers of unique users that login to the system in the previous 30 days. Thus, for a user to be counted towards your MAU, they  
will need to be first identified using userpilot.identify or userpilot.anonymous for being unique users that have logged into the product  
in the past 30 days
7. How can I push a flow live?
First, make sure the Userpilots script is installed in your web application. Y ou can verify that Userpilot is receiving data from your  
web app in the installation tab
Then navigate to the flow settings and click on the “Publish” button.
8. How many NPS surveys can I create?
For now , you can create one NPS survey per account. And you can localize the NPS  to appear in dif ferent languages.
9. What happens if the user closed the browser without engaging with the flow?
If the flow is set to trigger “Only once” and the user has seen the flow , the flow will not trigger again even if they haven’t interacted  
with it.

Q&A Knowledge Base
58Please keep in mind that Userpilot stores user information such as the user id, app token, hostname, and so on in the app's local  
storage. This ensures that Userpilot displays the right content to the user during their experience. Therefore, for Userpilot to function  
effectively and deliver the appropriate material to the appropriate user , we rely on browser storage mechanisms, which require the  
usage of Script/Cookie.  
88. Could we see the Github Repository URL?  
Of course, here is the link  
 https://github.com/Userpilot/userpilot-installation
89. What is the plan / how are you prepared to supply chain attacks / DoS attacks?  
On the infrastructure level, we have a mechanism to rate limit the number of requests/events (HTTP & W ebSocket) sourced  
from each IP address. Any unnatural traf fic incoming will be dropped and the IP address gets temporarily blocked from sending  
any requests. In addition, we are using A WS Shield which is a distributed service to safeguard our applications on the network  
level.  
90. Could you send the SOC II report?
https://s3-us-west-2.amazonaws.com/secure.notion-static.com/8dadb591-645a-49e7-a498-b15500cbe470/Userpilot_SOC_2
_Type_2_Report_2021_(1).pdf
91. Can the communication and pop-ups of Userpilot be categorized, and if a certain user chooses to not receive for example.  
marketing pop-ups, but wants to see user support pop-ups, they are able to do so?  
Yes, you can set Userpilot content to appear for a segment of users, one way to achieve this would be to create a UI that asks  
users what the would/wouldn't like to see, and based on their selection you will trigger the right content for them. I have attached  
screenshots below to show how this can be implemented

Q&A Knowledge Base
24
153. I'm looking to better understand the product. For a new SaaS app, I need to understand all the dif ferent components I need  
including analytics, guides/flows, emails, etc. I think you can provide some of this, but I want your suggestion on the full suite I need.
Userpilot is a cloud platform that helps product teams engage the right user with the right feature at the right time, personalize the  
user journey & run in-product growth experiments at scale, understand user sentiment, and drive user adoption & increase product  
engagement.
A brief summary of some of Userpilot components that would help in achieving the above:
-Flows: they consist of dif ferent UI patterns - useful for onboarding users and highlighting in-app features.  
-Checklists: Great to push users deep into your product and increase activation & engagement  
-Native tooltips & Resource Center: Educating your users by providing contextual help through a less-subtle form of in-app  
engagement.  
-NPS: Understanding user sentiment and taking actions upon it.  
All of the components come with powerful targeting and triggering settings as well as basic usage analytics. For more advanced  
analytics such as Creating Analytics funnels for your end-users based on their completion of Userpilot content, we integrate with  
product analytics in your stacks such as Google Analytics and many more.
I hope this was helpful and If you're interested in booking a demo & in-app tour with one of our CSMs, you can use this link:
https://userpilot.com/get-a-demo
154. I have a question regarding Setting Goals for Flows. For the User Identification -> Has Engagement ->What are the 3  
differences between Seen, Engaged, and Completed?  
These terms are related to Flow level analytics and the explanation of each is as follows:
Seen means that the user started the flow/spotlight by seeing the first step.
Engaged means that the user interacted with the flow/spotlight by clicking a button, submitted a form, etc.
Completed means that the user finished each step interacting with the flow/spotlight all the way till the end
156. Can Userpilot be installed electron apps?

Q&A Knowledge Base
757. My MAU count is much higher than expected, what’ s going on?
Userpilot counts users on any page that it is installed. For this reason, we recommend that you don’t install Userpilot on public-
facing pages (e.g. your marketing site or login page). If your MAU count is significantly higher than you think it should be, make sure  
Userpilot isn’t installed on a public-facing page.
58. Do I need to pay for users that don't see Userpilot content?
Yes, Userpilot charges based on MAUs, not flows shown. So if a new user logs in they will be counted and considered against the  
MAU total on your plan, regardless if they have seen a flow or not. T argeting is one of our most powerful features, and allows you to  
target/exclude, current or historical users by relevant properties/events. MAU pricing empowers you to create the best content and  
ensure that content is shown to the specific people who you'd like to see it.  
59. Nothing shows up until you do a refresh?
It seems that you don’t have the reload method installed on that page. Since Userpilot depends on page reloads (refresh) to update  
and trigger new experiences, your developer team must call userpilot.reload() every time the URL is changed. we require to have  
userpilot.reload() installed on each URL change. This method will tell the script to update the queue of the content to be shown on  
that page.
60. I need a product tour/user guiding tool that is working via Internet Explorer , next to Google chrome or firefox. Is Userpilot  
suitable?
Yes, Userpilot is browser-friendly . The building will be done using our chrome extension. But Userpilot works on all browsers
61. How can I modify the x padding for a modal?
You can do that by using custom CSS.  
.userpilot-node-modal .userpilot-slide .up-x-to-close {
top: 20px;
right: 20px;
}
62. What are the key dif ferences vs intercom walk-throughs?
I completely agree that using one tool is easier to manage and Intercom is indeed a great tool to create a simple walkthrough,  
however , Userpilot does have many advantages and I promise I'm not being biased here, I'll list the dif ference between Intercom  
and Userpilot below just in case you're curious to know Userpilot has a more robust Chrome extension for building Flows, such  
as:Advanced customizability to make your content looks native to your appBranched flowsConnecting flows to other Userpilot  
engagement contentOut of the box LocalizationPowerful themes to define the look & feel once and use it across the app Features  
set
1. Spotlights, Checklists, Resource Center , Feature tags, Goals -- Are all features that are not included in Intercom
2. Userpilot provides more actionable analytics on Product T ours and many more targeting options such as Prioritization,  
Experimentation, etc.
We launched a new Events page that will more power to understanding how events are being interacted with inside your app -  
Intercom doesn't have thatGenerally speaking, Intercom is mainly a messaging tool and they only provide Product T ours as a  
secondary feature, whereas Userpilot specializes in providing all the tools needed to increase product adoption and growth.
63. Do you support custom URLs?
Human: What is Userpilot used for?

> Finished chain.

> Finished chain.
Userpilot is a cloud platform that helps product teams engage the right user with the right feature at the right time, personalize the user journey & run in-product growth experiments at scale, understand user sentiment, and drive user adoption & increase product engagement. It provides various components such as Flows, Checklists, Native tooltips & Resource Center, NPS, and more, all with powerful targeting and triggering settings as well as basic usage analytics. Userpilot is mainly used for user onboarding, feature adoption, and product growth.

[hodgesz]

After further digging into this, it looks like even the successful run isn't using my custom prompt template:

> Entering new LLMChain chain...
Prompt after formatting:
System: Use the following pieces of context to answer the users question. 
If you don't know the answer, just say that you don't know, don't try to make up an answer.

This also doesn't look like the default template, so does anyone know where this is coming from?

[davnords]

Thanks for this question, I would also like to know how to use this! Waiting for answer, please tell me if you solved it

[hodgesz]

I will definitely let you know if I solve it. Please do the same. I am still at a loss for where this mystery prompt is coming from.

[hodgesz]

Okay, finally got some time to revisit this and got a little further. As the error states, context is missing as an input variable. When I change the variable to 'chat_history' it works using my custom prompt.

prompt = PromptTemplate(
        input_variables=["question", "chat_history"], template=template
    )

However, that isn't what I want. The whole point is to add the closest indexed docs to the query to the context via the vector store retriever. So this may come down to just knowing what this input variable is called or better understanding how the ConversationalRetrievalChain constructs the prompt with the relevant chunks from Pinecone.

Anyone with more experience with ConversationalRetrievalChain have any ideas?

[ShantanuNair]

@hodgesz Let me save you some headache :)

 qa = ConversationalRetrievalChain.from_llm(
        llm=llm,
        chain_type="stuff",
        retriever=doc_db.as_retriever(),
        memory=memory,
        verbose=True,
        condense_question_prompt=prompt,
        max_tokens_limit=4097
    )

Here you are setting condense_question_prompt which is used to generate a standalone question using previous conversation history.

What you want to do is:

qa = ConversationalRetrievalChain.from_llm(
        llm=llm,
        chain_type="stuff",
        retriever=doc_db.as_retriever(),
        memory=memory,
        verbose=True,
#        condense_question_prompt=prompt,
        max_tokens_limit=4097,
        combine_docs_chain_kwargs={"prompt": prompt}
    )

[hodgesz]

@ShantanuNair I really appreciate your help! I was not understanding what was happening, but your response makes perfect sense now that I see it.