Doubts about rest api and pre-trained model

[SoniaMola]

I am using the haystack rest api integrated in a rasa chatbot. My doubt is the following: is it possible to use the haystack rest api with an empty document store, using only the pre-trained model (like roberta-base-squad2)? Can the chatbot answer the questions that the language model chosen in the pipeline has been trained on (defined in pipelines.haystack-pipeline.yml)? Or is the haystack rest api only able to answer questions relating to the documents contained in the document store?

[sjrl]

For generating answers I would recommend looking at the nodes RAGenerator, Seq2SeqGenerator, or the OpenAIAnswerGenerator in Haystack. You can find more info on our docs pages here for answer generation https://docs.haystack.deepset.ai/docs/answer_generator. By default, these pipelines still expect a retriever b/c the answer generator takes both the retrieved documents and question into account to answer the question.

However, it should be possible to set up the Seq2SeqGenerator and OpenAIAnswerGenerator in such a way that they can work without even if they are given zero retrieved documents. For example, you could try something like

result = generator.predict(
    query='What are the best party games for adults?',
    documents=[],
    top_k=top_k
)

[SoniaMola]

I tried to use the Seq2SeqGenerator generator as you suggested but I get an error "AttributeError: generator needs documents to predict the answer", so it's probably not possible to work without zero retrieved documents. In the haystack documentation the list of documents is not empty:
result = generator.predict (
query = 'What are the best party games for adults?',
documents = [doc1, doc2, doc3 ...],
top_k = top_k
)
So is context (or documents) always needed?

[sjrl]

It looks like the Haystack node is designed to only work with Documents right now. This could be changed so it could also work with an empty list of Documents, but this would require a pull request with a code change to the Seq2SeqGenerator in Haystack. If this is something you would like I'd recommend opening an issue on Haystack requesting the feature :)

[sjrl]

Also, the OpenAI node should already work without any Documents.

[SoniaMola]

I tried with the OpenAI node but it doesn't seem to work without documents.
This is the script for answer generation:

import logging
logging.basicConfig(format="%(levelname)s - %(name)s -  %(message)s", level=logging.WARNING)
logging.getLogger("haystack").setLevel(logging.INFO)

from typing import List
import requests
import pandas as pd
from haystack import Document
from haystack.document_stores import FAISSDocumentStore
from haystack.nodes import DensePassageRetriever, OpenAIAnswerGenerator
from haystack.utils import print_answers, fetch_archive_from_http


def answer_generator():

    document_store = FAISSDocumentStore(faiss_index_factory_str="Flat", return_embedding=True)
    
    # Initialize DPR Retriever to encode documents, encode question and query documents
    retriever = DensePassageRetriever(
        document_store=document_store,
        query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
        passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
        use_gpu=True,
        embed_title=True,
    )
    
    generator = OpenAIAnswerGenerator(api_key="****************")
    
    # Delete existing documents in documents store
    document_store.delete_documents()

    # Now ask your questions
    # We have some sample questions
    QUESTIONS = [
        "who got the first nobel prize in physics",
        "when is the next deadpool movie being released",
        "which mode is used for short wave broadcast service",
        "who is the owner of reading football club",
        "when is the next scandal episode coming out",
        "when is the last time the philadelphia won the superbowl",
        "what is the most current adobe flash player version",
        "how many episodes are there in dragon ball z",
        "what is the first step in the evolution of the eye",
        "where is gall bladder situated in human body",
        "what is the main mineral in lithium batteries",
        "who is the president of usa right now",
        "where do the greasers live in the outsiders",
        "panda is a national animal of which country",
        "what is the name of manchester united stadium",
    ]

    # Now generate answer for question
    for question in QUESTIONS:
        # Retrieve related documents from retriever
        retriever_results = retriever.retrieve(query=question)

        # Now generate answer from question and retrieved documents
        predicted_result = generator.predict(query=question, documents=[], top_k=1)

        # Print you answer
        answers = predicted_result["answers"]
        print(f" -> Generated answer is '{answers[0].answer}' for the question = '{question}'")

    # Or alternatively use the Pipeline class
    from haystack.pipelines import GenerativeQAPipeline

    pipe = GenerativeQAPipeline(generator=generator, retriever=retriever)
    for question in QUESTIONS:
        res = pipe.run(query=question, params={"Generator": {"top_k": 1}, "Retriever": {"top_k": 5}})
        print_answers(res, details="minimum")


if __name__ == "__main__":
    answer_generator()

This is the error output:

INFO - haystack.modeling.model.language_model -  Auto-detected model language: english
WARNING - haystack.nodes.answer_generator.openai -  Skipping all of the provided Documents, as none of them fits the maximum t                           oken limit of 2048. The generated answers will therefore not be conditioned on any context.
Traceback (most recent call last):
  File "/user/smola/project/haystack/tutorials/answer_generator.py", line 75, in <module>
    answer_generator()
  File "/user/smola/project/haystack/tutorials/answer_generator.py", line 59, in answer_generator
    predicted_result = generator.predict(query=question, documents=[], top_k=1)
  File "/user/smola/.local/lib/python3.8/site-packages/haystack/nodes/answer_generator/openai.py", line 153, in predict
    answers = self._create_answers(generated_answers, input_docs)
  File "/user/smola/.local/lib/python3.8/site-packages/haystack/nodes/answer_generator/base.py", line 77, in _create_answers
    "doc_ids": flat_docs_dict["id"],
KeyError: 'id'

[sjrl]

Ahh, you are right. I'm sorry about that. It also looks like this would need to be fixed in the source code to allow for answer generation without documents.