How to use Generative QA with other language (like french).

[ghost]

I have found in your tutorial how to use Generative QA but can I just take a model that work with french (BERT variant maybe ?) here ?

generator = RAGenerator( model_name_or_path="facebook/rag-token-nq")

Do you have any advice on wich one ?

[julian-risch]

Unfortunately, to the best of my knowledge, there is no french RAG model available. As an alternative, you could use an English model together with our TranslationWrapper node. It's described in the end of the following tutorial: https://haystack.deepset.ai/tutorials/question-generation
@vblagoje are you aware of any LFQA models working with french texts? Probably not?

[ghost]

@tstadel You would probably laugh, but I used the wrong model to translate as my docs are french also I do not have a good hardware env so I shut down the prog when the computer started to show some lag I will retry late when the temperature will be less.

btw im using my cpu but i dont know how to specify to haystack how to use the gpu

also it may be a good thing to add some progess bar for the translation like how you do it when you update the embeddings

[tstadel]

For GPU support check out https://haystack.deepset.ai/guides/optimization#gpu-acceleration. Haystack will make use of it automatically if you're environment is correctly set up. You'll need a nvidia GPU, installed cuda toolkit and a working pytorch environment. For pytorch and CUDA setup checkout https://pytorch.org/get-started/locally/.

Progressbars are being addressed in #2580.

[sjrl]

Hi, @azetoy I was able to successfully run the tutorial https://haystack.deepset.ai/tutorials/retrieval-augmented-generation in Google colab. The link for trying it in colab is here: https://colab.research.google.com/github/deepset-ai/haystack/blob/master/tutorials/Tutorial7_RAG_Generator.ipynb

To help us debug your error could you provide us with the code of the modified version of the tutorial that you are using?

[sjrl]

Hi @azetoy could you clarify what you mean that you tested it with generator = Seq2SeqGenerator(model_name_or_path="deepset/roberta-base-squad2") and it worked perfectly? Do you mean that using Seq2SeqGenerator instead of RAGenerator in your above script ran successfully?

[ghost]

@sjrl exactly with this code it work perfectly

generator = Seq2SeqGenerator(model_name_or_path="vblagoje/bart_lfqa")

pipe = GenerativeQAPipeline(generator, retriever)

res = pipe.run(
    query="Thanks for your reply.", params={"Retriever": {"top_k": 5}}
)

print_answers(res, details="all")

[ghost]

@sjrl after editing my script to be that (it seem with RAG i cant load the data of the DPR i need to init a new one)

document_store.load("data/Fr_data.faiss", config_path="data/Fr_data_config")

retriever = DensePassageRetriever(
    document_store=document_store,
    query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
    passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base")

doc_dir = "data/Fr"

docs = convert_files_to_docs(dir_path=doc_dir, split_paragraphs=True)

generator = RAGenerator(
    model_name_or_path="facebook/rag-token-nq",
    top_k=5,
    max_length=200,
    min_length=2,
    embed_title=False,
    num_beams=2,
)

pipe = GenerativeQAPipeline(generator, retriever)

question = translator_model.translate(query="What are the most beautiful place ?")

res = pipe.run(query=question, params={"Generator": {"top_k": 1}, "Retriever": {"top_k": 1}})
print_answers(res, details="minimum")

and the warning now is from an assert error

C:\Users\zined\PycharmProjects\pythonProject\venv\lib\site-packages\transformers\models\bart\configuration_bart.py:179: UserWarning: Please make sure the config includes `forced_bos_token_id=0` in future versions. The config can simply be saved and uploaded again to be fixed.
  warnings.warn(
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'RagTokenizer'. 
The class this function is called from is 'DPRQuestionEncoderTokenizer'.
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'RagTokenizer'. 
The class this function is called from is 'DPRQuestionEncoderTokenizerFast'.
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'RagTokenizer'. 
The class this function is called from is 'BartTokenizer'.
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'RagTokenizer'. 
The class this function is called from is 'BartTokenizerFast'.
Some weights of the model checkpoint at facebook/rag-token-nq were not used when initializing RagTokenForGeneration: ['rag.question_encoder.question_encoder.bert_model.pooler.dense.weight', 'rag.question_encoder.question_encoder.bert_model.pooler.dense.bias']
- This IS expected if you are initializing RagTokenForGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RagTokenForGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RagTokenForGeneration were not initialized from the model checkpoint at facebook/rag-token-nq and are newly initialized: ['rag.generator.lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
INFO - haystack.modeling.utils -  Using devices: CPU
INFO - haystack.modeling.utils -  Number of GPUs: 0
INFO - haystack.modeling.utils -  Using devices: CPU
INFO - haystack.modeling.utils -  Number of GPUs: 0
Traceback (most recent call last):
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\pipelines\base.py", line 509, in run
    node_output, stream_id = self.graph.nodes[node_id]["component"]._dispatch_run(**node_input)
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\base.py", line 172, in _dispatch_run
    return self._dispatch_run_general(self.run, **kwargs)
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\base.py", line 216, in _dispatch_run_general
    output, stream = run_method(**run_inputs, **run_params)
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\retriever\base.py", line 278, in run
    output, stream = run_query_timed(
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\retriever\base.py", line 109, in wrapper
    ret = fn(*args, **kwargs)
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\retriever\base.py", line 327, in run_query
    documents = self.retrieve(
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\retriever\dense.py", line 312, in retrieve
    documents = self.document_store.query_by_embedding(
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\document_stores\faiss.py", line 607, in query_by_embedding
    score_matrix, vector_id_matrix = self.faiss_indexes[index].search(query_emb, top_k)
  File "C:\Users\zined\PycharmProjects\pythonProject\venv\lib\site-packages\faiss\__init__.py", line 341, in replacement_search
    assert d == self.d
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\zined\PycharmProjects\pythonProject\run.py", line 44, in <module>
    res = pipe.run(query="What are the most beautiful place ?", params={"Generator": {"top_k": 1}, "Retriever": {"top_k": 1}})
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\pipelines\standard_pipelines.py", line 377, in run
    output = self.pipeline.run(query=query, params=params, debug=debug)
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\pipelines\base.py", line 512, in run
    raise Exception(
Exception: Exception while running node `Retriever` with input `{'root_node': 'Query', 'params': {'Generator': {'top_k': 1}, 'Retriever': {'top_k': 1}}, 'query': 'What are the most beautiful place ?', 'node_id': 'Retriever'}`: , full stack trace: Traceback (most recent call last):
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\pipelines\base.py", line 509, in run
    node_output, stream_id = self.graph.nodes[node_id]["component"]._dispatch_run(**node_input)
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\base.py", line 172, in _dispatch_run
    return self._dispatch_run_general(self.run, **kwargs)
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\base.py", line 216, in _dispatch_run_general
    output, stream = run_method(**run_inputs, **run_params)
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\retriever\base.py", line 278, in run
    output, stream = run_query_timed(
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\retriever\base.py", line 109, in wrapper
    ret = fn(*args, **kwargs)
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\retriever\base.py", line 327, in run_query
    documents = self.retrieve(
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\nodes\retriever\dense.py", line 312, in retrieve
    documents = self.document_store.query_by_embedding(
  File "C:\Users\zined\PycharmProjects\pythonProject\haystack\haystack\document_stores\faiss.py", line 607, in query_by_embedding
    score_matrix, vector_id_matrix = self.faiss_indexes[index].search(query_emb, top_k)
  File "C:\Users\zined\PycharmProjects\pythonProject\venv\lib\site-packages\faiss\__init__.py", line 341, in replacement_search
    assert d == self.d
AssertionError

Some warning of unexpected tokenizer and bart configuration but still cant find out why this is not working :(

[sjrl]

Hi @azetoy this definitely sounds frustrating. Unfortunately, I can't reproduce your error at the moment because I can't see your entire script. Could you provide me with the following:

• The entire script including the translator nodes and the original document writes to the document store
• Example files of the original French documents that are being translated

Thanks in advance!

[ghost]

Hi @sjrl sure here is the full code to train (generate the SQL DB and translate the doc from french to eng)

from haystack.document_stores import FAISSDocumentStore
from haystack.nodes import TransformersTranslator
from haystack.nodes import DensePassageRetriever
from haystack.utils import convert_files_to_docs


def train_save():
    document_store = FAISSDocumentStore(sql_url="sqlite:///Fr_data.db", faiss_index_factory_str="Flat")

    # Let's first get some files that we want to use
    doc_dir = "data/Fr"

    # Convert files to dicts
    docs = convert_files_to_docs(dir_path=doc_dir, split_paragraphs=True)

    # Now, let's write the dicts containing documents to our DB.

    retriever = DensePassageRetriever(
        document_store=document_store,
        query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
        passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base"
    )
    document_store.delete_documents()
    document_store.write_documents(translate(docs))
    document_store.update_embeddings(retriever=retriever)

    document_store.save("data/Fr_data.faiss", config_path="data/Fr_data_config")
    retriever.save("data/Fr_data_retriever.pt")


# translate function
def translate(doc):
    translator = TransformersTranslator(model_name_or_path="Helsinki-NLP/opus-mt-fr-en")
    return translator.translate_batch(documents=doc)


def main():
    train_save()


if __name__ == '__main__':
    main()

and here the full code to ask the question

from haystack.document_stores import FAISSDocumentStore
from haystack.nodes import DensePassageRetriever
from haystack.nodes import RAGenerator
from haystack.nodes import TransformersTranslator
from haystack.pipelines import GenerativeQAPipeline
from haystack.utils import print_answers, convert_files_to_docs

language_data = "Fr"

document_store = FAISSDocumentStore.load(index_path="data/Fr_data.faiss",
                                         config_path="data/Fr_data_config")

document_store.load("data/Fr_data.faiss",
                    config_path="data/Fr_data_config")

retriever = DensePassageRetriever(
    document_store=document_store,
    query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
    passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base")

doc_dir = "data/Fr"

docs = convert_files_to_docs(dir_path=doc_dir, split_paragraphs=True)

generator = RAGenerator(
    model_name_or_path="facebook/rag-token-nq",
    top_k=5,
    max_length=200,
    min_length=2,
    num_beams=2
)

pipe = GenerativeQAPipeline(generator, retriever)

input_t = TransformersTranslator(model_name_or_path="Helsinki-NLP/opus-mt-fr-en")

question = input_t.translate(query="Quel est le plus beau lieux ?")

res = pipe.run(query=question, params={"Generator": {"top_k": 1}, "Retriever": {"top_k": 1}})
print_answers(res, details="minimum")

as for the doc here a sample of the french doc i use https://we.tl/t-HcknFdE5lW hope it help and thank you for your help

[sjrl]

Hi @azetoy I think I've identified your issue. When you first declare your FAISSDocumentStore you need to make sure to pass the return_embedding=True to the initializer so

document_store = FAISSDocumentStore(
  sql_url="sqlite:///Fr_data.db", faiss_index_factory_str="Flat", return_embedding=True
)

because your error was originally something like this

AttributeError: _prepare_passage_embeddings need a DPR instance as self.retriever to embed document

which was thrown by the RAGenerator complaining that it could not find the embeddings for the documents in your document store. By default, the document store won't return the embeddings of documents which is why you need to pass the return_embedding=True.

[sjrl]

I would also recommend looking at the quality of translations of your documents. When I ran the example you provided me I realized the format of the file is actually csv and not pure txt. And convert_files_to_docs does not handle csv files well so I would recommend splitting the csv file into smaller pure txt files before running them through convert_files_to_docs.

[ghost]

hi @sjrl thanks for your reply everything work except the fact that when i append all my doc to a list like this
documents.append(Document( content=df.iloc[i]['text'], meta={"title": df.iloc[i]['title'], "type": df.iloc[i]['type'], "sub-title": df.iloc[i]['sub-title']}

and try to translate the documents i get a DefaultCPUAllocator: not enough memory: you tried to allocate 21142437888 bytes. i guess i need to convert the documents to text file like before with convert_files_to_docs and then translate them.

its not really a big problem so i will juste take the csv and make a more correct text file now i will make this post as resolved when i finish if you have any tips in how to translate only small per small batch of document tell me (hope the batchsize of the Translator will come soon).

As you guessed the original file is a csv that i convert to text to use convert_files_to_docs.

[sjrl]

Hi @azetoy yes I would recommend converting them into text files and using the convert_files_to_docs. You can try providing a list of list of documents to reduce memory usage. So something like

documents = [
[Doc1, Doc2 ], 
[Doc3, Doc4],
...
]

instead of

documents = [Doc1, Doc2, Doc3, Doc4, ...]

because translate_batch will only translate one list of documents at a time.

[ghost]

@sjrl thanks for eveything i still need to search for optimum configuration (like number of beam, generator and retriever top k as i have 20k line in my sql lite db and with small number the reply of the rag is empty or dont mean anything)