How can I switch PromptNode in the MultiHop tutorial to use a local model?

[recursionbane]

I'm following this tutorial: https://haystack.deepset.ai/tutorials/23_answering_multihop_questions_with_agents
Using v1.15 of farm-haystack[gpu] on a local machine with GPU.

It works great when using OpenAI's API.

If I switch this definition of the PromptNode:

    # Use OpenAI API as PromptNode
    from getpass import getpass
    api_key_prompt = "Enter OpenAI API key:"
    api_key = getpass(api_key_prompt)
    prompt_node = PromptNode(model_name_or_path="text-davinci-003", api_key=api_key, stop_words=["Observation:"])
    agent = Agent(prompt_node=prompt_node)

To this one:

    # Use a local model for PromptNode
    prompt_node = PromptNode(model_name_or_path="google/flan-t5-large", stop_words=["Observation:"])
    agent = Agent(prompt_node=prompt_node)

I see this error:

Agent zero-shot-react started with {'query': 'What year was the 1st president of the USA born?', 'params': None}
Traceback (most recent call last):
  File "./sample_agent_qa.py", line 48, in <module>
    result = agent.run("What year was the 1st president of the USA born?")
  File "/opt/conda/lib/python3.8/site-packages/haystack/agents/base.py", line 288, in run
    agent_step = self._step(agent_step, params)
  File "/opt/conda/lib/python3.8/site-packages/haystack/agents/base.py", line 312, in _step
    prompt_node_response = self.prompt_node(
  File "/opt/conda/lib/python3.8/site-packages/haystack/nodes/prompt/prompt_node.py", line 763, in __call__
    return self.prompt(self.default_prompt_template, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/haystack/nodes/prompt/prompt_node.py", line 802, in prompt
    output = self.prompt_model.invoke(prompt, **kwargs_copy)
  File "/opt/conda/lib/python3.8/site-packages/haystack/nodes/prompt/prompt_node.py", line 495, in invoke
    output = self.model_invocation_layer.invoke(prompt=prompt, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/haystack/nodes/prompt/providers.py", line 271, in invoke
    sw = StopWordsCriteria(tokenizer=self.pipe.tokenizer, stop_words=stop_words)
  File "/opt/conda/lib/python3.8/site-packages/haystack/nodes/prompt/providers.py", line 130, in __init__
    self.stop_words = tokenizer.encode(stop_words, add_special_tokens=False, return_tensors="pt")
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2291, in encode
    encoded_inputs = self.encode_plus(
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2699, in encode_plus
    return self._encode_plus(
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 502, in _encode_plus
    batched_output = self._batch_encode_plus(
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 429, in _batch_encode_plus
    encodings = self._tokenizer.encode_batch(
TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

I've also tried with a few different models "flan-t5-base", and without specifying a model.

Any suggestions?

[ZanSara]

Hey @recursionbane this looks like a bug. Let me convert it into an issue so we can track it better

[ZanSara]

Nevermind, I see you already did 😄

[dk008652]

is this bug resolved?

[bogdankostic]

Yes, should be fixed according to #4587.