Error using llama2

[santoshbs]

I am trying to use the basic example provided with llama2 model. I get an error. Not sure what I am doing wrong. Request your help.

Here's the config:

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["COMPLIMENT", "INSULT"]

[components.llm.model]
@llm_models = "spacy.Llama2.v1"
name = "Llama-2-13b-hf"

Here's the code:

from spacy_llm.util import assemble
nlp = assemble("config.cfg")
doc = nlp("You look gorgeous!")
print(doc.cats)

The error message:

Loading checkpoint shards: 100%|██████████| 3/3 [00:03<00:00,  1.31s/it]
Traceback (most recent call last):
  File "/home/santoshbs/.local/share/JetBrains/Toolbox/apps/PyCharm-C/ch-0/223.8617.48/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 7, in <module>
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/spacy/language.py", line 1047, in __call__
    error_handler(name, proc, [doc], e)
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/spacy/util.py", line 1724, in raise_error
    raise e
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/spacy/language.py", line 1042, in __call__
    doc = proc(doc, **component_cfg.get(name, {}))  # type: ignore[call-arg]
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/spacy_llm/pipeline/llm.py", line 145, in __call__
    docs = self._process_docs([doc])
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/spacy_llm/pipeline/llm.py", line 199, in _process_docs
    responses_iters = tee(self._model(prompts_iters[0]), n_iters)
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/spacy_llm/models/hf/llama2.py", line 43, in __call__
    return [
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/spacy_llm/models/hf/llama2.py", line 44, in <listcomp>
    self._model(pr, generation_config=self._hf_config_run)[0]["generated_text"]
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/transformers/pipelines/text_generation.py", line 204, in __call__
    return super().__call__(text_inputs, **kwargs)
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/transformers/pipelines/base.py", line 1129, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/transformers/pipelines/base.py", line 1136, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/transformers/pipelines/base.py", line 1035, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/transformers/pipelines/text_generation.py", line 265, in _forward
    generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/transformers/generation/utils.py", line 1423, in generate
    self._validate_model_kwargs(model_kwargs.copy())
  File "/home/santoshbs/anaconda3/envs/_llm/lib/python3.9/site-packages/transformers/generation/utils.py", line 1243, in _validate_model_kwargs
    raise ValueError(
ValueError: The following `model_kwargs` are not used by the model: ['use_auth_token'] (note: typos in the generate arguments will also show up in this list)

[rmitsch]

Hi @santoshbs! Our documentation is incomplete for this, we'll improve on it.

So the thing with Llama 2 is that Meta published this conditional on a custom license you'll have to agree on, otherwise you aren't allowed to - and can't - download the model from Hugging Face. You can see the license at the top of this page. So in order to be able to use Llama 2 via HF (and consequently in spacy-llm, as we wrap HF for local models), you need to do the following:

1. Create a HF account, if you don't have one yet.
2. Request access from Meta by following the steps on one of the HF Llama 2 model pages.
3. Log in with your HF user on your machine using huggingface-cli login (see here)
4. Once you get notified that Meta has granted you access: try running the example again.

These steps are not obvious to users not familiar with this process, so we'll update our docs correspondingly. Thanks for bringing this up!

[rmitsch]

Hm, I can run this exact example without issues, so it's most likely the HF authentication. Did you generate and use an access token to log in with huggingface-cli login?

[santoshbs]

Yes, I used the access token using that command. BTW, in the very same conda environment, the following HF code works fine:

from transformers import AutoTokenizer
import transformers
import torch
torch.cuda.is_available()

model = "meta-llama/Llama-2-13b-chat-hf"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

sequences = pipeline(
    'I liked "Breaking Bad" and "Band of Brothers". Do you have any recommendations of other shows I might like?\n',
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=200,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

So, I believe that access is not an issue.

[rmitsch]

Which version of spacy-llm are you using?

[santoshbs]

spacy-llm 0.4.3

[svlandeg]

Hm. So comparing your code snippet with the spacy-llm implementation, I see two main differences:

• The HF snippet uses "Llama-2-13b-chat-hf", the spacy-llm one "Llama-2-13b-hf"
• spacy-llm provides use_auth_token=True in the call to transformers.pipeline().

If you update your code snippet to replicate the spacy-llm setting, does it still work? Or do you then also get the error around use_auth_token?

[mzeidhassan]

Hi @rmitsch,
So, how to pass "use_auth_token=" to config.cfg?

Also, does this configuration look OK?

[components.llm.model]
@llm_models = "spacy.Llama2.v1"
name = "Llama-2-7b-hf"

[rmitsch]

Hi @mzeidhassan, use_auth_token doesn't have to be included in the config. The auth token is available from your local Hugging Face transformers installation, if you are logged in.

Your configuration looks ok.

[rmitsch]

For posterity: further discussion of this in #324.

[innocent-charles]

can the Hugging face models(LLMs) used in spacy-llm be quantized ? ... i mean instead of loading the whole raw models from hugging face which cause the computation issues and memory...can you just do this Quantization process automatically so that the Hugging face models used.

[rmitsch]

That's an interesting idea, thanks for the suggestion! This is currently not supported, but it wouldn't be too difficult to implement this following this approach. I'll put that on our backlog.

[innocent-charles]

Thanks much @rmitsch