Error when running Hugging Face models on M2 Pro - prompting LLM in order to create training data

[SilasFriby]

Hi, I wish to create training data for a NER model. My plan was to do so by using spacy-llm, more specifically by using the Prodigy manual "ner.llm.correct". This seems to be working with ChatGPT and Cohere. However with ChatGPT I ran out of credit and with Cohere I hit a wall due to the limitation on the free trial API key.

Therefore I decided to have a look at the models distributed through Hugging Face (HF). I started by looking at Llama2 and got permission to the Llama2 repo through HF. However, when running my code I receive the error:

"TypeError: BFloat16 is not supported on MPS"

As far as I can tell this error originates from PyTorch see here and based on this thread pytorch/pytorch#99272 the issue does not seem to be resolved.

My question is whether or not there is a know workaround or solution to this problem, which enables me to use spacy-llm?

FYI, I have read the blog post https://explosion.ai/blog/metal-performance-shaders. However it is not clear to me if this approach could somehow be used with spacy-llm.

CONFIG FILE: fewshots.cfg

[paths]
examples = null

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v3"
labels = ["COMPANY"]
description = Entities are the company names.
    Adjectives, verbs, adverbs are not entities.
    Pronouns are not entities.

[components.llm.task.label_definitions]
COMPANY = "A company name found in the text"

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "llm/examples.json"

[components.llm.model]
@llm_models = "spacy.Llama2.v1"
name = "Llama-2-7b-hf"

MAIN FILE: main.py

import prodigy
from huggingface_hub import login

# Initialize
recipe_name = 'ner.llm.correct'
dataset_name = "titles_annotated"
data_input_path = "data/statement_titles_sub.jsonl"
config_path = "llm/fewshot.cfg"

# Login to hugging face
key_token = 'hf...EK'
login(token=key_token)

# Prodigy serve
prodigy_str = (
    "prodigy"
    + " "
    + recipe_name
    + " "
    + dataset_name
    + " "
    + config_path
    + " "
    + data_input_path
)

prodigy.serve(prodigy_str)

DATA FILE: statement_titles_sub.jsonl

{"text": "Digizuite publish the half-year report for 2023-H1"}
{"text": "Caravelle International Group Announces Audited Financial Results for 2022 with 51.9% Revenue Growth and 118.2% Increase in EPS"}
{"text": "Top Strike Announces 2023 Fourth Quarter and Annual Financial Results ending April 30, 2023 and Corporate Update"}
{"text": "Top Strike Announces 2023 Fourth Quarter and Annual Financial Results ending April 30, 2023 and Corporate Update"}
{"text": "Scandinavian Tobacco Group A/S: Interim Report, 1 January - 30 June 2023 "}
{"text": "Scandinavian Tobacco Group A/S revises outlook and reports key financial data for the second quarter and first half of 2023"}
...

Thank you in advance.

Best regards
Silas

[rmitsch]

Hi Silas, unfortunately this is a PyTorch limitation. You could use a different model that uses float16 as its dtype, but those are usually hard to come by as bfloat16 has become the de-facto standard for training LLMs.

My question is whether or not there is a know workaround or solution to this problem, which enables me to use spacy-llm?

One workaround could be to modify the spacy-llm code directly and cast the model (model.to(torch.float16)), but this will most likely affect its performance. We'll consider whether to include this in an upcoming patch release.
Alternatively, you can pin the model to the CPU, but it'll be too slow to perform any useful inference. We suggest to run this code on a GPU machine, if you can.

FYI, I have read the blog post https://explosion.ai/blog/metal-performance-shaders. However it is not clear to me if this approach could somehow be used with spacy-llm.

This only applies to models trained with Thinc, which the Hugging Face models haven't.

[SilasFriby]

Hi Raphael, this is what I suspected. Hopefully the limitation i PyTorch will be solved in the near future.