How to specify "offload_folder" in config file

[glasserd]

Hi,

I'm using the spacy llm pipeline to try and summarize some text. I'm using the spacy.Dolly.v1 model and am running out of memory. I'm trying to use a configuration file with spacy. The error message suggests that I set an offload_folder however, I'm not sure how I can do this within the configuration file. Any ideas?

Error message:

ValueError: The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format.

Configuration file:

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.Summarization.v1"


[components.llm.model]
@llm_models = "spacy.Dolly.v1"
name = "dolly-v2-12b"

[rmitsch]

Hi @glasserd, you can pass offline_folder as an argument for your model, just as every other Hugging Face transformers argument:

[components.llm.model]
@llm_models = "spacy.Dolly.v1"
name = "dolly-v2-12b"
config_init = {"offload_folder": ...}

or

[components.llm.model]
@llm_models = "spacy.Dolly.v1"
name = "dolly-v2-12b"

[components.llm.model.config_init]
offload_folder = ...

[RasoolAbbas-AM]

I think you mean offload_folder?

Also, this is not working for me.

[rmitsch]

Sorry for the typo, fixed it. Not working how?

[RasoolAbbas-AM]

I am running 16 Gbs of RAM and a decent CPU, however no NVIDIA GPU. I am attempting to run any model locally however I keep receiving the error mentioned in the beginning of this thread, and this configuration code does not fix it for me. I have read around that it could potentially be due to low memory and I think that is ultimately the root of the issue, but not sure.

[rmitsch]

If specifying offload_folder doesn't work, I recommend consulting the guys over at Hugging Face. spacy-llm just wraps HF/transformers models and config, so ultimately they'll be able to give you better advice.