Caused by: org.apache.spark.api.python.PythonException: 'ValueError: [E109] Component 'tok2vec' could not be run. Did you forget to call initialize()?'.

[rafaeloliveirabd]

How to reproduce the behaviour

I'm trying to apply lemmatization on several languages on azure databricks in a dataframe, however, when I use this to do the lemmatiztion:

spark = SparkSession.builder.appName('text-cleaning').getOrCreate()
sql_selection = spark.read.table("hive_metastore.default.table")
df_final = ps.DataFrame(sql_selection)

... (I call a function that cleans the text here and the next line runs) ...

out['clean_text'] = out['clean_text'].apply(lambda row: " ".join([w.lemma_.lower() for w in nlp(row)]))

I keep getting this error (in azure databricks notebook):
Caused by: org.apache.spark.api.python.PythonException: 'ValueError: [E109] Component 'tok2vec' could not be run. Did you forget to call initialize()?'.

What does this mean? I can't even access the lemmatized text by doing dataframe.iloc[2,:] for example (if throws the same error)

Also, I'm downloading the language on the same notebook doing this:

%sh python -m spacy download sv_core_news_lg

It seems like this is a spacy issue? Or am I not loading something on the azure databricks?

Your Environment

• Operating System: Windows 11
• Python Version Used:3.8.10
• spaCy Version Used: latest version
• Environment Information: azure databricks, 10.4 runtime

[polm]

Normally I would expect to see that error if you were using an uninitialized tok2vec, like one that hadn't been trained. How are you creating your nlp object?

Also, I'm not familiar with databricks. Is your code running in a Windows environment there, or is that your local dev environment? Probably not relevant either way, just wanted to check.

[rafaeloliveirabd]

Normally I would expect to see that error if you were using an uninitialized tok2vec, like one that hadn't been trained. How are you creating your nlp object?

Also, I'm not familiar with databricks. Is your code running in a Windows environment there, or is that your local dev environment? Probably not relevant either way, just wanted to check.

Hey, the code is being run in the cloud, not locally. I wonder if the installation of spacy or the language being used (Swedish, for example) is not being properly installed in azure databricks?

I don't have this problem locally, only on the cloud (locally I also never had to initialize tok2vec for example to get the lemmatizations going)

I'm initializing it like this:

        nlp = sp.load(f'{language}_core_web_lg')
        #save default spacy stopwords for the language loaded under nlp
        spacy_stopwords = nlp.Defaults.stop_words 
        #remove all stop words - essentially resetting the spacy set of stop words
        nlp.Defaults.stop_words -= {word for word in spacy_stopwords} 
        #adding our own english stop words
        nlp.Defaults.stop_words |= en_stop_words
        stopwords = nlp.Defaults.stop_words

[rafaeloliveirabd]

It worked after I installed the spacy languages using the .whl on the azure databricks