Spacy textblob sentiment analysis very slow on a 500 rows dataset

[marcello-calabrese]

Hi,

I was wondering how I can speed up the sent analysis with spacy textblob on a dataset with 500 rows, The sentiment analysis I applied is on the 'Review'' column. It takes a while, more than 10 minutes. I will show below the code I applied:

`# Sentiment Analysis with spacy
def sentiment_analysis(text):
nlp = en_core_web_lg.load()
nlp.add_pipe("spacytextblob")
doc = nlp(text)

# get the sentiment score
sentiment_score = doc._.blob.polarity
# get the sentiment label
if sentiment_score > 0.4:
    sentiment_label = "Positive"
elif sentiment_score < 0:
    sentiment_label = "Negative"
else:
    sentiment_label = "Neutral"
return sentiment_score, sentiment_label`

`# test the sentiment analysis function creating a new column with the sentiment score and label

sample_data['sentiment_score'], sample_data['sentiment_label'] = zip(*sample_data['Review'].apply(sentiment_analysis))`

Hope there is a way to speed up the implementation. I tried to use the multiple cores processing but nothing, still slow.

Thanks,

Marcello

[danieldk]

You are loading the model in each call to sentiment_analysis and loading the model takes a fair amount of time. I'd suggest taking the model load out of the function and passing the model in as an argument:

def sentiment_analysis(nlp, text):
    doc = nlp(text)
    
    # get the sentiment score
    sentiment_score = doc._.blob.polarity
    # get the sentiment label
    if sentiment_score > 0.4:
        sentiment_label = "Positive"
    elif sentiment_score < 0:
        sentiment_label = "Negative"
    else:
        sentiment_label = "Neutral"
    return sentiment_score, sentiment_label`

nlp = en_core_web_lg.load()
nlp.add_pipe("spacytextblob")

sample_data['sentiment_score'], sample_data['sentiment_label'] = zip(*sample_data['Review'].apply(sentiment_analysis, args=(nlp,)))`

This should already speed things up a fair bit (I am assuming that apply prepends the given args, otherwise you could reverse the argument order).

A further optimization would be using Language.pipe in place of Language.__call__, which allows you to pass in multiple documents at once and is generally more efficient.

[marcello-calabrese]

Thanks a lot, I will try your suggestion.