Change in documentation code example in Rule-Based Matching emoji

[igormorgado]

The example for custom extensions based on emoji descriptions used in the (third example of rule-based matching)[https://spacy.io/usage/rule-based-matching#example3] is failing because the used library Emojipedia isn't correctly working. Last library update was around 5 years ago, probably the website changed and the http requests implemented in the lib are not correctly updated.

Because of that I propose the use of another library emoji, that do not make any requests to external sources, resulting in a more consistent example. Below you can find the whole code working.

I have also added the printing of created extension (emoji description) in the for loop used in the "main" part of the code. Since the attribute was not being displayed in the given example

Here follows a full working source code based on original proposal.

import emoji
from spacy.tokens import Span  # Get the global Span object
from spacy.lang.en import English
from spacy.matcher import Matcher

nlp = English()  # We only want the tokenizer, so no need to load a pipeline
matcher = Matcher(nlp.vocab)

pos_emoji = ["😀", "😃", "😂", "🤣", "😊", "😍"]  # Positive emoji
neg_emoji = ["😞", "😠", "😩", "😢", "😭", "😒"]  # Negative emoji

# Add patterns to match one or more emoji tokens
pos_patterns = [[{"ORTH": emoji}] for emoji in pos_emoji]
neg_patterns = [[{"ORTH": emoji}] for emoji in neg_emoji]

Span.set_extension("emoji_desc", default=None, force=True)  # Register the custom attribute

def label_sentiment(matcher, doc, i, matches):
    match_id, start, end = matches[i]
    if doc.vocab.strings[match_id] == "HAPPY":  # Don't forget to get string!
        doc.sentiment += 0.1  # Add 0.1 for positive sentiment
    elif doc.vocab.strings[match_id] == "SAD":
        doc.sentiment -= 0.1  # Subtract 0.1 for negative sentiment
    span = doc[start:end]
    # Verify if it is an emoji and set the extension attribute correctly.
    if emoji.is_emoji(span[0].text):
        span._.emoji_desc = emoji.demojize(span[0].text, delimiters=("", ""), language=doc.lang_).replace("_", " ")

matcher.add("HAPPY", pos_patterns, on_match=label_sentiment)  # Add positive pattern
matcher.add("SAD", neg_patterns, on_match=label_sentiment)  # Add negative pattern

# Add pattern for valid hashtag, i.e. '#' plus any ASCII token
matcher.add("HASHTAG", [[{"ORTH": "#"}, {"IS_ASCII": True}]])

doc = nlp("Hello world 😀 #MondayMotivation")
matches = matcher(doc)
for match_id, start, end in matches:
    string_id = doc.vocab.strings[match_id]  # Look up string ID
    span = doc[start:end]
    print(string_id, span.text, span._.emoji_desc)

Hope it helps.

[shadeMe]

Thanks for the report and the example code! We'll look into this.

[shadeMe]

I did a quick test of the example code on the website, and it runs correctly for on my end. Could it be that the errors that you were experiencing were intermittent?

EDIT: Apologies, the example I tested didn't actually invoke the library. Using the subsequent example on that page does raise an error.

[shadeMe]

I've opened an issue to track this, so let's close this discussion.