Addition of "entity_ruler" in spacy 3.2 - Portuguese

[pmoniz7]

good morning gentlemen!
I would like to have your help. I'm new to Spacy and I have countless (millions) documents in Portuguese and like most of these documents have several important metadata. for example "CPF" , "TELEFONE" and DATE in dd/mm/yyyy format are strings that have numbers , "-" , and "/" and I would like to classify them as new entities to be able to capture them.

I tested it in English and it worked perfectly, but when I use Portuguese, spacy doesn't recognize it.

Am I doing the thing wrong? I think I'm doing that I'm following what's in the documentation.

Please, does anyone have any clues what I'm doing wrong?

Thanks in advance for the help

Below is the code I am using:

import spacy
#from spacy.lang.en import English
from spacy.lang.pt import Portuguese
from spacy.pipeline import EntityRuler
nlp = Portuguese()
#nlp = English()
nlp = spacy.blank("en")
#nlp = spacy.blank("pt")
#nlp = spacy.load("en_core_web_lg")
nlp = spacy.load("pt_core_news_lg")
print(nlp.pipe_names)

#Criando uma entity_rule
ruler = nlp.add_pipe("entity_ruler")

#List of Entities and Patterns (source: https://spacy.io/usage/rule-based-matching)
patterns = [
                {"label": "TELEFONE", "pattern": [{"SHAPE": "dddd"}, {"ORTH": "-"}, {"SHAPE": "dddd"}]},         
                {"label": "CPF", "pattern": [{"SHAPE": "dddd", "LENGTH": 9}, {"ORTH": "-"}, {"SHAPE": "dd"}]},
                {"label": "DATA", "pattern": [{"SHAPE": "dd"}, {"ORTH": "/"}, {"SHAPE": "dd"},
                                              {"ORTH": "/"}, {"SHAPE": "dddd"}]} 
            ]    
ruler.add_patterns(patterns)
print(nlp.pipe_names)

text = """Meu nome é Maria da Silva cpf 090985210-21 numero  meu telefone  3232-1347  Fiz plano preferencial Unimed data
         de nascimento 21/02/2020 sem obstetricia estou gravida de 7 meses e um mes na empresa desta forma busco atendimendo 
         aqui no Rio de Janeiro ou São Paulo no endereço rua Baiana 27 cep 242412-65"""

doc = nlp(text)

for ent in doc.ents:
     
    print (ent.text,  ent.label_)

[svlandeg]

Hi, welcome to spaCy and to the forums!

It would be helpful to us if you format your post with appropriate markdown formatting. It makes your question easier to understand, so we can help you quicker. As it's your first post, I've gone ahead and formatted your question myself, but please be mindful of this in the future.

good morning gentlemen!

I'd also like to point out that definitely not everyone on the spaCy team (or in the community) are men, so you might want to make sure to use more inclusive phrasing ;-)

That said, we'll have a look at your question and get back to you!

[pmoniz7]

thank you all

[pmbaumgartner]

@pmoniz7 Thanks for the question. I think the issue is that the document is being tokenized differently than how you expect it to and the shape attribute is something different than you expect.

To debug match rules, you can always print out the tokens and any attributes you want to check from the doc, for example:

for token in doc:
    print(token.orth_, token.shape_)

If you do this, you'll see the SHAPE is a bit different for a few of your rules. For example, a matching rule for CPF would be:

{"label": "CPF", "pattern": [{"SHAPE": "dddd-dd", "LENGTH": 9}]}

I had to double check this myself on the token attribute docs, but it is correct - since this is a single token, the "d" in the shape is truncated after 4 repeats until the next unique character (-).

[pmoniz7]

Thanks @pmbaumgartner for your help.
I believe my lack of experience with Spacy contributed to my difficulty understanding Spacy's matching rules.

But your answer was enlightening and I've already read the corresponding documentation https://space.io/api/token#attributes.

Many thanks to you and all of the Spacy team