when I use "benepar", My result changes

[pejmannavi]

Hi

When I use this :

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The rules of the game help you imagine what happens.")
for sent in list(doc.sents): 
  if len(sent) > 1 : 
    print('TXT:', [token.text for token in sent]) 
    print('TAG:', [token.tag_ for token in sent]) 
    print('POS:', [token.pos_ for token in sent])

My result will be this :

TXT: ['The', 'rules', 'of', 'the', 'game', 'help', 'you', 'imagine', 'what', 'happens', '.']
TAG: ['DT', 'NNS', 'IN', 'DT', 'NN', 'NN', 'PRP', 'VB', 'WP', 'VBZ', '.']
POS: ['DET', 'NOUN', 'ADP', 'DET', 'NOUN', 'NOUN', 'PRON', 'VERB', 'PRON', 'VERB', 'PUNCT']

But when I use this :

import benepar, spacy
nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("benepar", config={"model": "benepar_en3"}) 
doc = nlp("The rules of the game help you imagine what happens.")
for sent in list(doc.sents): 
  if len(sent) > 1 : 
    print('TXT:', [token.text for token in sent]) 
    print('TAG:', [token.tag_ for token in sent]) 
    print('POS:', [token.pos_ for token in sent]) 

My result will be this :

TXT: ['The', 'rules', 'of', 'the', 'game', 'help', 'you', 'imagine', 'what', 'happens', '.']
TAG: ['DT', 'NNS', 'IN', 'DT', 'NN', 'VBP', 'PRP', 'VB', 'WP', 'VBZ', '.']
POS: ['DET', 'NOUN', 'ADP', 'DET', 'NOUN', 'NOUN', 'PRON', 'VERB', 'PRON', 'VERB', 'PUNCT']

Why in my first script "help tag" is "NN", but in my second one is "VBP"?
Why in second one, "help pos" is "NOUN"? It must be "VERB".
For correct result, Why I have to import "benepar"?

[adrianeboyd]

I'm not sure I understand your question about the annotation. If you add another component like benepar to the end of your pipeline, it can overwrite or modify annotation in the doc. Depending on what annotation you need, you many only need benepar and not other components from en_core_web_sm.

If the benepar package doesn't provide the registered factory through an entry point, then you need to import the package so spacy knows about the custom factory so you can use nlp.add_pipe("benepar").

[pejmannavi]

Hi Adrian
In my first sample, "help pos" must be VERB, but it has NOUN pos. Why?
In my second sample, "help pos" is NOUN, but "help tag" is VBP. Why?

[adrianeboyd]

benepar is a third-party component and I'm not 100% sure what it annotates. If it's annotating tag and not pos, then you might be seeing the pos that was created with the tag->pos map in attribute_ruler from the tagger tags rather than the benepar tags.

If you want the default tag->pos rules in en_core_web_sm to apply, then add benepar before attribute_ruler in the pipeline.