Unexpected result with attribute_ruler

[pb-jeff-oneill]

My goal is to add some constraints to the tagger so I have a mix of rule-based and model tagging. For example, suppose I want the word "foo" to always have a POS/TAG of DET/DT.

I added an attribute ruler before the tagger to set POS/TAG for the word "foo".

The TAG is set correctly to DT but the POS is set to PRON.

Below is example code and results. Any suggestions for getting this to work correctly?

import spacy

nlp = spacy.load('en_core_web_sm')

nlp.rename_pipe("attribute_ruler", "attribute_ruler_std")
ruler = nlp.add_pipe("attribute_ruler", first=True)
patterns = [[{"ORTH": "foo"}]]
attrs = {"TAG": "DT"} # Same result when also adding "POS": "DET"
ruler.add(patterns=patterns, attrs=attrs)

doc = nlp('the dog')
print('the dog:', doc[0].pos_, doc[0].tag_, doc[1].pos_, doc[1].tag_)
doc = nlp('foo')
print('foo:', doc[0].pos_, doc[0].tag_)

the dog: DET DT NOUN NN
foo: PRON DT

I'm on Mac with Python 3.7 and Spacy 3.3.2.

[adrianeboyd]

I think you're running into an existing attribute ruler rule:

  {
    "patterns":[
      [
        {
          "DEP":{
            "NOT_IN":[
              "det",
              "quantmod",
              ""
            ]
          },
          "TAG":{
            "IN": [
              "DT",
              "WDT"
            ]
          }
        }
      ]
    ],
    "attrs":{
      "POS":"PRON"
    },
    "index":0
  }

There are some determiner/pronoun distinctions that are different in OntoNotes vs. UD and this is trying to improve the tag->pos mapping in these cases based on the parse.

You might need to look at all the attribute ruler rules as a whole and figure out which ones make sense for your task?

[pb-jeff-oneill]

Thank you, I didn't realize that a DEP could change how a TAG is mapped to a POS. This raises another question that I'll post separately.