UserWarning: [W108] The rule-based lemmatizer did not find POS annotation for one or more tokens. Check that your pipeline includes components that assign token.pos, typically 'tagger'+'attribute_ruler' or 'morphologizer'.

[micmizer]

I am getting the warning in the title. When I look at the referenced file spacy/pipeline/lemmatizer.py:211, it appears this warning means that a tokens POS is an empty string ""?

    def rule_lemmatize(self, token: Token) -> List[str]:
        """Lemmatize using a rule-based approach.

        token (Token): The token to lemmatize.
        RETURNS (list): The available lemmas for the string.

        DOCS: https://spacy.io/api/lemmatizer#rule_lemmatize
        """
        cache_key = (token.orth, token.pos, token.morph.key)  # type: ignore[attr-defined]
        if cache_key in self.cache:
            return self.cache[cache_key]
        string = token.text
        univ_pos = token.pos_.lower()
        if univ_pos in ("", "eol", "space"):
            if univ_pos == "":
                warnings.warn(Warnings.W108)
            return [string.lower()]

I added this to my function that creates my training data and it never prints anything, so I am a bit confused why I keep getting this warning?

for token in doc:
    if token.pos_ == "":
        print(f"Token with empty POS tag: {token.text}")

My pipeline looks like this:

'tok2vec', 'tagger', 'attribute_ruler', 'lemmatizer', 'ner'

[micmizer]

I think I just figured this out. My attribute ruler config was like this

[components.attribute_ruler]
factory = "attribute_ruler"
scorer = {"@scorers":"spacy.attribute_ruler_scorer.v1"}
validate = false

And after changing that to this I no longer get the warnings:

[components.attribute_ruler]
source = "en_core_web_lg"