I would like to know if the token.tag for training in v3.0 is equal to token.tag or token.pos in v2.3?

[qingyun1988]

I've already consulted you @svlandeg in this post: #8075

There are a few things I am still not clear about.

You said this in the above post.

it only assigns token.tag since v3.0 for training data.

I would like to know if the token.tag in v3.0 is equal to token.tag or token.pos in v2.0? In other words, are the tags of token.tag in v3.0 'VBD,VBG,VBN...etc' or 'ADJ,NOUN,VERB...etc'?

In addition, can I save the model I trained to the current directory which is the same one of the model I am training?

Thank you!

Originally posted by @svlandeg in #8075 (comment)

[polm]

Token.tag in 2.0 is still token.tag in 3.0.

tag is language-specific, detailed part of speech tags. pos is more general Universal Dependencies part of speech tags. This is completely unchanged between v2 and v3.

In addition, can I save the model I trained to the current directory which is the same one of the model I am training?

Yes, you can save a model anywhere, you just specify the path when calling spacy train.

[qingyun1988]

Thanks for your help! Is it necessary to base on only token.tag but not token.pos or base on both token.tag and token.pos for training tagger? Since there is a bit of confusion in v.2.3 about how to train tagger, I don't know what the actual situation is in v3.0.

[svlandeg]

You definitely want to train the tagger on token.tag. Given that that's also the most detailed annotation, you'll be able to deduce the token.pos from it afterwards.

[qingyun1988]

Got it. Thank you~

[svlandeg]

I did reply, but perhaps you couldn't see it because you have to hit the "see more" link at the bottom of a thread if it's becoming too long: #8075 (reply in thread)

[svlandeg]

Yea, I can see how that's confusing. But the documentation there really aims to be simple, using easily-understandable annotations like "NOUN" instead of "NNP" or such, which wouldn't be clear to everyone.

The rule is this: whatever data you feed into the tagger training, will be trained as token.tag labels. It depends on the corpus. For the pretrained English pipelines, these tags are the fine-grained NN, NNP, NNPS, NNS etc, and then we use an attribute_ruler to convert those trained tags to token.pos.

[qingyun1988]

Thank you! In other words, I only need to prepare fine-grained tags data but not coarse-grained pos data for tagger training, right? But why are these tags coarse-grained NOUN, VERB , ADJ, etc in official examples? Do these tags have to be fine-grained for English pipelines? Are both fine-grained tag and coarse-grained pos appropriate? What happens if I provide coarse-grained pos or both fine-grained tag and coarse-grained pos for it? for instance:

words = ["I", "like", "stuff"]
pos = ["PRON", "VERB", "NOUN"]
tags = ["PRP", "VBP", "NN"]
predicted = Doc(nlp.vocab, words=words)
example = Example.from_dict(predicted, {"tags": tags,"pos":pos})

[qingyun1988]

@svlandeg

[svlandeg]

Hi @qingyun1988, please don't ping maintainers on various issues - it makes our GH mentions difficult to use.

[qingyun1988]

I'm sorry. I'm a little anxious to haven't got a reply for so long. Would you like to help me and answer the questions above, please? Thank you very much!