Characterization of PoS accuracy

[dandiep]

Has there been any characterization of where the PoS tagging is inaccurate with any of the models? It is definitely helpful to know that something is "95%" accurate, but it would be also helpful if that was broken down by part of speech. E.g. verbs are 99% accurate, but nouns are only 92%. I have searched but couldn't find anything. Thanks!

[svlandeg]

I don't think we have those numbers readily available for the POS tags. Our standard evaluation does evaluate NER and the dependency links per type though. You'll find that in the accuracy.json file of any pretrained pipeline, e.g.:

  "dep_las": 0.9019185362,
  "dep_las_per_type": {
    "prep": {
      "p": 0.8577165591,
      "r": 0.8671219882,
      "f": 0.8623936301
    },
    "det": {
      "p": 0.9790087226,
      "r": 0.97932806,
      "f": 0.9791683653
    },
    ...

For the tagger, we only include the tag_acc which is the result of calling Scorer.score_token_attr(examples, "tag"), but it should be straightforward to add a "per entity" scoring to that by calling score_token_attr_per_feat as well.

[dandiep]

Appreciate the helpful response. I will have to try it, but it may be a little bit before I'm able to get to this. Cheers!