n-best POS tags

[sjmielke]

Does the option to output n-best POS tags (or even all tags with score > 0) exist with some hidden option or in some branch?
If not, would such an additional function be welcome or should I just hack it into my own fork? Technically it should be very easy (basically just replace the arg_max_if_true in spacy/tagger.pyx#L211 with an appropriate collection of results), but I have a feeling it will be more difficult to expose it nicely...

[honnibal]

I've been thinking about this recently, especially in context of the parser and NER.

Currently, here's what I would recommend:

• Create a new Language subclass MyLanguage
• Specialise MyLanguage.Defaults.create_tagger to return a Tagger subclass MyTagger
• Specialise MyTagger.__call__. Paste in the current Tagger.__call__ body
• Before tagging, add a numpy array in doc.user_data['tagger_scores'], with shape (len(doc), n_tags)
• Write the scores for word i in row i of your array

Maybe the tagger, parser etc should support an optional callback. This would be tricky on the parser because of the multi-threading, but easy on the tagger. If we make the callback take (i, doc, scores), we'll even be able to modify the scores in arbitrary ways --- so the user can interfere with the model during tagging.

[SauravMaheshkar]

If nobody is working on this, I'd like to contribute. @honnibal