How tagger.predict works?

[dglopes]

I was looking for some way to measure the POS tagging confidence and found this answer:
https://stackoverflow.com/questions/65218606/is-it-possible-to-find-uncertainties-of-spacy-pos-tags
But after some research I didnt figured out how the tagger.predict method works or measure the tag probability/confidence.

[adrianeboyd]

You can get the raw scores from the tagger's internal model:

nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a sentence.")
nlp.get_pipe("tagger").model.predict([doc])

The scores for each token correspond to the labels in nlp.get_pipe("tagger").labels.

There's a bit of cython, but the pair of predict / set_annotations methods should help you understand the internals:

spaCy/spacy/pipeline/tagger.pyx

Lines 124 to 172 in 7df328f

	def predict(self, docs):
	"""Apply the pipeline's model to a batch of docs, without modifying them.
	docs (Iterable[Doc]): The documents to predict.
	RETURNS: The models prediction for each document.
	DOCS: https://spacy.io/api/tagger#predict
	"""
	if not any(len(doc) for doc in docs):
	# Handle cases where there are no tokens in any docs.
	n_labels = len(self.labels)
	guesses = [self.model.ops.alloc((0, n_labels)) for doc in docs]
	assert len(guesses) == len(docs)
	return guesses
	scores = self.model.predict(docs)
	assert len(scores) == len(docs), (len(scores), len(docs))
	guesses = self._scores2guesses(scores)
	assert len(guesses) == len(docs)
	return guesses
	def _scores2guesses(self, scores):
	guesses = []
	for doc_scores in scores:
	doc_guesses = doc_scores.argmax(axis=1)
	if not isinstance(doc_guesses, numpy.ndarray):
	doc_guesses = doc_guesses.get()
	guesses.append(doc_guesses)
	return guesses
	def set_annotations(self, docs, batch_tag_ids):
	"""Modify a batch of documents, using pre-computed scores.
	docs (Iterable[Doc]): The documents to modify.
	batch_tag_ids: The IDs to set, produced by Tagger.predict.
	DOCS: https://spacy.io/api/tagger#set_annotations
	"""
	if isinstance(docs, Doc):
	docs = [docs]
	cdef Doc doc
	cdef bint overwrite = self.cfg["overwrite"]
	labels = self.labels
	for i, doc in enumerate(docs):
	doc_tag_ids = batch_tag_ids[i]
	if hasattr(doc_tag_ids, "get"):
	doc_tag_ids = doc_tag_ids.get()
	for j, tag_id in enumerate(doc_tag_ids):
	if doc.c[j].tag == 0 or overwrite:
	doc.c[j].tag = self.vocab.strings[labels[tag_id]]

[dglopes]

So it returns the accuracy of the model for that token labels array based on its attempt to classify/predict the corret label?
ref: https://spacy.io/api/pipe

[adrianeboyd]

I'm not sure what you mean? It returns the model's score for each possible tag for each token in the doc.

[dglopes]

the score is given by model.predict, but what calculation occurs in the predict method? I imagine it is the model's accuracy measure calculated by training...but I'm not sure. Will the accuracy be measured based on your attempt x gold-standard?

[dglopes]

Ah, its ok! I found the explanation here:
https://spacy.io/usage/training

:)