EL evaluation recall and precission

[Joozty]

I was exploring the evaluation part of EntityLinker pipe and found out that when a wrong prediction is found it is both counted as false negative as well as false positive. I kind of cannot get it. It feels like it's correct but on the other hand shouldn't it be only false positive? Maybe you guys have an easy explanation so I could understand it better. Thanks 🙂

[svlandeg]

Let's imagine you have a sentence "Laura went shopping", which has "Laura" annotated as a PERSON entity.
First, let's assume Laura is a random person not included in any knowledge base, so NEL should annotate the link as NIL, this is the gold label. Let's now assume that the NEL instead predicted Q18225863. I hope you'll agree that this a one False Positive (and nothing else).

Now let's assume that the sentence reads "Laura was singing" and from the context we know that the PERSON Laura is in fact Q3229201, this is now the gold label. When instead the NEL predicts Q18225863, it is not only wrong (False Positive) but it's also missing one true label (False Negative). If the NEL would have predicted NIL instead, it would have only been a False Negative.

Intuitively, to me this means that the NEL algorithm is getting "punished" for "guessing wrong" in the face of ambiguity. In fact, I would definitely prefer the NEL to have predicted NIL, rather than having predicted an entirely wrong link.

So - that is at least my reasoning behind that implementation.