Why nlp.evaluate does not update examples in-place?

[narayanacharya6]

I'd like to inspect/save the predictions made during evaluation. But, nlp.evaluate runs on a copy of the examples? Is there is a better way to run evaluations and save the predictions in the same run?

[adrianeboyd]

The reason that nlp.evaluate runs on a copy is because many components don't overwrite existing annotation or make different predictions if some annotation is already saved in the predicted docs, so running evaluate twice might result in two different sets of predicted docs and scores, which could get really confusing:

scores1 = nlp.evaluate(examples)
scores2 = nlp.evaluate(examples)

You should be able to do something like this, which is just a simplified excerpt of Language.evaluate:

docs = nlp.pipe((eg.predicted for eg in examples))
for eg, doc in zip(examples, docs):
    eg.predicted = doc
scorer = Scorer(nlp=nlp)
scores = scorer.score(examples)