Support for custom scoring

[thejamesmarq]

I haven't come across any easy way to implement a custom scorer. For me, there are three metrics I am concerned with:

• Micro averaged F score for text categorizers
• Mean reciprocal rank for text categorizers
• Matthews correlation coefficient for text categorizers

Each of these are for different applications; just mentioning them to highlight that there might be gaps in scoring as is.

It's not a huge deal to run a separate script after training to do the evaluation, but given that the API / config system as a whole is pretty sleek, I was thinking it would be nice to be able to specify a scorer as part of a config.

I've seen in https://spacy.io/api/language#evaluate that there is already a way to specify a scorer in code. I'm thinking if we could source custom code in the config, it would be pretty useful.

I'm thinking this might be an additional field in the config like

[training]
lang = "en"
...

[training.scorer]
@evaluate = my_custom_scorer

Benefit of all this could be ensuring future evaluation of the pipeline is consistent according to the user's use case.

[adrianeboyd]

In spaCy v3.x each component provides its own scoring method, so there's not a single config setting that can be overridden to use, e.g., within nlp.evaluate.

Currently, I think the best way to implement this with spacy train and a config would be to extend the TextCategorizer and override the score method. Use that component (with custom spacy train --code) in your config to use those scoring methods during training rather the default methods.

The other main alternative is to create a custom training loop (as an alternative to using spacy train), where you would modify the evaluation callback to call a method other than nlp.evaluate, but it's a big drawback if you end up having to maintain your own separate training loop.

In general, I think that scoring separately for custom scoring methods is going to be easier in a lot of cases, but obviously if you want these scores to be used for early stopping, then you'd need to use custom code during training.

Luckily extending TextCategorizer and MultiLabel_TextCategorizer is pretty easy because they're implemented in plain python rather than cython, so it's easy to import the custom class with --code. For components implemented in cython, I think you would typically need to create + install a separate package with the right entry points to make the custom component available for the config. The pip packaging is the hardest part here. I created a demo package that shows how to do this for a tokenizer, but if you wanted to customize the scoring for Tagger or DependencyParser, it would look similar:

https://github.com/adrianeboyd/custom-cython-tokenizer

[adrianeboyd]

Wait, you're right, I totally forgot about the extra scorer option for Language.evaluate since we use all the defaults internally. You could extend Scorer and override Scorer.score to not use the components' built-in scoring methods for certain cases. I'd have to think about whether/how this could make sense as part of a new config setting...

I still think that overriding TextCategorizer.score is going to be the easiest option for the current release.

[thejamesmarq]

Makes sense. I do wonder about subclassing for each new combination of metrics, but I'll try that out for now.

Looking at the TextCategorizer code, maybe score could just be passed as an optional param? Like

class TextCategorizer(TrainablePipe):
    def __init__( self, vocab: Vocab, model: Model, name: str = "textcat", *, threshold: float, score: Callable) -> None:
        self.vocab = vocab
        self.model = model
        self.name = name
        self.score = score
        self._rehearsal_model = None
        cfg = {"labels": [], "threshold": threshold, "positive_label": None}
        self.cfg = dict(cfg)

...

so you could pass a custom function through the config like

[components.textcat]
factory = "textcat"
score = my_custom_score_function