Wrapping transformer models for text classification

[adrianeboyd]

Questions related to the development of spacy-wrap, which wraps transformer text classification models for inference:

• Haven't actually tested whether the model would be able to train. Ideally, the user should be able to further fine-tune the model. I assume setting the update and predict extensions would solve this issue. I have created an issue for it
• Will make sure to add a license
• Regarding the maintenance, this is actually a reworked version of a version in DaCy which broke with the spacy-transformers update, however, what I understand from your suggestion is that I copy-paste over the code to avoid breaking changes with changes in spacy-transformers? Or am I misunderstanding?
• If you have the time while developing this I got a few additional questions about whether it is possible to set the assigns dynamically? Similarly, I was also wondering what the dim "n0" is used for?

Originally posted by @KennethEnevoldsen in #10168 (comment)

[adrianeboyd]

Well, that wasn't 100% what I was going for in the conversion, but here we have a discussion at least, since I think this may be interesting for others interested in using similar models.

• The transformer component is only updated if other components like a tagger are listening to it. If you want to make it trainable like a typical annotating pipeline component, there's a lot of machinery in Transformer that's not really set up the right way.

  It might be easier not to extend Transformer but instead extend TrainablePipe and set up the model as an inline transformer in the model architecture (something like spacy-transformers.TransformerModel.v3) and then implement predict, set_annotations, and get_loss to handle the extensions and training. The inline transformer model serialization should just work (we hope, as of spacy-transformers v1.1, it has been a total pain to implement). I think your component would look a lot more like textcat and less like transformer in the end.

  (And this is a side issue, but it was a bad choice to have the transformer config settings in the [components] block. They really should be in the [initialize] block, but we don't want to change the design for transformer at this point.)

• Currently there isn't any templating in the component factory definitions, which is an issue we're aware of, e.g., for the scores for spancat which actually depend on the spans_key setting. This would be a pretty non-trivial extension (I think, anyway, maybe I'm overestimating it) and it's not something that we're currently working on in any way.

  There are also some components where assigns could potentially be determined dynamically, but a single static factory definition isn't really possible (like for the attribute_ruler), so the static pipe analysis tools don't really work like you would want.

• The listener details are not my specialty, but I think that nO is used in turn to initialize the dimensions for the TransformerListener that's listening to the transformer in another component.

• It's kind of unclear which code exactly is copied and pasted (also from a licensing standpoint).

Edited to try to straighten out the listener terminology.

[KennethEnevoldsen]

Hi @adrianeboyd, long time since I last looked at this, but have gotten some time so wanted to take a look at it again?

Seems like it is not possible to use the AutoModel approach (at least I far as I could figure out). Might it be possible to add an option to change the loaders using a syntax like (HFShim):

    def __init__(
        self,
        model: HFObjects,
        config=None,
        optimizer: Any = None,
        mixed_precision: bool = False,
        grad_scaler_config: dict = {},
        load_config_fn: Callable = AutoConfig.from_pretrained,
        load_tokenizer_fn: Callable = AutoTokenizer.from_pretrained,
        load_model_from_config_fn: Callable = AutoModel.from_config
    )

It should be backward compatible and allow passing these arguments forward in the HFWrapper.

def HFWrapper(
    hf_model: "HFObjects",
    convert_inputs: Optional[Callable] = None,
    convert_outputs: Optional[Callable] = None,
    mixed_precision: bool = False,
    grad_scaler_config: dict = {},
    load_config_fn: Callable = AutoConfig.from_pretrained,
    load_tokenizer_fn: Callable = AutoTokenizer.from_pretrained,
    load_model_from_config_fn: Callable = AutoModel.from_config
)

if so I can create a PR on this. My alternative solution currently is to use a monkey patch or define two classes called HFClassificationWrapper and an HFClassificationShim.

[adrianeboyd]

That is an interesting suggestion that I hadn't thought of. Is it straightforward to pass the settings to HFShim?

[KennethEnevoldsen]

That is at least what I have done as the temporary solution for spacy-wrap for the HFWrapper and HFShim. Which is called here.

[adrianeboyd]

Here's a PR to do this in spacy-transformers so it's easier to wrap other model types:

explosion/spacy-transformers#332

Here's a sketch of what spacy-wrap could like after this PR:

https://github.com/adrianeboyd/spacy-wrap/tree/feature/spacy-transformers-auto

(The spacy-transformers requirement would be >=1.1.7 later if this PR is merged, but this works for testing with PR branch.)

[KennethEnevoldsen]

Thanks for this @adrianeboyd! Sorry for getting to it a bit late. It is a wonderful update - will definitely make it easier to maintain and extend spacy-wrap. I have updated . it with the changes.