Transformers/Tok2Vec and listeners

[BramVanroy]

Hi

I am looking into training my own models and I have a question about the interaction between the Tok2Vec component and listeners. I have read the documentation here.

The Tok2Vec is always connected to its listener(s), also in the backward pass. If we train a tok2vec layer from-scratch, that makes sense because the Tok2Vec layer needs to be "learned". But using pretrained Transformers this is less straight-forward and it is not clear to me in how far Transformers operate within such a training pipeline:

• Is the Transformer model fine-tuned alongside the other components (end to end), or does it only serve as a feature extractor?
• If trainable, since the Transformers is pretrained, and the other components are not, there is bound to be an inconsistency issue when learning: the new components would need a larger learning rate than the LM. Typically in NLP we first freeze the LM and train the additional layers on top and then gradually unfreeze the LM to squeeze out some more performance.
  • Can you modify the config in such a way that it is not just one learning rate for the whole pipeline?
  • Can you freeze/unfreeze gradually? (If a hook is available to the training loop I can implement that manually, that is okay)
• Can you use a pretrained tok2vec layer as-is without training (i.e. as feature extractor for the rest of the pipeline)?

Thanks! (And apologies in advance for other questions that I will undoubtedly have.)

[BramVanroy]

As an update for the last question: I found an answer in the docs. It's here if you scroll down a bit.

If I'm correct, you can do something like

[nlp]
lang = "en"
pipeline = ["tok2vec","parser"]

[training]
frozen_components = ["tok2vec"]

[components.tok2vec]
source = "en_core_web_sm"
component = "tok2vec"

This should load the tok2vec layer from the en_core_web_sm model and use it as-is.