Freezing the transformer component for use with NER

[iashaheen]

I am trying to do a simple thing: freeze the transformer component for a set number of epochs to train the NER component only then resume training with both of them. I tried the following but it didn't work:

• Placed the transformer component in frozen_components under [training] configs but I received a value error. The way I understand it is that frozen components do not work during the training process so the NER component is not getting any embeddings.
• made grad_factor = 0.0 under [model.tok2vec] for the transformer listener config for a few epochs (first training run), then resumed the training by setting the source under the NER component config to be loaded from the best model (second training run). I kept the transformer config as is to be initialized. In the first training run the transformer loss was 0 the entire time, which is expected as the grad_factor is set to 0 and the model is not updated. In the second training run the transformer loss should change as the model is now updated but it did not and it remained 0. I also modified the value of grad_factor from the config file in the best _model directory before the second training run to be 1 but with no luck.

Any idea how can I freeze the transformer model initially then resume the training process to fine tune it?

[thomashacker]

Hello,
Unfortunately, freezing transformers currently don't work properly due to some issues.
We will be addressing them in the future.

[thomashacker]

Update: We've opened an issue for this problem.