Assemble Transformer Ner with trained Spancat model #11775
Replies: 3 comments 8 replies
-
|
Reading other discussions I tried one solution to train the pipeline with freezing components other than transformer and spancat. But i getting ValueError: Cannot get dimension 'nO' for model 'transformer-listener': value unset Error stack: |
Beta Was this translation helpful? Give feedback.
-
|
This is because freezing the Transformer doesn't work at the moment, see #11547. You can set |
Beta Was this translation helpful? Give feedback.
-
|
Hi @polm |
Beta Was this translation helpful? Give feedback.
-
|
I did chagne the grad_factor=0 but still getting the same error. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
I think this is a duplicate? |
Beta Was this translation helpful? Give feedback.
-
|
The reason your config based solution isn't working is that you aren't using See the docs on shared embeddings for more information about how to share embeddings and the tradeoffs involved. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah this worked, but still assemble pipeline model size is big since i have completely different trained model and assembling that with transformers. Correct my understanding, ideally i should be training only one model and freeze the other components to have smaller memory foot print. |
Beta Was this translation helpful? Give feedback.
-
|
To be clear, in your other comment you said you got the same error, but here you say it worked. I guess something was off and it actually worked with Anyway, there's no "ideal" here, there's different tradeoffs between having independent feature sources per component, see the embedding docs for more info. Your current configuration definitely takes more memory and disk space. Freezing the transformer and training another component is one alternative, another would be training multiple components at once, though setting up training data for that is trickier. For some discussion see this recent thread: #11767. For a project that uses the freezing strategy, see the coref project. |
Beta Was this translation helpful? Give feedback.
-
|
@polm Here is my basically, I need pre trained en_core_web_trf with all components enabled and add custom trained spancat-transformer model in the end and as mentioned earlier currently this is 1GB model and help my understanding if i train spancat with |
Beta Was this translation helpful? Give feedback.
-
That wouldn't work because you'd modify the transformer and break all the frozen components.
In this case what I would do is make a config with just the transformer and spancat. Source the transformer from Note that freezing the transformer like that usually results in lower accuracy, so I would recommend you also try training with an unfrozen transformer and compare your spancat scores so you know how big the difference is. It may not be that big but you should check. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I have a Spancat-transformer trained model and I am trying to assemble it with Pre-trained "en_core_web_trf" with NER and the assembled pipeline is not giving right results.
Here is what my spancat transformer model looks like which is producing results as expected
Here is the config for assembling it together
But my assembled pipeline is not working as expected -
however if i try to do outside the config like below I am getting the right results:
But my assumption the about code will increase the memory footprint and I need to assemble them all together as one model hence need the assemble_config to be working.
Beta Was this translation helpful? Give feedback.
All reactions