-
Notifications
You must be signed in to change notification settings - Fork 18
Closed
Description
ModernBERT is currently not supported by the train system, raising a ValueError here:
cnlp_transformers/src/cnlpt/CnlpModelForClassification.py
Lines 248 to 254 in 84e784d
| raise ValueError( | |
| f"Encoder config does not have an attribute" | |
| f' "{ke.args[0]}"; this is likely because the API of' | |
| f" the chosen encoder differs from the BERT/RoBERTa" | |
| f" API and the DistilBERT API. Encoders with different" | |
| f" APIs are not yet supported (#35)." | |
| ) |
It looks like the ModernBERT config doesn't define hidden_dropout_prob, and instead has four different dropout config attributes: attention_dropout, embedding_dropout, mlp_dropout, and classifier_dropout. There might be other changes needed to support ModernBERT, this is just the first error that surfaces when running the train system.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels