Skip to content

Support ModernBERT #225

@ianbulovic

Description

@ianbulovic

ModernBERT is currently not supported by the train system, raising a ValueError here:

raise ValueError(
f"Encoder config does not have an attribute"
f' "{ke.args[0]}"; this is likely because the API of'
f" the chosen encoder differs from the BERT/RoBERTa"
f" API and the DistilBERT API. Encoders with different"
f" APIs are not yet supported (#35)."
)

It looks like the ModernBERT config doesn't define hidden_dropout_prob, and instead has four different dropout config attributes: attention_dropout, embedding_dropout, mlp_dropout, and classifier_dropout. There might be other changes needed to support ModernBERT, this is just the first error that surfaces when running the train system.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions