Skip to content

Add support for converting RMSNorm when using transformer-engine #20804

@cyanic-selkie

Description

@cyanic-selkie

Description & Motivation

Recently, models (e.g., Qwen3/Llama) have replaced the LayerNorm layers with the RMSNorm layers because they perform identically but are slightly more efficient.

To support it, a new branch almost identical to the LayerNorm would have to be added to the conversion function.

cc @lantiga @Borda

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureIs an improvement or enhancementprecision: teTransformer Engine

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions