Skip to content

Feature Request: Add OLMoEΒ #9317

@natolambert

Description

@natolambert

Prerequisites

  • I am running the latest code. Mention the version if possible as well.
  • I carefully followed the README.md.
  • I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Add this model (and other variants) https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct

Motivation

We recently released the OLMoE model at Ai2. 1.3b active / 6.9b total param MoE model. Seems solid, and we'd love people to use it.

Possible Implementation

Should be able to quickly use mix of existing OLMo implementation + Transformers version https://github.com/huggingface/transformers/blob/main/src/transformers/models/olmoe/modeling_olmoe.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions