-
Notifications
You must be signed in to change notification settings - Fork 13.4k
Closed
Labels
Description
Prerequisites
- I am running the latest code. Mention the version if possible as well.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
Add this model (and other variants) https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct
Motivation
We recently released the OLMoE model at Ai2. 1.3b active / 6.9b total param MoE model. Seems solid, and we'd love people to use it.
Possible Implementation
Should be able to quickly use mix of existing OLMo implementation + Transformers version https://github.com/huggingface/transformers/blob/main/src/transformers/models/olmoe/modeling_olmoe.py
Nacholmo, brittlewis12, suncloudsmoon, davanstrien, Dampfinchen and 13 more