🎥 Watch our introduction video to learn more about the project!
Check out our speedrun leaderboard!
Our goals:
- GPT-1 Level by Dec 20 2025 ✓ Watch
- GPT-2 Level by Jan 20 2026
- GPT-3 Level by Feb 20 2026
- Top 150 in LMArena (GPT-4o-mini level) by April 2026
- Top 50 by Dec 2026
- Top 10 by April 2027
- We could aim for Top 1 by 2028, TBD
We will partner for compute while keeping all research/engineering/code fully open source.
Current research projects and experiments:
- Squared ReLU Research 🧪 TASK (relu branch) | Discussion
It is best to read the Quick Start directly in the tasks linked above.
git clone https://github.com/Open-Superintelligence-Lab/5-dollar-llm
cd 5-dollar-llm
pip install -r requirements.txt
python data/download_hf_data.py # Downloads 40M token subset
python train_llm.py --target_train_loss 4.5👉 Full Setup Guide | Leaderboard | Contributing Guide
We will partner with compute providers while keeping all research/engineering/code fully open source.
Potential partners include: Hugging Face, NVIDIA, Microsoft, Google, Amazon, Meta, IBM, Oracle, Alibaba, Tencent, Huawei, Baidu, CoreWeave, Lambda Labs, Hyperbolic, Stability AI, OpenAI, Anthropic, xAI, Cohere, Mistral AI, Graphcore, Tenstorrent, Intel, AMD, Dell Technologies, ai2, a16z, Sequoia Capital, and more.
If you or someone you know has extensive research experience and wants to contribute to this open source initiative, contact me.