| Paper | Documentation | Ask DeepWiki | ๐ค Models & Data |
WeChat (ๅพฎไฟก) Group |
AReaL is an open-source fully asynchronous reinforcement learning training system for large reasoning and agentic models, developed by the AReaL Team at Ant Group. Built upon the open-source project ReaLHF, we are fully committed to open-source principles by providing training details, data, and infrastructure required to reproduce our results along with the models themselves. AReaL aims to help everyone build their own AI agents easily and affordably. Our team loves milk tea because it's delicious, customizable, and affordable. We hope you enjoy our project just as you enjoy real-world milk tea (cheers).
AReaL Highlights
- โก Flexibility: Seamless customization for multi-turn agentic rollout workflows within a single file, and smooth integration with other agentic tooling frameworks.
- ๐ Scalability: Through algorithm-system co-design, AReaL delivers stable fully asynchronous RL training with industry-leading speed. AReaL seamlessly adapts to diverse computational environments, scaling from a single node to 1,000+ GPUs.
- ๐ช Cutting-Edge Performance: AReaL produces state-of-the-art math, coding, and search agents with exceptional capabilities.
[2025/08/30] Introducing ASearcher, a state-of-the-art search agent built with AReaL's end-to-end asynchronous RL training. Check out the paper and the open-source repository!
[2025/07/31] (AReaL-lite) We introduce AReaL-lite, a lightweight version of AReaL designed specifically for AI researchers and rapid prototyping. AReaL-lite features an algorithm-first API design that prioritizes ease of use and algorithm development, while natively supporting fully asynchronous agentic RL. With 80% fewer lines of code, AReaL-lite maintains 90% of AReaL's performance and core functionality. Check out our AReaL-lite design documentation and the quickstart guide to begin your journey with AReaL-lite!
๐ Previous Releases
[2025/06/03] (v0.3, bobaยฒ) We release bobaยฒ (double-boba) for fully asynchronous RL training, which achieves 2.77ร speedup while delivering comparable or superior training performance compared to synchronous systems. Furthermore, asynchronous RL significantly simplifies multi-turn agentic RL training setup! Check out our v0.3 overview blog and the research paper.
[2025/03/31] (v0.2, boba) Introducing our milestone releaseโboba! Please call it A-ReaL-boba! This release features significantly faster training with SGLang support and state-of-the-art 7B and 32B models for mathematical reasoning. Check out our v0.2 technical blog.
[2025/02/24] (v0.1) Our initial release includes reproducible results for 1.5B and 7B Large Reasoning Models (LRMs). Check out our v0.1 technical blog.
Our training scripts automatically download the required dataset (openai/gsm8k) and model (Qwen/Qwen2-1.5B-Instruct). To run on a single node:
python3 -m areal.launcher.local \
examples/math/gsm8k_grpo.py \
--config examples/math/gsm8k_grpo.yamlTo run on a Ray cluster with 2 nodes and 8 GPUs per node (remember to update paths in the YAML file to point to your shared storage):
python3 -m areal.launcher.ray \
examples/math/gsm8k_grpo.py \
--config examples/math/gsm8k_grpo.yaml \
cluster.n_nodes=2 \
cluster.n_gpus_per_node=8For comprehensive setup instructions, see our quickstart guide.
| Task | Description | Performance |
|---|---|---|
| Math | GSM8K mathematical problem solving | TBA |
| VLM Math | CLEVR visual counting tasks | TBA |
| Simple Reasoning | Countdown numbers game with custom rewards | Training Curve |
| Search Agent | An agent with end-to-end search, browsing, and summarization capabilities | ASearcher Repo |
- Customize dataset with AReaL-lite
- Customize Agentic/RVLR rollout workflows with AReaL-lite
- Customize algorithms with AReaL-lite
AReaL is under active development with planned minor releases weekly and major releases monthly. We warmly welcome community engagement and contributions. We are also actively hiring interns and full-time employees with open positions in both the US and China.
We gratefully acknowledge that major contributors are from the AReaL Team at Ant Group and the Institute for Interdisciplinary Information Sciences, Tsinghua University.
We have also received invaluable assistance from the following groups (listed alphabetically):
-
The Data Intelligence Lab at Ant Research for their data support
-
The Relaxed System Lab from HKUST for seamless collaboration on numerous system-related aspects
-
The SGLang team for supporting custom weight update features and their contributions during AReaL-lite development
-
The Super Computing Technology (SCT) team at Ant Group for their expertise in large-scale cluster operations and maintenance
-
Special thanks to @Lyken17 for providing valuable suggestions throughout our development process
We also deeply appreciate all pioneering work from the community, particularly the ReaLHF project from OpenPsi Inc. and other outstanding projects, including but not limited to DeepScaleR, Open-Reasoner-Zero, OpenRLHF, VeRL, SGLang, QwQ, Light-R1, and DAPO.
@inproceedings{mei2025real,
author = {Mei, Zhiyu and Fu, Wei and Li, Kaiwei and Wang, Guangju and Zhang, Huanchen and Wu, Yi},
title = {ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation},
booktitle = {Proceedings of the Eighth Conference on Machine Learning and Systems,
MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025},
publisher = {mlsys.org},
year = {2025},
}@misc{fu2025areal,
title={AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning},
author={Wei Fu and Jiaxuan Gao and Xujie Shen and Chen Zhu and Zhiyu Mei and Chuyi He and Shusheng Xu and Guo Wei and Jun Mei and Jiashu Wang and Tongkai Yang and Binhang Yuan and Yi Wu},
year={2025},
eprint={2505.24298},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2505.24298},
}
