Refine

Author: yanxi-chen

README.md

@@ -62,10 +62,55 @@ It is designed to support diverse application scenarios and serve as a unified p
 
 <p align="center">
   <img src="https://img.alicdn.com/imgextra/i2/O1CN01H3UbpF1yP7E1OCLbi_!!6000000006570-2-tps-1334-638.png" alt="Trinity-RFT">
-  <em>Fig: The high-level design of Trinity-RFT</em>
+  <em>Figure: The high-level design of Trinity-RFT</em>
 </p>
 
 
+<details>
+<summary>Figure: The architecture of RFT-core</summary>
+
+
+<p align="center">
+  <img src="https://img.alicdn.com/imgextra/i1/O1CN01BFCZRV1zS9T1PoH49_!!6000000006712-2-tps-922-544.png" alt="Trinity-RFT-core-architecture">
+</p>
+
+</details>
+
+
+<details>
+<summary>Figure: Some RFT modes supported by Trinity-RFT</summary>
+
+<p align="center">
+  <img src="https://img.alicdn.com/imgextra/i3/O1CN01E7NskS1FFoTI9jlaQ_!!6000000000458-2-tps-1458-682.png" alt="Trinity-RFT-modes">
+</p>
+
+
+</details>
+
+
+<details>
+<summary>Figure: The architecture of data processors</summary>
+
+<p align="center">
+  <img src="https://img.alicdn.com/imgextra/i3/O1CN01hR1LCh25kpJMKmYR4_!!6000000007565-2-tps-1474-740.png" alt="Trinity-RFT-data-pipeline-buffer">
+</p>
+
+</details>
+
+
+<details>
+<summary>Figure: The high-level design of data pipelines in Trinity-RFT</summary>
+
+<p align="center">
+  <img src="https://img.alicdn.com/imgextra/i4/O1CN01UvyfcZ1WoTv5t3pCp_!!6000000002835-2-tps-1166-274.png" alt="Trinity-RFT-data-pipelines">
+</p>
+
+</details>
+
+
+
+
+
 
 ## 🛠️ What can I use Trinity-RFT for?
 
@@ -84,23 +129,23 @@ It is designed to support diverse application scenarios and serve as a unified p
 
 * **Low-Code Usage:**
 
-  Use graphical interfaces for easy monitoring and tracking of the learning process.
+  Use graphical interfaces for easy monitoring and tracking of the learning process. 
 
 
 ---
 
 ## Table of contents
 
+
 - [Getting started](#getting-started)
   - [Step 1: preparations](#step-1-preparations)
   - [Step 2: prepare dataset and model](#step-2-prepare-dataset-and-model)
   - [Step 3: configurations](#step-3-configurations)
   - [Step 4: run the RFT process](#step-4-run-the-rft-process)
-- [Further examples](#further-examples)
+- [Further tutorials](#further-tutorials)
 - [Documentation](#documentation)
   - [Advanced usage and full configurations](#advanced-usage-and-full-configurations)
   - [Programming guide for developers](#programming-guide-for-developers)
-  - [Details: design and implementations](#details-design-and-implementations)
 - [Upcoming features](#upcoming-features)
 - [Contribution guide](#contribution-guide)
 - [Acknowledgements](#acknowledgements)
@@ -109,6 +154,7 @@ It is designed to support diverse application scenarios and serve as a unified p
 
 
 
+
 ## Getting started
 
 
@@ -119,7 +165,7 @@ It is designed to support diverse application scenarios and serve as a unified p
 ### Step 1: preparations
 
 
-Installation from source (recommended):
+**Installation from source (recommended):**
 
 ```shell
 # Pull the source code from GitHub
@@ -151,13 +197,13 @@ pip install -e .\[flash_attn\]
 # pip install flash-attn -v --no-build-isolation
 ```
 
-Installation using pip:
+**Installation using pip:**
 
 ```shell
 pip install trinity-rft==0.2.0
 ```
 
-Installation from docker:
+**Installation from docker:**
 we have provided a dockerfile for Trinity-RFT (trinity)
 
 ```shell
@@ -174,7 +220,7 @@ docker run -it --gpus all --shm-size="64g" --rm -v $PWD:/workspace -v <root_path
 ```
 
 
-Trinity-RFT requires
+**Requirements:**
 Python version >= 3.10,
 CUDA version >= 12.4,
 and at least 2 GPUs.
@@ -196,7 +242,7 @@ huggingface-cli download {model_name} --local-dir $MODEL_PATH/{model_name}
 modelscope download {model_name} --local_dir $MODEL_PATH/{model_name}
 ```
 
-For more details about model downloading, please refer to [Huggingface](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli) or  [ModelScope](https://modelscope.cn/docs/models/download).
+For more details about model downloading, see [Huggingface](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli) or  [ModelScope](https://modelscope.cn/docs/models/download).
 
 
 
@@ -210,29 +256,31 @@ huggingface-cli download {dataset_name} --repo-type dataset --local-dir $DATASET
 modelscope download --dataset {dataset_name} --local_dir $DATASET_PATH/{dataset_name}
 ```
 
-For more details about dataset downloading, please refer to [Huggingface](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli#download-a-dataset-or-a-space) or [ModelScope](https://modelscope.cn/docs/datasets/download).
+For more details about dataset downloading, see [Huggingface](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli#download-a-dataset-or-a-space) or [ModelScope](https://modelscope.cn/docs/datasets/download).
 
 
 
 ### Step 3: configurations
 
 
-For convenience, Trinity-RFT provides a web interface for configuring your RFT process.
+Trinity-RFT provides a web interface for configuring your RFT process.
 
 > [!NOTE]
 > This is an experimental feature, and we will continue to improve it.
 
 
 To enable *minimal* features (mainly for trainer), you can run
+
 ```bash
 trinity studio --port 8080
 ```
+
 Then you can configure your RFT process in the web page and generate a config file. You can save the config for later use or run it directly as described in the following section.
 
-Advanced users can also configure the RFT process by editing the config file directly.
-We provide a set of example config files in [`examples`](examples/).
+Advanced users can also edit the config file directly.
+We provide example config files in [`examples`](examples/).
 
-To enable *complete* visualization features, please refer to the monorepo for [Trinity-Studio](https://github.com/modelscope/Trinity-Studio).
+For *complete* GUI features, please refer to the monorepo for [Trinity-Studio](https://github.com/modelscope/Trinity-Studio).
 
 
 <details>
@@ -250,7 +298,7 @@ To enable *complete* visualization features, please refer to the monorepo for [T
 ### Step 4: run the RFT process
 
 
-First, start a ray cluster with the following command:
+Start a ray cluster:
 
 ```shell
 # On master node
@@ -260,35 +308,36 @@ ray start --head
 ray start --address=<master_address>
 ```
 
-Optionally, we can login into [wandb](https://docs.wandb.ai/quickstart/) to better monitor the RFT process:
+(Optional) Log in to [wandb](https://docs.wandb.ai/quickstart/) for better monitoring:
 
 ```shell
 export WANDB_API_KEY=<your_api_key>
 wandb login
 ```
 
-Then, for command-line users, run the RFT process with the following command:
+For command-line users, run the RFT process:
 
 ```shell
 trinity run --config <config_path>
 ```
 
-For example, below is the command for fine-tuning Qwen2.5-1.5B-Instruct on GSM8k dataset using GRPO algorithm:
+For example, below is the command for fine-tuning Qwen2.5-1.5B-Instruct on GSM8k with GRPO:
+
 ```shell
 trinity run --config examples/grpo_gsm8k/gsm8k.yaml
 ```
 
-For studio users, just click the "Run" button in the web page.
+For studio users, click "Run" in the web interface.
 
 
 ## Further tutorials
 
 
 Tutorials for running different RFT modes:
 
-+ [A quick example with GRPO and GSM8k](./docs/sphinx_doc/source/tutorial/example_reasoning_basic.md)
-+ [Off-policy mode of RFT](./docs/sphinx_doc/source/tutorial/example_reasoning_advanced.md)
-+ [Fully asynchronous mode of RFT](./docs/sphinx_doc/source/tutorial/example_async_mode.md)
++ [Quick example: GRPO on GSM8k](./docs/sphinx_doc/source/tutorial/example_reasoning_basic.md)
++ [Off-policy RFT](./docs/sphinx_doc/source/tutorial/example_reasoning_advanced.md)
++ [Fully asynchronous RFT](./docs/sphinx_doc/source/tutorial/example_async_mode.md)
 + [Offline learning by DPO or SFT](./docs/sphinx_doc/source/tutorial/example_dpo.md)
 
 
@@ -321,8 +370,6 @@ Please refer to [this document](./docs/sphinx_doc/source/tutorial/trinity_config
 
 
 
-
-
 ### Programming guide for developers
 
 
@@ -331,40 +378,41 @@ Please refer to [this document](./docs/sphinx_doc/source/tutorial/trinity_progra
 
 
 
-### Details: design and implementations
+<!-- ### Details: design and implementations -->
 
 <!--
+[TBC]
 **The architecture of RFT-core** is shown below, demonstrating the interplay between the explorer, buffer and trainer:  -->
 
-<p align="center">
+<!-- <p align="center">
   <img src="https://img.alicdn.com/imgextra/i1/O1CN01BFCZRV1zS9T1PoH49_!!6000000006712-2-tps-922-544.png" alt="Trinity-RFT-core-architecture">
   <em>Fig: The architecture of RFT-core</em>
-</p>
+</p> -->
 
 <!-- ![](./docs/sphinx_doc/assets/trinity-architecture.png) -->
 
 
-<p align="center">
+<!-- <p align="center">
   <img src="https://img.alicdn.com/imgextra/i3/O1CN01E7NskS1FFoTI9jlaQ_!!6000000000458-2-tps-1458-682.png" alt="Trinity-RFT-modes">
   <em>Fig: Some RFT modes supported by Trinity-RFT</em>
-</p>
+</p> -->
 
 <!-- ![](./docs/sphinx_doc/assets/trinity-mode.png) -->
 
 
 
-<p align="center">
+<!-- <p align="center">
   <img src="https://img.alicdn.com/imgextra/i3/O1CN01hR1LCh25kpJMKmYR4_!!6000000007565-2-tps-1474-740.png" alt="Trinity-RFT-data-pipeline-buffer">
   <em>Fig: The architecture of data processors</em>
-</p>
+</p> -->
 
 <!-- ![](./docs/sphinx_doc/assets/trinity-data-pipeline-buffer.png) -->
 
 
-<p align="center">
+<!-- <p align="center">
   <img src="https://img.alicdn.com/imgextra/i4/O1CN01UvyfcZ1WoTv5t3pCp_!!6000000002835-2-tps-1166-274.png" alt="Trinity-RFT-data-pipelines">
   <em>Fig: The high-level design of data pipelines in Trinity-RFT</em>
-</p>
+</p> -->