Skip to content

Commit 9660ab2

Browse files
authored
Merge pull request #41 from MigoXLab/feat/dataset
Feat/dataset
2 parents 0586bdd + 74d8196 commit 9660ab2

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+505202
-8359
lines changed

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
# 默认目标
44
help:
5-
@echo "LLMeter 项目管理命令:"
5+
@echo "LMeterX 项目管理命令:"
66
@echo ""
77
@echo "全局命令:"
88
@echo " help - 显示此帮助信息"

README.md

Lines changed: 59 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,32 @@
1515

1616
# LMeterX
1717

18+
## Contents
19+
- [LMeterX](#lmeterx)
20+
- [Contents](#contents)
21+
- [📋 Project Overview](#-project-overview)
22+
- [✨ Core Features](#-core-features)
23+
- [Feature Comparison](#feature-comparison)
24+
- [🏗️ System Architecture](#️-system-architecture)
25+
- [🚀 Quick Start](#-quick-start)
26+
- [Environment Checklist](#environment-checklist)
27+
- [One-Click Deployment (Recommended)](#one-click-deployment-recommended)
28+
- [Data \& Volume Layout](#data--volume-layout)
29+
- [Usage Guide](#usage-guide)
30+
- [🔧 Configuration](#-configuration)
31+
- [Database Configuration](#database-configuration)
32+
- [Resource Configuration](#resource-configuration)
33+
- [🤝 Development Guide](#-development-guide)
34+
- [Technology Stack](#technology-stack)
35+
- [Project Structure](#project-structure)
36+
- [Development Environment Setup](#development-environment-setup)
37+
- [🗺️ Development Roadmap](#️-development-roadmap)
38+
- [In Development](#in-development)
39+
- [Planned](#planned)
40+
- [📚 Related Documentation](#-related-documentation)
41+
- [👥 Contributors](#-contributors)
42+
- [📄 Open Source License](#-open-source-license)
43+
1844
## 📋 Project Overview
1945

2046
LMeterX is a professional large language model performance testing platform that can be applied to model inference services based on large model inference frameworks (such as LiteLLM, vLLM, TensorRT-LLM, LMDeploy, and others), and also supports performance testing for cloud services like Azure OpenAI, AWS Bedrock, Google Vertex AI, and other major cloud providers. Through an intuitive Web interface, users can easily create and manage test tasks, monitor testing processes in real-time, and obtain detailed performance analysis reports, providing reliable data support for model deployment and performance optimization.
@@ -26,11 +52,11 @@ LMeterX is a professional large language model performance testing platform that
2652
## ✨ Core Features
2753

2854
- **Universal Framework Support** - Compatible with mainstream inference frameworks (vLLM, LiteLLM, TensorRT-LLM) and cloud services (Azure, AWS, Google Cloud)
29-
- **Full Model Compatibility** - Supports mainstream LLMs like GPT, Claude, and Llama with one-click stress testing
55+
- **Full Model Compatibility** - Supports mainstream LLMs like GPT, Claude, and Llama, also supports large document parsing models such as [MinerU](https://github.com/opendatalab/MinerU) and [dots.ocr](https://github.com/rednote-hilab/dots.ocr).
3056
- **High-Load Stress Testing** - Simulates high-concurrency requests to accurately detect model performance limits
31-
- **Multi-Scenario Coverage** - Supports streaming/non-streaming, supports text/multimodal/custom datasets<sup>![NEW](https://img.shields.io/badge/NEW-00C851?style=flat&labelColor=transparent)</sup>
32-
- **Professional Metrics** - Core performance metrics including first token latency, throughput(RPS、TPS), and success rate
33-
- **AI Smart Reports** - AI-powered performance analysis<sup>![NEW](https://img.shields.io/badge/NEW-00C851?style=flat&labelColor=transparent)</sup>, multi-dimensional model comparison and visualization
57+
- **Multi-Scenario Coverage** &nbsp;<img src="docs/images/badge-new.svg" alt="NEW" height="16" /> - Supports streaming/non-streaming, supports text/multimodal/custom datasets
58+
- **Professional Metrics** - Core performance metrics including first token latency, throughput(RPS、TPS), and success rate
59+
- **AI Smart Reports** &nbsp;<img src="docs/images/badge-new.svg" alt="NEW" height="16" /> - AI-powered performance analysis, multi-dimensional model comparison and visualization
3460
- **Web Console** - One-stop management for task creation, stopping, status tracking, and full-chain log monitoring
3561
- **Enterprise-level Deployment** - Docker containerization with elastic scaling and distributed deployment support
3662

@@ -58,63 +84,62 @@ LMeterX adopts a microservices architecture design, consisting of four core comp
5884

5985
## 🚀 Quick Start
6086

61-
### Environment Requirements
62-
- Docker 20.10.0+
63-
- Docker Compose 2.0.0+
64-
- At least 4GB available memory
65-
- At least 5GB available disk space
66-
67-
### One-Click Deployment (Recommended)
87+
### Environment Checklist
88+
- Docker 20.10.0+ with the daemon running
89+
- Docker Compose 2.0.0+ (`docker compose` plugin or standalone `docker-compose`)
90+
- At least 4GB free memory and 5GB disk space
6891

69-
> **Complete Deployment Guide**: See [Complete Deployment Guide](docs/DEPLOYMENT_GUIDE.md) for detailed instructions on all deployment methods
92+
> **Need more deployment options?** See the [Complete Deployment Guide](docs/DEPLOYMENT_GUIDE.md) for Kubernetes, air-gapped installs, and advanced tuning.
7093
71-
Use pre-built Docker images to start all services with one click:
94+
### One-Click Deployment (Recommended)
7295

7396
```bash
74-
# Download and run one-click deployment script
97+
# Download and run the one-click deployment script
7598
curl -fsSL https://raw.githubusercontent.com/MigoXLab/LMeterX/main/quick-start.sh | bash
7699
```
77100

78-
### Multi-Instance Deployment (Supports Concurrent Testing Tasks)
101+
After the script finishes:
102+
- Check container health: `docker compose ps`
103+
- Tail logs if needed: `docker compose logs -f`
104+
- Scale services (if needed): `docker compose up -d --scale backend=2 --scale engine=2`
105+
- Open the web UI at http://localhost:8080 (see [Usage Guide](#usage-guide))
79106

80-
```bash
81-
# Download the deployment file docker-compose.yml
82-
curl -fsSL -o docker-compose.yml https://raw.githubusercontent.com/MigoXLab/LMeterX/main/docker-compose.yml
83-
# Start multiple instances using the --scale
84-
# Start 2 backends + 2 engines (the number can be adjusted as needed)
85-
docker compose up -d --scale backend=2 --scale engine=2
86-
```
107+
### Data & Volume Layout
108+
- `./data` → mounted to `/app/data` in the `engine` service (large datasets are **not** baked into the image)
109+
- `./logs` → shared log output for backend and engine
110+
- `./upload_files` → user-supplied payloads and exported reports
111+
112+
For custom data, please refer to the [Dataset Usage Guide](docs/DATASET_GUIDE.md).
87113

88114
### Usage Guide
89115

90116
1. **Access Web Interface**: Open http://localhost:8080
91-
2. **Create Test Task**: Navigate to Test Tasks → Create Task, configure LLM API request information, test data, and request-response field mapping
92-
- 2.1 Basic Information: For `/chat/completions` API, you only need to configure API path, model, and response mode. You can also supplement the complete payload in request parameters.
93-
- 2.2 Data Payload: Select built-in text datasets/multimodal datasets as needed, or upload custom JSONL data files.
94-
- 2.3 Field Mapping: Configure the prompt field path in payload, and response data paths for model output content, reasoning_content fields, usage fields, etc. This field mapping is crucial for updating request parameters with datasets and correctly parsing streaming/non-streaming responses.
95-
3. **API Testing**: In Test Tasks → Create Task, click the "Test" button in the Basic Information panel to quickly test API connectivity
96-
**Note**: For quick API response, it's recommended to use simple prompts when testing API connectivity.
117+
2. **Create Test Task**: Navigate to Test Tasks → Create Task, configure API request information, test data, and request/response field mappings.
118+
- 2.1 Basic Information: For OpenAI-like and Claude-like APIs, you only need to configure API path, model, and response mode. You can also supplement the complete payload in request parameters.
119+
- 2.2 Data & load: Select the dataset type, concurrency, load testing time, etc., as needed.
120+
- 2.3 Field Mapping: For custom APIs, you need to configure the prompt field path in payload, and response data paths for model output fields, usage fields, etc. This field mapping is crucial for updating request parameters with datasets and correctly parsing streaming/non-streaming responses.
121+
> 💡 **Tip**: For custom multimodal dataset load tests, follow the [Dataset Guide](docs/DATASET_GUIDE.md) for data preparation, mounting, and troubleshooting.
122+
3. **API Testing**: In Test Tasks → Create Task, click the "Test" button in the Basic Information panel to quickly test API connectivity (use a lightweight prompt for faster feedback).
97123
4. **Real-time Monitoring**: Navigate to Test Tasks → Logs/Monitoring Center to view full-chain test logs and troubleshoot exceptions
98124
5. **Result Analysis**: Navigate to Test Tasks → Results to view detailed performance results and export reports
99125
6. **Result Comparison**: Navigate to Model Arena to select multiple models or versions for multi-dimensional performance comparison
100126
7. **AI Analysis**: In Test Tasks → Results/Model Arena, after configuring AI analysis service, support intelligent performance evaluation for single/multiple tasks
101127

102128
## 🔧 Configuration
103129

104-
### Environment Variable Configuration
130+
### Database Configuration
105131

106-
#### General Configuration
107132
```bash
108133
# ================= Database Configuration =================
109134
DB_HOST=mysql # Database host (container name or IP)
110135
DB_PORT=3306 # Database port
111136
DB_USER=lmeterx # Database username
112137
DB_PASSWORD=lmeterx_password # Database password (use secrets management in production)
113138
DB_NAME=lmeterx # Database name
139+
```
114140

115-
# ================= Frontend Configuration =================
116-
VITE_API_BASE_URL=/api # Base API URL for frontend requests (supports reverse proxy)
117-
141+
### Resource Configuration
142+
```bash
118143
# ================= High-Concurrency Load Testing Deployment Requirements =================
119144
# When concurrent users exceed this threshold, the system will automatically enable multi-process mode (requires multi-core CPU support)
120145
MULTIPROCESS_THRESHOLD=1000
@@ -132,7 +157,6 @@ deploy:
132157
limits:
133158
cpus: '2.0' # Recommended minimum: 2 CPU cores (4+ cores recommended for high-concurrency scenarios)
134159
memory: 2G # Memory limit — adjust based on actual load (minimum recommended: 2G)
135-
136160
```
137161

138162
## 🤝 Development Guide
@@ -182,6 +206,7 @@ LMeterX/
182206

183207
- [Deployment Guide](docs/DEPLOYMENT_GUIDE.md) - Detailed deployment instructions and configuration guide
184208
- [Contributing Guide](docs/CONTRIBUTING.md) - How to participate in project development and contribute code
209+
- [Dataset Guide](docs/DATASET_GUIDE.md) - Instructions for preparing and using custom multimodal datasets
185210

186211
## 👥 Contributors
187212

README_CN.md

Lines changed: 63 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,31 @@
1515

1616
# LMeterX
1717

18+
## 内容导航
19+
- [LMeterX](#lmeterx)
20+
- [内容导航](#内容导航)
21+
- [📋 项目简介](#-项目简介)
22+
- [✨ 核心特性](#-核心特性)
23+
- [工具对比](#工具对比)
24+
- [🏗️ 系统架构](#️-系统架构)
25+
- [🚀 快速开始](#-快速开始)
26+
- [环境检查清单](#环境检查清单)
27+
- [一键部署(推荐)](#一键部署推荐)
28+
- [数据目录与挂载说明](#数据目录与挂载说明)
29+
- [使用指南](#使用指南)
30+
- [🔧 配置说明](#-配置说明)
31+
- [数据库配置](#数据库配置)
32+
- [资源配置](#资源配置)
33+
- [🤝 开发指南](#-开发指南)
34+
- [技术栈](#技术栈)
35+
- [开发环境搭建](#开发环境搭建)
36+
- [🗺️ 发展路线图](#️-发展路线图)
37+
- [开发中](#开发中)
38+
- [规划中](#规划中)
39+
- [📚 相关文档](#-相关文档)
40+
- [👥 贡献者](#-贡献者)
41+
- [📄 开源许可](#-开源许可)
42+
1843
## 📋 项目简介
1944

2045
LMeterX 是一个专业的大语言模型性能测试平台,支持基于大模型推理框架(如 LiteLLM、vLLM、TensorRT-LLM、LMDeploy 等)的模型推理服务,同时也支持对 Azure OpenAI、AWS Bedrock、Google Vertex AI 等主流云服务进行性能测试。通过直观的 Web 界面,可以轻松创建和管理测试任务,实时监控测试过程,并获得详细的性能分析报告,为模型部署和性能优化提供可靠的数据支撑。
@@ -27,11 +52,11 @@ LMeterX 是一个专业的大语言模型性能测试平台,支持基于大模
2752
## ✨ 核心特性
2853

2954
- **通用框架支持** - 兼容主流推理框架(vLLM、LiteLLM、TensorRT-LLM)和云服务(Azure、AWS、Google Cloud)
30-
- **全模型兼容** - 支持 GPT、Claude、Llama 等主流大模型,一键发起压测
55+
- **全模型兼容** - 支持 GPT、Claude、Llama 等主流大模型,也支持[MinerU](https://github.com/opendatalab/MinerU)[dots.ocr](https://github.com/rednote-hilab/dots.ocr)等文档解析大模型
3156
- **高负载压测** - 模拟高并发请求,精准探测模型性能极限
32-
- **多场景覆盖** - 支持流式/非流式、文本/多模态/自定义数据集<sup>![NEW](https://img.shields.io/badge/NEW-00C851?style=flat&labelColor=transparent)</sup>
57+
- **多场景覆盖**&nbsp;<img src="docs/images/badge-new.svg" alt="NEW" height="16" /> - 支持流式/非流式、文本/多模态/自定义数据集
3358
- **专业指标统计** - 首Token延迟、吞吐量(RPS、TPS)、成功率等核心性能指标
34-
- **AI智能报告** - AI智能分析报告<sup>![NEW](https://img.shields.io/badge/NEW-00C851?style=flat&labelColor=transparent)</sup>,多维度多模型可视化结果对比
59+
- **AI智能报告**&nbsp;<img src="docs/images/badge-new.svg" alt="NEW" height="16" /> - 提供 AI 智能分析报告,多维度多模型可视化结果对比
3560
- **Web控制台** - 提供任务创建、停止、状态跟踪、全链路日志监控等一站式管理
3661
- **企业级部署** - Docker容器化,支持弹性扩展与分布式部署
3762

@@ -60,61 +85,61 @@ LMeterX 采用微服务架构,由四个核心组件构成:
6085

6186
## 🚀 快速开始
6287

63-
### 环境要求
64-
- Docker 20.10.0+
65-
- Docker Compose 2.0.0+
66-
- 至少 4GB 可用内存
67-
- 至少 5GB 可用磁盘空间
68-
69-
### 一键部署(推荐)
88+
### 环境检查清单
89+
- Docker 20.10.0+(确保 Docker 守护进程已启动)
90+
- Docker Compose 2.0.0+(支持 `docker compose``docker-compose`
91+
- 至少 4GB 可用内存、5GB 磁盘空间
7092

71-
> **完整部署指南**:查看 [完整部署指南](docs/DEPLOYMENT_GUIDE_CN.md) 了解所有部署方式的详细说明
93+
> **需要更多部署方式?** 请查阅 [完整部署指南](docs/DEPLOYMENT_GUIDE_CN.md),获取 Kubernetes、离线环境等高级方案。
7294
73-
使用预构建的 Docker 镜像,一键启动所有服务:
95+
### 一键部署(推荐)
7496

7597
```bash
76-
# 一键启动所有服务(默认各服务 1个实例)
98+
# 默认使用预构建镜像启动全套服务
7799
curl -fsSL https://raw.githubusercontent.com/MigoXLab/LMeterX/main/quick-start.sh | bash
78100
```
79-
### 多实例部署(支持并发压测任务)
80101

81-
```bash
82-
# 下载部署文件 docker-compose.yml
83-
curl -fsSL -o docker-compose.yml https://raw.githubusercontent.com/MigoXLab/LMeterX/main/docker-compose.yml
84-
# 使用 --scale 启动多实例
85-
# 启动 2 个 backend + 2 个 engine(可根据需要调整数量)
86-
docker compose up -d --scale backend=2 --scale engine=2
87-
```
102+
启动完成后可执行:
103+
- `docker compose ps` 查看容器状态
104+
- `docker compose logs -f` 追踪实时日志
105+
- `docker compose up -d --scale backend=2 --scale engine=2` 服务扩容(如需)
106+
- 在浏览器打开 http://localhost:8080(详见下方「使用指南」)
107+
108+
### 数据目录与挂载说明
109+
- `./data` → 挂载到 `engine` 容器的 `/app/data`(大规模数据集不会打包进镜像,便于更新)
110+
- `./logs` → 后端与压测引擎的统一日志输出目录
111+
- `./upload_files` → 用户上传的自定义文件及导出的报表
112+
113+
如需准备自定义数据,请参考 [数据集使用指南](docs/DATASET_GUIDE.md)
88114

89115
### 使用指南
90116

91117
1. **访问界面**: 打开 http://localhost:8080
92-
2. **创建任务**: 导航至 测试任务 → 创建任务,配置LLM API请求信息、测试数据以及请求响应字段映射
93-
- 2.1 基础信息: 对于 `/chat/completions` API只需要配置API路径、模型以及响应模式即可,也支持在请求参数中补充完整payload
94-
- 2.2 数据负载: 根据需要选择内置的纯文本数据集/多模态数据集,也支持自定义JSONL数据或文件等
95-
- 2.3 字段映射: 配置payload中prompt对应字段路径,以及响应数据中对应模型输出的content、reasoning_content字段路径、usage字段路径等,此字段映射对于使用数据集更新请求参数和正确解析流式/非流式响应至关重要,请注意仔细填写
96-
3. **API测试**: 在 测试任务 → 创建任务,点击基础信息面板的"测试"按钮,快速测试API连通性
97-
**注意**: 为快速得到API响应,建议使用简单的prompt测试API连通性
98-
4. **实时监控**: 导航至 测试任务 → 日志/监控中心,查看全链路测试日志,排查异常
99-
5. **结果分析**: 导航至 测试任务 → 结果,查看详细性能结果,导出报告
100-
6. **结果对比**: 导航至 模型擂台,选择多个模型或版本进行多维度性能对比
101-
7. **AI分析**: 在 测试任务 → 结果/模型擂台,配置AI分析服务后,支持对单个/多个任务进行智能性能评估
118+
2. **创建任务**: 导航至 测试任务 → 创建任务,配置 API 请求信息、测试数据以及请求响应字段映射
119+
- 2.1 基础信息: 对于 OpenAI-like 和 Claude-like API 只需填写 API 路径、模型与响应模式,也可在请求参数中补充完整 payload
120+
- 2.2 数据&负载: 根据需要选择数据集类型、并发数、压测时间等
121+
- 2.3 字段映射: 针对自定义 API 需要配置 payload 中 prompt 对应字段路径,以及响应数据中 content、reasoning_content、usage 等字段路径;该映射对于使用压测数据集和解析流式响应尤为关键
122+
> 💡 **提示**: 若需自定义图文数据集压测,请参考 [数据集使用指南](docs/DATASET_GUIDE.md) 了解数据准备、挂载与常见问题排查。
123+
3. **API 测试**: 在 测试任务 → 创建任务,点击基础信息面板的「测试」按钮,快速验证接口连通性(建议使用简短 prompt)
124+
4. **实时监控**: 访问 测试任务 → 日志/监控中心,查看全链路测试日志,快速定位异常
125+
5. **结果分析**: 进入 测试任务 → 结果,查看详细性能指标并导出报告
126+
6. **结果对比**: 模型擂台 模块选择多个模型/版本,进行多维度性能对比
127+
7. **AI 分析**: 在 测试任务 → 结果/模型擂台 中配置 AI 分析服务后,可对单个或多任务进行智能评估
102128

103129
## 🔧 配置说明
104130

105-
### 环境变量
106-
131+
### 数据库配置
107132
```bash
108-
=== 数据库配置 ===
133+
=== 数据库配置 ===
109134
DB_HOST=mysql
110135
DB_PORT=3306
111136
DB_USER=lmeterx
112137
DB_PASSWORD=lmeterx_password
113138
DB_NAME=lmeterx
139+
```
114140

115-
=== 前端配置 ===
116-
VITE_API_BASE_URL=/api
117-
141+
### 资源配置
142+
```bash
118143
=== 高并发压测 部署要求 ===
119144
# 当并发用户数超过此阈值,系统将自动启用多进程模式(需多核 CPU 支持)
120145
MULTIPROCESS_THRESHOLD: 1000
@@ -141,7 +166,6 @@ deploy:
141166
- **前端** - React + TypeScript + Ant Design + Vite
142167
- **部署** - Docker + Docker Compose + Nginx
143168

144-
145169
```
146170
LMeterX/
147171
├── backend/ # 后端服务
@@ -172,6 +196,7 @@ LMeterX/
172196

173197
- [部署指南](docs/DEPLOYMENT_GUIDE_CN.md) - 详细部署说明
174198
- [贡献指南](docs/CONTRIBUTING.md) - 参与开发指南
199+
- [数据集使用指南](docs/DATASET_GUIDE.md) - 自定义图文数据集准备与使用说明
175200

176201
## 👥 贡献者
177202

backend/app.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,8 @@
1717
from middleware.db_middleware import DBSessionMiddleware
1818

1919
app = FastAPI(
20-
title="LLMeter Backend API",
21-
description="LLMeter Backend",
20+
title="LMeterX Backend API",
21+
description="LMeterX Backend",
2222
version="1.0.0",
2323
)
2424

@@ -44,7 +44,7 @@ def health_check():
4444
@app.get("/")
4545
def read_root():
4646
"""Root endpoint."""
47-
return {"message": "LLMeter Backend API"}
47+
return {"message": "LMeterX Backend API"}
4848

4949

5050
# add api routers

0 commit comments

Comments
 (0)