Skip to content

Commit a267ab4

Browse files
docs: Comprehensive MTL validation framework documentation improvements
- Add interactive paths and clear directory requirements for all commands - Create validation_quickstart.md for rapid setup with step-by-step guide - Enhance validation_framework.md with root user requirements and device options - Add gen_frames.sh usage documentation with supported formats - Include specific test parameter examples and VF creation instructions - Update tests/validation/README.md with clickable config file links - Add comprehensive troubleshooting and multiple device specification methods - Improve documentation hierarchy following repository standards Addresses developer feedback for clearer setup instructions and eliminates common configuration issues that block new developers.
1 parent f9b22e3 commit a267ab4

File tree

4 files changed

+202
-18
lines changed

4 files changed

+202
-18
lines changed

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,3 +108,5 @@ tools/readpcap/readpcap_*
108108
swig
109109
openh264
110110
level-zero-*
111+
112+
venv*

doc/validation_framework.md

Lines changed: 85 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
The Media Transport Library (MTL) Validation Framework provides comprehensive testing capabilities for various aspects of the MTL, including protocol compliance, performance, and integration testing.
44

5+
> **🚀 Quick Start**: For rapid setup, see [Validation Quick Start Guide](validation_quickstart.md)
6+
57
## Overview
68

79
The validation framework uses pytest to organize and execute tests across various scenarios, protocols, and backend implementations. It supports both automated testing in CI/CD environments and manual testing for development and troubleshooting.
@@ -32,13 +34,37 @@ The `common/` directory contains shared utilities that provide fundamental funct
3234
- **Integrity Tools**: Provides functions for data integrity verification between source and received media
3335
- **Network Interface Control**: Manages network interfaces required for testing
3436

37+
#### gen_frames.sh
38+
39+
A shell script for generating test frames for video testing:
40+
41+
- Creates test patterns in various formats
42+
- Supports different resolutions and frame rates
43+
- Configurable color patterns and test signals
44+
45+
**Usage**:
46+
```bash
47+
cd tests/validation/common
48+
./gen_frames.sh
49+
```
50+
51+
**Supported Formats**:
52+
- Resolutions: 3840x2160, 1920x1080, 1280x720, 640x360
53+
- Pixel formats: yuv422p, yuv422p10le
54+
- Custom color patterns and test signals with timestamps
55+
- Configurable frame rates and durations
56+
3557
### Configuration Files
3658

3759
The `configs/` directory contains YAML files that specify:
3860

3961
- **Test Environment Settings**: Hardware specifications, media paths, and test parameters
4062
- **Network Topology**: Interface configuration, IP addressing, and routing information
4163

64+
#### [test_config.yaml](../tests/validation/configs/test_config.yaml)
65+
66+
Defines the test execution environment:
67+
4268
### MTL Engine
4369

4470
The `mtl_engine/` directory contains the core components of the framework:
@@ -106,20 +132,26 @@ For complete build instructions, see [doc/build.md](build.md).
106132

107133
### Environment Setup
108134

109-
1. Create and activate a virtual environment:
135+
> **⚠️ IMPORTANT**: Run all commands in the `tests/validation/` directory
136+
137+
1. Create and activate a Python virtual environment:
110138

111139
```bash
112-
cd tests/validation
140+
cd tests/validation # Must be in this directory!
113141
python3 -m venv venv
114142
source venv/bin/activate
115143
```
116144

117145
**Note**: If you're using VS Code or other development tools that auto-configure Python environments, ensure you're using the correct Python interpreter. The tests require the packages from `tests/validation/requirements.txt`.
118146

119-
2. Install dependencies:
147+
2. Install required dependencies:
120148

121149
```bash
150+
# Main framework requirements (run in tests/validation/)
122151
pip install -r requirements.txt
152+
153+
# Additional integrity test components (optional but recommended)
154+
pip install -r common/integrity/requirements.txt
123155
```
124156

125157
Verify installation:
@@ -131,7 +163,7 @@ python -m pytest --version
131163

132164
#### Critical Configuration Steps
133165

134-
1. **Update `configs/topology_config.yaml`** with your actual network interface details:
166+
1. **Update [`configs/topology_config.yaml`](../tests/validation/configs/topology_config.yaml)** with your actual network interface details:
135167

136168
```yaml
137169
---
@@ -149,14 +181,25 @@ hosts:
149181
connection_type: SSHConnection
150182
connection_options:
151183
port: 22
152-
username: root # Update with your username
184+
username: root # ⚠️ MUST be root for MTL validation
153185
password: None # Use key-based auth when possible
154-
key_path: /home/user/.ssh/id_rsa # Update path to your SSH key
186+
key_path: /root/.ssh/id_rsa # Update path to your SSH key
155187
```
156188
157-
**To find your PCI device ID**: `lspci | grep Ethernet`
189+
**Device Specification Options**:
190+
- **PCI device ID** (recommended): Find with `lspci | grep Ethernet` → use format like "0000:18:00.0"
191+
- **System interface name**: Find with `ip link show` → use format like "enp24s0f0"
192+
193+
**To find your options**:
194+
```bash
195+
# Find PCI device IDs
196+
lspci | grep Ethernet
197+
198+
# Find system interface names
199+
ip link show
200+
```
158201

159-
2. **Update `configs/test_config.yaml`** with your environment paths:
202+
2. **Update [`configs/test_config.yaml`](../tests/validation/configs/test_config.yaml)** with your environment paths:
160203

161204
```yaml
162205
build: /path/to/Media-Transport-Library/ # Update to your MTL root directory
@@ -182,14 +225,35 @@ ramdisk:
182225
- Set `media_path` to where your test media files are located
183226
- Ensure the paths exist and are accessible
184227

228+
#### Optional: Create VFs for Advanced Testing
229+
230+
For NIC testing with Virtual Functions:
231+
232+
```bash
233+
# First, identify your network devices
234+
lspci | grep Ethernet
235+
236+
# Create VFs (replace with your actual PCI device IDs or interface names)
237+
sudo ./script/nicctl.sh create_vf "0000:18:00.0" # Replace with your primary port
238+
sudo ./script/nicctl.sh create_vf "0000:18:00.1" # Replace with your secondary port
239+
```
240+
241+
**Examples of valid identifiers**:
242+
- PCI device ID: `"0000:18:00.0"`
243+
- Interface name: `"enp24s0f0"`
244+
- Environment variables: `"${TEST_PF_PORT_P}"` (if you set them)
245+
185246
## Running Tests
186247

187-
### Basic Usage
248+
> **⚠️ CRITICAL**: Tests must be run as **root user**, not regular user. MTL validation framework requires root privileges for network operations.
249+
250+
### Basic Test Execution
188251

189252
Run all tests:
190253

191254
```bash
192255
cd tests/validation
256+
source venv/bin/activate # Activate virtual environment
193257
python3 -m pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml
194258
```
195259

@@ -199,6 +263,18 @@ Run smoke tests:
199263
python3 -m pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml -m smoke
200264
```
201265

266+
### Running Specific Tests with Parameters
267+
268+
Run a specific test case with custom parameters:
269+
270+
```bash
271+
pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml "tests/single/st20p/fps/test_fps.py::test_fps[|fps = p60|-ParkJoy_1080p]"
272+
```
273+
274+
```bash
275+
python3 -m pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml -m smoke
276+
```
277+
202278
Run specific test modules:
203279

204280
```bash

doc/validation_quickstart.md

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# MTL Validation Framework - Quick Start Guide
2+
3+
This quick start guide helps you get the MTL validation framework running with minimal setup. For detailed information, see the [complete validation framework documentation](validation_framework.md).
4+
5+
## Prerequisites (Must Complete First!)
6+
7+
1. **🏗️ Build MTL** (CRITICAL - tests will fail without this):
8+
```bash
9+
cd /path/to/Media-Transport-Library
10+
./build.sh
11+
```
12+
> If this fails, see [detailed build instructions](build.md)
13+
14+
2. **📋 Basic Requirements**:
15+
- Python 3.9+
16+
- Root user access (MTL validation requires root privileges)
17+
- Network interfaces configured for testing
18+
19+
## Quick Setup (3 steps)
20+
21+
### 1. Install Dependencies
22+
**Run in tests/validation directory**:
23+
```bash
24+
cd tests/validation
25+
python3 -m venv venv
26+
source venv/bin/activate
27+
pip install -r requirements.txt # Main framework requirements
28+
pip install -r common/integrity/requirements.txt # Integrity test components
29+
```
30+
31+
### 2. Configure Environment
32+
Update two key files:
33+
34+
**[tests/validation/configs/topology_config.yaml](../tests/validation/configs/topology_config.yaml)**:
35+
```yaml
36+
# Key settings to update:
37+
username: root # Must be root for MTL operations
38+
key_path: /root/.ssh/id_rsa # Your SSH key path
39+
```
40+
41+
**[tests/validation/configs/test_config.yaml](../tests/validation/configs/test_config.yaml)**:
42+
```yaml
43+
# Replace MTL_PATH_PLACEHOLDER with your actual paths:
44+
build: /home/gta/Media-Transport-Library/
45+
mtl_path: /home/gta/Media-Transport-Library/
46+
```
47+
48+
### 3. Run Tests
49+
**Basic smoke test**:
50+
```bash
51+
cd tests/validation
52+
source venv/bin/activate
53+
python3 -m pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml -m smoke -v
54+
```
55+
56+
**Run specific test with parameters**:
57+
```bash
58+
pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml "tests/single/st20p/fps/test_fps.py::test_fps[|fps = p60|-ParkJoy_1080p]"
59+
```
60+
61+
## Optional: Create VFs for Advanced Testing
62+
63+
If you need VFs for NIC testing:
64+
```bash
65+
# Find your network device first
66+
lspci | grep Ethernet
67+
68+
# Create VFs (replace with your device identifier)
69+
sudo ./script/nicctl.sh create_vf ${TEST_PF_PORT_P}
70+
sudo ./script/nicctl.sh create_vf ${TEST_PF_PORT_R}
71+
```
72+
73+
## Quick Troubleshooting
74+
75+
| Error | Solution |
76+
|-------|----------|
77+
| `RxTxApp: command not found` | Build MTL first with `./build.sh` |
78+
| `Permission denied` | Use root user (not regular user) |
79+
| `Config path errors` | Update placeholder paths in config files |
80+
81+
## Generate Test Media (Optional)
82+
83+
For video testing, generate test frames:
84+
```bash
85+
cd tests/validation/common
86+
./gen_frames.sh
87+
```
88+
89+
The script supports:
90+
- Multiple resolutions (3840x2160, 1920x1080, 1280x720, 640x360)
91+
- Different pixel formats (yuv422p, yuv422p10le)
92+
- Configurable color patterns and test signals
93+
- Various frame rates
94+
95+
---
96+
97+
**Need more details?**[Complete Documentation](validation_framework.md)
98+
**Build issues?**[Build Guide](build.md)
99+
**Configuration help?**[Configuration Guide](configuration_guide.md)

tests/validation/README.md

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,17 +2,19 @@
22

33
The Media Transport Library (MTL) Validation Framework provides comprehensive testing capabilities for various aspects of the MTL, including protocol compliance, performance, and integration testing.
44

5-
> **For detailed documentation, please refer to [the main validation framework documentation](/doc/validation_framework.md)**
5+
> **📖 For detailed documentation, please refer to [the main validation framework documentation](../../doc/validation_framework.md)**
6+
>
7+
> **🚀 Quick Start**: See [Validation Quick Start Guide](../../doc/validation_quickstart.md)
68
7-
## Quick Start Guide
9+
## Quick Setup
810

911
### Prerequisites
1012

1113
- Python 3.9 or higher
1214
- **⚠️ CRITICAL**: Media Transport Library built and installed (see [build instructions](../../doc/build.md))
1315
- Test media files (typically on NFS)
1416
- Network interfaces configured for testing
15-
- Root privileges for network operations
17+
- **Root privileges required** (MTL validation must run as root user)
1618
- FFmpeg and GStreamer plugins (for integration tests)
1719

1820
### Setup in 3 Simple Steps
@@ -24,27 +26,32 @@ The Media Transport Library (MTL) Validation Framework provides comprehensive te
2426
```
2527
See [detailed build instructions](../../doc/build.md) if needed.
2628

27-
2. **Create a virtual environment and install dependencies**:
29+
2. **Create virtual environment and install dependencies** (run in `tests/validation/`):
2830
```bash
29-
cd tests/validation
31+
cd tests/validation # Must be in this directory!
3032
python3 -m venv venv
3133
source venv/bin/activate
32-
pip install -r requirements.txt
34+
pip install -r requirements.txt # Main framework requirements
35+
pip install -r common/integrity/requirements.txt # Integrity test components
3336
```
3437

3538
3. **Configure your environment**:
36-
- Update network interfaces in `configs/topology_config.yaml`
37-
- Set correct paths in `configs/test_config.yaml` (especially `build` and `mtl_path`)
39+
- Update network interfaces in [`configs/topology_config.yaml`](configs/topology_config.yaml)
40+
- Set correct paths in [`configs/test_config.yaml`](configs/test_config.yaml) (especially `build` and `mtl_path`)
3841
- Ensure media files are accessible at `media_path`
42+
- **Use root user** in topology_config.yaml (not regular user)
3943

4044
4. **Run tests**:
4145
```bash
42-
# Run smoke tests (quick validation)
46+
# Run smoke tests (quick validation) - MUST be run as root
4347
python3 -m pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml -m smoke
4448

4549
# Run specific test module
4650
python3 -m pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml tests/single/st20p/test_st20p_rx.py
4751

52+
# Run specific test with parameters
53+
pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml "tests/single/st20p/fps/test_fps.py::test_fps[|fps = p60|-ParkJoy_1080p]"
54+
4855
# Generate HTML report
4956
python3 -m pytest --topology_config=configs/topology_config.yaml --test_config=configs/test_config.yaml -m smoke --template=html/index.html --report=report.html
5057
```

0 commit comments

Comments
 (0)