Skip to content

Commit 410397c

Browse files
KarolinaPomianMateuszGrabuszynski
KarolinaPomian
and
MateuszGrabuszynski
authored
Scenarios 2.x descriptions for manual testing (#338)
* info from communication added to 2.2 doc * example 2.2 commands * schema update * change variables to generic * test cases for 2.2 * move generic info to SCENARIO * move generic steps to SCENARIO * info how to run blob and audio * blob update * corrections for blob not supported in ffmpeg * Packet Time value clarification * Json parameters tables * -r param in ffmpeg * Doc: (not finished) Adding more information in SCENARIO.md - parameters, JT-NM 2022 compliance * update tests based on testplan * linter fix * Update SCENARIO.md Signed-off-by: KarolinaPomian <[email protected]> * linter fix * copilot suggestions corrections * Update SCENARIO.md Signed-off-by: KarolinaPomian <[email protected]> * Update SCENARIO.md Missing spaces between the words Signed-off-by: KarolinaPomian <[email protected]> * graph correction to reverse changes --------- Signed-off-by: KarolinaPomian <[email protected]> Co-authored-by: MateuszGrabuszynski <[email protected]>
1 parent beca485 commit 410397c

10 files changed

+741
-0
lines changed

‎tests/validation/docs/test-scenarios/SCENARIO.md‎

Lines changed: 244 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,244 @@
1+
# Test Scenarios Documentation
2+
3+
> **Note:** This document provides generic information for all scenarios in the `/Media-Communications-Mesh/tests/validation/docs/test-scenarios/` directory. Provided test cases may not cover all supported parameters.
4+
5+
## Transmission Modes
6+
7+
* standalone - with user application built on API
8+
* FFmpeg - using FFmpeg plugin
9+
10+
## Payload Options
11+
12+
* Blob (random binary block)
13+
* Video – uncompressed (raw)
14+
* Audio
15+
16+
## Blob (random binary block) format
17+
18+
At least 100 MB data generated based on `/dev/random`.
19+
20+
Generation command example using `dd` tool:
21+
```bash
22+
dd if=/dev/random of=random_data.bin bs=1 count=100000000
23+
```
24+
25+
## Uncompressed video formats
26+
27+
| Parameter | Standalone | FFmpeg |
28+
|-----------------|----------------------|------------------------|
29+
| Resolutions | 1920x1080 (Full HD) | 1920x1080 (Full HD) |
30+
| | 3840x2160 (4k) | 3840x2160 (4k) |
31+
| Framerate (FPS) | 59.94 | 59.94 |
32+
| | 60 | 60 |
33+
| Color format | YUV 4:2:2 10-bit planar | YUV 4:2:2 10-bit planar |
34+
| Interlace | progressive | progressive |
35+
| Packing | gpm | *cannot be set in FFmpeg plugin* |
36+
| | bpm | *cannot be set in FFmpeg plugin* |
37+
| | gpm_sl | *cannot be set in FFmpeg plugin* |
38+
| Pacing | wide | *cannot be set in FFmpeg plugin* |
39+
| | narrow | narrow (default in FFmpeg plugin) |
40+
| | linear | *cannot be set in FFmpeg plugin* |
41+
42+
<!-- Note: The FFmpeg plugin uses "narrow" pacing as the default according to the SDK API examples.
43+
Packing mode is not directly configurable in the FFmpeg plugin as it uses the SDK's default behavior. -->
44+
45+
## Analyzing Media Files
46+
47+
To validate raw video files using FFmpeg:
48+
49+
```bash
50+
ffmpeg -f rawvideo -pixel_format <pixel_format> -video_size <width>x<height> -i <input_file> -vframes <number_of_frames> -f null -
51+
```
52+
53+
To analyze video metadata and content details:
54+
55+
```bash
56+
ffprobe -v verbose -show_format -show_streams -print_format json <input_file>
57+
```
58+
59+
To verify audio file parameters:
60+
61+
```bash
62+
ffprobe -v quiet -print_format json -show_streams -select_streams a:0 <audio_file>
63+
```
64+
65+
### Information about video transmission parameters
66+
67+
Parameters provided in brackets refer to SDP (Session Description Protocol) values.
68+
69+
Packing mode abbreviations (according to SMPTE ST 2110-20):
70+
* gpm - General Packing Mode (`PM=2110GPM`) - Multiple scan lines per packet, potentially more efficient use of bandwidth
71+
* bpm - Block Packing Mode (`PM=2110BPM`) - Packs scan lines in blocks, helps with compatibility
72+
* gpm_sl - General Packing Mode with single scan line - Only one scan line per packet, may reduce latency
73+
74+
Pacing (according to SMPTE ST 2110):
75+
* wide (`TP=2110TPW`) - More permissive traffic shaping, allows bursts within limits
76+
* narrow (`TP=2110TPN`) - Stricter traffic shaping with more uniform packet distribution over time
77+
* linear - narrow linear (`TP=2110TPNL`) - Most strict pacing with completely linear distribution
78+
79+
Note: When using FFmpeg plugin with MCM, the pacing is set to "narrow" by default, as specified in the SDK API examples.
80+
81+
This section is partially based on [*VSF TR-05 Version 1.0*](https://static.vsf.tv/download/technical_recommendations/VSF_TR-05_2018-06-23.pdf) ([CC-BY-ND](https://creativecommons.org/licenses/by-nd/4.0/)), accessed on 2025-03-03.
82+
83+
## Audio formats
84+
85+
| Parameter | Standalone | FFmpeg |
86+
|--------------------|---------------------|-----------------------|
87+
| Audio sample depth | PCM 8-bit Big-Endian (`pcm_s8be`) | *not available* |
88+
| | PCM 16-bit Big-Endian (`pcm_s16be`) | PCM 16-bit Big-Endian (`pcm_s16be`) |
89+
| | PCM 24-bit Big-Endian (`pcm_s24be`) | PCM 24-bit Big-Endian (`pcm_s24be`) |
90+
| Sample rate | 44100 Hz* | *not available* |
91+
| | 48000 Hz | 48000 Hz |
92+
| | 96000 Hz | 96000 Hz |
93+
| Packet time | 1 ms/1.09 ms* (48/96 samples per group) | 1 ms (48/96 samples per group) |
94+
| Number of channels | 1 (M - Mono) | 1 (M - Mono) |
95+
| | 2 (ST - Stereo) | 2 (ST - Stereo) |
96+
| Test mode | frame | frame |
97+
98+
*1 ms Packet Time is not available for 44100 Hz sample rate. It would require a non-natural number of samples (since 44100 samples/second *0.001 seconds = 44.1 samples*), which is not possible. Adjust the packet time to 1.09ms for 44100 Hz (48 samples per packet).
99+
100+
## Test Cases
101+
102+
### Video Test Cases
103+
104+
| # | Standalone / FFmpeg | Resolution | Framerate |
105+
|---|---------------------|------------|-----------|
106+
| 1 | Standalone | 1920x1080 (Full HD) | 59.94 (60000/1001) |
107+
| 2 | Standalone | 1920x1080 (Full HD) | 60 |
108+
| 3 | FFmpeg | 1920x1080 (Full HD) | 59.94 (60000/1001) |
109+
| 4 | FFmpeg | 1920x1080 (Full HD) | 60 |
110+
| 5 | Standalone | 3840x2160 (4k) | 59.94 (60000/1001) |
111+
| 6 | Standalone | 3840x2160 (4k) | 60 |
112+
| 7 | FFmpeg | 3840x2160 (4k) | 59.94 (60000/1001) |
113+
| 8 | FFmpeg | 3840x2160 (4k) | 60 |
114+
115+
### Audio Test Cases
116+
117+
| # | Standalone / FFmpeg | Sample depth | Sample rate | Packet time | Channels |
118+
|---|---------------------|--------------|------------|------------|----------|
119+
| 1 | Standalone | PCM 8-bit Big-Endian (pcm_s8be) | 44100 Hz | 1.09 ms | 1 (M - Mono) |
120+
| 2 | Standalone | PCM 8-bit Big-Endian (pcm_s8be) | 44100 Hz | 1.09 ms | 2 (S - Stereo) |
121+
| 3 | Standalone | PCM 16-bit Big-Endian (pcm_s16be) | 44100 Hz | 1.09 ms | 1 (M - Mono) |
122+
| 4 | Standalone | PCM 16-bit Big-Endian (pcm_s16be) | 44100 Hz | 1.09 ms | 2 (S - Stereo) |
123+
| 5 | Standalone | PCM 24-bit Big-Endian (pcm_s24be) | 44100 Hz | 1.09 ms | 1 (M - Mono) |
124+
| 6 | Standalone | PCM 24-bit Big-Endian (pcm_s24be) | 44100 Hz | 1.09 ms | 2 (S - Stereo) |
125+
| 7 | Standalone | PCM 8-bit Big-Endian (pcm_s8be) | 48000 Hz | 1 ms | 1 (M - Mono) |
126+
| 8 | Standalone | PCM 8-bit Big-Endian (pcm_s8be) | 48000 Hz | 1 ms | 2 (S - Stereo) |
127+
| 9 | Standalone | PCM 16-bit Big-Endian (pcm_s16be) | 48000 Hz | 1 ms | 1 (M - Mono) |
128+
| 10 | Standalone | PCM 16-bit Big-Endian (pcm_s16be) | 48000 Hz | 1 ms | 2 (S - Stereo) |
129+
| 11 | Standalone | PCM 24-bit Big-Endian (pcm_s24be) | 48000 Hz | 1 ms | 1 (M - Mono) |
130+
| 12 | Standalone | PCM 24-bit Big-Endian (pcm_s24be) | 48000 Hz | 1 ms | 2 (S - Stereo) |
131+
| 13 | Standalone | PCM 8-bit Big-Endian (pcm_s8be) | 96000 Hz | 1 ms | 1 (M - Mono) |
132+
| 14 | Standalone | PCM 8-bit Big-Endian (pcm_s8be) | 96000 Hz | 1 ms | 2 (S - Stereo) |
133+
| 15 | Standalone | PCM 16-bit Big-Endian (pcm_s16be) | 96000 Hz | 1 ms | 1 (M - Mono) |
134+
| 16 | Standalone | PCM 16-bit Big-Endian (pcm_s16be) | 96000 Hz | 1 ms | 2 (S - Stereo) |
135+
| 17 | Standalone | PCM 24-bit Big-Endian (pcm_s24be) | 96000 Hz | 1 ms | 1 (M - Mono) |
136+
| 18 | Standalone | PCM 24-bit Big-Endian (pcm_s24be) | 96000 Hz | 1 ms | 2 (S - Stereo) |
137+
| 19 | FFmpeg | PCM 16-bit Big-Endian (pcm_s16be) | 48000 Hz | 1 ms | 1 (M - Mono) |
138+
| 20 | FFmpeg | PCM 16-bit Big-Endian (pcm_s16be) | 48000 Hz | 1 ms | 2 (S - Stereo) |
139+
| 21 | FFmpeg | PCM 24-bit Big-Endian (pcm_s24be) | 48000 Hz | 1 ms | 1 (M - Mono) |
140+
| 22 | FFmpeg | PCM 24-bit Big-Endian (pcm_s24be) | 48000 Hz | 1 ms | 2 (S - Stereo) |
141+
| 23 | FFmpeg | PCM 16-bit Big-Endian (pcm_s16be) | 96000 Hz | 1 ms | 1 (M - Mono) |
142+
| 24 | FFmpeg | PCM 16-bit Big-Endian (pcm_s16be) | 96000 Hz | 1 ms | 2 (S - Stereo) |
143+
| 25 | FFmpeg | PCM 24-bit Big-Endian (pcm_s24be) | 96000 Hz | 1 ms | 1 (M - Mono) |
144+
| 26 | FFmpeg | PCM 24-bit Big-Endian (pcm_s24be) | 96000 Hz | 1 ms | 2 (S - Stereo) |
145+
146+
### Blob Test Cases
147+
148+
| # | Standalone / FFmpeg | Data size | Packet size |
149+
|---|---------------------|-----------|-------------|
150+
| 1 | Standalone | 100 MB | 100 kB |
151+
152+
## Tested Parameters and Metrics
153+
154+
### Configuration Parameters
155+
* **Buffer Size**: Size of memory buffers allocated for media transmission
156+
* **Metadata Size**: Maximum size of metadata attached to each frame
157+
* **Queue Capacity**: Number of buffers in allocated queue (configurable with `bufferQueueCapacity` parameter)
158+
* **Connection Delay**: Time between connection establishment and first frame transmission (configurable with `connCreationDelayMilliseconds`)
159+
160+
### Performance Metrics
161+
* **Frame Rate Accuracy**: Verifying that transmitted frames per second (fps) match the configured rate
162+
* **Reception Rate**: Confirming that received fps keeps up with sent fps without drops
163+
* **Frame Integrity**: Ensuring the number of frames received closely matches the number of frames sent
164+
* **Latency**: Measuring end-to-end delay between transmission and reception
165+
166+
### Testing Tools
167+
* Built-in logging and statistics from Media Proxy
168+
* FFmpeg stats output (`-stats` option)
169+
* Custom test applications in the `tests/` directory
170+
171+
## Manual Test Execution
172+
173+
1. Start one `mesh-agent` per cluster:
174+
```bash
175+
mesh-agent
176+
```
177+
178+
2. Start `media_proxy` for the transmitter and receiver (always only a single instance per node):
179+
180+
Receiver side:
181+
```bash
182+
sudo media_proxy \
183+
-d <pci_device> \
184+
-i <st2110_ip> \
185+
-r <rdma_ip> \
186+
-p 9200-9299 \
187+
-t 8002
188+
```
189+
190+
Transmitter side:
191+
```bash
192+
sudo media_proxy \
193+
-d <pci_device> \
194+
-i <st2110_ip> \
195+
-r <rdma_ip> \
196+
-p 9100-9199 \
197+
-t 8001
198+
```
199+
200+
3. Start the receivers (e.g., using FFmpeg for video reception):
201+
```bash
202+
sudo MCM_MEDIA_PROXY_PORT=8002 ffmpeg -f mcm \
203+
-conn_type st2110 \
204+
-transport st2110-20 \
205+
-ip_addr <source_ip> \
206+
-port <port> \
207+
-frame_rate <frame_rate> \
208+
-video_size <width>x<height> \
209+
-pixel_format <pixel_format> \
210+
-i - -f <output_format> <output_destination>
211+
```
212+
213+
4. Start the transmitter (e.g., using FFmpeg for video transmission):
214+
```bash
215+
sudo MCM_MEDIA_PROXY_PORT=8001 ffmpeg -i <input_file> -f mcm \
216+
-conn_type st2110 \
217+
-transport st2110-20 \
218+
-ip_addr <destination_ip> \
219+
-port <port> \
220+
-frame_rate <frame_rate> \
221+
-video_size <width>x<height> \
222+
-pixel_format <pixel_format> -
223+
```
224+
225+
This sequence ensures that the receiver is ready to receive the data as soon as possible. Even then, there is a connection initialization delay, resulting in a few packets missing at the beginning of the transmission. Such behavior should be considered normal, except when that delay is longer than usual (more than a few frames).
226+
227+
## JT-NM Compliance Testing
228+
229+
Media transmitted with SMPTE ST 2110 protocols is received on a test system with a `tcpdump` session capturing network traffic to a `*.pcap` file. This compliance capture is performed separately from the main tests to avoid performance issues that might affect the measurements.
230+
231+
To capture network traffic for compliance testing:
232+
233+
```bash
234+
sudo tcpdump -i <interface_name> -w capture.pcap 'udp and port <port_number>'
235+
```
236+
237+
The resulting `*.pcap` file is later uploaded to the [EBU LIST tool](https://github.com/ebu/pi-list/) to verify compliance with JT-NM (Joint Task Force on Networked Media) specifications. The LIST tool analyzes various aspects including:
238+
239+
- Packet timing compliance (according to selected pacing profile)
240+
- RTP header consistency
241+
- Payload format compliance
242+
- Media quality parameters
243+
244+
For more information about JT-NM compliance testing and specifications, visit the [Joint Task Force on Networked Media website](https://www.jt-nm.org/jt-nm-tested).

‎tests/validation/docs/test-scenarios/scenario-set-1-local-single-node/SCENARIO-1.1-SINGLE-NODE-SINGLE-RECEIVER.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,7 @@ flowchart
2020
* Blob
2121
* Video – Uncompressed
2222
* Audio
23+
24+
### Test Cases
25+
26+
For detailed test cases, refer to the centralized [Test Cases documentation](../SCENARIO.md#test-cases).

‎tests/validation/docs/test-scenarios/scenario-set-1-local-single-node/SCENARIO-1.2-SINGLE-NODE-3X-RECEIVERS.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,7 @@ flowchart
2424
* Blob
2525
* Video – Uncompressed
2626
* Audio
27+
28+
### Test Cases
29+
30+
For detailed test cases, refer to the centralized [Test Cases documentation](../SCENARIO.md#test-cases).

‎tests/validation/docs/test-scenarios/scenario-set-2-rdma/SCENARIO-2.1-2X-NODES-SINGLE-RECEIVER-DIRECT-CABLE.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,7 @@ flowchart LR
2424
* Blob
2525
* Video – Uncompressed
2626
* Audio
27+
28+
### Test Cases
29+
30+
For detailed test cases, refer to the centralized [Test Cases documentation](../SCENARIO.md#test-cases).

0 commit comments

Comments
 (0)