Skip to content

Commit bdf388b

Browse files
MateuszGrabuszynskiMateuszGrabuszynski
authored
Merge branch 'main' into private/mlinkiew/trivy-related-fixes
2 parents b6d8469 + eacc252 commit bdf388b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+615
-4304
lines changed

‎docs/LatencyMeasurement.md‎

Lines changed: 297 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,297 @@
1+
# End-to-End Latency Measurement — Media Communications Mesh
2+
3+
This document describes a simple solution for measuring end-to-end latency in Media Communications Mesh.
4+
5+
## Overview
6+
7+
The solution is based on the FFmpeg ability to print current timestamps on the sender side (Tx) and the receiver side (Rx), and the use of Optical Character Recognition (OCR) to read the timestamps out of each received video frame and calculate the delta. The choice of OCR is determined by the fact that the text can be effectively recognized even if the picture is affected by any sort of a lossy video compression algorithm somewhere in the transmission path in the Mesh. To achieve proper accuracy of the measurement, both Tx and Rx host machines should be synchronized using Precision Time Protocol (PTP).
8+
9+
> Only video payload is supported.
10+
11+
```mermaid
12+
flowchart LR
13+
tx-file((Input
14+
video file))
15+
tx-ffmpeg(Tx
16+
FFmpeg)
17+
subgraph mesh [Media Communications Mesh]
18+
direction LR
19+
proxy1(Proxy1)
20+
proxy2a(. . .)
21+
proxy2b(. . .)
22+
proxy2c(. . .)
23+
proxy3(ProxyN)
24+
proxy1 --> proxy2a --> proxy3
25+
proxy1 --> proxy2b --> proxy3
26+
proxy1 --> proxy2c --> proxy3
27+
end
28+
rx-ffmpeg(Rx
29+
FFmpeg)
30+
rx-file((Output
31+
video file))
32+
33+
tx-file --> tx-ffmpeg --> mesh --> rx-ffmpeg --> rx-file
34+
```
35+
36+
## How it works
37+
38+
1. Tx side – The user starts FFmpeg with special configuration to stream video via the Mesh.
39+
1. Rx side – The user starts FFmpeg with special configuration to receive the video stream from the Mesh.
40+
1. Tx side – FFmpeg prints the current timestamp as a huge text at the top of each video frame and transmits it via the Mesh.
41+
1. Rx side – FFmpeg prints the current timestamp as a huge text at the bottom of each video frame received from the Mesh and saves it on the disk.
42+
1. After transmission is done, there is a resulting MPEG video file on the disk on the Rx side.
43+
1. The user runs the solution script against the MPEG file that recognizes the Tx and Rx timestamps in each frame, and calculates the average latency based on the difference between the timestamps. Additionally, the script generates a latency diagram and stores it in JPEG format on the disk.
44+
45+
## Sample latency diagram
46+
47+
<img src="_static/ffmpeg-based-latency-solution-diagram.jpg" width="520">
48+
49+
## Important notice on latency measurement results
50+
51+
> Please note the calculated average latency is highly dependent on the hardware configuration and CPU background load, and cannot be treated as an absolute value. The provided solution can only be used for comparing the latency in different Mesh configurations and video streaming parameters, as well as latency stability checks.
52+
53+
54+
## Build and install steps
55+
56+
> It is assumed that Media Communications Mesh is installed on the Tx and Rx host machines according to [Setup Guide](SetupGuid.md).
57+
58+
If [FFmpeg Plugin](FFmpegPlugin.md) was installed earlier, remove its directory before proceeding with the following.
59+
60+
1. Clone the FFmpeg 7.0 repository and apply patches.
61+
62+
```bash
63+
./clone-and-patch-ffmpeg.sh
64+
```
65+
66+
1. Run the FFmpeg configuration tool with special features enabled
67+
68+
```bash
69+
./configure-ffmpeg.sh 7.0 --enable-libfreetype --enable-libharfbuzz --enable-libfontconfig
70+
```
71+
72+
1. Build and install FFmpeg with the Media Communications Mesh FFmpeg plugin
73+
74+
```bash
75+
./build-ffmpeg.sh
76+
```
77+
78+
1. Install Tesseract OCR
79+
```bash
80+
apt install tesseract-ocr
81+
```
82+
1. Install Python packages
83+
```bash
84+
pip install opencv-python~=4.11.0 pytesseract~=0.3.13 matplotlib~=3.10.3
85+
```
86+
87+
1. Setup time synchronization on host machines
88+
89+
> Make sure `network_interface_1` and `network_interface_2` are connected to the same network.
90+
91+
* __host-1 Controller clock__
92+
```bash
93+
sudo ptp4l -i <network_interface_1> -m 2
94+
sudo phc2sys -a -r -r -m
95+
```
96+
97+
* __host-2 Worker clock__
98+
```bash
99+
sudo ptp4l -i <network_interface_2> -m 2 -s
100+
sudo phc2sys -a -r
101+
```
102+
103+
## Example – Measuring transmission latency between two FFmpeg instances on the same host
104+
105+
This example demonstrates sending a video file from the 1st FFmpeg instance to the 2nd FFmpeg instance via Media Communications Mesh on the same host, and then calculation of transmission latency from the recorded video.
106+
107+
108+
1. Run Mesh Agent
109+
```bash
110+
mesh-agent
111+
```
112+
113+
1. Run Media Proxy
114+
115+
```bash
116+
sudo media_proxy \
117+
-d 0000:32:01.1 \
118+
-i 192.168.96.11 \
119+
-r 192.168.97.11 \
120+
-p 9200-9299 \
121+
-t 8002
122+
```
123+
124+
1. Start the Receiver side FFmpeg instance
125+
126+
```bash
127+
sudo MCM_MEDIA_PROXY_PORT=8002 ffmpeg \
128+
-f mcm \
129+
-conn_type multipoint-group \
130+
-frame_rate 60 \
131+
-video_size 1920x1080 \
132+
-pixel_format yuv422p10le \
133+
-i - \
134+
-vf \
135+
"drawtext=fontsize=40: \
136+
text='Rx timestamp %{localtime\\:%H\\\\\:%M\\\\\:%S\\\\\:%3N}': \
137+
x=10: y=70: fontcolor=white: box=1: boxcolor=black: boxborderw=10" \
138+
-vcodec mpeg4 -qscale:v 3 recv.mp4
139+
```
140+
1. Start the Sender side FFmpeg instance
141+
142+
```bash
143+
sudo MCM_MEDIA_PROXY_PORT=8002 ffmpeg -i <video-file-path> \
144+
-vf \
145+
"drawtext=fontsize=40: \
146+
text='Tx timestamp %{localtime\\:%H\\\\\:%M\\\\\:%S\\\\\:%3N}': \
147+
x=10: y=10: fontcolor=white: box=1: boxcolor=black: boxborderw=10" \
148+
-f mcm \
149+
-conn_type multipoint-group \
150+
-frame_rate 60 \
151+
-video_size 1920x1080 \
152+
-pixel_format yuv422p10le -
153+
```
154+
155+
When sending a raw video file, e.g. of the YUV format, you have to explicitly specify the file format `-f rawvideo`, the pixel format `-pix_fmt`, and the video resolution `-s WxH`:
156+
157+
```bash
158+
ffmpeg -f rawvideo -pix_fmt yuv422p10le -s 1920x1080 -i <video-file-path> ...
159+
```
160+
161+
It is also recommended to provide the read rate `-readrate` at which FFmpeg will read frames from the file:
162+
163+
```bash
164+
ffmpeg -f rawvideo -readrate 2.4 -pix_fmt yuv422p10le -s 1920x1080 -i <video-file-path> ...
165+
```
166+
167+
The `-readrate` value is calculated from the `-frame_rate` parameter value using the following equation: $readrate=framerate\div25$. Use the pre-calculated values from the table below.
168+
169+
| frame_rate | readrate |
170+
|------------|-------------------|
171+
| 25 | 25 / 25 = 1 |
172+
| 50 | 50 / 25 = 2 |
173+
| 60 | 60 / 25 = 2.4 |
174+
175+
1. Run the script against the recorded MPEG file. The first argument is the input video file path. The second argument is the optional latency diagram JPEG file path to be generated.
176+
177+
```bash
178+
python text-detection.py recv.mp4 recv-latency.jpg
179+
```
180+
181+
Console output
182+
```bash
183+
...
184+
Processing Frame: 235
185+
Processing Frame: 236
186+
Processing Frame: 237
187+
Processing Frame: 238
188+
Processing Frame: 239
189+
Processing Frame: 240
190+
Saving the latency chart to: recv-latency.jpg
191+
File: recv.mp4 | Last modified: 2025-06-02 13:49:54 UTC
192+
Resolution: 640x360 | FPS: 25.00
193+
Average End-to-End Latency: 564.61 ms
194+
```
195+
196+
See the [Sample latency diagram](#sample-latency-diagram).
197+
198+
199+
## Example – Measuring transmission latency between two FFmpeg instances on different hosts
200+
201+
This example demonstrates sending a video file from the 1st FFmpeg instance to the 2nd FFmpeg instance via Media Communications Mesh on the same host, and then calculation of transmission latency from the recorded video.
202+
203+
1. Run Mesh Agent
204+
```bash
205+
mesh-agent
206+
```
207+
208+
1. Start Media Proxy on the Receiver host machine
209+
210+
```bash
211+
sudo media_proxy \
212+
-d 0000:32:01.1 \
213+
-i 192.168.96.11 \
214+
-r 192.168.97.11 \
215+
-p 9200-9299 \
216+
-t 8002
217+
```
218+
219+
1. Start the Receiver side FFmpeg instance
220+
221+
```bash
222+
sudo MCM_MEDIA_PROXY_PORT=8002 ffmpeg \
223+
-f mcm \
224+
-conn_type st2110 \
225+
-transport st2110-20 \
226+
-ip_addr 192.168.96.10 \
227+
-port 9001 \
228+
-frame_rate 60 \
229+
-video_size 1920x1080 \
230+
-pixel_format yuv422p10le \
231+
-i - \
232+
-vf \
233+
"drawtext=fontsize=40: \
234+
text='Rx timestamp %{localtime\\:%H\\\\\:%M\\\\\:%S\\\\\:%3N}': \
235+
x=10: y=70: fontcolor=white: box=1: boxcolor=black: boxborderw=10" \
236+
-vcodec mpeg4 -qscale:v 3 recv.mp4
237+
```
238+
239+
1. Start Media Proxy on the Sender host machine
240+
241+
```bash
242+
sudo media_proxy \
243+
-d 0000:32:01.0 \
244+
-i 192.168.96.10 \
245+
-r 192.168.97.10 \
246+
-p 9100-9199 \
247+
-t 8001
248+
```
249+
250+
1. Start the Sender side FFmpeg instance
251+
252+
```bash
253+
sudo MCM_MEDIA_PROXY_PORT=8001 ffmpeg -i <video-file-path> \
254+
-vf \
255+
"drawtext=fontsize=40: \
256+
text='Tx timestamp %{localtime\\:%H\\\\\:%M\\\\\:%S\\\\\:%3N}': \
257+
x=10: y=10: fontcolor=white: box=1: boxcolor=black: boxborderw=10" \
258+
-f mcm \
259+
-conn_type st2110 \
260+
-transport st2110-20 \
261+
-ip_addr 192.168.96.11 \
262+
-port 9001 \
263+
-frame_rate 60 \
264+
-video_size 1920x1080 \
265+
-pixel_format yuv422p10le -
266+
```
267+
268+
1. Run the script against the recorded MPEG file. The first argument is the input video file path. The second argument is the optional latency diagram JPEG file path to be generated.
269+
270+
```bash
271+
python text-detection.py recv.mp4 recv-latency.jpg
272+
```
273+
274+
Console output
275+
```bash
276+
...
277+
Processing Frame: 235
278+
Processing Frame: 236
279+
Processing Frame: 237
280+
Processing Frame: 238
281+
Processing Frame: 239
282+
Processing Frame: 240
283+
Saving the latency chart to: recv-latency.jpg
284+
File: recv.mp4 | Last modified: 2025-06-02 13:49:54 UTC
285+
Resolution: 640x360 | FPS: 25.00
286+
Average End-to-End Latency: 564.61 ms
287+
```
288+
289+
See the [Sample latency diagram](#sample-latency-diagram).
290+
291+
## Customization
292+
When modifying FFmpeg commands if you change parameters of `drawtext` filter, especialy `fontsize`, `x`, `y` or `text`, you have to adjust python script __text-detection.py__ too, please refer to function `extract_text_from_region(image, x, y, font_size, length)`
293+
294+
295+
<!-- References -->
296+
[license-img]: https://img.shields.io/badge/License-BSD_3--Clause-blue.svg
297+
[license]: https://opensource.org/license/bsd-3-clause

‎docs/SDK_API_Definition.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -441,10 +441,9 @@ NOTE: The codes are negative integer values.
441441
| `-MESH_ERR_BAD_CONFIG_PTR` | Bad configuration pointer | The configuration pointer is NULL. |
442442
| `-MESH_ERR_BAD_BUF_PTR` | Bad buffer pointer | The buffer pointer is NULL. |
443443
| `-MESH_ERR_BAD_BUF_LEN` | Bad buffer length | **Rx connection**: The buffer length is corrupted.<br>**Tx connection**: The buffer length is bigger than maximum. |
444-
| `-MESH_ERR_CLIENT_FAILED` | Client creation failed | An error occurred while creating an SDK client. |
445444
| `-MESH_ERR_CLIENT_CONFIG_INVAL` | Invalid client config | JSON client configuration string is malformed. |
446445
| `-MESH_ERR_MAX_CONN` | Reached max connections number | An attempt to create a connection failed due to reaching the maximum number of connections defined in `"maxMediaConnections"`. |
447-
| `-MESH_ERR_FOUND_ALLOCATED` | Found allocated resources | When deleting an SDK client, some connections were found not closed. Delete all connections explicitly before deleting the client. |
446+
| `-MESH_ERR_FOUND_ALLOCATED` | Found allocated resources | When deleting a client, some connections were found not closed. Delete all connections explicitly before deleting the client. |
448447
| `-MESH_ERR_CONN_FAILED` | Connection creation failed | An error occurred while creating a connection. |
449448
| `-MESH_ERR_CONN_CONFIG_INVAL` | Invalid connection config | JSON connection configuration string is malformed or one of parameters has an incorrect value. |
450449
| `-MESH_ERR_CONN_CONFIG_INCOMPAT` | Incompatible connection config | Incompatible parameters found in the JSON connection configuration string. |

‎docs/_static/ffmpeg-based-latency-solution-diagram.jpg‎

140 KB
Loading

‎ffmpeg-plugin/mcm_audio_rx.c‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -133,17 +133,19 @@ static int mcm_audio_read_packet(AVFormatContext* avctx, AVPacket* pkt)
133133
s->first_frame = false;
134134

135135
err = mesh_get_buffer_timeout(s->conn, &buf, timeout);
136-
if (err == -MESH_ERR_CONN_CLOSED)
137-
return AVERROR_EOF;
138-
136+
if (err == -MESH_ERR_CONN_CLOSED) {
137+
ret = AVERROR_EOF;
138+
goto error_close_conn;
139+
}
139140
if (err) {
140141
if (mcm_shutdown_requested()) {
141-
return AVERROR_EXIT;
142+
ret = AVERROR_EXIT;
142143
} else {
143144
av_log(avctx, AV_LOG_ERROR, "Get buffer error: %s (%d)\n",
144145
mesh_err2str(err), err);
145-
return AVERROR(EIO);
146+
ret = AVERROR(EIO);
146147
}
148+
goto error_close_conn;
147149
}
148150

149151
if (mcm_shutdown_requested()) {
@@ -164,14 +166,21 @@ static int mcm_audio_read_packet(AVFormatContext* avctx, AVPacket* pkt)
164166
if (err) {
165167
av_log(avctx, AV_LOG_ERROR, "Put buffer error: %s (%d)\n",
166168
mesh_err2str(err), err);
167-
return AVERROR(EIO);
169+
ret = AVERROR(EIO);
170+
goto error_close_conn;
168171
}
169172

170173
return len;
171174

172175
error_put_buf:
173176
mesh_put_buffer(&buf);
174177

178+
error_close_conn:
179+
err = mesh_delete_connection(&s->conn);
180+
if (err)
181+
av_log(avctx, AV_LOG_ERROR, "Delete mesh connection failed: %s (%d)\n",
182+
mesh_err2str(err), err);
183+
175184
return ret;
176185
}
177186

0 commit comments

Comments
 (0)