Latency exceeding 1000 milliseconds

[iceriver97]

Generated by Generative AI

No response

Operating System:

Linux integration02 5.15.148-rt-tegra #1 SMP PREEMPT_RT Mon Oct 14 17:05:40 PDT 2024 aarch64 aarch64 aarch64 GNU/Linux

ROS version or commit hash:

humble

RMW implementation (if applicable):

rmw_zenoh && rmw_fastrtps_cpp

RMW Configuration (if applicable):

router:
{
mode: "router",
listen: {
endpoints: ["tcp/0.0.0.0:7447#so_rcvbuf=134217728;so_sndbuf=134217728"],
},
transport: {
shared_memory: {
enabled: true,
transport_optimization: {
enabled: true,
pool_size: 536870912,
},
},
link: {
tx: {
queue: {
size: { data: 16 }, // 转发队列深度
},
},
rx: {
buffer_size: 134217728, // 路由器接收缓冲区
},
},
},
adminspace: {
enabled: true,
permissions: { read: true, write: false },
},
}
session:
{
mode: "peer",
connect: {
endpoints: ["tcp/localhost:7447#so_rcvbuf=16777216;so_sndbuf=16777216"],
timeout_ms: -1,
},
transport: {
shared_memory: {
enabled: true,
mode: "init",
transport_optimization: {
enabled: true,
pool_size: 268435456,
message_size_threshold: 512,
},
},
link: {
tx: {
queue: {
size: {
data: 16,
},
congestion_control: {
drop: {
wait_before_drop: 100000,
},
},
},
},
rx: {
buffer_size: 33554432,
},
},
},
qos: {
publication: [
{
key_exprs: [
"**"
],
config: {
congestion_control: "drop",
priority: "data_high",
reliability: "best_effort",
},
}
]
}
}

Client library (if applicable):

No response

'ros2 doctor --report' output

ros2 doctor --report

<COPY OUTPUT HERE>

Steps to reproduce issue

The latency measurement is calculated by the subscriber. After the subscriber enters the callback function, it prints the difference between the current timestamp and the message header timestamp.
The message type is point cloud data. The publisher and subscriber run in different Docker containers. Under the same conditions, compare the latency of FastRTPS and Zenoh.

Expected behavior

The two should be quite similar or have only minor differences.

Actual behavior

Zenoh has experienced latency exceeding 1000ms on multiple occasions.

Additional information

No response

[JEnoch]

I believe the shared memory is likely not used between your containers. If Zenoh cannot use shared memory, it automatically falls back to TCP communication.

How are configured your containers for SHM ?
Did you try to set the ZENOH_SHM_ALLOC_SIZE environment variable to a bigger size than the default (50331648 bytes which is 48MiB) ?

[iceriver97]

I believe the shared memory is likely not used between your containers. If Zenoh cannot use shared memory, it automatically falls back to TCP communication.

How are configured your containers for SHM ? Did you try to set the ZENOH_SHM_ALLOC_SIZE environment variable to a bigger size than the default (50331648 bytes which is 48MiB) ?

The containers all started with --privileged. The aforementioned latency anomaly is intermittent, but it will definitely occur in long-term testing. In contrast, FastRTPS did not exhibit similar issues. A latency of less than 100ms should be normal, including LiDAR scanning time and data transmission time. ZENOH_SHM_ALLOC_SIZE was not set. Additionally, could you please tell me when Zenoh will support zero-copy? In comparative tests, FastRTPS's data streaming latency performance is better. I previously saw that Zenoh API support was updated, and its latency performance is quite excellent, but I haven't conducted long-term testing yet. Will there be further updates?