Skip to content

[Bug] Message Duplication and Out-of-Order Delivery in Multi-Bridge Setup #542

@jj7258

Description

@jj7258

Describe the bug

Issue Description

When running zenoh-bridge-ros2dds on multiple machines connected to the same Zenoh router, ROS 2 messages are being duplicated extensively and delivered out of order. A single message published by a talker node results in dozens of duplicate messages received by the listener, with sequence numbers appearing in random order.

Setup Architecture

Desktop (192.168.0.XXX):          Laptop (192.168.0.x):
├── Zenoh Router (port 7447)      ├── zenoh-bridge-ros2dds
├── zenoh-bridge-ros2dds          └── ROS 2 listener node
└── ROS 2 talker node

Configuration Files

Zenoh Router Config (zenoh_router_config.json5)

{
  mode: "router",
  listen: {
    endpoints: ["tcp/0.0.0.0:7447"]
  },
  scouting: {
    multicast: {
      enabled: true
    }
  }
}

Bridge Config (zenoh_bridge_config.json5) - Used on both machines

{
  mode: "client",
  connect: {
    endpoints: ["tcp/127.0.0.1:7447"]  // "tcp/Desktop_IP:7447" on laptop
  },
  scouting: {
    multicast: {
      enabled: false
    },
    gossip: {
      enabled: false
    }
  },
  plugins: {
    ros2dds: {
      domain: 70,
      ros_localhost_only: false,
      reliable_routes_blocking: true,
      transient_local_cache_multiplier: 0,
    }
  }
}

Docker Compose (docker-compose.yml)

services:
  zenoh_router:
    image: eclipse/zenoh:latest
    container_name: zenoh_router
    volumes:
      - ./zenoh_router_config.json5:/zenoh/zenoh_router_config.json5:ro
    command: ["-c", "/zenoh/zenoh_router_config.json5"]
    network_mode: host

  zenoh_bridge:
    image: eclipse/zenoh-bridge-ros2dds:latest
    network_mode: host
    init: true
    stdin_open: true
    tty: true
    volumes:
      - ./zenoh_bridge_config.json5:/zenoh/zenoh_bridge_config.json5:ro
    command: ["-c", "/zenoh/zenoh_bridge_config.json5"]
    depends_on:
      - zenoh_router

Problem Demonstration

Expected Behavior

[INFO] [listener]: I heard: [Hello World: 1]
[INFO] [listener]: I heard: [Hello World: 2]
[INFO] [listener]: I heard: [Hello World: 3]

Actual Behavior

[INFO] [1758189864.780930432] [listener]: I heard: [Hello World: 599]
[INFO] [1758189864.781051947] [listener]: I heard: [Hello World: 600]
[INFO] [1758189864.781274747] [listener]: I heard: [Hello World: 600]
[INFO] [1758189864.781420818] [listener]: I heard: [Hello World: 599]
[INFO] [1758189864.781503294] [listener]: I heard: [Hello World: 601]
[INFO] [1758189864.782485288] [listener]: I heard: [Hello World: 601]
[INFO] [1758189864.782609293] [listener]: I heard: [Hello World: 599]
[INFO] [1758189864.782696983] [listener]: I heard: [Hello World: 600]
[INFO] [1758189864.782779420] [listener]: I heard: [Hello World: 599]
[INFO] [1758189864.782878196] [listener]: I heard: [Hello World: 601]

Topic Info Shows Multiple Publishers

$ ros2 topic info /chatter -v
Publisher count: 3

Node name: talker
Node namespace: /
Topic type: std_msgs/msg/String
Endpoint type: PUBLISHER
GID: 01.10.02.12.87.1a.46.c5.2c.86.3f.9a.00.00.15.03.00.00.00.00.00.00.00.00

Node name: zenoh_bridge_ros2dds  # Bridge 1
Node namespace: /
Topic type: std_msgs/msg/String
Endpoint type: PUBLISHER
GID: 01.10.c1.4d.02.99.e2.07.d9.c1.17.41.00.00.07.03.00.00.00.00.00.00.00.00

Node name: zenoh_bridge_ros2dds  # Bridge 2
Node namespace: /
Topic type: std_msgs/msg/String
Endpoint type: PUBLISHER
GID: 01.10.fa.3d.fa.a7.e8.7e.b0.a5.51.c4.00.00.80.03.00.00.00.00.00.00.00.00

Root Cause Analysis

The issue appears to be caused by:

  1. Routing Loop: Both bridges can discover each other over DDS (due to ros_localhost_only: false)
  2. Multiple Data Paths: Messages travel through multiple routes:
    • Direct DDS between bridges
    • Via Zenoh router between bridges
  3. No Duplicate Detection: The same message gets re-published multiple times through different paths

Bridge Startup Logs

Desktop Bridge

2025-09-18T10:16:02.552176Z  INFO main ThreadId(01) zenoh::net::runtime: Using ZID: b7c8364e1b8d5c1f1e4a5e7c8d9f2a3b
2025-09-18T10:16:02.552400Z  INFO main ThreadId(01) zenoh::api::loader: Starting required plugin "ros2dds"
2025-09-18T10:16:02.553200Z  INFO tokio-runtime-worker ThreadId(16) zenoh_plugin_ros2dds: ROS2 plugin Config { 
  domain: 70, 
  ros_localhost_only: false, 
  reliable_routes_blocking: true,
  transient_local_cache_multiplier: 0
}

Laptop Bridge

2025-09-18T10:16:08.445123Z  INFO main ThreadId(01) zenoh::net::runtime: Using ZID: d72192711af68788a44fe79a5b2215a3
2025-09-18T10:16:08.446789Z  INFO tokio-runtime-worker ThreadId(10) zenoh_plugin_ros2dds: Discovered ROS node: talker -> ZID b7c8364e1b8d5c1f1e4a5e7c8d9f2a3b

Questions

  1. Is this expected behavior when running multiple bridges with ros_localhost_only: false?

  2. What is the recommended configuration for preventing routing loops in multi-bridge setups while maintaining connectivity?

  3. Are there configuration options to enable duplicate message detection/filtering in zenoh-bridge-ros2dds?

  4. Should different bridge instances use different ROS domains or namespaces to prevent interference?

Attempted Solutions

  1. Setting ros_localhost_only: true: Prevents the bridge from discovering local ROS nodes
  2. Disabling multicast/gossip: Did not resolve the duplication issue
  3. Using reliable_routes_blocking: true: Did not prevent out-of-order delivery

Expected Solution

A configuration that allows:

  • Bridge on desktop to discover local ROS nodes
  • Bridge on laptop to receive messages via Zenoh only
  • No routing loops or message duplication
  • Preserved message ordering

To reproduce

Steps to Reproduce

Prerequisites

  • Two Linux machines on the same network
  • Docker installed on both machines
  • ROS 2 Humble installed on both machines

Step 1: Setup Desktop (192.168.0.XXX)

  1. Create configuration files:

    mkdir zenoh-test && cd zenoh-test
    
    # Create router config
    cat > zenoh_router_config.json5 << EOF
    {
      mode: "router",
      listen: {
        endpoints: ["tcp/0.0.0.0:7447"]
      },
      scouting: {
        multicast: {
          enabled: true
        }
      }
    }
    EOF
    
    # Create bridge config
    cat > zenoh_bridge_config.json5 << EOF
    {
      mode: "client",
      connect: {
        endpoints: ["tcp/127.0.0.1:7447"]
      },
      scouting: {
        multicast: {
          enabled: false
        },
        gossip: {
          enabled: false
        }
      },
      plugins: {
        ros2dds: {
          domain: 70,
          ros_localhost_only: false,
          reliable_routes_blocking: true,
          transient_local_cache_multiplier: 0,
        }
      }
    }
    EOF
  2. Create docker-compose.yml:

    services:
      zenoh_router:
        image: eclipse/zenoh:latest
        container_name: zenoh_router
        volumes:
          - ./zenoh_router_config.json5:/zenoh/zenoh_router_config.json5:ro
        command: ["-c", "/zenoh/zenoh_router_config.json5"]
        network_mode: host
    
      zenoh_bridge:
        image: eclipse/zenoh-bridge-ros2dds:latest
        network_mode: host
        init: true
        stdin_open: true
        tty: true
        volumes:
          - ./zenoh_bridge_config.json5:/zenoh/zenoh_bridge_config.json5:ro
        command: ["-c", "/zenoh/zenoh_bridge_config.json5"]
        depends_on:
          - zenoh_router
  3. Start services:

    docker compose up -d
  4. Run talker node:

    export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
    export ROS_DOMAIN_ID=70
    ros2 run demo_nodes_cpp talker

Step 2: Setup Laptop

  1. Create bridge config (modify endpoint to point to desktop):

    cat > zenoh_bridge_config.json5 << EOF
    {
      mode: "client",
      connect: {
        endpoints: ["tcp/Desktop_IP:7447"]
      },
      scouting: {
        multicast: {
          enabled: false
        },
        gossip: {
          enabled: false
        }
      },
      plugins: {
        ros2dds: {
          domain: 70,
          ros_localhost_only: false,
          reliable_routes_blocking: true,
          transient_local_cache_multiplier: 0,
        }
      }
    }
    EOF
  2. Start bridge:

    docker run --rm --network=host \
      -v $(pwd)/zenoh_bridge_config.json5:/zenoh/config.json5 \
      eclipse/zenoh-bridge-ros2dds:latest -c /zenoh/config.json5
  3. Run listener node:

    export RMW_IMPLEMENTATION=rmw_cyclonedx_cpp
    export ROS_DOMAIN_ID=70
    ros2 run demo_nodes_cpp listener

Step 3: Observe the Issue

  1. Check topic info:

    ros2 topic info /chatter -v

    Expected: 1 publisher (talker)
    Actual: 3 publishers (talker + 2 bridge instances)

  2. Monitor listener output:

    ros2 topic echo /chatter

    You will see multiple copies of each message delivered out of order.

Expected vs Actual Results

Expected: Sequential message delivery with no duplicates
Actual: Extensive message duplication with out-of-order delivery (10+ copies per message)

System info

  • Zenoh Version: 1.5.1
  • zenoh-bridge-ros2dds Version: 1.5.1
  • ROS 2 Distribution: Humble
  • RMW Implementation: rmw_cyclonedds_cpp
  • Operating System: Linux (Ubuntu)
  • Network Setup: Two machines on same LAN (192.168.0.x)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions