mDNS discovery: nodes visible but fail to establish connections

[doogie-bigmack]

Description

Running a 4-node M3 Ultra Mac Studio cluster, nodes successfully discover each other via mDNS but fail to establish actual connections for distributed inference.

Environment

• OS: macOS Sonoma 15.x
• Hardware: 4x M3 Ultra Mac Studios (192GB RAM each)
• Network: Same LAN, mDNS enabled
• Node names: cluster-1, cluster-3, cluster-4, cluster-6

Observed Behavior

1. Start EXO on all 4 nodes
2. Nodes appear in each other's peer lists (mDNS discovery works)
3. When initiating distributed inference, connections between nodes fail/timeout
4. Work is not distributed - only local node processes the request

Expected Behavior

Once nodes discover each other, they should successfully establish connections and distribute inference workload.

Debug Observations

When investigating the discovery mechanism in exo/networking/discovery.rs, we noticed the TTL configuration:

Duration::from_secs(2_500)  // This equals ~41 minutes

We suspected this might be a typo and should be from_millis(2_500) (2.5 seconds), which is more typical for mDNS refresh intervals. After making this change locally, nodes were able to connect successfully.

Questions

1. Is the 2500-second TTL intentional? Seems very long for dynamic peer discovery
2. Could the long TTL cause stale peer information that breaks connection establishment?
3. Are there recommended network/firewall settings we should verify?

Workaround

Building from source with modified TTL values appeared to resolve the issue, but want to understand if this is the actual root cause or if something else is happening.

Logs

Happy to provide debug logs if you can point me to how to enable verbose mDNS/networking logging.

---

Related to closed PR #1297 - opening as issue per maintainer request to investigate further.

[Evanev7]

Work is not distributed - only local node processes the request
Nodes appear in each other's peer lists

when you say peer lists, do you mean within the EXO dashboard? if you are able to place models across multiple nodes and that is failing, then mDNS is working fine and the issue is downstream. Can you tell me what network configuration the nodes have? We use an epehemeral port so a harsh firewall will block traffic.

[doogie-bigmack]

Network Details

Thanks for the quick response! Let me clarify:

What I Mean by "Peer Lists"

Yes, the EXO web dashboard (http://localhost:52415). When I open the topology view:

• All 4 nodes appear with their hostnames
• The UI shows them as discovered/connected
• Model placement dropdown shows all 4 nodes as available targets

So mDNS discovery is definitely working — nodes see each other.

Network Configuration

Hardware:

• 4x M3 Ultra Mac Studios
• All on same LAN: 192.168.1.0/24
• Connected via UniFi managed switch (1Gbps)

Firewall Status:

# On each node:
$ /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate
Firewall is disabled. (State = 0)

macOS firewall is disabled on all nodes. No third-party firewall software installed.

UniFi Network Settings:

• mDNS reflection: enabled
• IGMP snooping: disabled
• No VLANs — all devices on default network
• No inter-device firewall rules

The Actual Failure Mode

When I initiate inference:

1. Local node starts processing
2. Dashboard shows the job started
3. Remote nodes never receive work — their logs stay quiet
4. All compute happens on the originating node
5. No explicit error message about failed connections

It's as if the orchestrator decides not to distribute, or the connection handshake silently fails.

What Fixed It (For Us)

When we built from source and changed this in exo/networking/discovery.rs:

// Before:
Duration::from_secs(2_500)  // 41+ minutes

// After:
Duration::from_millis(2_500)  // 2.5 seconds

After this change, distributed inference worked immediately across all nodes.

Questions

1. Ephemeral ports — What port range does EXO use for inter-node communication? I can verify those are open (though with firewall disabled, they should be)

2. Verbose logging — Is there a way to enable debug logging for the networking/connection layer? Would help pinpoint where exactly the handshake fails

3. TTL value — Is the 2500-second TTL intentional? Curious about the design decision since most mDNS implementations use TTLs in the 1-60 second range

Happy to run any diagnostic commands or test builds. Have all 4 nodes available right now.

[Evanev7]

regarding your questions:

1. exo runs with the default macos firewall happily, we use ephemeral ports which you can see in the logs - look for hosts_by_node or ibv_coordinator depending on if you use ring or jaccl.
2. -v adds verbose logging - your issue doesn't touch rust code, but you can also configure rust logs with the RUSTLOG env var.
3. the 2500 second TTL was an intentional decision - while not mine - and it has not caused issues so far.

Are you certain you're loading the model across all macs? How are you launching the instance?