Yamux Listener Bug: IncompleteReadError when handling incoming connections

[acul71]

Summary

Python yamux has a listener-specific bug when handling incoming connections, specifically with TCP transport. The bug manifests as an IncompleteReadError in the handle_incoming method after successful connection establishment and ping/pong exchange.

Evidence

Test Results Pattern

Test	Direction	Transport	Result
jvm-v1.2 x python-v0.4 (tcp, noise, yamux)	JVM dialer → Python listener	TCP	❌ FAILS
python-v0.4 x jvm-v1.2 (tcp, noise, yamux)	Python dialer → JVM listener	TCP	✅ SUCCEEDS

Key Observations:

• Python yamux listener (TCP): 1/1 failures (100%)
• Python yamux dialer (TCP): 0/1 failures (0%)
• Mplex works in both directions, confirming this is yamux-specific

Error Details

Error Location: libp2p/stream_muxer/yamux/yamux.py - handle_incoming method

Error Message:

Error in handle_incoming for peer <peer_id>: Transport: unknown. IncompleteReadError: Connection closed during read operation: expected 2 bytes but received 0 bytes. This may indicate the peer closed the connection prematurely, a network issue, or a transport-specific problem (e.g., WebSocket message boundary handling).

Sequence of Events:

1. ✅ Connection established successfully
2. ✅ Noise handshake completes
3. ✅ Ping/pong exchange succeeds
4. ❌ Error occurs in handle_incoming (yamux frame parsing)
   • Error: IncompleteReadError: expected 2 bytes but received 0 bytes
   • This happens after the ping/pong succeeds
   • Suggests yamux is trying to read the next frame header but the connection is closed

Root Cause Hypothesis

The error occurs in Python's yamux handle_incoming method when it tries to read the next yamux frame header (2 bytes). The connection appears to be closed prematurely, possibly because:

1. The peer closes the connection after sending the pong, but Python yamux listener is still trying to read the next frame
2. Python yamux listener doesn't properly handle the case where the connection is closed while waiting for the next frame
3. Frame boundary issue: The yamux frame header read fails because the connection was closed at the wrong time

The fact that it works when Python is the dialer suggests the issue is specific to how Python yamux handles incoming connections (listener side) vs outgoing connections (dialer side).

Investigation Recommendations

1. Check Python Yamux Listener Implementation

File: libp2p/stream_muxer/yamux/yamux.py

Focus Areas:

• Incoming stream handling when Python is listener
• Yamux frame parsing for incoming connections
• Stream initialization for inbound streams
• Error handling during listener operations

2. Compare with Working Dialer Implementation

Since Python yamux dialer works correctly:

• Compare dialer vs listener code paths
• Identify differences in stream initialization
• Check for missing error handling in listener path

3. Test with Other Implementations

To confirm if this is implementation-specific or general:

• Test Python yamux listener with other implementations (Go, JS, Nim, Rust v0.54+)
• If it fails with multiple implementations, it's a Python yamux listener bug
• If it only fails with specific implementations, it may be a compatibility issue

Logs

Full logs are available from the transport-interop test suite:

• Failing test: jvm-v1.2 x python-v0.4 (tcp, noise, yamux)
• Successful test: python-v0.4 x jvm-v1.2 (tcp, noise, yamux)

Priority

Medium-High

• Not blocking (only affects specific implementation combinations)
• But fixing would improve Python yamux reliability
• Could prevent similar issues with other implementations

Related

This issue was discovered during interoperability testing. Given yamux's buggy history in py-libp2p, this listener bug should be investigated and fixed.

[acul71]

@sumanjeet0012 @seetadev

Log Analysis

Failing Test: jvm-v1.2 x python-v0.4 (tcp, noise, yamux)

Log File: logs/jvm-v1-2-x-python-v0-4--tcp--noise--yamux-.log

Key Error (from Python listener):

listener-1  | Error in handle_incoming for peer 12D3KooWEz7xLNJeDJXfajaAK7cgZZdqDKeUWuh5NQx5EV2fzm2L: Transport: unknown. IncompleteReadError: Connection closed during read operation: expected 2 bytes but received 0 bytes. This may indicate the peer closed the connection prematurely, a network issue, or a transport-specific problem (e.g., WebSocket message boundary handling).

Sequence of Events:

1. ✅ Connection established successfully
2. ✅ Noise handshake completes
3. ✅ Ping/pong exchange succeeds:
   • listener-1 | received ping from 12D3KooWEz7xLNJeDJXfajaAK7cgZZdqDKeUWuh5NQx5EV2fzm2L
   • listener-1 | responded with pong to 12D3KooWEz7xLNJeDJXfajaAK7cgZZdqDKeUWuh5NQx5EV2fzm2L
   • dialer-1 | Ping latency 2 ms
4. ❌ Error occurs in handle_incoming (yamux frame parsing)
   • Error: IncompleteReadError: expected 2 bytes but received 0 bytes
   • This happens after the ping/pong succeeds
   • Suggests yamux is trying to read the next frame header but the connection is closed

Analysis:

• The connection and handshake work correctly
• The ping/pong protocol works
• The failure occurs in yamux's handle_incoming method when trying to read the next yamux frame
• The error message shows "Transport: unknown" which suggests the transport type isn't being properly identified in the error context
• The IncompleteReadError with "expected 2 bytes but received 0" suggests the connection was closed while yamux was trying to read a frame header

Successful Test: python-v0.4 x jvm-v1.2 (tcp, noise, yamux)

Log File: logs/python-v0-4-x-jvm-v1-2--tcp--noise--yamux-.log

Key Observations:

• ✅ Full yamux debug logs showing successful frame parsing
• ✅ Stream creation and data exchange work correctly
• ✅ Window updates are sent and received properly
• ✅ Stream closure is handled gracefully
• ✅ No errors in handle_incoming

Yamux Frame Flow (from successful test):

dialer-1    | 2025-12-03 00:31:08,757 - libp2p.stream_muxer.yamux - DEBUG - Sending SYN header for stream 1
dialer-1    | 2025-12-03 00:31:08,764 - libp2p.stream_muxer.yamux - DEBUG - Received header for peer ...:type=0, flags=0, stream_id=1,length=20
dialer-1    | 2025-12-03 00:31:08,765 - libp2p.stream_muxer.yamux - DEBUG - Received header for peer ...:type=1, flags=2, stream_id=1,length=0
dialer-1    | 2025-12-03 00:31:08,765 - libp2p.stream_muxer.yamux - DEBUG - Received window update for stream...:1, increment: 0

Comparison

Aspect	Failing (Python Listener)	Successful (Python Dialer)
Connection	✅ Works	✅ Works
Noise Handshake	✅ Works	✅ Works
Ping/Pong	✅ Works	✅ Works
Yamux Frame Parsing	❌ Fails in handle_incoming	✅ Works
Error Location	handle_incoming method	No errors
Error Type	IncompleteReadError: expected 2 bytes but received 0	N/A
Timing	After ping/pong succeeds	N/A

[bomanaps]

Hello @acul71 can I take a look at this

[acul71]

Hello @acul71 can I take a look at this

You're going to have to setup test-plan transport or a local jvm-v1.2 to test with
Can you do that ?

[bomanaps]

Yes I can if I get any issues I canj ping you for directions

[acul71]

Yes I can if I get any issues I canj ping you for directions

How you doing?

[bomanaps]

Yes I can if I get any issues I canj ping you for directions

How you doing?

Am still working on it will drop an update on on friday or if am stuck at any point