Suggested Features for Initial Release of kad-dht

[seetadev]

@sumanjeet0012 : Thank you for the update — this is excellent progress!

Having FIND_NODE, PUT_VALUE, and GET_VALUE operational is a strong indicator that the core routing and value storage mechanisms of the DHT are working as intended. Local testing with multiple nodes is a great first step to validate functionality and ensure that node-to-node communication, record persistence, and routing table updates behave correctly in a controlled environment.

Suggested Features for Initial Release

For the initial release of Kademlia DHT, wish to share some additional capabilities and refinements:

Core Feature Completeness

1. Provider Records (Content Routing)

   • Implement ADD_PROVIDER / GET_PROVIDERS functionality.
   • Essential for content discovery, especially in networks like IPFS where content addressing is key.

2. Record Expiration & Replication

   • Implement expiration timers and republish/replicate mechanisms for stored records.
   • Ensures data availability and resilience over time.

3. Routing Table Maintenance

   • Refresh buckets periodically (k-bucket refreshing).
   • Implement logic to handle stale peers or ping-timeout detection (via PING or passive timeout).

4. Parallel Querying / α-Parameter Implementation

   • Implement α (alpha) concurrency in queries to improve robustness and latency in real-world conditions.

5. Handling NATed and Unreachable Peers

   • Introduce a strategy for discovering and working around NATed peers, or simulate these scenarios locally.

Advanced / Stretch Goals (optional for initial release, but helpful)

• DHT Query Logging and Diagnostics

  • Add detailed logging of queries to analyze routing path lengths, convergence times, and peer distributions.

• Simulated Large-Scale Testing

  • If real-world testing with 100+ nodes is difficult, consider using a simulation or emulation environment (e.g., libp2p-sim, mininet, or custom Docker setups).

• DHT Bootstrapping Enhancements

  • Robust bootstrap peer configuration with retry/backoff strategies to handle real network variability.

Testing Ideas for Large Networks

You're absolutely right that it’s hard to locally test scenarios involving hundreds of peers. Here are a few ways to approach this:

1. Simulated Network via Docker or Containers

   • Run 100+ containerized nodes using Docker Compose or Kubernetes. Assign different subnets or virtual IPs to simulate a more realistic environment.

2. Emulators / DHT Sim Tools

   • Tools like libp2p-sim, shadow, or building a mock transport layer can let you emulate DHT behavior at scale without needing real TCP/UDP connections.

3. Testnets or Remote Deployments

   • Deploy nodes across a cloud or distributed setup (e.g., DigitalOcean droplets or testnet environments) to better reflect production-like latency and routing behavior.

4. Metrics Collection

   • Implement lightweight telemetry (e.g., Prometheus or log aggregation) to observe behavior as the number of nodes increases.

Again, kudos on reaching this point — it’s a big milestone. With a few more utilities for data persistence, peer routing hygiene, and testing at scale, your DHT will be in great shape for an initial release!

[seetadev]

@sumanjeet0012 : That’s an excellent milestone — congratulations on getting yamux working in Python libp2p! Getting nodes to connect to mino bootstrapper nodes is a solid sign that your transport and multiplexing layers are functioning properly.

Now, regarding your observation:

“Nodes are now able to connect to the mino bootstrapper nodes but they are not responding to DHT queries.”

Wish to share some suggestions:

What is working

This is significant progress. You’ve overcome one of the major hurdles by achieving connection-level compatibility. The fact that connections are being established with nodes means:

• Transport (probably TCP or WebSocket) is working.
• Stream multiplexing (yamux) is working well enough to establish initial communication.
• Identity and encryption handshake (secio/noise) are likely working, or the nodes wouldn’t maintain a connection.

So now we are hitting the protocol interoperability layer, which is a very common next pain point when connecting different language implementations of libp2p.

🧠 Likely Cause

The issue you're encountering — where DHT queries get no response from mino bootstrapper nodes — is very likely due to incompatibility at the protocol level, not a transport or muxer issue. Here are a few possible root causes:

1. DHT protocol version mismatch

• Go-libp2p often uses /ipfs/kad/1.0.0.
• Python-libp2p might be using a different version string (or not implementing the same semantics).

Check:

# Verify this in your Python node
'/ipfs/kad/1.0.0' in protocols_supported

💡 Suggestion: Use identify protocol to introspect supported protocols on both sides and compare.

2. Your node doesn’t respond correctly to DHT protocol messages

• Even if both nodes support /ipfs/kad/1.0.0, they might interpret message formats or expectations differently.
• For example, a FIND_NODE or GET_PROVIDERS request may not be serialized/deserialized the same way.

💡 Suggestion: Log raw DHT messages and compare byte-level or protobuf-level encoding with Go-libp2p.

3. Stream not being properly opened

• Your node may open a stream to the peer but fail to write/read data over it.
• If the Python node doesn’t write the query or the remote doesn’t understand it, it will appear as "no response".

💡 Suggestion: Add detailed logging for stream open, write, read, close events for the /ipfs/kad/1.0.0 protocol.

4. Missing or partial DHT support in Python libp2p

• Some Python implementations don’t fully support DHT (especially provider or value records), and only handle routing table or limited record types.

💡 Suggestion: Double-check what parts of Kademlia your Python node supports. If only FIND_NODE is implemented, you won't get responses to GET_PROVIDERS.

---

🔧 Suggestions and Next Steps

✅ 1. Use identify protocol

• Check what protocols each node reports supporting. If the DHT protocol doesn’t match exactly, you won’t get responses.

✅ 2. Cross-test with a Go or JS node

• Run your Python node and try DHT queries to known public DHT nodes (or local Go-libp2p node).
• Use the Go-libp2p DHT CLI tool to check behavior:

libp2p-relay dht findpeer <peer-id>

✅ 3. Enable verbose DHT logging

• On the bootstrapper side (e.g., Go), enable DHT debug logs.
• On your side, log inbound/outbound messages and stream events.

✅ 4. Test with a Python-to-Python network

• Set up two Python nodes with DHT enabled and see if they can respond to each other.
• If yes → it’s likely an interop bug.
• If not → issue may lie in your own DHT handler.

---

🛠️ Long-Term Fixes or Actions

If you confirm this is an interop issue, consider:

• Filing a detailed issue in the py-libp2p GitHub repo.
• Sharing logs (with raw messages) and protocol negotiations.
• Collaborating with libp2p maintainers (especially if this is a blocking issue for your project like Filecoin integration).

---

🧭 Great Progress

Getting multiplexer-level compatibility and seeing successful peer connections is a sign that the core libp2p plumbing is solid. Now you’re at the nuance of cross-impl protocol semantics, which is where most real interop work lives. Keep debugging and you’ll likely uncover something valuable not just for you, but for the ecosystem.

[sumanjeet0012]

@seetadev, @acul71

I’m happy to share that the Python node can now communicate successfully with Amino bootstrapper nodes and execute DHT queries. Huge thanks to @acul71 and @paschal533 for implementing Yamux in py-libp2p — this wouldn’t have been possible without their contribution!

As I continue working toward the initial release of the Kademlia DHT, I’d like to confirm whether I’m on the right track with a few design and implementation aspects:

---

1. Node Modes

Currently, there’s no distinction between different types of nodes in the implementation. I was wondering if it would make sense to introduce separate server and client modes.

From my understanding:

• A server node would include full DHT functionality — routing table, value store, provider store, and handle incoming DHT queries.
• A client node would not maintain any state (no routing table, value store, or provider store), and would not register stream handlers for DHT queries.

Would this separation be appropriate for the initial release? And if yes, should the client mode include any additional constraints or behaviors?

---

2. Query Optimization

• Parallel Queries: Running up to α = 3 parallel queries to peers during lookups seems like a good balance between speed and network efficiency. This is straight forward and easy to implement.

• Caching & Query Prioritization: Are these features necessary for the first release? If so, could you provide some guidance or pseudo-code to help implement them?

---

3. Error Recovery and Reporting

I'm not yet sure whether structured error handling and reporting are required for the initial release. Would you recommend including them at this stage? If yes, any high-level guidance or examples would be greatly appreciated.

---

4. Testing Plan

Here is the tentative list of test cases I’ve planned. Could you please confirm whether these are sufficient, or suggest any additions?

Unit Testing

• Routing Table

  • Add a peer
  • Add a duplicate peer
  • Remove a peer
  • Remove a non-existent peer
  • Add a peer to a fully filled routing table

• Value Store

  • Store a value for a key
  • Retrieve a value for a key
  • Retrieve a non-existent key
  • Overwrite an existing key
  • Retrieve expired values

• Provider Store

  • Add a provider for a key
  • Retrieve providers for a key
  • Multiple providers for the same key
  • Remove a provider
  • Expired providers
  • Add a duplicate provider
  • Missing key should return empty list

Integration Testing (using two Python nodes)

• FIND_NODE
• ADD_VALUE
• GET_VALUE
• ADD_PROVIDER
• GET_PROVIDER

Would you recommend adding any additional test scenarios or edge cases?

---

Looking forward to your feedback to ensure I’m heading in the right direction.

[sumanjeet0012]

@seetadev I have uploaded demo video and a developer documentation.
Developer docs might need some corrections, I will do it soon.

Developer documentation: https://docs.google.com/document/d/1VCAdMtZjtb-Po0_74Uhm6VZuWF52OLK3/edit?usp=sharing&ouid=108233186261105252601&rtpof=true&sd=true

Kademlia-demo video: https://drive.google.com/file/d/1FbeLUtnCXNBiM_zp_LPK7lxwwaijflrP/view?usp=drive_link