Noise module feature addition + revamp

[varun-r-mallya]

The noise module in py-libp2p is not updated to the spec, so I will be describing in detail the architecture of go-libp2p so we can replicate this in py-libp2p to make sure that the noise module is interoperable. I think a great understanding of the design initially will save me a lot of time while actually implementing it.

I hope this can also be used as documentation for the noise module in go-libp2p as well. I plan to work on that too.

@pacrob @seetadev @acul71 @sukhman-sukh
Please give me pointers that you feel are necessary.

My course of action will be as follows:

1. Identify differences between the two implementations.
2. Find out why they are different and if that has to be fixed or not.
3. Fix it.
4. Move on to adding newer features in the following order
   a. extensions
   b. early data handling

I will be providing a detailed data flow analysis of the go-libp2p here. It took a hell lot of time to make, so I really hope it saves time later. Also, I used no AI to make sure all my analysis is not hallucinated.

go-libp2p noise module data flow analysis

This module implements the Noise Protocol Framework for libp2p, providing secure transport layer encryption. It uses the XX handshake pattern with Curve25519 key exchange, ChaCha20-Poly1305 encryption, and SHA256 hashing.

Transport layer link

type Transport struct {
    protocolID protocol.ID     // "/noise"
    localID    peer.ID         // Local peer identifier
    privateKey crypto.PrivKey  // Local identity key
    muxers     []protocol.ID   // Supported stream multiplexers
}

This is the main transport struct. There is a diff between this and python. Especially that the Python implementation contains early data field here, but this does not.

Constructor

New(id protocol.ID, privkey crypto.PrivKey, muxers []tptu.StreamMuxer) -> (*Transport, error)

Takes in protocol ID, private key and muxer list.

Outbound connection function

SecureOutbound(ctx, insecure, peerID) -> (SecureConn, error)

1. SecureOutbound() called with insecure connection and target peer ID
2. Creates transportEarlyDataHandler for muxer negotiation (THIS IS IMPORTANT FOR US!!!)
3. Calls newSecureSession() with initiator=true, checkPeerID=true
4. Returns session wrapped with connection state via SessionWithConnState()

Outbound connection function

SecureInbound(ctx, insecure, peerID) -> (SecureConn, error)

1. SecureInbound() called with insecure connection and expected peer ID
2. Creates transportEarlyDataHandler for muxer negotiation
3. Calls newSecureSession() with initiator=false, checkPeerID=(peerID != "")
4. Returns session wrapped with connection state

Session Management link

type secureSession struct {
    // Identity & Connection
    initiator   bool           // Is this the connection initiator?     important
    checkPeerID bool           // Should we verify peer identity?  important
    localID     peer.ID        // Our peer ID  important
    localKey    crypto.PrivKey // Our private key  important
    remoteID    peer.ID        // Remote peer ID  impotant
    remoteKey   crypto.PubKey  // Remote public key  important
    
    // I/O Management
    readLock      sync.Mutex      // useless for me
    writeLock     sync.Mutex      // uselsss
    insecureConn  net.Conn        // underlying connection  imporant
    insecureReader *bufio.Reader  // buffered reader  idk what this is
    
    // Message Buffering
    qseek int      // Queue seek position  important, but for high loads only ig. So low priority
    qbuf  []byte   // Queued message buffer  same as above
    rlen  [2]byte  // Length prefix work buffer  same as above
    
    // Cryptographic State
    enc *noise.CipherState  // Encryption cipher  IMPORTANT
    dec *noise.CipherState  // Decryption cipher IMPORTANT
    
    // Protocol Extensions
    prologue                  []byte           // Handshake prologue                    ABSOLUTELY IMP
    initiatorEarlyDataHandler EarlyDataHandler // Client early data     ABSOLUTELY IMP
    responderEarlyDataHandler EarlyDataHandler // Server early data    ABSOLUTELY IMP
    connectionState           network.ConnectionState // Connection metadata   ABSOLUTELY IMP
}

interfaces implemented: sec.SecureConn, net.Conn

newSecureSession(tpt, ctx, insecure, remote, prologue, initiatorEDH, responderEDH, initiator, checkPeerID) -> (*secureSession, error)

1. Creates session struct with parameters
2. Launches goroutine to run runHandshake(ctx)
3. Waits for handshake completion or context cancellation
4. Returns configured session or error

Methods

• LocalPeer() -> peer.ID - Returns local peer ID
• RemotePeer() -> peer.ID - Returns remote peer ID
• LocalPublicKey() -> crypto.PubKey - Returns local public key
• RemotePublicKey() -> crypto.PubKey - Returns remote public key
• ConnState() -> network.ConnectionState - Returns connection metadata
• net.Conn methods (SetDeadline, Close, etc.)

func SessionWithConnState(s *secureSession, muxer protocol.ID) *secureSession {
	if s != nil {
		s.connectionState.StreamMultiplexer = muxer
		s.connectionState.UsedEarlyMuxerNegotiation = muxer != ""
	}
	return s
}

The above method is something I don't understand yet, so I'd pointers on it. @pacrob

Handshake Protocol link

This method was the actual one that was run in session.go, so this is more important than session.go and it's just a wrapper to this.

var cipherSuite = noise.NewCipherSuite(
    noise.DH25519,        // Curve25519 key exchange
    noise.CipherChaChaPoly, // ChaCha20-Poly1305 AEAD
    noise.HashSHA256      // SHA256 hash function
)

then the runHandshake function is described.

Initiator path

Functions work like the following:

• Generate static keypair
• Init handshake with XX
• Stage 0
  • send ephemeral public key of 32 bytes
• Stage 1 - Received response
  • receive response
  • get ephemeral + static + payload
  • handleRemoteHandshakePayload
  • call initiatorEarlyDataHandler.Received with NoiseExtensions in the early data field. Here, we need to keep it empty.
• Stage 2 - Final message
  • Some Notes: the interface is defined in session, but the actual code is written in noise_early_data.go under webtransport for some reason.
  • generateHandshakePayload() creates signed payload
  • sendHandshakeMessage(hs, payload, hbuf)
  • Sends static key + signed payload
  • setCipherStates() initializes encryption

Responder path

Functions work like the following:

• Generate static keypair
• Same config initialize handshake
• Stage 0: receive ephemeral
• Stage 1: Send response
  • generateHandshakePayload() with early data
  • sendHandshakeMessage() sends ephemeral + static + payload
  • Note for above:

  func (s *secureSession) generateHandshakePayload(localStatic noise.DHKey, ext *pb.NoiseExtensions) ([]byte, error)

  • here early data is just noise extensions and it just literally used to initialise payload.
  • Send the ephemeral + static + payload after this.
• Stage 2: Receive final
  • readHandshakeMessage(hs) gets final message
  • handleRemoteHandshakePayload() validates identity
  • responderEarlyDataHandler.Received() processes data
  • Notes about the above: Implemented as abstract function that needs to be passed into it's parent. havent yet managed to find it's actual implementation.
  • setCipherStates() initializes encryption

Payload generation & Validation

Payload creation

generateHandshakePayload(localStatic noise.DHKey, ext *pb.NoiseExtensions) -> ([]byte, error)

1. Marshal local libp2p public key: crypto.MarshalPublicKey()
2. Create signature payload: payloadSigPrefix + localStatic.Public
3. Sign with identity key: s.localKey.Sign(toSign)
4. Create protobuf payload with key, signature, extensions
5. Marshal to bytes: proto.Marshal()

Payload Validation

handleRemoteHandshakePayload(payload []byte, remoteStatic []byte) -> (*pb.NoiseExtensions, error)

1. Unmarshal protobuf: proto.Unmarshal(payload, &NoiseHandshakePayload{})
2. Extract public key: crypto.UnmarshalPublicKey()
3. Derive peer ID: peer.IDFromPublicKey()
4. Verify peer ID matches expected (if checking enabled)
5. Verify signature: remotePubKey.Verify(msg, sig)
6. Set remote identity and return extensions (IMPORTANT: Just return the extensions)

Cryptographic Ops link

Just encrypt and decrypt functions with

• encrypt input as plaintext in bytes

  • output is 16 bytes Polu1305 text

• decrypt input as ciphertext and output is plaintext with removed tags
  Note: for both of them we need to do something like the following

	enc *noise.CipherState
	dec *noise.CipherState

And CipherState is a go noise library thing.

I/O ops and helper IO function, I will not be documenting or reading

Early Data Handling and Extensions (Were actually the issue was supposed to solve how the problem is more and more examples. link

EarlyDataHandler Interface

gotype EarlyDataHandler interface {
    Send(context.Context, net.Conn, peer.ID) -> *pb.NoiseExtensions
    Received(context.Context, net.Conn, *pb.NoiseExtensions) -> error
}

Transport Early Data Handler

gotype transportEarlyDataHandler struct {
    transport      *Transport
    receivedMuxers []protocol.ID
}

• Note: MatchMuxers(isInitiator bool): Finds common multiplexer

Muxer Negotiation Flow:

• Initiator sends: List of supported muxers in handshake
• Responder receives: Stores muxer list, sends own list back
• Initiator receives: Responder's muxer list
• Match determination: matchMuxers(local, remote) finds first common muxer (this has some weird algo associated with it)
• Connection state: Negotiated muxer stored in ConnectionState

EarlyDataHandling interface:

• EarlyData configures handlers for sending and receiving early data during the Noise handshake.
• It takes two EarlyDataHandler parameters:
• initiator: handles early data when this transport acts as the handshake initiator (client)
• responder: handles early data when this transport acts as the handshake responder (server)
  Early data allows application payloads to be sent in handshake messages:
• Responder early data goes in the 2nd handshake message (encrypted but peer not yet authenticated)
• Initiator early data goes in the 3rd handshake message (encrypted and peer authenticated)
  This returns a SessionOption that can be passed to configure a SessionTransport.
  DisablePeerIDCheck() function in this Equivalent of calling SecureInbound with empty ID. Susceptible to MITM, so be careful of such issues. Also sets the handlers

Now, Looking at this class design, Ill be able to make a similar thing for Python after matching spec first

[acul71]

Response to Varun's Noise Module Proposal

Overview

Great analysis, Varun! Your detailed breakdown of the go-libp2p architecture is excellent and will be very valuable for implementation.

Current State Reality Check

Based on discussion #810, the current Python noise implementation is actually in good shape:

✅ What's Working

• Full interoperability with go-libp2p (all transport tests pass)
• Core protocol implementation is solid
• Security guarantees are identical to Go
• XX Pattern implementation is correct

❌ What's Actually Missing

• Extensions Support (WebTransport compatibility)
• Early Data Handler interface
• Enhanced error handling

Key Corrections

Common Misconceptions to Address:

• Noise Pipes: Not needed - IK pattern has been removed from the spec
• Rekey Support: Doesn't exist in Go-libp2p (was incorrectly mentioned)
• Current Implementation: More aligned with current spec than some discussions suggest

Recommended Approach: Targeted Improvements

Rather than a complete rewrite, I suggest:

Phase 1: Add Missing Features

1. Extensions Support - Add NoiseExtensions protobuf and handshake payload updates
2. Early Data Handler - Implement the interface for early data processing
3. Clean up noise pipes references - Remove IK pattern support (no longer in spec)

Phase 2: Enhance Existing Code

1. Improve error handling - Add specific error types
2. Update session management - Enhance connection state handling
3. Add comprehensive tests - Ensure interoperability

Why Not Rewrite?

Pros of Improving Existing:

• Faster progress - incremental improvements
• Lower risk - smaller, focused changes
• Preserved work - existing improvements remain valuable
• Community stability - no breaking changes

Cons of Rewriting:

• Massive time investment (months of development)
• Risk of regressions while fixing old issues
• Lost work - existing improvements would be discarded

Next Steps

1. Start with extensions support - highest priority missing feature
2. Add early data handling - enables advanced use cases
3. Use your analysis as blueprint - perfect guide for what to implement

Your analysis provides the ideal roadmap for enhancing the solid foundation that's already there. Let's build on what works rather than starting from scratch.

---

[varun-r-mallya]

I am not proposing a full rewrite, but a remodel of the classes. A full rewrite is not required. I'll make the changes slowly and add more comprehensive tests