py-libp2p Multibase Integration Status and Improvement Opportunities

[acul71]

py-libp2p Multibase Integration Status and Improvement Opportunities

Executive Summary

This document analyzes the current usage of base encoding functionality in py-libp2p and identifies opportunities to leverage py-multibase (v2.0.0+) to improve code quality, interoperability, and maintainability.

Key Finding: py-libp2p currently uses direct base58 and base64 encoding without multibase prefixes. The codebase does not import or use the py-multibase library. Integrating py-multibase can provide self-describing encodings, better error handling, encoding detection, and support for multiple encoding formats.

---

py-multibase Overview

What is Multibase?

Multibase is a protocol for distinguishing base encodings and ensuring full compatibility with program interfaces. It answers the question: Given data d encoded into string s, how can I tell what base d is encoded with?

Multibase solves the problem by prefixing encoded data with a single character that identifies the encoding used, making the encoding self-describing.

py-multibase Features (v2.0.0)

Core Functions

1. encode(encoding, data) - Encode data with a multibase prefix

   >>> from multibase import encode
   >>> encode('base58btc', 'hello world')
   b'zStV1DL6CwTryKyV'  # 'z' prefix indicates base58btc
   >>> encode('base64', 'hello world')
   b'mGhlbGxvIHdvcmxk'  # 'm' prefix indicates base64

2. decode(data, return_encoding=False) - Decode multibase-encoded data

   >>> from multibase import decode
   >>> decode('mGhlbGxvIHdvcmxk')
   b'hello world'
   >>> encoding, data = decode('mGhlbGxvIHdvcmxk', return_encoding=True)
   >>> print(f'Encoded with {encoding}: {data}')

3. is_encoded(data) - Check if data is multibase-encoded

4. get_codec(data) - Get encoding information from multibase data

5. is_encoding_supported(encoding) - Check if encoding is supported

6. list_encodings() - List all supported encodings

7. get_encoding_info(encoding) - Get detailed encoding information

Reusable Classes

• Encoder(encoding) - Reusable encoder for a specific encoding
• Decoder() - Reusable decoder with composition support
• ComposedDecoder - Try multiple decoders in sequence

Supported Encodings

py-multibase supports 24+ encodings including:

• base2, base8, base10, base16, base16upper
• base32, base32upper, base32pad, base32padupper
• base32hex, base32hexupper, base32hexpad, base32hexpadupper
• base32z, base36, base36upper
• base58flickr, base58btc (most common for libp2p)
• base64, base64pad, base64url, base64urlpad
• base256emoji
• identity (no encoding, just prefix)

Error Handling

• MultibaseError - Base exception
• UnsupportedEncodingError - Encoding not supported
• InvalidMultibaseStringError - Invalid multibase format
• DecodingError - Decoding failure

---

Current State Analysis

1. Peer ID Encoding (Base58 Without Multibase Prefix)

Location: libp2p/peer/id.py

Current Implementation:

import base58

class ID:
    @functools.cached_property
    def base58(self) -> str:
        return base58.b58encode(self._bytes).decode()
    
    def to_base58(self) -> str:
        return self.base58
    
    @classmethod
    def from_base58(cls, b58_encoded_peer_id_str: str) -> "ID":
        peer_id_bytes = base58.b58decode(b58_encoded_peer_id_str)
        return cls(peer_id_bytes)

Issues:

• Peer IDs are encoded without multibase prefix (not self-describing)
• Cannot detect encoding automatically - assumes base58
• No way to support alternative encodings (base32, base64, etc.)
• Direct dependency on base58 library instead of standardized multibase
• Inconsistent with other libp2p implementations that may use multibase

Usage Locations:

• libp2p/peer/peerinfo.py - info_from_p2p_addr(), peer_info_from_bytes()
• libp2p/relay/circuit_v2/transport.py - Multiple peer ID parsing locations
• libp2p/discovery/mdns/listener.py - Peer ID parsing
• libp2p/discovery/bootstrap/bootstrap.py - Peer ID parsing
• libp2p/transport/quic/transport.py - Peer ID parsing
• libp2p/rcmgr/manager.py - Peer ID parsing
• libp2p/security/tls/certificate.py - Certificate CN encoding
• libp2p/security/tls/transport.py - Peer ID handling

2. DHT Key Encoding (Base58 Utility)

Location: libp2p/kad_dht/utils.py

Current Implementation:

import base58

def bytes_to_base58(data: bytes) -> str:
    """
    Convert bytes to base58 encoded string.
    """
    return base58.b58encode(data).decode("utf-8")

Issues:

• Utility function that could benefit from multibase
• No encoding detection capability
• Hardcoded to base58 only

Usage Locations:

• Used in Kademlia DHT for key encoding/decoding

3. Pubsub Message ID Encoding (Base64 Without Multibase Prefix)

Location: libp2p/pubsub/pubsub.py

Current Implementation:

import base64

def get_content_addressed_msg_id(msg: rpc_pb2.Message) -> bytes:
    return base64.b64encode(hashlib.sha256(msg.data).digest())

Issues:

• Message IDs use base64 without multibase prefix
• Not self-describing - cannot detect encoding automatically
• Direct dependency on standard library base64 instead of multibase
• Inconsistent encoding approach compared to peer IDs (base58 vs base64)

Additional Base58 Usage in Pubsub:

# Line 973: Encoding peer ID for logging/comparison
base58.b58encode(msg.from_id).decode()

# Line 992: Comparing peer IDs
base58.b58encode(msg.from_id).decode() == self_id

4. Test Utilities

Location: tests/core/peer/test_peerid.py

Current Implementation:

import base58

def test_id_to_base58():
    random_id_string = ""
    for _ in range(10):
        random_id_string += random.choice(ALPHABETS)
    expected = base58.b58encode(random_id_string).decode()
    actual = ID(random_id_string.encode()).to_base58()
    assert actual == expected

Issues:

• Tests use direct base58 encoding
• Could test multibase integration instead

5. Example Code

Locations:

• examples/pubsub/pubsub.py
• examples/pubsub/floodsub.py

Current Implementation:

import base58
# Direct base58 usage in examples

Issues:

• Examples demonstrate non-standard encoding approach
• Could showcase multibase usage for better interoperability

---

Improvement Opportunities

1. Peer ID Encoding with Multibase (High Priority)

Current Problem: Peer IDs are encoded as plain base58 strings without multibase prefix, making them not self-describing and limiting encoding flexibility.

Proposed Solution: Use multibase for peer ID encoding/decoding while maintaining backward compatibility.

Benefits:

• Self-describing encodings (can detect encoding automatically)
• Support for multiple encodings (base58btc, base32, base64, etc.)
• Better interoperability with other libp2p implementations
• Standardized encoding approach

Implementation Approach:

# Option 1: Add multibase support while keeping base58 as default
from multibase import encode, decode, is_encoded

class ID:
    def to_base58(self) -> str:
        """Return base58 encoded peer ID (backward compatible)."""
        return base58.b58encode(self._bytes).decode()
    
    def to_multibase(self, encoding: str = 'base58btc') -> str:
        """Return multibase-encoded peer ID."""
        return encode(encoding, self._bytes).decode()
    
    @classmethod
    def from_base58(cls, b58_encoded_peer_id_str: str) -> "ID":
        """Parse from base58 (backward compatible)."""
        peer_id_bytes = base58.b58decode(b58_encoded_peer_id_str)
        return cls(peer_id_bytes)
    
    @classmethod
    def from_multibase(cls, multibase_str: str) -> "ID":
        """Parse from multibase-encoded string."""
        if not is_encoded(multibase_str):
            # Fallback to base58 for backward compatibility
            return cls.from_base58(multibase_str)
        peer_id_bytes = decode(multibase_str)
        return cls(peer_id_bytes)
    
    @classmethod
    def from_string(cls, peer_id_str: str) -> "ID":
        """Smart parser that tries multibase first, then base58."""
        if is_encoded(peer_id_str):
            return cls.from_multibase(peer_id_str)
        else:
            # Assume base58 for backward compatibility
            return cls.from_base58(peer_id_str)

Migration Strategy:

1. Add new multibase methods alongside existing base58 methods
2. Update from_base58 to try multibase decoding first, fallback to base58
3. Gradually migrate internal code to use multibase
4. Update documentation and examples

Files to Update:

• libp2p/peer/id.py - Core peer ID class
• libp2p/peer/peerinfo.py - Peer info parsing
• libp2p/relay/circuit_v2/transport.py - Relay peer ID parsing
• libp2p/discovery/mdns/listener.py - mDNS peer ID parsing
• libp2p/discovery/bootstrap/bootstrap.py - Bootstrap peer ID parsing
• libp2p/transport/quic/transport.py - QUIC peer ID parsing
• libp2p/security/tls/certificate.py - TLS certificate handling
• libp2p/rcmgr/manager.py - Resource manager peer ID handling

2. DHT Key Encoding with Multibase (Medium Priority)

Current Problem: DHT keys use a utility function that only supports base58.

Proposed Solution: Replace bytes_to_base58 with multibase-aware functions.

Benefits:

• Support for multiple encodings
• Self-describing keys
• Better error handling

Implementation:

from multibase import encode, decode, is_encoded

def bytes_to_multibase(data: bytes, encoding: str = 'base58btc') -> str:
    """
    Convert bytes to multibase-encoded string.
    
    Args:
        data: Bytes to encode
        encoding: Encoding to use (default: base58btc for compatibility)
    
    Returns:
        Multibase-encoded string
    """
    return encode(encoding, data).decode()

def multibase_to_bytes(multibase_str: str) -> bytes:
    """
    Convert multibase-encoded string to bytes.
    
    Args:
        multibase_str: Multibase-encoded string
    
    Returns:
        Decoded bytes
    
    Raises:
        InvalidMultibaseStringError: If string is not valid multibase
        DecodingError: If decoding fails
    """
    if not is_encoded(multibase_str):
        # Fallback to base58 for backward compatibility
        return base58.b58decode(multibase_str)
    return decode(multibase_str)

# Keep old function for backward compatibility
def bytes_to_base58(data: bytes) -> str:
    """Deprecated: Use bytes_to_multibase instead."""
    return bytes_to_multibase(data, 'base58btc')

Files to Update:

• libp2p/kad_dht/utils.py - DHT utility functions
• libp2p/kad_dht/kad_dht.py - DHT key handling

3. Pubsub Message ID Encoding with Multibase (Medium Priority)

Current Problem: Message IDs use base64 without multibase prefix.

Proposed Solution: Use multibase for message IDs.

Benefits:

• Self-describing message IDs
• Consistent encoding approach with peer IDs
• Support for different encodings if needed

Implementation:

from multibase import encode

def get_content_addressed_msg_id(msg: rpc_pb2.Message, encoding: str = 'base64') -> bytes:
    """
    Generate content-addressed message ID using multibase encoding.
    
    Args:
        msg: Pubsub message
        encoding: Encoding to use (default: base64)
    
    Returns:
        Multibase-encoded message ID
    """
    digest = hashlib.sha256(msg.data).digest()
    return encode(encoding, digest)

Files to Update:

• libp2p/pubsub/pubsub.py - Message ID generation
• Update message ID comparison logic to handle multibase

4. Enhanced Error Handling (Low Priority)

Current Problem: Direct base58/base64 usage provides limited error information.

Proposed Solution: Use multibase's exception hierarchy.

Benefits:

• Better error messages
• Distinguish between encoding errors and data errors
• More informative debugging

Implementation:

from multibase import (
    decode,
    InvalidMultibaseStringError,
    DecodingError,
    UnsupportedEncodingError,
)

def parse_peer_id(peer_id_str: str) -> ID:
    """Parse peer ID with enhanced error handling."""
    try:
        return ID.from_multibase(peer_id_str)
    except InvalidMultibaseStringError as e:
        # Try base58 fallback
        try:
            return ID.from_base58(peer_id_str)
        except Exception:
            raise ValueError(f"Invalid peer ID format: {e}") from e
    except DecodingError as e:
        raise ValueError(f"Failed to decode peer ID: {e}") from e

5. Encoding Detection and Validation (Low Priority)

Current Problem: No way to detect or validate encoding of existing data.

Proposed Solution: Use multibase's detection capabilities.

Benefits:

• Validate peer IDs and other encoded data
• Detect encoding of unknown data
• Better debugging and logging

Implementation:

from multibase import is_encoded, get_codec, get_encoding_info

def detect_encoding(data: str) -> str | None:
    """Detect encoding of multibase-encoded data."""
    if is_encoded(data):
        codec = get_codec(data)
        return codec.encoding
    return None

def validate_peer_id(peer_id_str: str) -> tuple[bool, str | None]:
    """
    Validate peer ID encoding.
    
    Returns:
        Tuple of (is_valid, encoding_name)
    """
    if is_encoded(peer_id_str):
        try:
            codec = get_codec(peer_id_str)
            # Try to decode
            decode(peer_id_str)
            return True, codec.encoding
        except Exception:
            return False, None
    # Assume base58 for backward compatibility
    try:
        base58.b58decode(peer_id_str)
        return True, 'base58'  # Implied, not multibase
    except Exception:
        return False, None

6. Support for Multiple Encodings (Low Priority)

Current Problem: Hardcoded to base58/base64 only.

Proposed Solution: Use multibase's encoding support.

Benefits:

• Support for base32 (more URL-friendly)
• Support for base64url (URL-safe)
• Support for other encodings as needed
• Future-proof for new encoding requirements

Use Cases:

• URL-safe peer IDs: Use base64url or base32
• Compact peer IDs: Use base32z (z-base-32)
• Human-readable: Use base36 or base58btc

7. Reusable Encoder/Decoder Classes (Low Priority)

Current Problem: Encoding/decoding logic is scattered and not reusable.

Proposed Solution: Use multibase's Encoder and Decoder classes.

Benefits:

• Reusable encoding/decoding instances
• Better performance for repeated operations
• Composable decoders for fallback strategies

Implementation:

from multibase import Encoder, Decoder

class PeerIDEncoder:
    """Reusable encoder for peer IDs."""
    
    def __init__(self, encoding: str = 'base58btc'):
        self._encoder = Encoder(encoding)
        self._decoder = Decoder()
    
    def encode(self, peer_id: ID) -> str:
        """Encode peer ID."""
        return self._encoder.encode(peer_id.to_bytes()).decode()
    
    def decode(self, peer_id_str: str) -> ID:
        """Decode peer ID with fallback."""
        try:
            return ID.from_multibase(peer_id_str)
        except Exception:
            # Fallback to base58
            return ID.from_base58(peer_id_str)

---

Integration Plan

Phase 1: Add Multibase Support (Non-Breaking)

1. Add py-multibase dependency

   • Update pyproject.toml to include py-multibase>=2.0.0
   • Ensure compatibility with Python 3.10+

2. Extend Peer ID Class

   • Add to_multibase() method
   • Add from_multibase() method
   • Add from_string() smart parser
   • Keep existing to_base58() and from_base58() for backward compatibility

3. Update Utility Functions

   • Add multibase-aware utility functions
   • Keep old functions with deprecation warnings

4. Add Tests

   • Test multibase encoding/decoding
   • Test backward compatibility
   • Test encoding detection

Phase 2: Internal Migration

1. Update Internal Code

   • Gradually migrate internal code to use multibase
   • Update peer ID parsing to try multibase first
   • Update DHT utilities

2. Update Pubsub

   • Migrate message ID generation to multibase
   • Update peer ID handling in pubsub

3. Update Examples

   • Show multibase usage in examples
   • Demonstrate encoding detection

Phase 3: External API (Optional)

1. Consider Making Multibase Default

   • After sufficient testing, consider making multibase the default
   • Keep base58 support for backward compatibility
   • Update documentation

2. Add Configuration

   • Allow users to choose default encoding
   • Support encoding preferences

---

Dependencies

Current Dependencies

• base58>=1.0.3 - Direct base58 encoding

Proposed Addition

• py-multibase>=2.0.0 - Multibase encoding support

Note: py-multibase internally uses:

• python-baseconv>=1.2.0,<2.0 - Base conversion utilities
• morphys>=1.0,<2.0 - Bytes/string handling

Compatibility

• py-multibase supports Python 3.10+ (matches py-libp2p requirements)
• No conflicts with existing dependencies
• Can coexist with base58 library during migration

---

Testing Considerations

Test Cases to Add

1. Multibase Encoding/Decoding

   def test_peer_id_multibase_encode_decode():
       peer_id = ID(b'\x12\x34\x56')
       multibase_str = peer_id.to_multibase('base58btc')
       assert multibase_str.startswith('z')  # base58btc prefix
       decoded = ID.from_multibase(multibase_str)
       assert decoded == peer_id

2. Backward Compatibility

   def test_backward_compatibility():
       # Old base58 strings should still work
       old_format = base58.b58encode(b'\x12\x34\x56').decode()
       peer_id = ID.from_string(old_format)  # Should work
       assert peer_id.to_bytes() == b'\x12\x34\x56'

3. Encoding Detection

   def test_encoding_detection():
       peer_id = ID(b'\x12\x34\x56')
       multibase_str = peer_id.to_multibase('base64')
       assert multibase_str.startswith('m')  # base64 prefix
       encoding = detect_encoding(multibase_str)
       assert encoding == 'base64'

4. Error Handling

   def test_invalid_multibase():
       with pytest.raises(InvalidMultibaseStringError):
           ID.from_multibase('invalid')

5. Multiple Encodings

   def test_multiple_encodings():
       peer_id = ID(b'\x12\x34\x56')
       base58btc = peer_id.to_multibase('base58btc')
       base64 = peer_id.to_multibase('base64')
       base32 = peer_id.to_multibase('base32')
       
       assert ID.from_multibase(base58btc) == peer_id
       assert ID.from_multibase(base64) == peer_id
       assert ID.from_multibase(base32) == peer_id

---

Interoperability Considerations

libp2p Specification

The libp2p specification does not mandate multibase for peer IDs, but many implementations are moving toward multibase for better interoperability. Using multibase would:

1. Improve Interoperability: Other libp2p implementations (Go, Rust, JS) increasingly support multibase
2. Future-Proof: Aligns with multiformats standards
3. Self-Describing: Makes data formats more robust

Backward Compatibility

• Critical: Must maintain backward compatibility with existing base58 peer IDs
• Migration Path: Support both formats during transition period
• Detection: Use is_encoded() to detect multibase vs plain base58

---

Performance Considerations

Encoding/Decoding Performance

• Multibase Overhead: Minimal - just adds one-byte prefix
• Base58 Performance: py-multibase uses efficient base58 implementation
• Caching: Can use Encoder/Decoder classes for repeated operations

Memory Impact

• Negligible: Multibase prefix adds only 1 byte (or 4 bytes for base256emoji)
• No Additional Buffers: Decoding is done in-place

---

Documentation Updates Needed

1. API Documentation

   • Document new to_multibase() and from_multibase() methods
   • Document encoding options
   • Update examples

2. Migration Guide

   • Guide for migrating from base58 to multibase
   • Backward compatibility notes
   • Encoding selection recommendations

3. Examples

   • Show multibase usage in examples
   • Demonstrate encoding detection
   • Show multiple encoding options

---

Summary

Current State

• ✅ Direct base58/base64 encoding works
• ❌ Not self-describing (cannot detect encoding)
• ❌ Limited to single encoding per use case
• ❌ Inconsistent encoding approaches
• ❌ No standardized error handling

With py-multibase Integration

• ✅ Self-describing encodings
• ✅ Support for 24+ encoding formats
• ✅ Automatic encoding detection
• ✅ Standardized error handling
• ✅ Better interoperability
• ✅ Backward compatible migration path

Priority Recommendations

1. High Priority: Peer ID multibase support (most impactful)
2. Medium Priority: DHT key encoding, Pubsub message IDs
3. Low Priority: Enhanced error handling, encoding detection utilities, multiple encoding support

Estimated Effort

• Phase 1 (Add support): 2-3 days
• Phase 2 (Internal migration): 3-5 days
• Phase 3 (External API): 1-2 days
• Testing: 2-3 days
• Documentation: 1-2 days

Total: ~2 weeks for complete integration

---

References

• py-multibase GitHub
• py-multibase Documentation
• Multibase Specification
• libp2p Specification

[asmit27rai]

@acul71 I raised a pr regarding this improvement. Currently, phase 1 is implemented.
I will improve the commit and next phase implementation based on your feedback on this.
PR : #1179

CCing : @seetadev