libp2p-v0.52.0

[thomaseizinger]

Automatic kademlia client/server mode

Let's get the biggest one out the way first, I promise the other points are easier explained but equally exciting.
The tl;dr is: Healthier Kademlia routing tables and an improved developer experience.

If you don't know about Kademlia's client/server mode, checkout the specs.

With the v0.52 release, rust-libp2p automatically configures Kademlia in client or server mode depending on our external addresses.
If we have a confirmed, external address, we will operate in server-mode, otherwise client-mode.
This is entirely configuration-free (yay!) although follow-up work is under-way to allow setting this manually in certain situations: #4074.

We can now do the following:

1. As soon as we learn about an external address (e.g. via AutoNAT), we activate server mode of Kademlia.
2. Activating server-mode means we allow inbound requests, this is a change in our set of supported protocols.
3. The change is detected automatically and reported to all protocols as ConnectionEvent::LocalProtocolsChange.
4. libp2p-identify picks up this change and pushes it to all connected remote nodes.
5. Remote nodes can instantly put us into their routing table.

To implement this, several other features/issues had to be fixed.
If you are interested in the details, read on:

• Simplify the scoring mechanism of external addresses: #3954

  As a consequence, the observed address reported by identify is no longer considered an external address but just an address candidate.
  Checkout the changelog-entry for a way of restoring the old behaviour.

• Changes to the supported protocols are now detected at runtime and communicated to all protocols: #3651.

  Previously, a protocol could retrieve the supported protocols via PollParameters::supported_protocols.
  This list however was computed at start-up and was static.
  Now, ConnectionEvent has two new variants:

  pub enum ConnectionEvent<'a> {
    // existing variants omitted ...
  
    /// The local [`ConnectionHandler`] added or removed support for one or more protocols.
    LocalProtocolsChange(ProtocolsChange<'a>),
    /// The remote [`ConnectionHandler`] now supports a different set of protocols.
    RemoteProtocolsChange(ProtocolsChange<'a>),
  }
  
  pub enum ProtocolsChange<'a> {
    Added(ProtocolsAdded<'a>),
    Removed(ProtocolsRemoved<'a>),
  }

  ProtocolsAdded and ProtocolsRemoved are iterators over a (new) type called StreamProtocol.

• Protocols are now enforced to be valid UTF-8 strings: #3746.

  This was always the case in the specs but the rust-libp2p implementation was lagging behind here and improperly represented them as bytes internally.

• Local changes to our protocols (i.e. a node going from Kademlia server to client mode) are now immediately pushed to the remote via the /ipfs/id/push/1.0.0 protocol: #3980.

Not only does this work out-of-the-box and thus improves the developer experience of rust-libp2p, it should also result in a much more useful and up-to-date routing table for all nodes on a Kademlia DHT.

Type-safe /p2p multiaddresses

The /p2p protocol of multiaddresses specifies the identity of a peer in the form of a PeerId such as 12D3KooWETLZBFBfkzvH3BQEtA1TJZPmjb4a18ss5TpwNU7DHDX6.
Yet for the longest time, the type-definition of the /p2p protocol looked like this:

pub enum Protocol<'a> {
    // omitted other variants ...
    P2p(Multihash),
}

Every PeerId is a valid Multihash but not vice-versa.
Dang!
That is a lot of "impossible" errors that the type-system should avoid.

Thanks to a lot of work, it now looks like this:

pub enum Protocol<'a> {
    // omitted other variants ...
    P2p(PeerId),
}

More consistent event/command naming

Naming is hard and humans are creatures of habit.
Thus, once familiar with certain names, it is often hard to see how they make absolutely no sense at all to newcomers.

In the v0.52 release we renamed several associated types and enums which hopefully make the message-passing system implemented in rust-libp2p easier to grasp.

A quick recap:

• A NetworkBehaviour represent a protocol's state across all peers and connections.
• A ConnectionHandler represents a protocol's state for a single connection to a peer.
• Multiple NetworkBehaviours are composed into a tree using #[derive(NetworkBehaviour)] and run inside a Swarm

NetworkBehaviours can emit events to the Swarm.
This used to be called OutEvent.
Now, its aptly named ToSwarm:

pub trait NetworkBehaviour {
    type ToSwarm;
  
    // functions and other types omitted ...
}

Returning one of these events was previously done with a NetworkBehaviourAction::GenerateEvent, a type-name so long you were grateful for autocomplete.
These actions are essentially commands that are issued to the swarm.
What could possibly be a good name for that?
I present:

pub enum ToSwarm<TOutEvent, TInEvent> {
    GenerateEvent(TOutEvent),
    
    // other variants omitted ..
}

We followed the same strategy for ConnectionHandler.
A ConnectionHandler can receive messages from a NetworkBehaviour via ToSwarm::NotifyHandler (gosh, that was so much easier to write, why didn't we do this earlier?) and send message to its NetworkBehaviour.
The associated types defining these messages are now called ToBehaviour and FromBehaviour, representing where the message is going / coming from.
Previously, they carried the generic names InEvent and OutEvent which had me utterly confused when I first started working on rust-libp2p.

To wrap it all up, the ConnectionHandlerEvent::Custom variant is now called ConnectionHandlerEvent::NotifyBehaviour, making it clear what happens to types returned here.

Improved ergonomics around stream errors

Oh, isn't this one of my favourites!

Ever wondered why ConnectionHandlerUpgrErr had what felt like 10 layers of nested enums?
So did we and went ahead and fixed this in #3882.

This is what it looked like before:

pub enum ConnectionHandlerUpgrErr<TUpgrErr> {
    Timeout,
    Timer,
    Upgrade(UpgradeError<TUpgrErr>),
}

pub enum UpgradeError<E> {
  Select(NegotiationError),
  Apply(E),
}

pub enum NegotiationError {
  ProtocolError(ProtocolError),
  Failed,
}

pub enum ProtocolError {
  IoError(io::Error),
  InvalidMessage,
  InvalidProtocol,
  TooManyProtocols,
}

Now, it looks like this:

pub enum StreamUpgradeError<TUpgrErr> {
    Timeout,
    Apply(TUpgrErr),
    NegotiationFailed,
    Io(io::Error),
}

But that is not it!

Previously, we would give you one of these ConnectionHandlerUpgrErr when an inbound stream failed.
But, what exactly does ProtocolError::InvalidMessage for example mean for an inbound stream?
How would we even figure out, which protocol (read ConnectionHandler) this should be dispatched to if we failed to negotiate the protocols?

Well, you might have guessed it.
It is impossible to dispatch this to the correct one, so we just informed all protocols.
But that was pretty useless.
If we don't know, which stream a protocol belongs to, we shouldn't just inform all of them.
Thus, in #3605 we stopped this which removes several "impossible" error paths.

Simpler noise interface

Since its introduction, the libp2p-noise crate supported a wide range of handshake patterns.
The libp2p specs however only documents the XX handshake: libp2p/specs/noise.

Supporting additional patterns introduced significant complexity to the codebase.
We decided to deprecate and remove them.
This significantly reduced the complexity of the libp2p-noise implementation (-1000 LoC!).
Additionally, it allowed us to finally adopt the naming conventions we have been pursuing across the workspace for libp2p-noise.

Using the noise handshake is now as simple as:

let config = libp2p::noise::Config::new(&keypair).unwrap();

Request-response protocols using serde

A simple, yet impactful contribution was made by @dgarus in #3952:

• libp2p::request_response::cbor::Behaviour<Req, Res>
• libp2p::request_response::json::Behaviour<Req, Res>

These two, new type-aliases come with a pre-configured CBOR/JSON codec and only need to be parameterised by the Req and Res types.
The only requirement is that the types implement serde::{Serialize,Deserialize}.
Defining a request-response protocol has never been easier in rust-libp2p!

As with any code that serializes data structures, be aware that any changes to it can easily be breaking and thus not backwards-compatible with older versions.

Hole-punching for QUIC

Despite still being in alpha, our QUIC implementation can now establish direct connections across certain NATs, i.e. hole-punching!
This was contributed by @arpankapoor in #3964.

You don't need to configure anything special besides the dcutr behaviour.
See the dcutr-example in our repository for details.

Thanks!

A big thanks to all people that contributed to this release.
In alphabetical order:

• @AgeManning
• @arpankapoor
• @b0xtch
• @b-zee
• @creativcoder
• @dariusc93
• @dgarus
• @DougAnderson444
• @drHuangMHT
• @elenaf9
• @felinira
• @galargh
• @hanabi1224
• @helloimalemur
• @jsantell
• @kckeiks
• @leviathanbeak
• @linoscope
• @melekes
• @nathanielc
• @obi1kenobi
• @oblique
• @ozkanonur
• @ozwaldorf
• @PopBogdan97
• @shamil-gadelshin
• @StemCll
• @stonecharioteer
• @stormshield-pj50
• @tcoratger
• @tthebst
• @umgefahren
• @uwejan
• @vnermolaev
• @wizeguyy
• @yellowred

Closing

Well done for making it to the end!

As always, a detailed log of changes for every release can be found in our repository: CHANGELOG.md.
If you are struggling with an upgrade, feel free to tag Max (@mxinden) or myself (@thomaseizinger) in a PR and we'll try to help.

Happy coding!

---

This discussion was created from the release libp2p-v0.52.0.

[p-kraszewski]

[...] If we have a confirmed, external address, we will operate in server-mode [...]

Yay! I hope it is done better than in Go port. They have hardcoded network classes that are always considered private, regardless of the configuration (and forgot to mention it in the docs), including TEST-NET-* ranges which are not to be treated as private networks. This costed me a week of debugging my code, when the error wasn't within it.

[thomaseizinger]

We don't hardcode any network ranges. If you manually add an external address via Swarm::add_external_address it will work. Even for 127.0.0.1.

Note that there is a caveat: If you happen to listen on two interfaces, we will operate in server mode on both, even if one is effectively not publicly routable.

We might add a feature at some point to detect this.

[p-kraszewski]

Thank you.

[joshuef]

Hey! After the merge of the Distance kbucket pr (#4109), our only blocker to moving up to 0.52 now is a patch release... Is there plan for one in the near future? Are there any blockers to that which we might be able to help in?

Thanks! 🙇

[thomaseizinger]

We can definitely do a patch-release of libp2p-kad. Can you prepare a release PR? See https://github.com/libp2p/rust-libp2p/blob/master/docs/release.md#releasing-one-or-more-crates.

[mxinden]

@joshuef I already did that yesterday, see #4109 (comment). Am I missing something?

[joshuef]

Ah wow, No I am missing something :D

i did not realise that had happened. I was looking at the releases page, thinking we had to get 0.52.1 out the door of libp2p!

thanks @thomaseizinger @mxinden 🙇 (and everyone else). This should help us a fair bit with some wee issues we've been seeing 💪

[tmpfs]

Hi, thanks for the 0.52 release, I have a small issue with converting from an ed25519 key stored elsewhere in my program into a PublicKey and then into a PeerId that I can then use as a device identifier.

Previously, I was using the enum variants to achieve this and they have been removed but it seems to me that I should be able to convert the ed25519 public key bytes into a PeerId.

Any thoughts on the best way to support this use case?

[tmpfs]

Unfortunately I can't use Keypair::try_from_bytes, as I only have the public key bytes in my TrustedDevice type and I don't want to store the keypair in that type, only the public key bytes.

I am already using the PublicKey::try_from_bytes (previously decode) which works fine but I don't seem to be able to convert from the ed25519::PublicKey to the libp2p_identity::PublicKey in order to call to_peer_id().

This is the code I was using:

        let p2p_public_key =
            identity::ed25519::PublicKey::try_from_bytes(public_key.as_slice())?;

        #[allow(deprecated)]
        let p2p_public_key = identity::PublicKey::Ed25519(p2p_public_key);
        let peer_id = p2p_public_key.to_peer_id();

Am I missing something or is there no way now to go from ed25519::PublicKey to libp2p_identity::PublicKey?

[tmpfs]

If you are open to implementing From for the specific PublicKey types into the generic libp2p_identity::PublicKey type I am happy to submit a PR?

[thomaseizinger]

If you are open to implementing From for the specific PublicKey types into the generic libp2p_identity::PublicKey type I am happy to submit a PR?

That already exists: https://github.com/libp2p/rust-libp2p/blob/master/identity/src/keypair.rs#L714-L721

Are you on the latest version of libp2p-identity?

[tmpfs]

Sorry, my bad for missing that @thomaseizinger - been a long day, thanks! 🙏

[thomaseizinger]

Sorry, my bad for missing that @thomaseizinger - been a long day, thanks! 🙏

No worries, glad it is resolved and take care! :)