Protocol Middleware

[b5]

There are two major use cases for wanting to hook into the connection lifecycle before connection construction is passed to ProtocolHandler.on_connecting:

• Endpoint-Level Authorization - user needs to screen all connections to a node with the same authorization logic.
• Endpoint Activate/Deactivation - user needs to stop accepting new connections based on some state such as application-level backpressure management, downstream service degradation, etc.

Both could operate off a hook with the same function signature as on_connecting, It would just need to be called before being passed to a protocol handler

async fn on_connecting(&self, connecting: Connecting) -> Result<Connection, AcceptError> {

Advantages

It would be really nice to have an endpoint on-off switch.

Drawbacks

• For authorization, the allowed logic would be be inherently limited, because the procotol itself can't be modified, so any access-token-based approach to authorization would still need to be on a per-protocol basis.
• This probably means growing the Endpoint API, which is a big deal. Given that most folks will probably want this to be the place where they do access-token based access control, they'll likely be left feeling sad that this middleware doesn't go far enough.

Alternatives

• ALPN wrappers - it's in theory possible to write a generic ProtocolHandler wrapper, and wrap all registered protocols. My concern here is this has all the problems of an endpoint-level middleware function (can't affect the byte stream), but is now spiritually breaking the value of the ALPN, which really should mean "this alpn == that function"
• Router that wraps ProtcolHandlers internally - we could mitigate the above with a new Router that does all the wrapping internally, and boxes users out of an API that breaks the protocol contract.=
• Protocol Encapsulation Middleware - We could write actual middleware in userland that when it wraps an existing protocol, changes the ALPN by prefixing it to convey that the protocol must be wrapped on both sides. This would allow users to affect the wire protocol, and then delegate to downstream ProtocolHandlers.

[b5]

Chatted with @dignifiedquire, a few key points from that discussion:

Wrapping existing protocols is the solution we recommend

We already have code that does exactly this: the AccessLimit struct from iroh::protocols wraps an existing protocol with a function that limits access.

We can't put middleware between the protocol and the endpoint because ProtocolHandler::on_connecting is already as early as possible/safe in the connection lifecycle, the only earlier alternative would be to pause the endpoint entirely. So all solutions do need to wrap protocols.

With that said, this leaves the router that wraps ProtocolHandlers internally on the table. We can write a custom router that accepts a single screening function, and wraps all protocols with it.

We should write more example routers

My recommended remedy is to ship more examples, and later consider if one of these routers becomes used commonly enough to give it a separate crate.

The iroh::protocol::Router is intentionally a limited thing, and we should be encouraging folks to write custom routers, much in the same vein that we do for protocols themselves.

Recommendations:

1. Convert feat(iroh): add new protocols to router at runtime #3355 into an example. It's lower stakes, can be merged quickly, and gives us an example to point to for "here! look! screw with routers! it's ok!"
2. Convert docs(examples): screening-connection example #3360 into a router that accepts a single function & wraps any protocol. Now we have to examples of custom routers.