resource-manager: "conn" limits are not applied to connLimiter

[hsanjuan]

The documentation says:

How to disable limits

Sometimes disabling all limits is useful when you want to see how much resources you use during normal operation. You can then use this information to define your initial limits. Disable the limits by using InfiniteLimits.

https://pkg.go.dev/github.com/libp2p/go-libp2p@v0.45.0/p2p/host/resource-manager#section-readme

InfiniteLimits sets the conn limit to infinite: https://github.com/libp2p/go-libp2p/blob/v0.45.0/p2p/host/resource-manager/limit_defaults.go#L867-L878

Yet, the connLimiter, which is in charge of limiting connections, is created bare and does not know about the limits: https://github.com/libp2p/go-libp2p/blob/v0.45.0/p2p/host/resource-manager/rcmgr.go#L139-L149 (I´d say same for connRateLimiter).

The connLimiter has a default limit of 8 concurrent connections for an IP: https://github.com/libp2p/go-libp2p/blob/v0.45.0/p2p/host/resource-manager/conn_limiter.go#L35-L38

So, the limits are not disabled. The work around of using a NullResourceManager works, it is just not documented.

As a side note, GOLOG_LOG_LEVEL="rcmgr=debug" is no help to discover this issue, as this is logged in the upgrader, which means someone monitoring for resource manager errors will miss these unless debug logging is enabled in the upgrader.

As a second side note, GOLOG_LOG_LEVEL doesn't work without special setup anymore via gologshim.

[MarcoPolo]

The connLimiter is about limiting the number of connections from a specific network prefix or a generic subnet. See https://pkg.go.dev/github.com/libp2p/go-libp2p@v0.45.0/p2p/host/resource-manager#WithLimitPerSubnet and https://pkg.go.dev/github.com/libp2p/go-libp2p@v0.45.0/p2p/host/resource-manager#WithNetworkPrefixLimit.

An update to the "How to disable limits" section makes sense.

The connLimiter has a default limit of 8 concurrent connections for an IP: https://github.com/libp2p/go-libp2p/blob/v0.45.0/p2p/host/resource-manager/conn_limiter.go#L35-L38

Do you need more than 8 concurrent connections to the same IP? Are there multiple peers behind a single IP address?

As a side note, GOLOG_LOG_LEVEL="rcmgr=debug" is no help to discover this issue, as this is logged in the upgrader, which means someone monitoring for resource manager errors will miss these unless debug logging is enabled in the upgrader.

The error should propagate up to the caller. Did that not happen in your case?

As a second side note, GOLOG_LOG_LEVEL doesn't work without special setup anymore via gologshim.

Can you create a new issue and explain? This works for me:

GOLOG_LOG_LEVEL=rcmgr=debug go test -v ./p2p/host/resource-manager/

[hsanjuan]

Do you need more than 8 concurrent connections to the same IP? Are there multiple peers behind a single IP address?

Happens when there's a loop where an application starts up, makes a request to a libp2p-server and exists. The server does not clean up the close connection fast enough. While the limit is 8, the issue appears after 12 sequential requests.

The error should propagate up to the caller. Did that not happen in your case?

In the form of connection closed sure. But that doesn't say much as to why.

The error should propagate up to the caller. Did that not happen in your case?

As a second side note, GOLOG_LOG_LEVEL doesn't work without special setup anymore via gologshim.

Can you create a new issue and explain? This works for me:

GOLOG_LOG_LEVEL=rcmgr=debug go test -v ./p2p/host/resource-manager/

That doesn't work when go-libp2p is used in an application without doing https://github.com/ipfs/go-log/releases/tag/v2.9.0. At least it didn't for me.

[hsanjuan]

The connLimiter is about limiting the number of connections from a specific network prefix or a generic subnet. See https://pkg.go.dev/github.com/libp2p/go-libp2p@v0.45.0/p2p/host/resource-manager#WithLimitPerSubnet and https://pkg.go.dev/github.com/libp2p/go-libp2p@v0.45.0/p2p/host/resource-manager#WithNetworkPrefixLimit.

It would make sense to have a Limit in the LimitsConfig for this because that is the way limits are defined/scaled, and currently this is done differently in a non-obvious way.

[MarcoPolo]

The error should propagate up to the caller. Did that not happen in your case?

In the form of connection closed sure. But that doesn't say much as to why.

That's not the expected error. It should be

go-libp2p/p2p/host/resource-manager/rcmgr.go

Line 389 in 4dc576a

return nil, fmt.Errorf("connections per ip limit exceeded for %s", endpoint)

.

Or maybe this is the error the client sees, but it's the server that is limiting the connections. So the server sees the error, but only logs it as debug.

It would make sense to have a Limit in the LimitsConfig for this because that is the way limits are defined/scaled, and currently this is done differently in a non-obvious way.

This is a good point. I really do not expect most users to need to fiddle with these numbers because they shouldn't need to scale like the rest of the limits. Note that your scenario of a client closing and opening connections is very similar to a resource exhaustion attack.

[hsanjuan]

Note that your scenario of a client closing and opening connections is very similar to a resource exhaustion attack.

Not asking to change defaults, my point is that it was not obvious what was happening despite having InfiniteLimits configured, enabling debug logging on the resource manager and reading the documentation.