Caddy behind AWS Network Load Balancer with proxyprotocol enabled leads to tls errors

[kellertobias]

We are trying to pass the client IP through our load balancing stack. The stack is NLB => Caddy (on Docker/Fargate) => ALB.

My Understanding based on the documentation of your caddy plugin, caddies documentations and AWS's documentation of the Network Load Balancer (NLB) (https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#proxy-protocol), I need to enable proxy protocol v2 in the NLB and install your plugin in caddy as well as configure it. I've written about my configuration in the Caddy Forum: https://caddy.community/t/caddy-behind-nlb/13988

when I enable all the stuff mentioned above, I get the following lines in caddy if I set the log level to debug:

`{"level":"debug","ts":1634749313.4648879,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.20.201.113:12064: invalid length"}``

where the 10.20.201.113 is the private IP of our NLB.

What can I do to further debug this issue? Might there be a problem in your plugin when working with the NLB?

[ermiaqasemi]

Having the same issue!

[ermiaqasemi]

@kellertobias I read your details in Caddy Forum, I am using the same design as yours but the difference is that we deployed Caddy as an ingress in Kubernetes and I just succeed to get the client IP by enabling preserve Client IP address at NLB level, then in order to get the Client IP you can get it from {http.request.remote.host} placeholder and the pass this as X-forward-for header to ALB and then reset will be handle by AWS.
hope this helps you!

[toreym]

@kellertobias, curious if you've found a resolution to this or if you changed your setup to work around it. Thanks!

[kellertobias]

@toreym not yet. we kind of just ignored the problem for now

[toreym]

Will update you if I find a solution.

[ChrisLahaye]

Did anyone manage to get this working?

[toreym]

No, sorry @ChrisLahaye. We're not that familiar with this protocol so it's been difficult to debug.

[toreym]

@ChrisLahaye @kellertobias
We were able to create a listener that works. As long as you are using JSON to configure your server this will work. I assume you could point xcaddy build to our module (I'm not a go person so don't know how their package management works).

xcaddy build --with github.com/rocketreferrals/caddy-go-proxyproto

And the json config would look like this:

"listener_wrappers": [{"wrapper": "go_proxyproto", "timeout": "5s"},{"wrapper": "tls"}]

At the end of the day this is just a simple wrapper for https://github.com/pires/go-proxyproto which is a different lib, but it seems to work better with AWS ELBs.

Hope this helps!

[ChrisLahaye]

@toreym
We applied your module behind an NLB with proxy protocol enabled, and didn't encounter any request errors. Unfortunately we found out that IP preservation does not work when using Global Accelerator with NLB, and therefore, have no need for proxy protocol. Either way, I would like to thank you for your efforts as your module resolved the errors we were originally encountering. You should try to get your module listed in the official documentation as I am sure it will be helpful to others. I really like the simplicity of it.

Thank you.

[kellertobias]

@toreym sounds good, will give it a try; we are building caddy ourselves too at the moment, so adding another module should be no problem and the configuration is generated at startup of the container also as a json file.

Do you know if IP Preservation works if we are not using Global Accelerator in the NLB - as this is the main reason for us as well to use proxy protocol.

[ChrisLahaye]

@toreym sounds good, will give it a try; we are building caddy ourselves too at the moment, so adding another module should be no problem and the configuration is generated at startup of the container also as a json file.

Do you know if IP Preservation works if we are not using Global Accelerator in the NLB - as this is the main reason for us as well to use proxy protocol.

I am not completely certain but when preservation is enabled and registering the targets by instance ID, then targets receive the client's IP address, else, i.e. if targets are registered by IP address, targets receive the NLB's IP address and you can get the client's IP address from the proxy protocol.

[toreym]

@kellertobias We're not using Global Accelerator and do not have IP Preservation turned on. So, that's why we need to get the IP from the Proxy Protocol. We've had this setup running in production for a couple of weeks now and have not had any issues. I don't imagine you'd have any problem running the module with IP Preservation on though.