Skip to content

Explain issue related to L7 load balancers with apm-server #614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 3, 2025
Merged

Explain issue related to L7 load balancers with apm-server #614

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
64a6a38
explain issue related to L7 load balancers with apm-server
rubvs Feb 27, 2025
be6b8b3
Merge branch 'main' into apm-server-load-balancers
rubvs Feb 28, 2025
89365e6
resolve comments
rubvs Mar 3, 2025
1fc13c7
Merge branch 'main' into apm-server-load-balancers
colleenmcginnis Mar 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion solutions/observability/apps/upstream-opentelemetry-collectors-language-sdks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,11 @@ If you use the OTLP/gRPC protocol, requests to Elastic must use either HTTP/2 ov

When using a layer 7 (L7) proxy like AWS ALB, requests must be proxied in a way that ensures requests to Elastic follow the rules outlined above. For example, with ALB you can create rules to select an alternative backend protocol based on the headers of requests coming into ALB. In this example, you’d select the gRPC protocol when the `"Content-Type: application/grpc"` header exists on a request.

Many L7 load balancers handle HTTP and gRPC traffic separately and rely on explicitly defined routes and service configurations to correctly proxy requests. Since APM Server serves both protocols on the same port, it may not be compatible with some L7 load balancers. For example, to work around this issue in [Ingress NGINX Controller for Kubernetes](https://github.com/kubernetes/ingress-nginx), either:

* Use the `otlp` exporter in the OTel collector. Set annotation `nginx.ingress.kubernetes.io/backend-protocol: "GRPC"` on the K8s Ingress object proxying to APM Server.
* Use the `otlphttp` exporter in the OTel collector. Set annotation `nginx.ingress.kubernetes.io/backend-protocol: "HTTP"` (or `"HTTPS"` if APM Server expects TLS) on the K8s Ingress object proxying to APM Server.

For more information on how to configure an AWS ALB to support gRPC, see this AWS blog post: [Application Load Balancer Support for End-to-End HTTP/2 and gRPC](https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/).

For more information on how APM Server services gRPC requests, see [Muxing gRPC and HTTP/1.1](https://github.com/elastic/apm-server/blob/main/dev_docs/otel.md#muxing-grpc-and-http11).
Expand All @@ -266,4 +271,4 @@ For more information on how APM Server services gRPC requests, see [Muxing gRPC

* [Collect metrics](../../../solutions/observability/apps/collect-metrics.md)
* Add [Resource attributes](../../../solutions/observability/apps/resource-atrributes.md)
* Learn about the [limitations of this integration](../../../solutions/observability/apps/limitations.md)
* Learn about the [limitations of this integration](../../../solutions/observability/apps/limitations.md)
Loading