OpenTelemetry ingestion using gRPC behind reverse proxy

[ypedegroot]

I am currently self-hosting Seq in a Docker container on a Linux VM. The Seq UI is accessible through a reverse proxy (caddy) which terminates the SSL (caddy takes care of the SSL certificates) and then forwards traffic to the seq container on a local IP inside the Docker environment. This all works fine.

However, I am moving to OpenTelemetry exporters in all my .NET apps. That means, logging and tracing (and metrics in the future, I hope!) are exported using the OTLP exporters available in .NET 10. I want to use gRPC since this is way more efficient in a logging context than HTTP protobuf. And I want to minimize egress data of course.

My current .NET exporters configuration uses protobuf, and that works perfectly fine. But gRPC does not. I have read that gRPC only works when HTTPS is enabled. I have also seen some blog posts on your blog demonstrating gRPC (around May 2023).

(1) How can I know whether HTTPS is enabled? Normally, TLS/HTTPS terminates at the reverse proxy, so HTTPS on the Seq container itself is not required (and also not desirable. Letting a reverse proxy managing certificates is more convenient). This is also what you mention here: https://datalust.co/docs/https-tlsssl. That is exactly what I have done.

(2) What is the correct endpoint to use? In your blogpost (https://datalust.co/blog/opentelemetry-logs-protocol-ingestion-preview), you use https://localhost:45341. But in the documentation (https://datalust.co/docs/ingestion-with-opentelemetry), there is written that for the endpoint I should use "The HTTPS root URL identifying your Seq server". That is, without port specification.

The documentation also states:

Seq can ingest logs using the gRPC protocol flavor:

- on endpoints served using HTTPS (TLS), and
- where all intermediate network infrastructure fully supports HTTP 2.0.

How can we know for sure if all these conditions are met? I have tried increasing the log level in hopes of deducing something from the logs, but to no avail.

I know this it more than one question, but I hope someone can point me in the right direction. I would really like to keep using Seq, especially with Metrics coming too! In that perspective, I think it makes sense to invest in gRPC infrastructure because it is very efficient in terms of bandwidth usage. Any help is greatly appreciated!

[liammclennan]

Hi @ypedegroot

(1) How can I know whether HTTPS is enabled?

Ultimately the test is if you can connect via a HTTPS connection, but you can also check the api.listenUris server configuration variable, or look at Seq's log when it starts (there will be something like

[07:36:08 INF] Seq listening on ["http://localhost:5341/"]

(2) What is the correct endpoint to use?

To connect with OTLP gRPC "The HTTPS root URL identifying your Seq server" is correct. This may include the port. Without specifying a port 443 is implied for HTTPS.

For your setup to work you need to setup secure connections to Seq, or use HTTP/protobuf.

If you stick with gRPC you will need to enable caddy's--change-host-header flag. I'm not sure what else is necessary to get caddy to handle HTTP/2 connections correctly. If you are having problems make sure you check caddy's logs for clues.

[nblumhardt]

Hi @ypedegroot,

Just a quick question about:

I want to use gRPC since this is way more efficient in a logging context than HTTP protobuf. And I want to minimize egress data of course.

In OTLP, the performance gap between HTTP/2 + protobuf and gRPC shouldn't be especially pronounced. Behind the scenes these are very similar. Despite there being a lot of information online suggesting that gRPC is faster for this because of streaming, OpenTelemetry doesn't use this, and as far as we're aware, support for concurrent requests in both configurations is hampered by lack of SDK support rather than by protocol limitations (though I haven't dug deeply into this).

Is there an aspect of your usage scenario we're missing? Any background info you can share would be awesome. Otherwise, HTTP/proto is probably the way to go. HTH!

[ypedegroot]

Hi Nicholas,

Thanks for your insight. It is much appreciated. My usage scenario is relatively simple: I use OpenTelemetry logging and tracing in my .NET apps and send that to Seq using the OtlpExporters in .NET. It was not known to me that OpenTelemetry doesn't use streaming in its gRPC implementation, which makes it the performance "gap" between HTTP/proto and gRPC smaller than one would initially think! I will stick to HTTP/proto for now. Thanks!