OpenTelemetry exporter endpoint won't respond

[fetchezar]

Hi! We've been having an issue with our Ocelot gateway.

We want to add metrics to it but we're unable to make the endpoint "/metrics" work. We're using the OpenTelemetry Prometheus exporter package, as explained here https://learn.microsoft.com/en-us/dotnet/core/diagnostics/observability-prgrja-example#5-configure-opentelemetry-with-the-correct-providers

But the mapped "/metrics" endpoint always responds with a 404.

Is there a way to let Ocelot know it shouldn't run requests to that metrics endpoint through the pipeline and rather delegate to the exporter?

I tried mapping the endpoint (with MapPrometheusScrapingEndpoint) first in the pipeline, last in the pipeline, to no avail. It's very much like this issue here #2183 where the OP added their GET mapping before UseOcelot and after UseOcelot, still didn't work in either case.

[raman-m]

Dear @fetchezar ! What's your full name and LinkedIn?

You haven't forked the Ocelot repository into your account repos, so it doesn't seem like you're a C# developer. Why should I answer your question?

About #2183... have you attempted implementing the C# recipes mentioned in it?
Mixing the function of a reverse proxy with that of third-party metrics software isn't recommended. Currently, Ocelot does not integrate with the OpenTelemetry library, though it might theoretically be possible.

To continue this discussion, you must upload the solution or at least show complete Program.cs and ocelot.json.

---

We want to add metrics to it but we're unable to make the endpoint "/metrics" work. We're using the OpenTelemetry Prometheus exporter package, as explained here

Ocelot+OpenTelementry+MS.NET
Finding a representative to take responsibility for an integration solution can be challenging 😉
Do we need to reach out directly to the OpenTelemetry team for assistance? Alternatively, could we open a support ticket on the Microsoft boards?! 😄

---

Is there a way to let Ocelot know it shouldn't run requests to that metrics endpoint through the pipeline and rather delegate to the exporter?

So, if I understand correctly, you're looking to consume the service directly? Then what's the purpose of Ocelot in this setup? If you prefer a direct connection, just connect straight to the service from the client! 😉

---

But the mapped "/metrics" endpoint always responds with a 404.
Is there a way to let Ocelot know it shouldn't run requests to that metrics endpoint through the pipeline and rather delegate to the exporter?

It seems that the OpenTelemetry library is currently incompatible with the Ocelot pipeline. Fixing this issue might require understanding the internal implementation of the OpenTelemetry library. You may need to explore their code or reach out to the OpenTelemetry team for assistance.

[fetchezar]

I've updated my profile with my LinkedIn, you're free to add me if you want. We don't develop public applications so I can't share my C# work with you, sorry. But it ranges from APIs to backend services, data processing processes, web sites, and so on. ~10 YOE give or take.

The Prometheus exporter middleware collects metrics internally and then exposes them via an endpoint, /metrics by default. That's why it's required to have an endpoint that the Ocelot pipeline wont touch. This needs to exist on the service with Ocelot since the metrics are collected there, the service self-reports its metrics, and Prometheus consumes them polling this endpoint.

I gave MapWhenOcelotPipelinea shot like you suggested here but I think it's a no-go because the middleware seems to be internal, so replicating what MapPrometheusScrappingEndpoint does here doesn't seems to be possible.

Something like a blacklist of paths Ocelot won't process would be necessary I think, rather than a way to manually handle an endpoint like with MapWhenOcelotPipeline.

[raman-m]

I've updated my profile with my LinkedIn, you're free to add me if you want.

Great, thanks, Facundo! I usually review a contributor's LinkedIn profile to verify they are a genuine developer and assess their programming expertise for potential contributions to our project. Additionally, I don't connect with all contributors, but I maintain connections exclusively with Ocelot team members.

---

The Prometheus exporter middleware collects metrics internally and then exposes them via an endpoint, /metrics by default. That's why it's required to have an endpoint that the Ocelot pipeline won't touch. This needs to exist on the service with Ocelot since the metrics are collected there, the service self-reports its metrics, and Prometheus consumes them polling this endpoint.

Alright, makes sense! From Ocelot's perspective, the URL path /metrics isn't treated as an endpoint; instead, it's seen as a static route with the UpstreamPathTemplate value set to /metrics, like this →

{
  "Routes": [
    {
      "UpstreamPathTemplate": "/metrics",
      // other props...
    }
  ]
}

You can transform the route request using a custom delegating handler or middleware injection, but the downstream will point to an external endpoint, not Ocelot's. As I mentioned, Ocelot's design relies on URL path analysis and HTTP request transformations rather than Endpoint objects.

From what I understand, the OpenTelemetry+Prometheus endpoint must be part of the current Ocelot application to gather metrics for each registered endpoint. This creates a significant issue since Ocelot's pipeline does not depend on Microsoft.AspNetCore.Http.Endpoint objects.
In theory, we can register an endpoint object using the HttpContext.SetEndpoint(Endpoint) method within custom middleware injected into Ocelot's pipeline. This approach is demonstrated in PR #2188, where a temporary Endpoint object is registered to leverage ASP.NET functionality. A potential future solution might involve utilizing the HttpContext.SetEndpoint(Endpoint) method and/or a custom delegating handler, as both encapsulate a RequestDelegate object for invocation.

Finally, have you tried a Controller-based solution to define endpoint without the /metrics route in ocelot.json configuration, similar to this code? 👇

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
namespace MyNamespace;

[Authorize] // authorized or anonymous, it is up to you
[Route("metrics")] // -> similar to /metrics as Ocelot's route
public class MetricsController : Controller
{ /* your integration */ }

It should work, and I'll be surprised if it doesn't.

---

I gave MapWhenOcelotPipeline a shot like you suggested here but I think it's a no-go because the middleware seems to be internal, so replicating what MapPrometheusScrappingEndpoint does here doesn't seems to be possible.

Regarding MapWhenOcelotPipeline, I can't provide a working draft solution at the moment, as it requires actual coding and research. However, the #416 author has mentioned that branching Ocelot's pipeline is feasible. I'm currently occupied with the highly anticipated Globality release (v24.1), so we can revisit this discussion after the release.

what MapPrometheusScrappingEndpoint does here doesn't seems to be possible. 👇

builder.UseMiddleware<PrometheusExporterMiddleware>(meterProvider);

Very interesting... And middleware injection is possible in Ocelot. Have you tried debugging the code for your integration app to see what happens?

Something like a blacklist of paths Ocelot won't process would be necessary I think, rather than a way to manually handle an endpoint like with MapWhenOcelotPipeline.

Ocelot doesn't use "blacklists" because, for the /metrics path, there is no matching between the path and the configured routes in ocelot.json. If there's no match, Ocelot simply returns a 404. This matching process is handled by the DownstreamRouteFinder service, which processes static routes as part of the DownstreamRouteFinderMiddleware.
The issue is that the middleware is executed before the branching pipes in MapWhenOcelotPipeline take effect, causing the DownstreamRouteFinderMiddleware to return an error prematurely, which prevents the MapWhenOcelotPipeline configuration from being applied →

Ocelot/src/Ocelot/Middleware/OcelotPipelineExtensions.cs

Lines 54 to 71 in 265f9b4

	// Then we get the downstream route information
	app.UseMiddleware<DownstreamRouteFinderMiddleware>();
	// Multiplex the request if required
	app.UseMiddleware<MultiplexingMiddleware>();
	// This security module, IP whitelist blacklist, extended security mechanism
	app.UseMiddleware<SecurityMiddleware>();
	//Expand other branch pipes
	if (configuration.MapWhenOcelotPipeline != null)
	{
	foreach (var pipeline in configuration.MapWhenOcelotPipeline)
	{
	// todo why is this asking for an app app?
	app.MapWhen(pipeline.Key, pipeline.Value);
	}
	}

Another option is to override the DownstreamRouteFinder service, enabling a whitelist of predefined paths to be reused by the associated endpoints.

[fetchezar]

Thank you, Raman. This information is very helpful.

I think the problem is that even if I could map the /metrics endpoint with a controller for example, since the middleware that has the metrics is internal to the Prometheus library, I cant return the metrics from that controller. I was thinking along the lines of handling the endpoint manually somehow, grabbing the prometheus middleware, and returning the metrics information manually... but it's all internal and I cant invoke it.

Mapping the endpoint is one problem, and returning the metrics is another.

From what I understand, the OpenTelemetry+Prometheus endpoint must be part of the current Ocelot application to gather metrics for each registered endpoint

Exactly and even if I managed to make up some kind of controller that can return the metrics as the Prometheus server expects them, I don't even know if the prometheus middleware will successfully collect any metrics from requests handled by Ocelot. Things like memory usage or kestrel's counters should be doable but things like average response time? I'm not sure.

Thanks a lot, with this context I think I have enough information to research a bit more. Maybe I can ask the Prometheus exporter devs to make a public interface for the middleware so it can be handled in a more personalized fashion.

[raman-m]

I understand the challenges with the Ocelot+OpenTelemetry+Prometheus integration. This is a complex setup involving products we currently don't support.
I think we should prioritize adding OpenTelemetry support first and then consider integrating Prometheus. This will likely require significant coding efforts.
Please prepare a draft solution, conduct some coding research, and we can revisit this discussion after the v24.1 release when I have more free time to review your draft solution.

[raman-m]

We currently don't support OpenTelemetry, but we're open to accepting pull requests from contributors for integration.
Please stick to the officially supported logging and tracing methods.