[Promethus Exporter] Failed to gathering metrics: collected metric "http_server_active_requests"

[ychaydhari]

Hello,
There are two services with otel-java-agent running on my machine. I'm getting one error. Can someone help me to figure out root cause of this issue.

2022-09-28T11:56:38.079+0530 error [email protected]/log.go:34 error gathering metrics: collected metric "http_server_active_requests" { label:<name:"http_flavor" value:"1.1" > label:<name:"http_host" value:"localhost:7171" > label:<name:"http_method" value:"GET" > label:<name:"http_scheme" value:"http" > label:<name:"job" value:"cm_service" > gauge:<value:16 > } was collected before with the same name and label values {"kind": "exporter", "data_type": "metrics", "name": "prometheus"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*promLogger).Println github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/log.go:34 github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1 github.com/prometheus/[email protected]/prometheus/promhttp/http.go:139 net/http.HandlerFunc.ServeHTTP net/http/server.go:2084 net/http.(*ServeMux).ServeHTTP net/http/server.go:2462 go.opentelemetry.io/collector/config/confighttp.(*decompressor).wrap.func1 go.opentelemetry.io/[email protected]/config/confighttp/compression.go:162 net/http.HandlerFunc.ServeHTTP net/http/server.go:2084 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:204 go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP go.opentelemetry.io/[email protected]/config/confighttp/clientinfohandler.go:39 net/http.serverHandler.ServeHTTP net/http/server.go:2916 net/http.(*conn).serve net/http/server.go:1966

Command line :-
Service 1 :- -javaagent:"C:\Workspace\api\service\opentelemetry-javaagent.jar" -Dotel.metrics.exporter=otlp -Dotel.traces.exporter=none -Dotel.exporter.otlp.metrics.endpoint=http://localhost:4317 -Dotel.service.name=service1

Service 2 :- -javaagent:"C:\Workspace\api\service\opentelemetry-javaagent.jar" -Dotel.metrics.exporter=otlp -Dotel.traces.exporter=none -Dotel.exporter.otlp.metrics.endpoint=http://localhost:4317 -Dotel.service.name=service2

[jack-berg]

I've recreated your setup as best as I can tell and can't recreate the error you're seeing. From what I can tell, you're running version 0.59.0 of the collector with a otlp receiver and prometheus exporter.

Could you post a small reproduction, or add details about any unusual settings in the collector?

BTW, given that the error message is from the prometheus exporter of the collector, you might have better luck posting this in opentelemetry-collector-contrib.

[ychaydhari]

Which Java agent version using?
-- 2.38.0

Please find below the config file which I'm using to start OTEL service

receivers:
otlp:
protocols:
grpc:
http:

exporters:
prometheus:
endpoint: "0.0.0.0:8889"

logging:

extensions:
health_check:

processors:
batch:

service:
extensions: [health_check]
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging, prometheus]

[jack-berg]

Which Java agent version using?
-- 2.38.0

Not sure what this means - there is not a 2.38.0 of the otel java agent.

I did my best attempt to recreate your situation locally and can't reproduce the error. Maybe you could share a small reproduction on github?

[ychaydhari]

yes, I'm using version 0.59.0 of collector

[ychaydhari]

Metrics value also comming as possitive and negative means 5 and -5