Error on InfluxDB exporter: failed to convert OTLP metric to line protocol: unsupported non-monotonic sum

[ziomill]

Hello,

We are trying to send metrics to InfluxDB through the influxdb exporter (provided by the contrib project) and we are experiencing this error:

2022-03-07T18:43:28.642Z	error	exporterhelper/queued_retry.go:186	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "name": "influxdb", "error": "Permanent error: failed to convert OTLP metric to line protocol: unsupported non-monotonic sum 'runtime.jvm.memory.pool'", "dropped_items": 35}
2022-03-07T18:43:28.642684400Z go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
2022-03-07T18:43:28.642692400Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:186
2022-03-07T18:43:28.642696700Z go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
2022-03-07T18:43:28.642700800Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:134
2022-03-07T18:43:28.642704800Z go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
2022-03-07T18:43:28.642709100Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry_inmemory.go:121
2022-03-07T18:43:28.642713100Z go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
2022-03-07T18:43:28.642717100Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:99
2022-03-07T18:43:28.642721100Z go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
2022-03-07T18:43:28.642725100Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:78

This is the OTel collector configuration:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  # Send Metrics to InfluxDB
  influxdb:
    endpoint: http://influxdb:8086
    timeout: 500ms
    org: mycompany
    bucket: otel-metrics
    token: 78z0IzKhi7ik_cT9Dx-whP7Qe5Q9LO30d5b420cOR_rzKDJnI3I5gVnHfMlKWzjkPoSmOAbwycW82ENeli5BHA==
    metrics_schema: telegraf-prometheus-v2

    sending_queue:
      enabled: true
      num_consumers: 3
      queue_size: 10

    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 3s
      max_elapsed_time: 10s


### Configure the Collector
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [influxdb]

Do you have any suggestions?

Thank you

[jpkrohling]

cc @jacobmarble @8none1

[jacobmarble]

Hi @ziomill . Thanks for this, I'll take a look today.

[jacobmarble]

@ziomill I have reviewed the OTel metrics data model spec. It looks like we should treat cumulative, non-monotonic sums similar to a gauge value, for schema telegraf-prometheus-v2.

Please follow along here: influxdata/influxdb-observability#50

[jacobmarble]

See also #8348

[ziomill]

Hi @jacobmarble

I have upgraded the otel-collector to the opentelemetry-collector-contrib:0.47.0 (where the problem should have been fixed):

Unfortunately, I still have errors about metrics sending:

Using the metrics_schema: telegraf-prometheus-v1:

2022-03-25T10:19:59.925Z	error	exporterhelper/queued_retry.go:183	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "name": "influxdb", "error": "Permanent error: failed to convert OTLP metric to line protocol: unsupported non-monotonic sum 'runtime.jvm.memory.pool'", "dropped_items": 40}
2022-03-25T10:19:59.926065000Z go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
2022-03-25T10:19:59.926072900Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:183
2022-03-25T10:19:59.926075900Z go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
2022-03-25T10:19:59.926080500Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:133
2022-03-25T10:19:59.926084400Z go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
2022-03-25T10:19:59.926086700Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry_inmemory.go:118
2022-03-25T10:19:59.926088900Z go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
2022-03-25T10:19:59.926091400Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:99
2022-03-25T10:19:59.926093800Z go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
2022-03-25T10:19:59.926096200Z 	go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:78

Using the metrics_schema: telegraf-prometheus-v2:

otel-clct-container            | 2022-03-25T10:30:18.241Z       error   exporterhelper/queued_retry.go:183      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "name": "influxdb", "error": "Perma
nent error: failed to convert OTLP metric to line protocol: unsupported non-monotonic sum 'runtime.jvm.memory.pool'", "dropped_items": 35}
otel-clct-container            | go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
otel-clct-container            |        go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:183
otel-clct-container            | go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
otel-clct-container            |        go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:133
otel-clct-container            | go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
otel-clct-container            |        go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry_inmemory.go:118
otel-clct-container            | go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
otel-clct-container            |        go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:99
otel-clct-container            | go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
otel-clct-container            |        go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:78

Any suggestion?

Thank you

[ziomill]

Thank you @jacobmarble

Anyway, in the v0.47.0 Release note the #8348 is reported as merged:

[pmm-sumo]

Yeah, apparently it's part of the v0.47.0 release looking at the component's go.mod

[jacobmarble]

My mistake x2.

#8462 was merged after v0.47.0, and is unrelated to this concern.

#8348 is, indeed, included in v0.47.0. The error unsupported non-monotonic sum persists because I tagged https://github.com/influxdata/influxdb-observability/releases/tag/otel2influx%2Fv0.2.15 one commit early.

commit 14db1a7807e294fd3fea47c084818cdb42f07710
Author: Jacob Marble <[email protected]>
Date:   Tue Mar 8 14:55:13 2022 -0800

    Merge pull request #51 from influxdata/jgm-non-monotonic-metric
    
    feat: add support for cumulative, non-monotonic sums

commit 04e1dbbfa96a856a9a916ef83df9fe7c3b04df2e (tag: otel2influx/v0.2.15, tag: influx2otel/v0.2.15, tag: common/v0.2.15)
Author: Jacob Marble <[email protected]>
Date:   Mon Mar 7 09:18:32 2022 -0800

    chore: update dependencies per dependabot suggestions

So, the fix for this issue is included in main, post-v0.47.0 due to #8462, but not in v0.47.0.

My apologies.

[pmm-sumo]

Thank you for checking this @jacobmarble

v0.48.0 is just around the corner so we should have the fix released in a few days

[ziomill]

Just to tell you that with 0.48.0 it works as expected. Thank you!

P.S.: Now I would use Grafana with InfluxDB data source. Do you know if there are prebuilt dashboards to import in order to visualize standard OpenTelemetry metrics?