Does the googlecloud / stackdriver support metrics and traces from opentelemetry?

[tyrone-anz]

We've been exporting our opentelemetry metrics to the collector (v0.23.0) using opentelemetry-go (v0.17.0) and the collector is configured to forward to google cloud. However, we are not successful in doing so and the collector is getting errors.

Depending on the metric distribution, it complains about a different error. This is a sample error from the collector.

2021-03-31T01:05:36.094Z	info	exporterhelper/queued_retry.go:276	Exporting failed. Will retry the request after interval.	{"component_kind": "exporter", "component_type": "googlecloud", "component_name": "googlecloud", "error": "rpc error: code = InvalidArgument desc = Field timeSeries[0].points[0].distributionValue had an invalid value: Distribution value has |bucket_counts| fields that sum to 0 which does not equal the |count| field value of 847.", "interval": "4.98550433s"}

Distribution value has |bucket_counts| fields that sum to 0 which does not equal the |count| field value of 847

When I inspected the googlecloud exporter code, I can see it is using opencensus libraries to send the metrics. Based on that, I am beginning to think that it can only export metrics if it from the opencensus receiver. I cannot find any documentation about it though.

My assumption before is any exporter in the collector can forward successfully any telemetry data the collector is receiving regardless of the receiver used. But in my experience with the googlecloud exporter, we are only able to forward metrics successfully if from opencensus receiver and not otlp receiver. Or is this just a matter of the otlp receiver not yet stable?

[bogdandrutu]

/cc @punya @dashpole @jsuereth

[dashpole]

Apologies for the slow response.
The googlecloud exporter export metrics from any receiver. Since the OTel metrics API/SDK isn't stable yet, our opentelemetry metrics exporter can't be stable, so the collector still uses the opencensus exporter. It should still be able to send metrics from opentelemetry.

This is likely a bug in the google cloud opencensus exporter, or the go SDK (but probably in our exporter).

If you can help us reproduce the issue, that would be helpful. I can think of a few approaches:

1. It would be helpful to know the details of metric that is causing problems. If you can enable the logging exporter (with logLevel: debug along-side the google-cloud one, we can figure out what the bucket counts are, and such. With that, we might be able to construct an easy repro.
2. If you already have a reproduction using opentelemetry-go that you can share, that would also work.

[tyrone-anz]

Hi @dashpole, sorry me again. Would you know if the Exact distribution will be supported eventually? And if so, when do we hope to be able to use it?

[dashpole]

@tyrone-anz We don't have plans to support it right now. Based on recent discussions in the opentelemetry-go SIG, the Exact distribution might be removed prior to the metrics API/SDK becoming stable, so we may not need to support it. If it isn't removed, we would likely have limited support for the Exact distribution, since we can't send the same metric to GCM more than once every 10 seconds. This would mean we are forced to drop some of the recorded metrics in an Exact distribution if metrics are recorded more frequently than every 10 seconds. At that point, I believe the Exact histogram becomes equivalent to a gauge with an CollectPeriod of 10 seconds.

[tyrone-anz]

Thanks for the prompt reply @dashpole.

[JCMais]

I am having a similar issue here, related to Duplicate TimeSeries encountered, is there any config on the otel collector that can help alleviate this?

[dashpole]

@JCMais there are a variety of issues that can cause Duplicate TimeSeries encountered. The error usually means metrics from two instances of an application are being written to the same monitored resource, and is solved by adding resource attributes to differentiate the two instances. Feel free to open a separate discussion/issue, and we can figure out what the issue is.