oltp grpc exporter failing in k8s deployment using helm charts

[parth-purohit]

Hi Team,

I have deployed both otel-collector as daemonset and standalone deployment as per the README docs in below link

https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector

agentCollector:
enabled: true
standaloneCollector:
enabled: true

The oltp grpc exporter in agent pod is failing with below error:

2022-01-25T05:53:25.907Z	info	service/telemetry.go:116	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "basic", "service.instance.id": "ebb82220-7804-4da1-af63-1bd1055120e8", "service.version": "latest"}
2022-01-25T05:53:25.907Z	info	service/collector.go:230	Starting otelcontribcol...	{"Version": "v0.37.1", "NumCPU": 4}
2022-01-25T05:53:25.907Z	info	service/collector.go:132	Everything is ready. Begin running and processing data.
2022-01-25T05:53:37.500Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "5.810290538s"}
2022-01-25T05:53:56.408Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "7.534661199s"}
2022-01-25T05:54:07.502Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "4.816822688s"}
2022-01-25T05:54:26.217Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "16.060087817s"}
2022-01-25T05:54:37.525Z	info	exporterhelper/queued_retry.go:231	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "failed to push metrics data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "4.261162066s"}

Steps followed:
helm install --set standaloneCollector.enabled=true my-opentelemetry-collector open-telemetry/opentelemetry-collector
kubectl logs -f daemonset/my-opentelemetry-collector-agent

Could you please help resolve this issue.

[jpkrohling]

Looks like I can't transfer this, but this probably belongs to open-telemetry/opentelemetry-helm-charts.

[asitkumar13]

@parth-purohit
I had similar issue. I was using javagant version 1.3.0 but then I updated to 1.10.1 and I dont see DeadlineExceeded error.