request to http://loki:3100/otlp/v1/logs/v1/logs responded with HTTP Status Code 404

[gustavosett]

I'm facing this issue trying to use Otel collector with loki inside docker...

2024-07-23 17:16:27 2024-07-23T20:16:27.911Z    error   exporterhelper/queue_sender.go:90       Exporting failed. Dropping data.        {"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = error exporting items, request to http://loki:3100/otlp/v1/logs/v1/logs responded with HTTP Status Code 404", "dropped_items": 1}
2024-07-23 17:16:27 go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
2024-07-23 17:16:27     go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:90
2024-07-23 17:16:27 go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
2024-07-23 17:16:27     go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52
2024-07-23 17:16:27 go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
2024-07-23 17:16:27     go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43

loki-local-config.yaml:

auth_enabled: false

limits_config:
  allow_structured_metadata: true

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: localhost
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4417  # GRPC endpoint for telemetry data
      http:
        endpoint: 0.0.0.0:4418  # HTTP endpoint for telemetry data

processors:
  batch:
    timeout: 5s                # Batch data for up to 5 seconds
    send_batch_size: 1000      # Number of spans or metrics per batch

exporters:
  otlphttp:
    endpoint: http://loki:3100/otlp/v1/logs # Endpoint for Loki to send logs
    tls:
      insecure: true
  otlp/tempo:
    endpoint: tempo:4317       # Endpoint for Tempo to send traces
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: prometheus:9090  # Prometheus remote write endpoint
    tls:
      insecure: true
    external_labels:
      source: otelcol
      cluster: kubernetes-cluster

extensions:
  health_check:                # Health check extension
  pprof:                       # Performance profiling data via pprof
  zpages:                      # Provides detailed status and span data

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

I've tryied with "http://loki:3100/loki/api/v1/push" endpoint but still the same problem:
2024-07-23 20:44:44 2024-07-23T23:44:44.250Z error exporterhelper/queue_sender.go:90 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = error exporting items, request to http://loki:3100/loki/api/v1/push/v1/logs responded with HTTP Status Code 404", "dropped_items": 1}

Without any endpoints, only with "http://loki:3100":
{"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = error exporting items, request to http://loki:3100/v1/logs responded with HTTP Status Code 404", "dropped_items": 1}

[td5r]

have you tried

otlphttp:
    endpoint: http://loki:3100/otlp

instead of

otlphttp:
    endpoint: http://loki:3100/otlp/v1/logs 

The OTEL connector already appends /v1/logs/ to the endpoint

[pszybiak]

I have the same problem. Even if I set endpoint: http://loki:3100/otlp I get error

{"kind": "exporter", "data_type": "logs", "name": "otlphttp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = error exporting items, request to http://loki:3100/otlp/v1/logs responded with HTTP Status Code 404", "dropped_items": 10}

Loki is working correctly:

level=info ts=2024-08-06T11:44:56.82896247Z caller=main.go:108 msg="Starting Loki" version="(version=2.9.9, branch=HEAD, revision=aaca205d6c)"
level=info ts=2024-08-06T11:44:56.829444Z caller=server.go:322 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"

[pszybiak]

OK, I found a source of problem.
I take docker image like this:

  loki:
    image: grafana/loki:latest
    ports:
       - 3100:3100

but this instruction returns image with version 2.9.9.
We need to set strict version number:

  loki:
    image: grafana/loki:3.1.0
    ports:
       - 3100:3100

Version 2.9.9 was published after version 3.1.0 and this is the problem.