Can't set logs service_name for docker logs

[fcortes]

I'm not really sure if this is a problem with this setup, Loki's OTLP endpoint, Alloy, me, or somewhere in-between, but I've been struggling for a while to make the following (seemingly simple) setup work

• Grafana LGTM stack using the grafana/otel-lgtm image
• Local services running on docker
• Query logs in grafana

For the log ingestion and service discoverability I'm using Alloy and everything seem to work fine except the service_name attribute in the stored logs. For some reason, loki is adding its own service_name=unknown_service label even if I try to explicitly set it. This is pretty important when trying to correlate traces as it is the only filter that grafana automatically applies when clicking on the "Go to logs" button (there's probably a way to configure this in grafana itself but following the philosophy of the repository, I don't want to start messing with each service config).

I found a couple of threads with similar problems but could't use any proposed solution successfully (like this one and this one)

Am I doing something wrong? Or is this too much to ask for this no-config repository?

Setup to reproduce the behavior

Use this compose file

services:
  logger:
    image: busybox
    command: sh -c 'while true; do echo "[$(date)] Infinite log message..."; sleep 5; done'
    depends_on:
      lgtm:
        condition: service_healthy
    labels:
      com.local.monitor-logs: "true"

  alloy:
    image: grafana/alloy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config.alloy:/etc/alloy/config.alloy
    depends_on:
      lgtm:
        condition: service_healthy

  lgtm:
    image: grafana/otel-lgtm
    environment:
    - OTEL_METRIC_EXPORT_INTERVAL=500
    ports:
      - 5000:3000
    healthcheck:
      test: test -e /tmp/ready
      interval: 5s
      timeout: 2s
      retries: 20

Things I have tried

Default options

config.alloy

discovery.docker "containers_with_logs" {
  host = "unix:///var/run/docker.sock"

	filter {
		name = "label"
		values = ["com.local.monitor-logs=true"]
	}
}

loki.source.docker "docker_compose_services_logs" {
  host = "unix:///var/run/docker.sock"

  targets = discovery.docker.containers_with_logs.targets
  labels = {"app" = "docker"}
  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "lgtm:4317"
    tls {
      insecure = true
      insecure_skip_verify = true
    }
  }
}

No service_name detected (expected)

Add a docker processor

Thinking that it may be able to infer some names from the integration

config.alloy

discovery.docker "containers_with_logs" {
  host = "unix:///var/run/docker.sock"

	filter {
		name = "label"
		values = ["com.local.monitor-logs=true"]
	}
}

loki.source.docker "docker_compose_services_logs" {
  host = "unix:///var/run/docker.sock"

  targets = discovery.docker.containers_with_logs.targets
  labels = {"app" = "docker"}
  forward_to = [loki.process.default.receiver]
}

loki.process "default" {
	stage.docker {}

  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "lgtm:4317"
    tls {
      insecure = true
      insecure_skip_verify = true
    }
  }
}

No service_name either (kind of expected too)

Explicitly set the label in the docker processor

config.alloy

discovery.docker "containers_with_logs" {
  host = "unix:///var/run/docker.sock"

	filter {
		name = "label"
		values = ["com.local.monitor-logs=true"]
	}
}

loki.source.docker "docker_compose_services_logs" {
  host = "unix:///var/run/docker.sock"

  targets = discovery.docker.containers_with_logs.targets
  labels = {"app" = "docker"}
  forward_to = [loki.process.default.receiver]
}

loki.process "default" {
	stage.docker {}

	stage.labels {
		values = {
			service_name = "some-service-name",
			some_other_label = "some-value",
		}
	}

  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "lgtm:4317"
    tls {
      insecure = true
      insecure_skip_verify = true
    }
  }
}

Completely ignored the service_name label. It did set the other non-standard label correctly

Explicitly set it as a static_label instead

config.alloy

discovery.docker "containers_with_logs" {
  host = "unix:///var/run/docker.sock"

	filter {
		name = "label"
		values = ["com.local.monitor-logs=true"]
	}
}

loki.source.docker "docker_compose_services_logs" {
  host = "unix:///var/run/docker.sock"

  targets = discovery.docker.containers_with_logs.targets
  labels = {"app" = "docker"}
  forward_to = [loki.process.default.receiver]
}

loki.process "default" {
	stage.docker {}

	stage.static_labels {
		values = {
			service_name = "some-service-name",
			some_other_label = "some-value",
		}
	}

  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "lgtm:4317"
    tls {
      insecure = true
      insecure_skip_verify = true
    }
  }
}

The label is added, but Loki appends _extracted to the key and sets its own service_name to unknown_service

Add relabeling rules in the discovery level

config.alloy

discovery.docker "containers_with_logs" {
  host = "unix:///var/run/docker.sock"

	filter {
		name = "label"
		values = ["com.local.monitor-logs=true"]
	}
}

discovery.relabel "docker_compose_services" {
    targets = discovery.docker.containers_with_logs.targets

    rule {
        source_labels = ["__meta_docker_container_label_com_docker_compose_project"]
        target_label  = "project"
    }

    rule {
        source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
        target_label  = "service"
    }

    rule {
        source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
        target_label  = "service_name"
    }
}

loki.source.docker "docker_compose_services_logs" {
  host = "unix:///var/run/docker.sock"

  targets = discovery.docker.containers_with_logs.targets
	relabel_rules = discovery.relabel.docker_compose_services.rules
  labels = {"app" = "docker"}
  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "lgtm:4317"
    tls {
      insecure = true
      insecure_skip_verify = true
    }
  }
}

Pretty much the same behavior. service_name being ignored and redefined.

Set attributes using a otelcol.processor.attributes

(this way I could set service.name instead of service_name which was proposed somewhere)

config.alloy

discovery.docker "containers_with_logs" {
  host = "unix:///var/run/docker.sock"

	filter {
		name = "label"
		values = ["com.local.monitor-logs=true"]
	}
}

loki.source.docker "docker_compose_services_logs" {
  host = "unix:///var/run/docker.sock"

  targets = discovery.docker.containers_with_logs.targets
  labels = {"app" = "docker"}
  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.processor.attributes.default.input]
  }
}

otelcol.processor.attributes "default" {
	action {
		key = "service.name"
		value = "abc"
		action = "upsert"
	}

	action {
		key = "other_attribute"
		value = "xyz"
		action = "upsert"
	}

  output {
    logs = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "lgtm:4317"
    tls {
      insecure = true
      insecure_skip_verify = true
    }
  }
}

Same thing

Try to disable Loki trying to infer the service name

https://grafana.com/docs/loki/latest/setup/upgrade/#service_name-label

config.alloy

discovery.docker "containers_with_logs" {
  host = "unix:///var/run/docker.sock"

	filter {
		name = "label"
		values = ["com.local.monitor-logs=true"]
	}
}

loki.source.docker "docker_compose_services_logs" {
  host = "unix:///var/run/docker.sock"

  targets = discovery.docker.containers_with_logs.targets
  labels = {"app" = "docker"}
  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.processor.attributes.default.input]
  }
}

otelcol.processor.attributes "default" {
	action {
		key = "service.name"
		value = "abc"
		action = "upsert"
	}

	action {
		key = "other_attribute"
		value = "xyz"
		action = "upsert"
	}

	action {
		key = "discover_service_name"
		value = false
		action = "insert"
	}

  output {
    logs = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "lgtm:4317"
    tls {
      insecure = true
      insecure_skip_verify = true
    }
  }
}

You get the point...

[zeitlinger]

@cyrille-leclerc can you give a recommendation?

[zeitlinger]

I think this is a question for https://github.com/grafana/alloy?tab=readme-ov-file#community

[jakubnoga]

@fcortes
Hi, I did some googling on a similar problem I had. Seems it's solved by using a discovery.relabel component. I found an example that works for me. No sure what exactly makes it work, but I'm guessing it's setting a "container" label which does the trick. Or maybe something implicit?

Basically define relabeling rules. Then use the exported fields in relabel_rules field of your loki.source.docker. I'm leaving you the link to the solution. But I didn't need all the rules - setting "container" label was enough for me.

Here's config snippet that did the trick for me:

discovery.docker "logs_integrations_docker" {
    host = "unix:///var/run/docker.sock"
    refresh_interval = "5s"
}

discovery.relabel "logs_integrations_docker" {
    targets = []
    rule {
        source_labels = ["__meta_docker_container_name"]
        regex = "/(.*)"
        target_label = "container"
    }
}

loki.source.docker "logs_integrations_docker" {
    host = "unix:///var/run/docker.sock"
    targets = discovery.docker.logs_integrations_docker.targets
    forward_to = [loki.write.grafana_cloud_loki.receiver]
    relabel_rules = discovery.relabel.logs_integrations_docker.rules
    refresh_interval = "5s"
}

Here's the backlink to the solution:
https://www.reddit.com/r/grafana/comments/1er3b18/alloy_and_docker_how_to_get_container_labels/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button