Consume logs from other containers running in docker compose

[paymog]

I'm interested in adopting vector. We run our workloads in kubernetes and run our local developer environments with docker compose. I can see a fairly easy path to integrating with kubernetes but I'm having trouble figuring out how to configure vector to collect the logs of other containers running in docker compose. Is there a pre-defined source which I can use for this? Is there a sample config I can follow to set this up?

I've tried using the following config

[api]
enabled = true
address = "0.0.0.0:8686"

[sources.demo_logs]
type = "demo_logs"
interval = 1.0
format = "json"

[sources.docker_logs]
type = "docker_logs"

[sinks.console]
inputs = ["demo_logs", "docker_logs"]
target = "stdout"
type = "console"
encoding.codec = "json"

but then when I start my docker compose script I only see the demo logs

 ❯❯❯ pnpm start | rg vector
[+] Running 12/0
 ⠿ Container e2e-tests-vector-1                 Created 0.0s
 ⠿ Container e2e-tests-ipfs-1                   Created 0.0s
 ⠿ Container e2e-tests-postgres-1               Created 0.0s
 ⠿ Container e2e-tests-stripe-cli-1             Created 0.0s
 ⠿ Container e2e-tests-rpc-node-proxy-1         Created 0.0s
 ⠿ Container e2e-tests-influxdb-1               Created 0.0s
 ⠿ Container e2e-tests-temporalite-1            Created 0.0s
 ⠿ Container e2e-tests-hasura-1                 Created 0.0s
 ⠿ Container e2e-tests-telegraf-1               Created 0.0s
 ⠿ Container e2e-tests-enterprise-hasura-1      Created 0.0s
 ⠿ Container e2e-tests-enterprise-graph-node-1  Created 0.0s
 ⠿ Container e2e-tests-graph-node-1             Created 0.0s
Attaching to e2e-tests-enterprise-graph-node-1, e2e-tests-enterprise-hasura-1, e2e-tests-graph-node-1, e2e-tests-hasura-1, e2e-tests-influxdb-1, e2e-tests-ipfs-1, e2e-tests-postgres-1, e2e-tests-rpc-node-proxy-1, e2e-tests-stripe-cli-1, e2e-tests-telegraf-1, e2e-tests-temporalite-1, e2e-tests-vector-1
e2e-tests-vector-1                 | <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
e2e-tests-vector-1                 | <jemalloc>: (This is the expected behaviour if you are running under QEMU)
e2e-tests-vector-1                 | 2023-05-15T13:45:32.802233Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
e2e-tests-vector-1                 | 2023-05-15T13:45:32.864250Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
e2e-tests-vector-1                 | 2023-05-15T13:45:33.083897Z  INFO source{component_kind="source" component_id=docker_logs component_type=docker_logs component_name=docker_logs}: vector::sources::docker_logs: Capturing logs from now on. now=2023-05-15T13:45:33.079407799+00:00
e2e-tests-vector-1                 | 2023-05-15T13:45:33.093584Z  INFO source{component_kind="source" component_id=docker_logs component_type=docker_logs component_name=docker_logs}: vector::sources::docker_logs: Listening to docker log events.
e2e-tests-vector-1                 | 2023-05-15T13:45:33.129386Z  INFO vector::topology::running: Running healthchecks.
e2e-tests-vector-1                 | 2023-05-15T13:45:33.130200Z  INFO vector::topology::builder: Healthcheck passed.
e2e-tests-vector-1                 | 2023-05-15T13:45:33.210172Z ERROR source{component_kind="source" component_id=docker_logs component_type=docker_logs component_name=docker_logs}: vector::sources::docker_logs: Listing currently running containers failed. error=error trying to connect: No such file or directory (os error 2)
e2e-tests-vector-1                 | 2023-05-15T13:45:33.226936Z  INFO vector: Vector has started. debug="false" version="0.29.1" arch="x86_64" revision="74ae15e 2023-04-20 14:50:42.739094536"
e2e-tests-vector-1                 | {"message":"{\"host\":\"47.118.136.150\",\"user-identifier\":\"shaneIxD\",\"datetime\":\"15/May/2023:13:45:33\",\"method\":\"PATCH\",\"request\":\"/apps/deploy\",\"protocol\":\"HTTP/1.0\",\"status\":\"400\",\"bytes\":7403,\"referer\":\"https://up.org/booper/bopper/mooper/mopper\"}","source_type":"demo_logs","timestamp":"2023-05-15T13:45:33.169230466Z"}
e2e-tests-vector-1                 | 2023-05-15T13:45:33.583346Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=http://0.0.0.0:8686/playground
e2e-tests-vector-1                 | {"message":"{\"host\":\"12.14.132.121\",\"user-identifier\":\"Karimmove\",\"datetime\":\"15/May/2023:13:45:34\",\"method\":\"POST\",\"request\":\"/do-not-access/needs-work\",\"protocol\":\"HTTP/2.0\",\"status\":\"500\",\"bytes\":38427,\"referer\":\"https://random.org/do-not-access/needs-work\"}","source_type":"demo_logs","timestamp":"2023-05-15T13:45:34.154350966Z"}

I have many services in my docker compose file outputting many logs. I guess that in this instance I haven't configured vector to correctly communicate with docker from within docker?

[paymog]

As is typical, I found the solution within minutes of posting the question. I mounted the host docker socket into the vector container like so in my docker compose:

      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock

As usual, there are security considerations.

[spencergilbert]

Yep, that was going to be my suggestion 👍