Can vector (inside a docker container as a service) connect to other docker services directly without HTTP Sink connection?

[phoenisx]

Hi

Is there a way to create a sink that can connect to Docker service directly, without exposing the other service as an HTTP to the outside world?

E.g. I have two docker services: vector and analytics, where analytics service is not exposed to the outside world, but lies in the same docker network where vector service lies.

So instead of creating an HTTP sink, like:

sinks:
   analytics:
      type: "http"
      inputs:
         - logs
      encoding:
         codec: "json"
      method: "post"
      request:
         retry_max_duration_secs: 10
      uri: "http://${SERVICE_ANALYTICS_NAME:-analytics}:${ANALYTICS_PORT:-3000}/api/logs?source_name=gotrue.logs.prod&api_key=your-super-secret-and-long-logflare-key"

Can I make the above work with service names directly.

[jszwedko]

This seems like less of a Vector question and more of a Docker DNS and networking question 🤔

[phoenisx]

I think the issue is with the docker labels. Removing the labels starts showing the logs, but also disables the filtering entirely.

I am using Docker labels and not Docker Container names, cause my Local and Production setups work very differently, and container names are dynamic.

[phoenisx]

Found one more thing, having a single label item works and vector detects it just fine, but the moment I add more than two labels in my .yml, vector service stops detecting the docker containers.
What I mean is that the following works

# vector.yml
...
sources:
   docker_host:
      type: docker_logs
      include_labels:
         - "com.foo.appname=foo-web-service"
...

But the following doesn't

# vector.yml
...
sources:
   docker_host:
      type: docker_logs
      include_labels:
         - "com.foo.appname=foo-web-app"
         - "com.foo.appname=foo-web-service"
...

I also tried using different label key-value pairs for different images, but it still didn't work.

[phoenisx]

This seems to be a workaround for now:

# vector.yml
...
sources:
   foo_web:
      type: docker_logs
      include_labels:
         - "com.foo.appname=foo-web-app"
   foo_service:
      type: docker_logs
      include_labels:
         - "com.foo.appname=foo-web-service"
...

[phoenisx]

@jszwedko Connecting directly to the service host inside Docker swarm doesn't seem to work with Vector.

I am getting the following error:

ERROR vector::topology::builder: msg="Healthcheck failed." error=Failed to make HTTP(S) request: error trying to connect: tcp connect error: Connection refused (os error 111) component_kind="sink" component_type="loki" component_id=

[jszwedko]

Apologies, this seems like a Docker networking issue rather than something specific to Vector. I might recommend following up with Docker or Docker Swarm documentation or community support to help with networking.