¿Vector > HAProxy > datadoghq.eu?

[julgsk]

Hello there,

Here is what I want to achieve:
vector (sink: datadog_logs) > (datadog-agent/)haproxy > datadoghq.eu

Some context:

• I have datadog agents (I am a Datadog customer) sending their metrics to a designated datadog agent with HAProxy (only one machine is authorized to send data to DD platform)
• I have Vector sending digested logs to a Kafka topic, works great

My objective:

• Send the same logs from Vector to the datadog "proxy" that will in its turn send these logs to DD platform

Is this achievable?

This vector config does not work:

sink_datadog:
  type: datadog_logs
    site: "datadoghq.eu"
    inputs:
      - my_great_input
    default_api_key: "super_secret_dd_api_key"
    proxy:
      enabled: true
      http: "http://fqdn_for_dd_agent:3838"

The 3838 port is found here: https://docs.datadoghq.com/agent/configuration/proxy/?tab=linux#haproxy
HAProxy is configured just like in the official docs, I use default ports.

This is excerpt from /etc/haproxy/haproxy.cfg:

# This declares the endpoint where your Agents connects for
# sending Logs (e.g the value of "logs.config.logs_dd_url")
# If sending logs with force_use_http: true
frontend logs_http_frontend
    bind *:3838
    mode http
    option tcplog
    default_backend datadog-logs-http

Thanks for your help!

[jszwedko]

Hi @jgresiak !

The proxy configuration in Vector is meant to be used with standard HTTP proxies, HAProxy is more of a load balancer rather than a generic proxy.. I think you want configuration more like:

sink_datadog:
  type: datadog_logs
    endpoint: http://fqdn_for_dd_agent:3838"
    inputs:
      - my_great_input
    default_api_key: "super_secret_dd_api_key"

(note the use of the HAProxy URL as endpoint)

Note that you'll need to configure HAProxy listen and forward the logs specifically. That means configuration like:

frontend logs_http_frontend
    bind *:3838 ssl crt <PATH_TO_PROXY_CERTIFICATE_PEM>
    mode http
    option tcplog
    default_backend datadog-logs-http

backend datadog-logs-http
    balance roundrobin
    mode http
    # The following configuration is for HAProxy 1.8 and newer (you'll need to use the EU endpoint)
    server-template mothership 5 agent-http-intake.logs.datadoghq.com:443  check port 443 ssl verify required ca-file <PATH_TO_DATADOG_CERTIFICATES_CRT> check resolvers my-dns init-addr none resolve-prefer ipv4
    # Uncomment the following configuration for older HAProxy versions
    # server datadog agent-http-intake.logs.datadoghq.com:443 check port 443 ssl verify required ca-file <PATH_TO_DATADOG_CERTIFICATES_CRT>

This is shown on https://docs.datadoghq.com/agent/configuration/proxy/?tab=linux#haproxy

Alternatively, rather than HAProxy, you could use an actual HTTP proxy like Squid which just accepts and forwards any HTTP requests. This could be used with Vector's proxy configuration.

Hope this helps!

[julgsk]

Could this be of help?
2024-07-12T16:56:15.445934Z DEBUG http: vector::internal_events::http_client: HTTP response. status=404 Not Found version=HTTP/1.1 headers={"content-type": "application/json", "content-length": "81", "cross-origin-resource-policy": "cross-origin", "accept-encoding": "identity,gzip,x-gzip,deflate,x-deflate,zstd", "x-content-type-options": "nosniff", "strict-transport-security": "max-age=31536000; includeSubDomains; preload", "date": "Fri, 12 Jul 2024 16:56:14 GMT", "via": "1.1 google", "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"} body=[81 bytes]

[jszwedko]

Hmm, that seems to be missing the path. Is there a line before that one that has the request details?

[julgsk]

2024-07-12T17:11:40.169804Z DEBUG http: hyper::client::pool: pooling idle connection for ("http", fqdn_agent:3838)
2024-07-12T17:11:40.170159Z DEBUG http: vector::internal_events::http_client: HTTP response. status=404 Not Found version=HTTP/1.1 headers={"content-type": "application/json", "content-length": "81", "cross-origin-resource-policy": "cross-origin", "accept-encoding": "identity,gzip,x-gzip,deflate,x-deflate,zstd", "x-content-type-options": "nosniff", "strict-transport-security": "max-age=31536000; includeSubDomains; preload", "date": "Fri, 12 Jul 2024 17:11:39 GMT", "via": "1.1 google", "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"} body=[81 bytes]```

[julgsk]

2024-07-12T17:11:39.988968Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=http://fqdn_agent:3838/api/v1/validate method=GET version=HTTP/1.1 headers={"dd-api-key": "api_key", "user-agent": "Vector/0.39.0 (x86_64-pc-windows-msvc 73da9bb 2024-06-17 16:00:23.791735272)", "accept-encoding": "identity"} body=[empty]
2024-07-12T17:11:39.989883Z DEBUG vector::topology::running: Configuring outputs for source. component=source_reception_appointement
2024-07-12T17:11:39.991232Z TRACE http: hyper::client::pool: checkout waiting for idle connection: ("http", fqdn_agent:3838)
2024-07-12T17:11:40.002599Z DEBUG vector::topology::running: Configuring output for component. component=source_reception_appointement output_id=None
2024-07-12T17:11:40.003220Z TRACE http: hyper::client::connect::http: Http::connect; scheme=Some("http"), host=Some("fqdn_agent"), port=Some(Port(3838))
2024-07-12T17:11:40.003399Z DEBUG vector::topology::running: Configuring outputs for source. component=source_transfert_echange
2024-07-12T17:11:40.004527Z  INFO vector::topology::builder: Healthcheck passed.
2024-07-12T17:11:40.004547Z DEBUG vector::topology::running: Configuring output for component. component=source_transfert_echange output_id=None
2024-07-12T17:11:40.004733Z DEBUG hyper::client::connect::dns: resolving host="fqdn_agent"
2024-07-12T17:11:40.005075Z  INFO vector::topology::builder: Healthcheck passed.
2024-07-12T17:11:40.005800Z DEBUG vector::topology::running: Configuring outputs for source. component=source_lmxt_communicator
2024-07-12T17:11:40.007614Z DEBUG vector::topology::running: Configuring output for component. component=source_lmxt_communicator output_id=None
2024-07-12T17:11:40.007832Z DEBUG http: hyper::client::connect::http: connecting to fqdn_ip:3838
2024-07-12T17:11:40.008183Z DEBUG vector::topology::running: Configuring outputs for transform. component=transform01_lmxt_communicator
2024-07-12T17:11:40.008402Z TRACE rdkafka::client: Create new librdkafka client 0x1fda6d58570
2024-07-12T17:11:40.009294Z DEBUG vector::topology::running: Configuring output for component. component=transform01_lmxt_communicator output_id=None
2024-07-12T17:11:40.010153Z DEBUG vector::topology::running: Configuring outputs for transform. component=transform01_transfert_echange
2024-07-12T17:11:40.009752Z TRACE rdkafka::client: Starting metadata fetch
2024-07-12T17:11:40.010234Z DEBUG http: hyper::client::connect::http: connected to fqdn_ip:3838

[jszwedko]

Ah, thanks for sharing the additional logs. I would expect the requests to http://fqdn_agent:3838/api/v1/validate, which are part of Vector's health checks, to fail unless you wired up that path in the HAProxy config. You can disable the health check with healthcheck.enabled: false in the sink configuration.