AWS managed prometheus integration with vector sink prometheus_remote_write

[jarfral]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I am trying to connect my AWS managed prometheus instance with my vector agent running on a ec2 instance with a ecs agent running. I am trying to send ecs_metrics to prometheus. I am able to do so with a prometheus server running on the same host machine and using the prometheus_exporter sink but not with the remote_write sink. I enabled VECTOR_LOG=aws_config=debug on my vector agent and have verified that I am able to authenticate with AWS to receive the correct credentials only when I pass the env variables with the access key, session token, secret key.

Is there a way for the vector agent to assume the role that is attached to the task that vector is running in?

I am running the docker image: timberio/vector:0.39.0-alpine

Configuration

#                                    __   __  __
#                                    \ \ / / / /
#                                     \ V / / /
#                                      \_/  \/
#
#                                    V E C T O R
#                                   Configuration
#
# ------------------------------------------------------------------------------
# Website: https://vector.dev
# Docs: https://vector.dev/docs
# Chat: https://chat.vector.dev
# ------------------------------------------------------------------------------

# Change this to use a non-default directory for Vector data storage:
# data_dir: "/var/lib/vector"

# Random Syslog-formatted logs
sources:
  docker_content:
    type: docker_logs
    # include_images:
    #   - nginx:latest
    include_labels: ["com.amazonaws.ecs.cluster=test"] 

  ecs_metrics:
    type: aws_ecs_metrics

# Parse Syslog logs

transforms:

  filter_out_backslashes:
    type: lua
    inputs:
      - docker_content
    version: "2"
    hooks:
      process: |-
        function (event, emit)
          -- Check if the message field contains backslashes
          if string.match(event.log.message, "\\") then
            -- Drop the log by not emitting it
            return
          end
          -- Pass the log through
          emit(event)
        end
sinks:
  prometheus_write:
    type: prometheus_remote_write
    inputs:
      - ecs_metrics
    endpoint: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-2a9dded1-5d43-43d0-9deb-77a78d86d1d9/api/v1/remote_write
    default_namespace: vector
    healthcheck: false
    auth:
      region: us-east-1
      strategy: aws
    aws:
      region: us-east-1
      endpoint: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-2a9dded1-5d43-43d0-9deb-77a78d86d1d9/api/v1/remote_write

Version

timberio/vector:0.39.0-alpine

Debug Output

`2024-07-30T12:20:46.740109Z DEBUG hyper::proto::h1::io: flushed 990 bytes
2024-07-30T12:20:47.222155Z DEBUG transform{component_kind="transform" component_id=filter_out_backslashes component_type=lua}: vector::utilization: utilization=0.00008934048952547165
2024-07-30T12:20:47.405741Z DEBUG source{component_kind="source" component_id=ecs_metrics component_type=aws_ecs_metrics}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://169.254.170.2/v3/6a7d9d39-5892-455c-923d-941891cb2220/task/stats method=GET version=HTTP/1.1 headers={"user-agent": "Vector/0.39.0 (x86_64-unknown-linux-musl 73da9bb 2024-06-17 16:00:23.791735272)", "accept-encoding": "identity"} body=[empty]
2024-07-30T12:20:47.405806Z DEBUG source{component_kind="source" component_id=ecs_metrics component_type=aws_ecs_metrics}:http: hyper::client::connect::http: connecting to 169.254.170.2:80
2024-07-30T12:20:47.405962Z DEBUG source{component_kind="source" component_id=ecs_metrics component_type=aws_ecs_metrics}:http: hyper::client::connect::http: connected to 169.254.170.2:80
2024-07-30T12:20:47.406046Z DEBUG hyper::proto::h1::io: flushed 209 bytes
2024-07-30T12:20:47.406558Z DEBUG sink{component_kind="sink" component_id=prometheus_write component_type=prometheus_remote_write}: vector::utilization: utilization=0.00000034015692348652186
2024-07-30T12:20:47.406579Z DEBUG hyper::proto::h1::io: parsed 7 headers
2024-07-30T12:20:47.406585Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1689 bytes)
2024-07-30T12:20:47.406612Z DEBUG sink{component_kind="sink" component_id=prometheus component_type=prometheus_exporter}: vector::utilization: utilization=0.0000004127891402505394
2024-07-30T12:20:47.406614Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-07-30T12:20:47.406633Z DEBUG source{component_kind="source" component_id=ecs_metrics component_type=aws_ecs_metrics}:http: hyper::client::pool: pooling idle connection for ("http", 169.254.170.2)
2024-07-30T12:20:47.406649Z DEBUG source{component_kind="source" component_id=ecs_metrics component_type=aws_ecs_metrics}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"content-type": "application/json", "x-rate-limit-duration": "1", "x-rate-limit-limit": "40.00", "x-rate-limit-request-forwarded-for": "", "x-rate-limit-request-remote-addr": "10.14.81.72:57828", "date": "Tue, 30 Jul 2024 12:20:47 GMT", "content-length": "1689"} body=[1689 bytes]
2024-07-30T12:20:47.515632Z DEBUG source{component_kind="source" component_id=prome_export component_type=prometheus_scrape}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://10.14.81.72:9091/metrics method=GET version=HTTP/1.1 headers={"accept": "text/plain", "user-agent": "Vector/0.39.0 (x86_64-unknown-linux-musl 73da9bb 2024-06-17 16:00:23.791735272)", "accept-encoding": "identity"} body=[empty]
2024-07-30T12:20:47.515698Z DEBUG source{component_kind="source" component_id=prome_export component_type=prometheus_scrape}:http: hyper::client::pool: reuse idle connection for ("http", 10.14.81.72:9091)
2024-07-30T12:20:47.515792Z DEBUG hyper::proto::h1::io: flushed 189 bytes
2024-07-30T12:20:47.515846Z DEBUG hyper::proto::h1::io: parsed 4 headers
2024-07-30T12:20:47.515862Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-07-30T12:20:47.515895Z DEBUG sink{component_kind="sink" component_id=prometheus component_type=prometheus_exporter}:http-request{method=GET path=/metrics}: vector::internal_events::http: Internal log [Received HTTP request.] is being suppressed to avoid flooding.
2024-07-30T12:20:47.516103Z DEBUG hyper::proto::h1::io: flushed 8262 bytes
2024-07-30T12:20:47.516141Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-07-30T12:20:47.516146Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2024-07-30T12:20:47.516158Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x1FBC (8124 bytes)
2024-07-30T12:20:47.516182Z DEBUG source{component_kind="source" component_id=prome_export component_type=prometheus_scrape}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"content-type": "text/plain; version=0.0.4", "transfer-encoding": "chunked", "date": "Tue, 30 Jul 2024 12:20:47 GMT"} body=[unknown]
2024-07-30T12:20:47.516220Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-07-30T12:20:47.516232Z DEBUG hyper::client::pool: pooling idle connection for ("http", 10.14.81.72:9091)
2024-07-30T12:20:48.408422Z DEBUG sink{component_kind="sink" component_id=prometheus_write component_type=prometheus_remote_write}:request{request_id=5}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Environment
2024-07-30T12:20:48.408524Z DEBUG sink{component_kind="sink" component_id=prometheus_write component_type=prometheus_remote_write}:request{request_id=5}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-2a9dded1-5d43-43d0-9deb-77a78d86d1d9/api/v1/remote_write method=POST version=HTTP/1.1 headers={"x-prometheus-remote-write-version": "0.1.0", "content-type": "application/x-protobuf", "content-encoding": "snappy", "x-amz-date": "20240730T122048Z", "authorization": Sensitive, "x-amz-security-token": Sensitive, "user-agent": "Vector/0.39.0 (x86_64-unknown-linux-musl 73da9bb 2024-06-17 16:00:23.791735272)", "accept-encoding": "identity"} body=[1331 bytes]
2024-07-30T12:20:48.408565Z DEBUG sink{component_kind="sink" component_id=prometheus_write component_type=prometheus_remote_write}:request{request_id=5}:http: hyper::client::pool: reuse idle connection for ("https", aps-workspaces.us-east-1.amazonaws.com)
2024-07-30T12:20:48.408817Z DEBUG hyper::proto::h1::io: flushed 2977 bytes
2024-07-30T12:20:48.441793Z DEBUG hyper::proto::h1::io: parsed 5 headers
2024-07-30T12:20:48.441815Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-07-30T12:20:48.441904Z DEBUG sink{component_kind="sink" component_id=prometheus_write component_type=prometheus_remote_write}:request{request_id=5}:http: hyper::client::pool: pooling idle connection for ("https", aps-workspaces.us-east-1.amazonaws.com)`

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Can you share the log output when you set VECTOR_LOG=aws_config=debug? Do you see it trying to fetch the role credentials from the metadata endpoint?