AWS Prometheus_remote_write failed to receive credentials

[aggholamyy]

Hi, first of all: I'm a total vector novice so bear with me,
I'm trying to send node_exporter metrics of an EC2 instance to a AWS managed prometheus on the same VPC,
I'm using docker image of the vector timberio/vector:0.39.0-debian
after configuring the vector by vector.yaml and running the container I receive the following logs:

ERROR sink{component_kind="sink" component_id=prometheus_remote_write component_type=prometheus_remote_write}:request{request_id=2}: vector::sinks::util::retries: Unexpected error type; dropping the request. error=the credential provider was not enabled internal_log_rate_limit=true
WARN sink{component_kind="sink" component_id=prometheus_remote_write component_type=prometheus_remote_write}:request{request_id=2}: vector::sinks::util::adaptive_concurrency::controller: Unhandled error response. error=the credential provider was not enabled internal_log_rate_limit=true
ERROR sink{component_kind="sink" component_id=prometheus_remote_write component_type=prometheus_remote_write}:request{request_id=2}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") })) request_id=2 error_type="request_failed" stage="sending" internal_log_rate_limit=true
ERROR sink{component_kind="sink" component_id=prometheus_remote_write component_type=prometheus_remote_write}:request{request_id=2}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=499 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

my vector.yaml is like:

sources:
  my_source_id:
    type: prometheus_scrape
    endpoints:
      - http://exporter:9100/metrics
    scrape_interval_secs: 15
    scrape_timeout_secs: 5

sinks:
  prometheus_remote_write:
    type: "prometheus_remote_write"
    inputs:
      - my_source_id
    endpoint: "https://aps-workspaces.eu-central-1.amazonaws.com/workspaces/ws-<the_WS_ID>/api/v1/remote_write"
    auth:
      strategy: "aws"
      region: "eu-central-1"
    aws:
      region: "eu-central-1"

all the policy, IAM Role and attaching Role to the EC2 is done, I send curl request like the following to verify that credentials are accessible in EC2:

curl http://169.254.169.254/latest/meta-data/iam/info
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/vector-prome-remote-write

the results are respectively:

{
  "Code" : "Success",
  "LastUpdated" : "2024-07-24T21:34:34Z",
  "InstanceProfileArn" : "arn:aws:iam::<XXXXX>:instance-profile/vector-prome-remote-write",
  "InstanceProfileId" : "AIPEF5D6XCCQYXYXYXY"
}

vector-prome-remote-write

{
  "Code" : "Success",
  "LastUpdated" : "2024-07-24T21:35:22Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASIAT5D6XCCDSFVGHIOD",
  "SecretAccessKey" : "bIjDVDVDVDVDVDVDkKpKbdwyU",
  "Token" : "<long_string_of_characters>"

The vector sink document for Sink Prometheus remote write said "aws.region" is optional but without it I received error on "if you are using aws authentication then you must have aws.region".

Did I miss anything in configuration file?

Thank you in advance for your time and help.

[jszwedko]

Hi @aggholamyy . I'd suggest looking at debug logs from the AWS SDK. You can do this by setting an environment variable: VECTOR_LOG=aws_config=debug. This should emit logs showing where the SDK attempted to pull credentials from.