Poblems trying to use AWS sink with assume_role

[agustin-munoz]

Hello,

On our setup in order to access S3 bucket placed on another account we use sts assume role with external-id and role-session-name, we have a provided credentials file and we are able to use sts assume role, then access the bucket.

I am trying to setup this using AWS S3 sink, having this:

Credentials file set in /etc/aws/credentials and environment variable AWS_CONFIG_FILE set to use that file
Environment variables ARN_EXTERNAL_ID and AWS_ROLE_SESSION_NAME

The sink section:

sinks:
  audit_bucket:
    type: aws_s3
    inputs:
      - squid-logs.shadow
    acknowledgements:
      enabled: false
    auth:
      assume_role: ${ARN}
      external_id: ${ARN_EXTERNAL_ID}
    bucket: s3sinkbucket
    encoding:
      codec: json
    healthcheck:
      enabled: false
    region: eu-central-1
    storage_class: STANDARD_IA
    filename_append_uuid: true
    filename_time_format: "%Y/%m/%d"
    key_prefix: "squid_log/maint5.eu-central-1.aws.openpaas-maint.axa-cloud.com/%F/"

When I start vector it is stopping with following error:

2023-09-27T08:17:34.874299Z ERROR vector::cli: Configuration error. error=data did not match any variant of untagged enum AwsAuthentication

If I remove the external_id, then it starts but it is not able to perform sts as the user is not allowed.

Is there any way to place together assume_role and external_id ??

[jszwedko]

Hi @agustin-munoz ,

What version of Vector are you running? external_id was added in v0.32.

[agustin-munoz]

Hello @jszwedko

I am running v0.31, I will check with latest available version (so far v0.33.0)

[agustin-munoz]

Hello @jszwedko ,

Once updated to v0.33.0 the error about configuration error has been gone but now it is not able to perform STS.

In our case, the remote AWS account has a role with this policy attached

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${accountid_one}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "${external_id}",
          "sts:RoleSessionName": "${role_session_name}"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${accountid_one}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "${external_id}",
          "sts:RoleSessionName": "${role_session_name}"
        }
      }
    }
  ]
}

I will try with another account to use a role checking only the external id condition.

[agustin-munoz]

After setting a similar environment in another account. Assume role works right if the condition for the sts:RoleSessionName is not set in the trusted policy, otherwise it is not able to perform sts:AssumeRole even if AWS_ROLE_SESSION_NAME environment variable is set.

[jszwedko]

Ah, we may need to add a configuration option for that. I think you are specifying AWS config options directly in the Vector config file then it won't also read the AWS_ authentication environment variables in addition. You could try dropping the AWS auth config from Vector and configuring it all via the environment, but I'm not seeing an env var for external id, just AWS_ROLE_ARN and AWS_ROLE_SESSION_NAME. Would you want to open a feature request to add a session_name option?

[agustin-munoz]

Definetely it will be superb to fit our needs and probably someone else needs. We can not set other environment variables for auth config as far as the file with credentials is periodically rotated.

We are currently able to work that way with fluentd, but we want to move over vector :)

Definetely I will create a feature request for it. Thanks for your support.