Loading OPA Bundles from S3 using EKS and OIDC

[CheyneBaker-EdApp]

Hello I am currently trying to set up OPA Bundles in S3 (not public) and I am having trouble getting the web_identity_credentials to work in OPA. I have set up the EKS with OIDC and enabled full S3 privileges (for testing). To confirm connectivity and permissioning I was able to download the bundle via the aws-cli using the OIDC s3 credentials without issue in a different container but when I use the OPA container I receive a 403 forbidden.
I have followed the documentation here and tired a few other things. Any advice would be welcomed.

Testing with this command
opa run --log-level debug -c /config/config.yaml

Current config is
`apiVersion: v1
kind: ConfigMap
metadata:
name: opa-istio-config
data:
config.yaml: |
decision_logs:
console: true

services:
  s3:
    url: https://redacted.s3.ap-southeast-2.amazonaws.com/
    credentials:
      s3_sigining:
        web_identity_credentials:
          aws_region: ap-southeast-2
          session_name: opa-bundle 

bundles:
  authz:
    service: s3
    resource: bundle.tar.gz`

Error is
{"level":"debug","msg":"Download starting.","time":"2023-08-01T06:52:25Z"} {"headers":{"Prefer":["modes=snapshot,delta"],"User-Agent":["Open Policy Agent/0.55.0 (linux, amd64)"]},"level":"debug","method":"GET","msg":"Sending request.","time":"2023-08-01T06:52:25Z","url":"https://redacted.s3.ap-southeast-2.amazonaws.com/bundle.tar.gz"} {"headers":{"Content-Type":["application/xml"],"Date":["Tue, 01 Aug 2023 06:52:25 GMT"],"Server":["AmazonS3"],"X-Amz-Id-2":["q7RsWaT2c4h1SCKYsCwL8GToyt7bqgfQ5m"],"X-Amz-Request-Id":["1CDYNYXX"]},"level":"debug","method":"GET","msg":"Received response.","response":"HTTP/1.1 403 Forbidden\r\nTransfer-Encoding: chunked\r\nContent-Type: application/xml\r\nDate: Tue, 01 Aug 2023 06:52:25 GMT\r\nServer: AmazonS3\r\nX-Amz-Id-2: KgAvAtDxY8q7RsWaT2c4h1SCKYsCwL8GToyt7bqgfQ5m\r\nX-Amz-Request-Id: G1CDYNYXX\r\n\r\nf3\r\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cError\u003e\u003cCode\u003eAccessDenied\u003c/Code\u003e\u003cMessage\u003eAccess Denied\u003c/Message\u003e\u003cRequestId\7TVDG1CDYNYXX\u003c/RequestId\u003e\u003cHostId\vAtDxY8q7RsWaT2c4h1SCKYsCwL8GToyt7bqgfQ5m\u003c/HostId\u003e\u003c/Error\u003e\r\n0\r\n\r\n","status":"403 Forbidden","time":"2023-08-01T06:52:25Z","url":"https://redacted.s3.ap-southeast-2.amazonaws.com/bundle.tar.gz"} {"level":"error","msg":"Bundle load failed: server replied with Forbidden","name":"authz","plugin":"bundle","time":"2023-08-01T06:52:25Z"} {"level":"debug","msg":"Waiting 4.42708873s before next download/retry.","time":"2023-08-01T06:52:25Z"}

[ashutosh-narkar]

Have you already followed the steps documented here? There was a related issue mentioning Web Identity Creds that could be helpful open-policy-agent/opa#5562.

[CheyneBaker-EdApp]

@ashutosh-narkar Thank you for your reply sadly the document provide was already the one I detailed that I had followed in my problem summary. The second URL link I had already stumbled across and ensured those items were set also in the OPA config I also tried setting the AWS Signature to 4a.

To provide more clarity I have provided some more information where possible also of note is I have an AWS CLI container in the same pod as OPA and it can access the bucket with the Web Credentials without error and I have opened up the IAM policy permissions to take it out of the equation.

AWS CLI Container
ENV

AWS_ROLE_ARN=arn:aws:iam::123456789:role/OpaBundleRole@clustername
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Bundle Load

bash-4.2# aws s3 cp s3://redacted-opa-bundles-poc/bundles.tar.gz ./
bash-4.2# aws s3 cp s3://redacted-opa-bundles-poc/bundle.tar.gz ./
download: s3://redacted-opa-bundles-poc/bundle.tar.gz to ./bundle.tar.gz
bash-4.2# ls
bundle.tar.gz

OPA Container
ENV

AWS_ROLE_ARN=arn:aws:iam::123456789:role/OpaBundleRole@clustername
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_REGION=ap-southeast-2

Bundle Load

opa run -c /config/config.yaml https://redacted-opa-bundles-poc.s3.ap-southeast-2.amazonaws.com/bundle.tar.gz

{"level":"info","msg":"Starting bundle loader.","name":"cli1","plugin":"bundle","time":"2023-08-01T23:32:40Z"}
{"level":"info","msg":"Starting bundle loader.","name":"authz","plugin":"bundle","time":"2023-08-01T23:32:40Z"}
{"level":"error","msg":"Bundle load failed: server replied with Forbidden","name":"cli1","plugin":"bundle","time":"2023-08-01T23:32:40Z"}
{"level":"error","msg":"Bundle load failed: server replied with Forbidden","name":"authz","plugin":"bundle","time":"2023-08-01T23:32:40Z"}

IAM Role
Trust entities

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::123456789:oidc-provider/[oidc.eks.ap-southeast-2.amazonaws.com/id/EXAMPLE](http://oidc.eks.ap-southeast-2.amazonaws.com/id/EXAMPLE)”
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.ap-southeast-2.amazonaws.com/id/EXAMPLE:sub": "system:serviceaccount:*:*",
                    "oidc.eks.ap-southeast-2.amazonaws.com/id/EXAMPLE:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::redacted-opa-bundles-poc"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::redacted-opa-bundles-poc/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Service Account

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/OpaBundleRole@clustername
  name: opa-service-role-s3
  namespace: default
automountServiceAccountToken: false

[ashutosh-narkar]

Thanks for sharing this. Is there any difference between a public and non-public S3 setup in terms of access? Also are you able to verify that the env variables are properly set and available to the OPA container?

[CheyneBaker-EdApp]

A public S3 bucket setup does not require IAM credentials so the bundle is able to be pulled down without error. I provided the ENV variables above or was there some other ENV's that you were looking for?

If you're asking if the OPA container has a JWT set in the token file location it does.

[CheyneBaker-EdApp]

After pulling apart the entire environment I have found the issue to not be related to OPA and it operates as documented.

Though I am still investigating it looks like Kyverno was affecting OPA in some way. I am happy to call this issue closed or I can report back on my findings after I complete my investigation.

Thanks

[anderseknert]

Please do! 😃 Always good to share that kind of knowledge should others find this when having similar problems.

[CheyneBaker-EdApp]

After investigation, the issue was found to be in the OPA config. s3_signing was spelt incorrectly causing OPA to skip over using or pulling in AWS credentials.

services:
  s3:
    url: https://redacted.s3.ap-southeast-2.amazonaws.com/
    credentials:
      s3_sigining:  <<< should be s3_signing
        web_identity_credentials:
          aws_region: ap-southeast-2
          session_name: opa-bundle 

Thanks all for your time on this.

[anderseknert]

Oh wow, thanks for reporting back! We really ought to have a better way of warning the user when that happens.

[ashutosh-narkar]

Thanks for reporting @CheyneBaker-EdApp. Something like open-policy-agent/opa#2745 would have helped in this case.