401 Unauthorized error when pushing Kafka Connect build to ECR despite correct IAM and Service Account setup

[comstering]

Hello,

I'm encountering a persistent 401 Unauthorized error when attempting to push an image to ECR using the build feature of the Strimzi KafkaConnect resource.

Strimzi Operator Version: [Strimzi version in use]

Kubernetes Environment: EKS [EKS version in use]

Error Log:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "{account-id}.dkr.{region}.amazonaws.com/strimzi/kafka-connect:dev": POST https://{account-id}.dkr.ecr.{region}.amazonaws.com/v2/strimzi/kafka-connect/blobs/uploads/: unexpected status code 401 Unauthorized: Not Authorized

Troubleshooting steps taken

1. IAM Role Permissions: I have verified that the IAM Role has all the necessary ECR push permissions, including ecr:GetAuthorizationToken, ecr:PutImage, and other related actions for layer uploads.

2. IRSA (IAM Roles for Service Accounts) Configuration: The KafkaConnect resource's template.buildServiceAccount.metadata.annotations field is correctly configured with the IAM Role ARN that has ECR push permissions.

spec:
  template:
    buildServiceAccount:
      metadata:
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::{account-id}:role/{role-name}

3. Service Account Verification: I have confirmed that the KafkaConnect Pod is using the correct Service Account and that this Service Account is properly linked to the IAM Role using kubectl describe sa.
4. ECR Repository Policy: The ECR repository policy does not contain any explicit Deny rules that would override the Allow permissions granted to the IAM Role. The policy is configured to allow access from the correct AWS account ID.

Given that all standard configurations seem correct, I suspect there might be another underlying issue. Could there be a problem with a VPC Endpoint policy, Private Link, or a specific Strimzi configuration that I might have overlooked? Any guidance on this would be greatly appreciated.

Thank you.

[im-konge]

Hey, I see all the points there, maybe one additional question - and TBH I never tried to push to ECR -> for the account you should have some creds, are you sure that those are used in the build? It can happen that even though everything is correctly configured, you don't have the creds set for the build (I suppose it's the Kaniko build on Kubernetes, or are you using OCP?).

[im-konge]

It's here https://strimzi.io/docs/operators/latest/configuring#type-Build-reference under the Using Docker registry

[comstering]

Sorry. I'm idlot, this probled solved to iam resource.

I set ECR arn like this

{
    "Statement": [
        {
                " Action": ["ecr:*"],
                "Effect": "Allow",
                "Resource": ["arn.aws.ecr.{region}.{account-id}.repository/*"] // this is problem
        }
    ]
}

I fix like this.

{
    "Statement": [
        {
                " Action": ["ecr:*"],
                "Effect": "Allow",
                "Resource": ["*"]
        }
    ]
}

I don't know why don't working I set resource.

But issue solved.

Thank U

[im-konge]

Ah okay, so something else actually :) great that you figured it out