NoCredentialProviders: no valid providers in chain error after booting my custom build AMI

[effy-coding]

👉 Question: My custom build bottlerocket cannot access to public ECR while booting EC2...

I added some features and successfully builded and published to my AWS account.

After launch EC2, NoCredentialProviders: no valid providers in chain error occurs even if target registry is PUBLIC.

System error log below

See 'systemctl status kubelet.service' for details.
[  OK  ] Reached target Multi-User System.
[   60.580411] host-ctr[1556]: time="2022-02-24T08:55:12Z" level=error msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" ref="ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4"
[   60.592938] host-ctr[1556]: time="2022-02-24T08:55:12Z" level=fatal msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
[   61.337685] host-ctr[1557]: time="2022-02-24T08:55:13Z" level=error msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" ref="ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5"
[   61.365208] host-ctr[1557]: time="2022-02-24T08:55:13Z" level=fatal msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
[  157.245747] host-ctr[1608]: time="2022-02-24T08:56:49Z" level=error msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" ref="ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5"
[  157.270728] host-ctr[1608]: time="2022-02-24T08:56:49Z" level=fatal msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
[  157.587903] host-ctr[1601]: time="2022-02-24T08:56:49Z" level=error msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" ref="ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4"
[  157.630292] host-ctr[1601]: time="2022-02-24T08:56:49Z" level=fatal msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
[  254.938238] host-ctr[1630]: time="2022-02-24T08:58:27Z" level=error msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" ref="ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5"
[  254.951812] host-ctr[1630]: time="2022-02-24T08:58:27Z" level=fatal msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
[  256.525479] host-ctr[1637]: time="2022-02-24T08:58:28Z" level=error msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" ref="ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4"
[  256.537827] host-ctr[1637]: time="2022-02-24T08:58:28Z" level=fatal msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
[  268.704557] metricdog[1652]: Error sending HTTP request to https://metrics.bottlerocket.aws/v1/metrics?sender=metricdog&event=health_ping&version=1.6.0&variant=aws-k8s-1.21&arch=x86_64&region=ap-northeast-2&seed=1569&version_lock=latest&ignore_waves=false&failed_services=kubelet%3A0&is_healthy=false: error sending request for url (https://metrics.bottlerocket.aws/v1/metrics?sender=metricdog&event=health_ping&version=1.6.0&variant=aws-k8s-1.21&arch=x86_64&region=ap-northeast-2&seed=1569&version_lock=latest&ignore_waves=false&failed_services=kubelet%3A0&is_healthy=false): operation timed out
[  350.128472] host-ctr[1674]: time="2022-02-24T09:00:02Z" level=error msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" ref="ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5"
[  350.200264] host-ctr[1674]: time="2022-02-24T09:00:02Z" level=fatal msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-control:v0.5.5\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
[  353.755990] host-ctr[1681]: time="2022-02-24T09:00:06Z" level=error msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" ref="ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4"
[  353.781605] host-ctr[1681]: time="2022-02-24T09:00:06Z" level=fatal msg="retries exhausted: failed to resolve reference \"ecr.aws/arn:aws:ecr:ap-northeast-2:328549459982:repository/bottlerocket-admin:v0.7.4\": NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

[effy-coding]

This is because EC2 doesnt have IAM role having AmazonEC2ContainerRegistryReadOnly