Upgrade AWS CDK to avoid downtime for applications using iam.OpenIdConnectProvider

[evgenyka]

Introduction

If you have deployed an AWS Cloud Development Kit (CDK) application which uses the OpenIdConnectProvider construct from the aws-iam module, you are at risk of an outage. CDK versions 2.51.0 and 1.181.0 contain an important fix to the aws-iam module, which eliminates the risk.
The AWS CDK can create an OpenIdConnectProvider by either accepting a certificate thumbprint or fetching a thumbprint from the issuer URL. When fetching the thumbprint, previous versions of the AWS CDK used the leaf certificate; however, according to AWS documentation, applications should use the root certificate, not the leaf certificate. Because the root certificate is the top-most certificate in the certificate chain, it is rotated much less frequently than leaf certificates, which are typically rotated once a year.

Note: this discussion is focused on the impact on applications deployed to Amazon EKS clusters, but this issue can impact any application that uses the AWS IAM OpenIdConnectProvider construct.

Applications running in containers deployed to Amazon EKS clusters can use AWS IAM’s implementation of the OpenIdConnect standard, OpenIdConnectProvider, to associate Kubernetes service accounts with AWS IAM Roles. This enables pods to use Kubernetes service account credentials to make AWS API calls. Pods running on clusters provisioned with the AWS CDK which use the aws-iam module’s implementation of the OpenIdConnectProvider construct with an Amazon EKS cluster will break when leaf certificates are rotated.

Note: if you only use the OpenIdConnectProvider from aws-eks, then your application is not at immediate risk of breakage. aws-eks hard-codes the thumbprint of the root certificate, which doesn't expire anytime soon. The OpenIdConnectProvider created by the Amazon EKS Cluster uses this construct by default.

How do I know if my application is impacted?

If you specified a custom OpenIdConnectProvider from aws-iam that uses an Amazon EKS Cluster’s clusterOpenIdConnectIssuerUrl (or any issuer url from another source), and you did not provide a thumbprint list in its props, your cluster is at risk. For example, an application containing the code below will break when the leaf certificates are rotated:

import * as iam from `aws-cdk-lib/aws-iam`;
import * as eks from `aws-cdk-lib/aws-eks`;

const cluster = new eks.Cluster(this, "EKSCluster", {
  version: eks.KubernetesVersion.V1_21,
});

new iam.OpenIdConnectProvider(this, "EKSOIDCProvider", {
  url: cluster.clusterOpenIdConnectIssuerUrl,
  clientIds: ["sts.amazonaws.com"]
});

Your cluster is not at risk if you did not specify an OpenIdConnectProvider from aws-iam. By default, clusters deployed with AWS CDK create an OpenIdConnectProvider from the aws-eks module, which hard-codes the thumbprint of the root certificate:

import * as eks from `aws-cdk-lib/aws-eks`;
const cluster = new eks.Cluster(this, "EKSCluster", {
  version: eks.KubernetesVersion.V1_21,
});

import * as eks from `aws-cdk-lib/aws-eks`;

const cluster = new eks.Cluster(this, "EKSCluster", {
  version: eks.KubernetesVersion.V1_21,
});

new eks.OpenIdConnectProvider(this, "EKSOIDCProvider", {
  url: cluster.clusterOpenIdConnectIssuerUrl,
});

How can I mitigate the issue?

You can change the thumbprint list of the provider via the AWS IAM Console. However, this introduces drift from your AWS CDK stack and is not recommended. To mitigate the issue, we strongly advise you to upgrade the AWS CDK to a version at least 2.51.0, or at least 1.181.0, if you’re still on v1, neither of which suffer from this issue. Once you’ve upgraded, perform a deployment by using cdk deploy or a deployment pipeline.

If you cannot upgrade your AWS CDK version, you can avoid the breakage by modifying the code that created the OpenIdConnectProvider to use the aws-eks module like so:

import * as iam from `aws-cdk-lib/aws-iam`;
import * as eks from `aws-cdk-lib/aws-eks`;

const cluster = new eks.Cluster(this, "EKSCluster", {
  version: eks.KubernetesVersion.V1_21,
});

- new iam.OpenIdConnectProvider(this, "OIDCProvider", {
+ new eks.OpenIdConnectProvider(this, "OIDCProvider", {
  url: cluster.clusterOpenIdConnectIssuerUrl,
});

Note: while this example focuses on Amazon EKS, any iam.OpenIdConnectProvider declared in this way is at risk.

Alternatively, if you are not using AWS EKS and must use aws-iam, you can hard-code the thumbprint of the root certificate:

import * as iam from `aws-cdk-lib/aws-iam`;
import * as eks from `aws-cdk-lib/aws-eks`;

const cluster = new eks.Cluster(this, "EKSCluster", {
  version: eks.KubernetesVersion.V1_21,
});

new iam.OpenIdConnectProvider(this, "OIDCProvider", {
  url: cluster.clusterOpenIdConnectIssuerUrl,
    clientIds: ["sts.amazonaws.com"],
  + thumbprints: ['9e99a48a9960b14926bb7f3b02e22da2b0ab7280'],
});

How can I validate that the impact was mitigated?

Go to the AWS IAM Console and ensure that you are in the same region as any impacted OpenIdConnectProviders. Then, under IAM resources, click on the number beneath Identity providers:

This will open a new page with a list of OpenIdConnectProviders. Click on each one, and verify that at least one of the thumbprints in the list is the EKS root certificate, 9e99a48a9960b14926bb7f3b02e22da2b0ab7280. You can find it under the Thumbprints heading on the right hand side:

If your provider is not Amazon EKS, then ensure that the root certificate matches the one from your provider instead of the Amazon EKS root certificate.

How was the issue fixed?

We changed the implementation of the aws-iam module’s OpenIdConnectProvider to retrieve the root certificate instead of the leaf certificate. Additionally, the aws-eks module no longer hard-codes the OpenIdConnectProvider’s thumbprint list; it is now only a wrapper on the corrected implementation of aws-iam module.
For more details, please see the Pull Requests that contain the fix:

• fix(iam): oidc provider fetches leaf certificate thumbprint instead of root #22802
• chore(eks): Integ Test for OIDCP Certificate Retrieval #22608

[takahiro-asano-uhuru]

Hi!
By when will it be necessary to respond?

[evgenyka]

I would recommend not to wait till leaf certificate will be rotated (currently valid till mid 2023), but upgrade to latest version and redeploy the relevant stacks no later than end of 2022. To be on the safe side.

[takahiro-asano-uhuru]

Thank you very much for your responses!!
I will use it as a reference!

[takahiro-asano-uhuru]

Is the deadline for correspondence correct in the document?
https://aws.amazon.com/jp/blogs/developer/version-1-of-the-aws-cloud-development-kit-aws-cdk-is-now-in-maintenance-mode/

On June 1, 2023, AWS CDK version 1 will reach the end-of-support.

[evgenyka]

No, the one that you've mentioned is referring to EOS (end-of-support) for CDK v1.

[takahiro-asano-uhuru]

I see, I understand!

[dbartholomae]

How does this interact with the recent change in GitHub certificate that caused the OIDC integration to fail? Will updating to the newest CDK fix existing certificate issues by updating the existing certificate, and prevent future problems with GitHub?