(eks.AlbController): 'UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress' when AlbController chart install is attempted

[Aidan-Pasquale99]

General Issue

Observing 'UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress' error when installing AlbController chart via construct

The Question

I am trying to install the ALB Controller Helm chart via my CDK code using the AlbController construct as follows:

Cluster myCluster = Cluster.Builder.create(this, "myEKS")
    .version(KubernetesVersion.V1_21)
    .vpc(vpc)
    .clusterName(name)
    .vpcSubnets(Arrays.asList(privateVpcSubnets))
    .defaultCapacity(2)
    .defaultCapacityInstance(InstanceType.of(InstanceClass.COMPUTE5, InstanceSize.XLARGE))
    .build();

 AlbController.Builder.create(this, "myAlbController")
    .cluster(myCluster)
    .version(AlbControllerVersion.V2_3_1)
    .repository("602401143452.dkr.ecr.us-west-1.amazonaws.com")
    .build();

I run the above CDK code and get the following Helm error in the CloudFormation events output, as well as in the Lambda function logs:

UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

I've looked through this and this but the solutions do not apply here, there appears to be no known state of the AlbController release and therefore I am unable to perform any rollback commands.

CDK CLI Version

2.17.0 (build f9cd009)

Framework Version

No response

Node.js Version

No response

OS

No response

Language

Java

Language Version

No response

Other information

No response

[asanka-x]

@Aidan-Pasquale99 Thank you for raising the issue. I am also facing the same issue.

[whitney]

Seeing the same thing in the Python CDK using e.g.

cluster = eks.Cluster(
            self,
            "some-id",
            cluster_name="some-cluster-name",
            vpc=vpc,
            vpc_subnets=[ec2.SubnetSelection(...)],
            default_capacity=4,
            version=eks.KubernetesVersion.V1_21,
            alb_controller=eks.AlbControllerOptions(version=eks.AlbControllerVersion.V2_3_1),
        )

[dimaria-tai]

Is there any workaround?

I'm struggling with this

[whitney]

@peterwoodworth noticed this was converted from an issue to a discussion. Is there additional information/exploration needed for it to be treated as an "issue"?

[andrewclevelandibm]

I am also struggling with this, has anyone found a solution yet?

[nilroy]

I am also facing the same issue and struggling with this. I am using kubernetes version 1.22

[IllarionovDimitri]

same problem. have tried different versions (V2_3_1 and V2_4_1). no chance. doesn't work.
aws-cdk-lib==2.37.1

Received response status [FAILED] from custom resource.
Message returned: Error: b'Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress\n'

[smislam]

Hi team, I am experiencing the same issue. Do we have any updates on this?

My Dependencies

• "aws-cdk-lib": "2.37.1",
• KubernetesVersion.V1_21
• AlbControllerVersion.V2_4_1

Error
Received response status [FAILED] from custom resource. Message returned: Error: b'Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress\n'

[toduythienluong]

hello,
I also got stuck here with this

"Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress"

• CDK 2.40.0 (build 56ba2ab)
• eks.KubernetesVersion.V1_21
• eks.AlbControllerVersion.V2_4_1

Any workaround please?

[zorrofox]

• CDK: 2.43.1
• eks.KubernetesVersion.V1_21
• eks.AlbControllerVersion.V2_4_1

Run same issue! Why this is not a bug?

[estorey11]

I'm having this same issue. Not sure why @peterwoodworth converted this from an issue to a discussion.

[pahud]

related to #22005

I will try reproduce this in my environment and see what I can find out.

[pahud]

I was able to deploy successfully with no error. #22005 (comment)

Is this issue still valid?

[k-ovidiu]

I think is still a valid issue, I just tried it with almoust all default parameters and still fails:
self.eks_cluster = eks.Cluster(self, "TruedatEKS", version=eks.KubernetesVersion.V1_21, default_capacity=3, default_capacity_instance=ec2.InstanceType.of(ec2.InstanceClass.T3A, ec2.InstanceSize.LARGE), alb_controller=eks.AlbControllerOptions(version=eks.AlbControllerVersion.V2_4_1) )
This is with python version 3.9, cdk 2.46.0 (build 5a0595e)
And same error Received response status [FAILED] from custom resource. Message returned: Error: b'Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress\n'

[hanoveel]

I had the same problem, solution to fix it with me:

1. Install the eks cluster without ALB. (reason in step 3).
2. Reinstall by enabling ALB. If failed move to step 3.
3. Because we had a running cluster from step 1, I checked the helm packages and found alb-load-balancer-controller as failed. when I uninstalled this package and run step 2, it succeeded.

[estorey11]

Does this result in the load balancer controller being successfully installed on the cluster? Or does this just allow the cluster to deploy without it?

[hanoveel]

Deployed with it

ALB is enabled and working

[hanoveel]

I only installed first without ALB to gain access to helm list and found the helm chart for alb-load-balancer-controller exists in failed status.

When I uninstalled the chart manually, and enabled alb in cdk and deployed again it worked

[pahud]

Anyone still having this issue now? Can someone share the full CDK code to reproduce it?

[m17kea]

The work around for me was to either specify some default capacity or call AddNodegroupCapacity on the cluster followed by new AlbController with dependency on the nodes i.e. in c#:

            var armNodes = eks.AddNodegroupCapacity("arm-node-group", new NodegroupOptions
            {
                InstanceTypes = new[]
                {
                    InstanceType.Of(InstanceClass.BURSTABLE4_GRAVITON, InstanceSize.MEDIUM),
                },
                MinSize = 2,
                MaxSize = 4,
                DiskSize = 20,
                AmiType = NodegroupAmiType.BOTTLEROCKET_ARM_64,
                NodegroupName = $"{props.Stage}-Eks-Arm"
            });

            var albController = new AlbController(this, "AlbController", new AlbControllerProps
            {
                Cluster = eks,
                Version = AlbControllerVersion.V2_4_1
            });
            
            albController.Node.AddDependency(armNodes);

[karatakos]

@m17kea I tried this approach unsuccessfully, i.e. 0 default capacity but add node group capacity and defer alb controller creation. Did you really manage to get this deployment to work without 0 default capacity?

    const cluster = new eks.Cluster(this, "cluster", {
      version: eks.KubernetesVersion.V1_27,
      clusterName: "cluster",  
      defaultCapacity: 0,
      vpc: this.vpc,
      vpcSubnets: [{ subnets: this.vpc.privateSubnets }],
      outputClusterName: true,
      outputMastersRoleArn: true,
      placeClusterHandlerInVpc: true
    });

    const eksNodeRole = new iam.Role(this, 'cluster-nodegroup-role', {
      assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
      managedPolicies: [
        iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSWorkerNodePolicy'),
        iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEC2ContainerRegistryReadOnly')
      ]
    });

    const nodeGroup = cluster.addNodegroupCapacity("cluster-nodegroup", {
      instanceTypes: [
        new ec2.InstanceType('t2.nano')],
      amiType: eks.NodegroupAmiType.AL2_X86_64,
      capacityType: eks.CapacityType.ON_DEMAND,
      desiredSize: 1,
      minSize: 1,
      maxSize: 2,
      nodeRole: eksNodeRole,
      subnets: { subnets: this.vpc.privateSubnets }
    });

    const albController = new eks.AlbController(this, "cluster-albcontroller", {
      cluster: cluster,
      version: eks.AlbControllerVersion.V2_5_1
    });

    albController.node.addDependency(nodeGroup);

[nerdlem]

The below Python 3.9 code triggers the behavior @pahud...

        self.eks_cluster[app] = eks.Cluster(self, f"{app}-eks",
            alb_controller=eks.AlbControllerOptions(
                version=eks.AlbControllerVersion.V2_4_1,
            ),
            cluster_logging=[eks.ClusterLoggingTypes.API,
                eks.ClusterLoggingTypes.AUDIT,
                eks.ClusterLoggingTypes.AUTHENTICATOR,
                eks.ClusterLoggingTypes.SCHEDULER],
            cluster_name=f"{app}",
            default_capacity = 0,
            output_cluster_name=True,
            output_masters_role_arn=True,
            place_cluster_handler_in_vpc=True,
            tags=addtl_tags,
            version=eks.KubernetesVersion.V1_21,
            vpc=self.vpc,
        )

Also tried:

• Providing explicitly created security groups
• Creating the AlbController after the EKS Cluster

For all cases, same results afterwards:

3:08:42 PM | CREATE_FAILED        | Custom::AWSCDK-EKS-HelmChart                | MyAlbControllerA08B1419
Received response status [FAILED] from custom resource. Message returned: Error: b'Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress\n'

Logs: /aws/lambda/mything-test-awscdkawseksKubec-Handler886CB40B-ib4KrQ81eAAF

at invokeUserFunction (/var/task/framework.js:2:6)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async onEvent (/var/task/framework.js:1:365)
at async Runtime.handler (/var/task/cfn-response.js:1:1543)

This is one error out of the many tests, taken from the CloudWatch log stream:

[ERROR] Exception: b'Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress\n'
Traceback (most recent call last):
  File "/var/task/index.py", line 17, in handler
    return helm_handler(event, context)
  File "/var/task/helm/__init__.py", line 88, in helm_handler
    helm('upgrade', release, chart, repository, values_file, namespace, version, wait, timeout, create_namespace)
  File "/var/task/helm/__init__.py", line 186, in helm
    raise Exception(output)

I suppose this is happening within the Lambda function—not sure on the calling order—and might be as simple as a timeout caused by some missing rule somewhere. However the documentation had me thinking that the construct would build proper security groups automatically.

The VPC was created at us-east-2. 6 subnets were defined—3 private and 3 public. The VPCs have an EC2 instance deployed. There are NAT Gateways and Aurora Serverless RDS instances created and attached to the VPC. I am fairly sure the VPC side is properly configured, except for the tags required for handler use. I was under the impression that this construct would apply the required tags to the VPCs.

If I attempt to create the EKS without the AlbController, construction succeeds.

I can provide more code if this would help.

Best regards

-lem

[pahud]

@nerdlem From what I can see in my recent test.

The following works if you specify some default capacity.

eks.Cluster(self, 'Cluster',
    vpc=vpc,
    alb_controller=eks.AlbControllerOptions(
        version=eks.AlbControllerVersion.V2_4_1,
    ),
    cluster_logging=[eks.ClusterLoggingTypes.API,
                     eks.ClusterLoggingTypes.AUDIT,
                     eks.ClusterLoggingTypes.AUTHENTICATOR,
                     eks.ClusterLoggingTypes.SCHEDULER],
    version=eks.KubernetesVersion.V1_24,
    default_capacity=2,
    output_cluster_name=True,
    output_masters_role_arn=True,
    place_cluster_handler_in_vpc=True,
    kubectl_layer=kubectl.KubectlLayer(self, 'KubectlLayer')
    )

However, if you set default_capacity=0 it will not complete the deployment.

Please try again with some default capacity or addNodeGroupCapacity() in the cluster.

[erikdebruin]

Today I tried to create a FargateCluster (v1.24) with ALB Controller (v2.4.1) - which forced me to add a kubectlLayer - and that failed with the above mentioned 'operation in progress'.

Next I tried to create a FargateCluster v1.21 with the same version ALB Controller, a combination which does not require an explicit kubectlLayer to be defined. All other settings and properties were the same. This combination gave me a successful setup.

I hope this information will help someone more knowledgable than me to debug the issue?

[pahud]

check out this working sample for the workaround:

#22005 (comment)

[karatakos]

@pahud thank you for the investigation. However, what is your proposed work around for scenarios that require fine grained control over the node group? As in, require a custom nodegroup as opposed to default capacity, e.g. if we need to specify a node group role, or change sizing, etc. From testing, it's not possible to zero out default capacity, and yet also not possible to simply add capacity after the fact (see comment). Thanks.