[Bug]: Update Lease failed on nginx-ingress update to 3.7.0

[thomas-schn]

Version

3.7.0

What Kubernetes platforms are you running on?

EKS Amazon

Steps to reproduce

I updated my nginx-ingress-controller in eks from Version 3.6.2 to 3.7.0. The eks-cluster is provisioned with terraform helm-provider.
During Update process I got the following error-message:

Error: Unable to continue with update: Lease "internal-ingress-controller-nginx-ingress-leader-election" in namespace
 "ingress-internal" exists and cannot be imported into the current release: invalid ownership metadata; label validation
 error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key 
 "meta.helm.sh/release-name": must be set to "internal-ingress-controller"; annotation validation error: missing key 
 "meta.helm.sh/release-namespace": must be set to "ingress-internal"
│ 
│   with module.cluster.module.nginx_ingress_controller_internal[0].helm_release.nginx_ingress_controller_helm,
│   on ../../modules/eks/nginx-ingress-controller/main.tf line 104, in resource "helm_release" "nginx_ingress_controller_helm":
│  104: resource "helm_release" "nginx_ingress_controller_helm" {
│ 
╵ 

The describe on my lease resource in my cluster looks like this:

~ kubectl -n ingress-internal describe lease/internal-ingress-controller-nginx-ingress-leader-election
Name:         internal-ingress-controller-nginx-ingress-leader-election
Namespace:    ingress-internal
Labels:       <none>
Annotations:  <none>
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2022-11-08T11:03:42Z
  Resource Version:    185034718
  UID:                 10111f40-61f1-41d2-894e-c47b4fb92841
Spec:
  Acquire Time:            2023-12-04T10:06:03.514514Z
  Holder Identity:         internal-ingress-controller-nginx-ingress-nginx-ingress-cont2lnm5
  Lease Duration Seconds:  30
  Lease Transitions:       25
  Renew Time:              2024-01-30T13:46:25.679644Z
Events:                    <none>

Did I miss some new configuration for my cluster?
For now, I fixed it, by manually adding the missing annotations and label to my existing lease-resource befor updating to new nginx-ingress-version.

[github-actions]

Hi @thomas-schn thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

[vepatel]

hi @thomas-schn , from release 3.7.0, leases are managed by helm and will be prefixed by helm release name (unless overridden using flag) so I wonder if this is because of conflict with older lease with the same name.
Can you provide us with the output of kubectl get leases to check if there's one leftover form last release?

[thomas-schn]

hi @vepatel

There are two leases. I had to edit internal-ingress-controller-nginx-ingress-leader-election manualy.

גּ kubectl -n ingress-internal get leases
NAME                                                        HOLDER                                                            AGE
internal-ingress-controller-nginx-ingress-leader-election   internal-ingress-controller-nginx-ingress-nginx-ingress-cokrqqq   539d
nginx-ingress-leader                                        internal-ingress-controller-nginx-ingress-nginx-ingress-coz7pbf   259d

Here is the output of kubectl -n ingress-internal describe lease/nginx-ingress-leader

Name:         nginx-ingress-leader
Namespace:    ingress-internal
Labels:       <none>
Annotations:  <none>
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2024-01-30T13:47:14Z
  Resource Version:    206270450
  UID:                 b2d55696-7bb6-436c-b2e3-95c360473a96
Spec:
  Acquire Time:            2024-09-10T09:28:32.347165Z
  Holder Identity:         internal-ingress-controller-nginx-ingress-nginx-ingress-coz7pbf
  Lease Duration Seconds:  30
  Lease Transitions:       14
  Renew Time:              2024-10-15T12:44:35.272016Z
Events:                    <none>

and this is after I edited the lease manualy and the update nginx-ingress to 3.7.0

~ kubectl -n ingress-internal describe lease/internal-ingress-controller-nginx-ingress-leader-election
Name:         internal-ingress-controller-nginx-ingress-leader-election
Namespace:    ingress-internal
Labels:       app.kubernetes.io/instance=internal-ingress-controller
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=nginx-ingress
              app.kubernetes.io/version=3.7.0
              helm.sh/chart=nginx-ingress-1.4.0
Annotations:  meta.helm.sh/release-name: internal-ingress-controller
              meta.helm.sh/release-namespace: ingress-internal
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2023-04-25T11:22:43Z
  Resource Version:    206357669
  UID:                 46bbc16e-ea8a-4cc5-b6f1-e6daf96bad7b
Spec:
  Acquire Time:            2024-10-15T12:45:20.790142Z
  Holder Identity:         internal-ingress-controller-nginx-ingress-nginx-ingress-cokrqqq
  Lease Duration Seconds:  30
  Lease Transitions:       18
  Renew Time:              2024-10-15T17:55:25.890838Z
Events:                    <none>

[vepatel]

okay looks like issue has nothing to do with the orphaned lease from last installation nginx-ingress-leader which can now be deleted. Can you please check if this issue only occurs when deploying helm via terraform but not with simple helm install commands?

[thomas-schn]

Sorry that's not possible. I already add all the label and annotation to the lease objects and then updated all my nginx-controller

[vepatel]

sorry meant to ask if you run into same issue if installing/upgrading just using helm on test cluster, I'm unable to repro it unfortunately using helm

[pdabelf5]

I have attempted to reproduce this issue using the terraform helm provider. However, I did not see the same results as you have. Here is my terraform configuration:

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

resource "helm_release" "nic" {
  name             = "tf-helm"
  namespace        = "nginx-ingress"
  version          = "1.3.2"
  repository       = "oci://ghcr.io/nginxinc/charts"
  chart            = "nginx-ingress"
  upgrade_install  = true
  wait             = true
  create_namespace = true
  values           = [
    "${file("values.yaml")}"
  ]
}

and helm values

controller:
  globalConfiguration:
    create: true
  enableSnippets: true
  enableCustomResources: true
  replicaCount: 3

With this configuration when updating to the 1.4.0 chart, the lease gets created with the correct Helm annotations. Are these settings much different from what you have tried?

[thomas-schn]

I didn't set upgrade_install = true, cause the documents from the helm provider is warning, to not use it in production. Maby that's the difference. Also I used repository = "https://helm.nginx.com/stable".

[pdabelf5]

I have updated my test setup to include the same settings you have described but am still unable to reproduce the issue. What seems unusual is that 3.7.0 changed the naming structure of the lease but you already had a lease of the same name. Was there a failed helm upgrade prior to this error?

[thomas-schn]

@pdabelf5 Yes, we used to have problems that were due to a misconfiguration. The ticket can probably be closed if you can't reproduce the error.

[pdabelf5]

No problem @thomas-schn thanks for the update 👍