[efs-controller] Newly created Filesystem CR remains in ACK.ResourceSynced = Unknown state

[cellux]

Describe the bug

After creating a Filesystem resource in the Kubernetes cluster, the EFS filesystem is properly created in AWS but the Filesystem resource never becomes ready on the Kubernetes side.

Steps to reproduce

Submit the following Filesystem CR via kubectl apply:

apiVersion: efs.services.k8s.aws/v1alpha1
kind: FileSystem
metadata:
  name: cratus-root
  namespace: cratus
spec:
  availabilityZoneName: eu-central-1a
  backupPolicy:
    status: ENABLED
  encrypted: true
  fileSystemProtection:
    replicationOverwriteProtection: ENABLED
  lifecyclePolicies:
  - transitionToArchive: ""
    transitionToIA: ""
    transitionToPrimaryStorageClass: AFTER_1_ACCESS
  performanceMode: generalPurpose
  policy: |
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::REDACTED:role/cratus-nonprod-aws-eu-central-1-aws-efs-csi-driver"
          },
          "Action": [
            "elasticfilesystem:ClientMount",
            "elasticfilesystem:ClientWrite",
            "elasticfilesystem:ClientRootAccess"
          ]
        }
      ]
    }
  tags:
  - key: Name
    value: cratus-nonprod-aws-eu-central-1
  throughputMode: elastic

Expected outcome

Within one minute after deploying the Filesystem resource, the status of its ACK.ResourceSynced condition switches to "True".

Here is what we actually get 3+ hours after creation:

apiVersion: efs.services.k8s.aws/v1alpha1
kind: FileSystem
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"efs.services.k8s.aws/v1alpha1","kind":"FileSystem","metadata":{"annotations":{},"name":"cratus-root","namespace":"cratus"},"spec":{"availabilityZoneName":"eu-central-1a","backupPolicy":{"status":"ENABLED"},"encrypted":true,"fileSystemProtection":{"replicationOverwriteProtection":"ENABLED"},"lifecyclePolicies":[{"transitionToArchive":"","transitionToIA":"","transitionToPrimaryStorageClass":"AFTER_1_ACCESS"}],"performanceMode":"generalPurpose","policy":"{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::REDACTED:role/cratus-nonprod-aws-eu-central-1-aws-efs-csi-driver\"\n      },\n      \"Action\": [\n        \"elasticfilesystem:ClientMount\",\n        \"elasticfilesystem:ClientWrite\",\n        \"elasticfilesystem:ClientRootAccess\"\n      ]\n    }\n  ]\n}\n","tags":[{"key":"Name","value":"cratus-nonprod-aws-eu-central-1"}],"throughputMode":"elastic"}}
  creationTimestamp: "2025-05-12T11:49:26Z"
  finalizers:
  - finalizers.efs.services.k8s.aws/FileSystem
  generation: 1
  name: cratus-root
  namespace: cratus
  resourceVersion: "77157240"
  uid: c20a0d5a-83df-40b3-8cbc-b8baa6b37a05
spec:
  availabilityZoneName: eu-central-1a
  backupPolicy:
    status: ENABLED
  encrypted: true
  fileSystemProtection:
    replicationOverwriteProtection: ENABLED
  lifecyclePolicies:
  - transitionToArchive: ""
    transitionToIA: ""
    transitionToPrimaryStorageClass: AFTER_1_ACCESS
  performanceMode: generalPurpose
  policy: |
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::REDACTED:role/cratus-nonprod-aws-eu-central-1-aws-efs-csi-driver"
          },
          "Action": [
            "elasticfilesystem:ClientMount",
            "elasticfilesystem:ClientWrite",
            "elasticfilesystem:ClientRootAccess"
          ]
        }
      ]
    }
  tags:
  - key: Name
    value: cratus-nonprod-aws-eu-central-1
  throughputMode: elastic
status:
  ackResourceMetadata:
    arn: arn:aws:elasticfilesystem:eu-central-1:REDACTED:file-system/fs-0f784e2f3f28e3ab9
    ownerAccountID: "REDACTED"
    region: eu-central-1
  availabilityZoneID: euc1-az2
  conditions:
  - lastTransitionTime: "2025-05-12T11:49:27Z"
    message: Unable to determine if desired resource state matches latest observed
      state
    reason: filesystem in 'creating' state, requeuing until filesystem is 'available'
    status: Unknown
    type: ACK.ResourceSynced
  creationTime: "2025-05-12T11:49:26Z"
  fileSystemID: fs-0f784e2f3f28e3ab9
  lifeCycleState: creating
  name: cratus-nonprod-aws-eu-central-1
  numberOfMountTargets: 0
  ownerID: "REDACTED"
  sizeInBytes:
    value: 0
    valueInArchive: 0
    valueInIA: 0
    valueInStandard: 0

ACK EFS controller logs:

{"level":"info","ts":"2025-05-12T11:29:47.432Z","logger":"setup","msg":"initializing service controller","aws.service":"efs"}
{"level":"info","ts":"2025-05-12T11:29:47.533Z","msg":"Waited for the caches to sync","synced":true}
{"level":"info","ts":"2025-05-12T11:29:47.943Z","logger":"setup","msg":"starting manager","aws.service":"efs"}
{"level":"info","ts":"2025-05-12T11:29:47.943Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2025-05-12T11:29:47.943Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"0.0.0.0:8080","secure":false}
{"level":"info","ts":"2025-05-12T11:29:47.944Z","msg":"Starting EventSource","controller":"adoptedresource","controllerGroup":"services.k8s.aws","controllerKind":"AdoptedResource","source":"kind source: *v1alpha1.AdoptedResource"}
{"level":"info","ts":"2025-05-12T11:29:47.944Z","msg":"Starting EventSource","controller":"mounttarget","controllerGroup":"efs.services.k8s.aws","controllerKind":"MountTarget","source":"kind source: *v1alpha1.MountTarget"}
{"level":"info","ts":"2025-05-12T11:29:47.944Z","msg":"Starting EventSource","controller":"fieldexport","controllerGroup":"services.k8s.aws","controllerKind":"FieldExport","source":"kind source: *v1alpha1.FieldExport"}
{"level":"info","ts":"2025-05-12T11:29:47.944Z","msg":"Starting EventSource","controller":"accesspoint","controllerGroup":"efs.services.k8s.aws","controllerKind":"AccessPoint","source":"kind source: *v1alpha1.AccessPoint"}
{"level":"info","ts":"2025-05-12T11:29:47.944Z","msg":"Starting EventSource","controller":"field-export.efs.services.k8s.aws/v1alpha1, Kind=MountTarget","controllerGroup":"efs.services.k8s.aws","controllerKind":"MountTarget","source":"kind source: *v1alpha1.MountTarget"}
{"level":"info","ts":"2025-05-12T11:29:47.945Z","msg":"Starting EventSource","controller":"filesystem","controllerGroup":"efs.services.k8s.aws","controllerKind":"FileSystem","source":"kind source: *v1alpha1.FileSystem"}
{"level":"info","ts":"2025-05-12T11:29:47.945Z","msg":"Starting EventSource","controller":"field-export.efs.services.k8s.aws/v1alpha1, Kind=AccessPoint","controllerGroup":"efs.services.k8s.aws","controllerKind":"AccessPoint","source":"kind source: *v1alpha1.AccessPoint"}
{"level":"info","ts":"2025-05-12T11:29:47.945Z","msg":"Starting EventSource","controller":"field-export.efs.services.k8s.aws/v1alpha1, Kind=FileSystem","controllerGroup":"efs.services.k8s.aws","controllerKind":"FileSystem","source":"kind source: *v1alpha1.FileSystem"}
{"level":"info","ts":"2025-05-12T11:29:48.038Z","msg":"starting server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2025-05-12T11:29:48.338Z","msg":"Starting Controller","controller":"accesspoint","controllerGroup":"efs.services.k8s.aws","controllerKind":"AccessPoint"}
{"level":"info","ts":"2025-05-12T11:29:48.338Z","msg":"Starting workers","controller":"accesspoint","controllerGroup":"efs.services.k8s.aws","controllerKind":"AccessPoint","worker count":1}
{"level":"info","ts":"2025-05-12T11:29:48.339Z","msg":"Starting Controller","controller":"mounttarget","controllerGroup":"efs.services.k8s.aws","controllerKind":"MountTarget"}
{"level":"info","ts":"2025-05-12T11:29:48.339Z","msg":"Starting Controller","controller":"field-export.efs.services.k8s.aws/v1alpha1, Kind=MountTarget","controllerGroup":"efs.services.k8s.aws","controllerKind":"MountTarget"}
{"level":"info","ts":"2025-05-12T11:29:48.339Z","msg":"Starting workers","controller":"field-export.efs.services.k8s.aws/v1alpha1, Kind=MountTarget","controllerGroup":"efs.services.k8s.aws","controllerKind":"MountTarget","worker count":1}
{"level":"info","ts":"2025-05-12T11:29:48.339Z","msg":"Starting Controller","controller":"adoptedresource","controllerGroup":"services.k8s.aws","controllerKind":"AdoptedResource"}
{"level":"info","ts":"2025-05-12T11:29:48.339Z","msg":"Starting workers","controller":"adoptedresource","controllerGroup":"services.k8s.aws","controllerKind":"AdoptedResource","worker count":1}
{"level":"info","ts":"2025-05-12T11:29:48.339Z","msg":"Starting Controller","controller":"fieldexport","controllerGroup":"services.k8s.aws","controllerKind":"FieldExport"}
{"level":"info","ts":"2025-05-12T11:29:48.339Z","msg":"Starting workers","controller":"fieldexport","controllerGroup":"services.k8s.aws","controllerKind":"FieldExport","worker count":1}
{"level":"info","ts":"2025-05-12T11:29:48.339Z","msg":"Starting workers","controller":"mounttarget","controllerGroup":"efs.services.k8s.aws","controllerKind":"MountTarget","worker count":1}
{"level":"info","ts":"2025-05-12T11:29:48.339Z","msg":"Starting Controller","controller":"filesystem","controllerGroup":"efs.services.k8s.aws","controllerKind":"FileSystem"}
{"level":"info","ts":"2025-05-12T11:29:48.340Z","msg":"Starting workers","controller":"filesystem","controllerGroup":"efs.services.k8s.aws","controllerKind":"FileSystem","worker count":1}
{"level":"info","ts":"2025-05-12T11:29:48.340Z","msg":"Starting Controller","controller":"field-export.efs.services.k8s.aws/v1alpha1, Kind=FileSystem","controllerGroup":"efs.services.k8s.aws","controllerKind":"FileSystem"}
{"level":"info","ts":"2025-05-12T11:29:48.340Z","msg":"Starting workers","controller":"field-export.efs.services.k8s.aws/v1alpha1, Kind=FileSystem","controllerGroup":"efs.services.k8s.aws","controllerKind":"FileSystem","worker count":1}
{"level":"info","ts":"2025-05-12T11:29:48.340Z","msg":"Starting Controller","controller":"field-export.efs.services.k8s.aws/v1alpha1, Kind=AccessPoint","controllerGroup":"efs.services.k8s.aws","controllerKind":"AccessPoint"}
{"level":"info","ts":"2025-05-12T11:29:48.340Z","msg":"Starting workers","controller":"field-export.efs.services.k8s.aws/v1alpha1, Kind=AccessPoint","controllerGroup":"efs.services.k8s.aws","controllerKind":"AccessPoint","worker count":1}
{"level":"info","ts":"2025-05-12T11:49:37.412Z","logger":"ackrt","msg":"desired resource state has changed","kind":"FileSystem","namespace":"cratus","name":"cratus-root","account":"REDACTED","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"diff":[{"Path":{"Parts":["Spec","LifecyclePolicies"]},"A":[{"transitionToArchive":"","transitionToIA":"","transitionToPrimaryStorageClass":"AFTER_1_ACCESS"}],"B":null},{"Path":{"Parts":["Spec","Policy"]},"A":"{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::REDACTED:role/cratus-nonprod-aws-eu-central-1-aws-efs-csi-driver\"\n      },\n      \"Action\": [\n        \"elasticfilesystem:ClientMount\",\n        \"elasticfilesystem:ClientWrite\",\n        \"elasticfilesystem:ClientRootAccess\"\n      ]\n    }\n  ]\n}\n","B":null}]}
{"level":"info","ts":"2025-05-12T11:49:37.643Z","logger":"ackrt","msg":"updated resource","kind":"FileSystem","namespace":"cratus","name":"cratus-root","account":"REDACTED","role":"","region":"eu-central-1","is_adopted":false,"generation":1}

Environment

• Kubernetes version: v1.31.7-eks-bcf3d70
• Using EKS (yes/no), if so version? Yes, we use EKS v1.31
• AWS service targeted (S3, RDS, etc.): EFS
• EFS controller version: 1.0.10 (deployed via Helm chart)
• EFS controller pod container image: public.ecr.aws/aws-controllers-k8s/efs-controller:1.0.10

[github-actions]

Hello @cellux 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[michaelhtm]

Hello @cellux
This issue should now be resolved in v1.0.13
Thanks :)
/close

[ack-prow]

@michaelhtm: Closing this issue.

In response to this:

Hello @cellux
This issue should now be resolved in v1.0.13
Thanks :)
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[cellux]

thx, I confirm that 1.0.13 resolved this issue