[APIGatewayV1] Integration resources go Terminal when racing Method resources

[arichtman-srt]

Describe the bug

API gateway's AWS resources have a combination of:

• Method
• Integration

Integrations depend on Methods, and when they're not created in the correct order, creation API calls will fail.
The Integration resource might be entirely valid but it becomes a race condition whether it'll appear AWS-side or not, with no controls over resource order application.
The resource then goes into a terminal state and seems to not retry.
It can be fixed by waiting for the Method method resource to be created, then deleting and recreating the Integration resource.

Lack of resource order application is a Kubernetes design - so not looking for that to be tackled.
But if these resources could attempt reconciliation even a handful of times it would be helpful.
Perhaps an annotation per resource that would allow some retries before reaching terminal status.

Steps to reproduce

1. Create API gateway (REST API was our case)
2. Note the root resource ID, and the API ID
3. Apply the following

apiVersion: apigateway.services.k8s.aws/v1alpha1
kind: Resource
metadata:
  name: openid-configuration-idp-nonprod-auth
spec:
  pathPart: openid-configuration
  parentID: $ROOT_RESOURCE_ID
  restAPIID: $API_ID
---
apiVersion: apigateway.services.k8s.aws/v1alpha1
kind: Method
metadata:
  name: openid-configuration-get
spec:
  authorizationType: NONE
  httpMethod: GET
  restAPIID: $API_ID
  resourceRef:
    from:
      name: openid-configuration-idp-nonprod-auth
  requestParameters:
    method.request.path.tenant: true
---
apiVersion: apigateway.services.k8s.aws/v1alpha1
kind: Integration
metadata:
  name: openid-configuration-get
spec:
  restAPIID: $API_ID
  resourceRef:
    from:
      name: openid-configuration-idp-nonprod-auth
  httpMethod: GET
  integrationHTTPMethod: GET
  type: MOCK
  requestParameters:
    integration.request.path.tenant: 'method.request.path.tenant'
  requestTemplates:
    "application/json": | 
      {"statusCode": 200}

Expected outcome

Integration is created successully.

Environment

• Kubernetes version v1.33.1
• Using EKS (yes/no), if so version? eks-595af52
• AWS service targeted (S3, RDS, etc.): API Gateway v1

[github-actions]

Hello @arichtman-srt 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.