Cloud Run: cannot reconcile service edited through console

[mattmoor]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

terraform -v
Terraform v1.2.3
on darwin_arm64
+ provider registry.terraform.io/chainguard-dev/ko v0.0.4
+ provider registry.terraform.io/hashicorp/google v4.47.0
+ provider registry.terraform.io/hashicorp/google-beta v4.47.0
...

Affected Resource(s)

• google_cloud_run_service

Terraform Configuration Files

This should affect virtually any Cloud Run service deployed through terraform.

Debug Output

N/A

Panic Output

N/A

Expected Behavior

terraform reconciles the service

Actual Behavior

After ~20 minutes it times out and prints an error with a 409 because the named revision already exists.

Steps to Reproduce

1. Deploy a service via terraform,
2. Edit it via the Console's editor (not yaml),
3. Deploy the service again via terraform.

Important Factoids

The Knative resource model used by Cloud Run supports "bring your own revision name" where you can use spec.template.metadata.name to name the revision that the Service will create. This is used by the Cloud Run console when edits are made.

If changes are made to the service without removing or updating this name, then things will fail to deploy.

cc @steren

References

b/272367711

[edwardmedia]

@mattmoor when you execute step 3, do you mean you want to re-deploy the terraform config which is the same as step 1?
Can you share your config and the debug log?

[mattmoor]

@edwardmedia it doesn't matter, it could be asking terraform to reconcile things back to how they were, or deploying something new.

The edit we made for 2. was to add a trivial env var to trigger a rollout, e.g. FOO=bar env var.

You can repro it with the examples in https://github.com/chainguard-dev/terraform-google-prober

[mattmoor]

I'd recommend the basic one as the complex one spins up GCLB, which is pricy.

https://github.com/chainguard-dev/terraform-google-prober/tree/main/examples/basic

[steren]

here are some more details:

• Step1: I deployed a service using Terraform using these steps
• Step2: I used the UI to add an anv var: https://cloud.google.com/run/docs/configuring/environment-variables#console

The YAML of my service is now:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: tf-test
  namespace: '607903476290'
  selfLink: /apis/serving.knative.dev/v1/namespaces/607903476290/services/tf-test
  uid: 5493a021-319a-446e-92eb-99e7bfd39d48
  resourceVersion: AAXxsHOMS4Y
  generation: 2
  creationTimestamp: '2023-01-07T18:09:16.917847Z'
  labels:
    cloud.googleapis.com/location: us-central1
  annotations:
    run.googleapis.com/client-name: cloud-console
    serving.knative.dev/creator: steren.giannini@gmail.com
    serving.knative.dev/lastModifier: steren.giannini@gmail.com
    client.knative.dev/user-image: us-docker.pkg.dev/cloudrun/container/hello
    run.googleapis.com/ingress: all
    run.googleapis.com/ingress-status: all
spec:
  template:
    metadata:
      name: tf-test-00002-juj
      annotations:
        run.googleapis.com/client-name: cloud-console
        autoscaling.knative.dev/maxScale: '100'
    spec:
      containerConcurrency: 80
      timeoutSeconds: 300
      serviceAccountName: 607903476290-compute@developer.gserviceaccount.com
      containers:
      - image: us-docker.pkg.dev/cloudrun/container/hello
        ports:
        - name: http1
          containerPort: 8080
        env:
        - name: FOO
          value: Bar
        resources:
          limits:
            cpu: 1000m
            memory: 512Mi
  traffic:
  - percent: 100
    latestRevision: true

Note the spec.template.metadata.name attribute set.

I run terraform plan note that it *does not call out that spec.template.metadata.name will be reset to null

terraform plan
data.google_iam_policy.noauth: Reading...
data.google_iam_policy.noauth: Read complete after 0s [id=3450855414]
google_cloud_run_service.default: Refreshing state... [id=locations/us-central1/namespaces/steren-playground/services/tf-test]
google_cloud_run_service_iam_policy.noauth: Refreshing state... [id=v1/projects/steren-playground/locations/us-central1/services/tf-test]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_cloud_run_service.default will be updated in-place
  ~ resource "google_cloud_run_service" "default" {
        id                         = "locations/us-central1/namespaces/steren-playground/services/tf-test"
        name                       = "tf-test"
        # (4 unchanged attributes hidden)

      ~ metadata {
          ~ annotations      = {
              - "client.knative.dev/user-image"     = "us-docker.pkg.dev/cloudrun/container/hello" -> null
              ~ "run.googleapis.com/client-name"    = "cloud-console" -> "terraform"
              - "run.googleapis.com/ingress"        = "all" -> null
                # (3 unchanged elements hidden)
            }
            # (6 unchanged attributes hidden)
        }

      ~ template {

          ~ spec {
                # (3 unchanged attributes hidden)

              ~ containers {
                    # (3 unchanged attributes hidden)

                  - env {
                      - name  = "FOO" -> null
                      - value = "Bar" -> null
                    }

                    # (2 unchanged blocks hidden)
                }
            }

            # (1 unchanged block hidden)
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

I run terraform apply and it indeed seems to hang.

2 things:

1. Is spec.template.metadata.name treated specially by Terraform? I would expect it to be reset to null in the plan and apply
2. Why does it hang?

[FabianFrank]

This problem also happens if you provision a cloud run service using terraform google_cloud_run_v2_service and then deploy revisions in CI/CD. Either Terraform will always detect and apply "changes" because the revision has changed or if you add the revision to lifecycle ignore_changes then you get the error Revision named ... with different configuration already exists.

[jamiezieziula]

I had this same issue - I deployed a cloudrun service via Terraform (terraform cloud) and then subsequently deployed new revisions with updated image tags via this GitHub actions workflow.

I was getting the same error:

Error 409: Revision named 'service-name-00048-fts' with different configuration already exists.

I was able to circumvent this error with 2 code changes. The first - adding the random_uuid terraform generator, and providing a unique revision name to the terraform resource.

resource "random_uuid" "cloudrun_revision_id" {
  keepers = {
    first = timestamp()
  }
}

resource "google_cloud_run_v2_service" "service" {
  name         = var.cloudrun_name

  template {
    revision = "${var.cloudrun_name}-${random_uuid.cloudrun_revision_id.result}"
  }
}

Secondly, ignoring the following lifecycle changes:

  lifecycle {
    ignore_changes = [
      annotations,
      client_version,
      client,
      labels,
      template.0.annotations,
      template.0.labels,
    ]
  }

Not ideal, but a fairly simple workaround that allowed me manage the service from two angles. I hope this helps!

[FabianFrank]

@jamiezieziula the problem with that solution though is that every time you've deployed a new revision without terraform then the next terraform run will create a new revision even if there are no changes.