read_path and read_search read from state instead of config on existing resources

[toadjaune]

Hi !

Problem statement

When modifying read_path or read_search, existing resources keep using their previous configuration, leading to surprising behaviors, especially when iterating on the configuration of said resource.

To illustrate the observed behavior, consider the following configuration :

terraform {
  backend "local" {
    path = "terraform.tfstate"
  }
  required_providers {
    restapi = {
      source  = "Mastercard/restapi"
      version = "2.0.1"
    }
  }
}

variable "API_TOKEN" {} # set with environment variable TF_VAR_API_TOKEN


provider "restapi" {
  uri                  = "https://<some_domain>"
  write_returns_object = true
  debug                = true

  headers = {
    accept       = "application/json"
    content-type = "application/json"
    Api-Token    = var.API_TOKEN
  }
}

resource "restapi_object" "webhook" {
  id_attribute = "webhook/uuid"
  path         = "/api/v1/webhooks"
  data = <something>
  debug                     = true
  ignore_all_server_changes = true
## There's no "read" endpoint for this specific resource, we have to work around this using the listing endpoint with the configuration below.
#   read_path                 = "/api/v1/webhooks/"
#   read_search = {
#     results_key  = "webhooks/"
#     id_attribute = "uuid"
#   }
}

Then, do the following sequence of operations :

• terraform apply
  • validate the plan, and create the webhook
• terraform plan
  • the refresh step attempts to update the state of the resource, fails to do so because the endpoint doesn't exist, terraform assumes the resource has been deleted on the server
  • terraform generates a plan re-creating the resource
• Uncomment the read_path and read_search options in the configuration
• terraform plan
  • terraform still generates a plan to create the resource
• (Delete the resource on the provider side)
• terraform apply
  • validate the plan again and create the resource again
• terraform plan
  • This time, the plan is empty

I'm unsure whether this is by design or an unintended behavior, but it is certainly quite unintuitive.

(Provider version v2.0.1)

Possible fixes

1. Make the refresh step use the declared configuration before the one stored in state (of course, the state-stored one is still needed for resource deletion)
2. Don't change anything, but explicitly document this behavior

I strongly lean towards 1, but I'm unsure whether there might be usecases where the current behavior is preferable. I can't think of any, anyway.

[DRuggeri]

Hey there again, @toadjaune. I agree with your expectation. Unfortunately, it is a limitation of the framework. The Read operation provides a ReadRequest which only has the current State available.

While migrating to the new plugin framework, I did experiment with the new ModifyPlan capability, but performing reads there ended up breaking other parts of the diff calculation. I will update the README to make this clear - thanks for the idea!