Allow definition of additional attributes in generator configuration

[AgustinBettati]

Context
I am facing a scenario in which I have a terraform schema which in majority conforms with the schema defined in the API Spec, but terraform defines additional attributes which are populated using an unrelated endpoint.

Enhancement request

For these cases, it would be useful to give users flexibility in the generator configuration to define any additional attributes that are needed to define the complete schema of the resource.

As a proposal, the attributes object could define a new field for defining new properties and require the user to provide the attribute definition using the provider code specification. Example:

schema:
  attributes:
    overrides:
      labels:
        description: other description
    additional: [
      {
        "name": "id",
        "string": {
          "computed_optional_required": "computed",
          "plan_modifiers": [
            {
              "custom": {
                "imports": [
                  {
                    "path": "github.com/hashicorp/terraform-plugin-framework/resource/schema/stringplanmodifier"
                  }
                ],
                "schema_definition": "stringplanmodifier.UseStateForUnknown()"
              }
            }
          ]
        }
      }
    ]

Edit from @austinvalle

Bringing over some information from #84 that would also fit in this request:

• Define plan modifiers for specific attributes. This is useful for common plan modifiers such as RequiresReplace and UseStateForUnkown.
• Define validators for specific attributes. Apart from validators that are generated from API Spec properties, there are other validations such as ConflictsWith which cannot be represented easily in an OpenAPI Spec.
• Allow overriding the resulting name of an attribute, giving flexibility in the case that terraform schema has a different name from the definition in the API.
• Marking an attribute as sensitive. This is information that likely cannot be inferred from the API Spec but highly relevant for terraform schema.
• Allow override of the computed_optional_required property for an attribute. While the tool has a certain logic to define this value for attributes, it would be helpful to give flexibility to the user to cover specific edge cases.

Also some from #85:

Currently schemas of "type" : "array" with nested objects are being defined as attribute lists. While this is correct, terraform has a broader set of possibilities:

• can be of type list_nested or set_nested.
• defined as an attribute or as a block.

[bflad]

Hi @AgustinBettati 👋 Thank you for raising this, its an interesting use case that we would love to hear more about.

terraform defines additional attributes which are populated using an unrelated endpoint.

Can you describe this scenario a little more? For example, is this endpoint also defined via OpenAPI? Is there a particular reason why the API models this information separately or, flipping the question around, is there a particular reason why the Terraform resource models this information from multiple API endpoints rather than separately?

I think we should be conscious in this project to make sure we're not trying to solve all possible generic specification or code generation use cases here, where a solution may potentially make more sense elsewhere (that might not exist yet). To describe what I mean a little more, if this is use case is not modeled well by OpenAPI alone, there feels like an opportunity to introduce tooling, etc. that is specifically meant for adding/overriding content in an existing specification. Thanks!

[AgustinBettati]

is there a particular reason why the Terraform resource models this information from multiple API endpoints rather than separately?

We had cases were certain functionality highly related to an existing resource was exposed in different endpoint and we decided to incorporate the feature as a new attribute over defining a new resource all together. This was mainly in terms of improving usability for the end user.

I would also keep in mind the case of the id attribute which currently has to be defined in the terraform schema due to testing framework limitations (hashicorp/terraform-plugin-testing#84 (comment)) and is not always present in API responses.

I think we should be conscious in this project to make sure we're not trying to solve all possible generic specification or code generation use cases here

It would seem logical that the configuration file focuses on providing information that complements the API Spec with terraform specific information or overrides (such as the description). Given that some properties may not be present in the API Spec, defining these additions in the generation configuration seems like an intuitive place.

[austinvalle]

Thanks for submitting this @AgustinBettati, this is an interesting proposal that, as Brian mentioned, not only applies to tfplugingen-openapi, but also to all other types of "provider spec generators" that may exist in the future (protobuf, smithy, etc.)

The general problem sounds like we need to have a good way to add/modify a provider code spec when a generator either:

• Can't add the terraform concepts because there is no viable translation (plan modifiers, computability (in some cases), etc.)
• Has limitations in what is defined, which need to be slightly adjusted (like our design decision to always favor attributes over blocks, which may not be desirable if a provider targets protocol v5)

Since this problem will likely exist in every provider spec generator in some form, we're going to do some discovery on if we can solve this problem more generally, rather then updating the generator config in the OpenAPI generator right now. I'm going to keep this issue open for convenience of discussing/tracking this functionality.

If you have more use-cases that fit into this "slight modifications, additions, deletions" that are unrelated to an OpenAPI spec, please share!

[AgustinBettati]

Hi @austinvalle. Just wanted to share an additional use case I encountered which is the use of the timeouts attribute, this is also a terraform specific concept that will not be present in the OpenAPI Spec.

[ethanzh]

Running into a related situation where I want to add a UseStateForUnknown() modifier to a handful of attributes on resources defined in my generator_config.yml. As far as I can tell, the only way I can accomplish this is by manually editing my provider-code-spec.json after its been generated, which will not scale

[artem-nefedov]

Plan modifiers are very important, especially RequiresReplace().
While this is not supported, using tfplugingen-openapi requires external spec hacking for most real world applications.