"Dynamic types inside of collections are not currently supported" but they actually behave as expected for custom functions

[mschuchard]

Module version

1.15.1

Use-cases

resp.Definition = function.Definition{
    Parameters: []function.Parameter{
        function.MapParameter{
            ElementType: types.DynamicType,
        },
    },
    Return: function.MapReturn{ElementType: types.DynamicType},
}

Error: Failed to load plugin schemas

Error while loading schemas for plugin components: Failed to obtain provider
schema: Could not load the schema for provider
registry.terraform.io/hashicorp/stdlib: failed to retrieve schema from
provider "registry.terraform.io/hashicorp/stdlib": 2 problems:

Invalid Function Definition: When validating the function definition, an
implementation issue was found. This is always an issue with the provider and
should be reported to the provider developers.

Return contains a collection type with a nested dynamic type.

Dynamic types inside of collections are not currently supported in
terraform-plugin-framework. If underlying dynamic values are required,
replace the return definition with DynamicReturn instead.
Invalid Function Definition: When validating the function definition, an
implementation issue was found. This is always an issue with the provider and
should be reported to the provider developers.

Parameter "map" at position 0 contains a collection type with a nested
dynamic type.

Dynamic types inside of collections are not currently supported in
terraform-plugin-framework. If underlying dynamic values are required,
replace the "map" parameter definition with DynamicParameter instead...

Attempted Solutions

It actually already works for custom functions, but is limited by schema restrictions.

Proposal

Allow this for custom functions if possible.

References

n/a

Summary: my unit test case matrix passes (but of course acceptance tests fail) for custom functions with dynamic types inside of collections, and everything behaves as expected as long as the type within the collection is uniform (otherwise the type validation fails which is expected behavior). I understand there is a schema restriction here, but everything is functioning correctly for custom functions, and so I was wondering if the restriction could be lifted solely for custom functions.

I also understand that dynamic types are not equivalent to generics, but rather more of an any type, but with type checking on UnderlyingValue this is not a roadblock.

Thank you.

[austinvalle]

Hey there @mschuchard 👋🏻, thanks for submitting the issue!

The portion that won't work for dynamics in collections today in Framework is actually related to the protocol conversions back to Terraform inside of the Map itself. The specific portion that wouldn't work is any conversions of null or unknown elements in a collection that has a dynamic element type. (It is very much possible to make the changes for that to work, it just currently isn't implemented in List, Map, or Set)

So given that, there really isn't a way for functions to fully work without making the necessary changes to the collection types themselves. There is already a feature request for that over in #973 which you can subscribe/upvote/add more info about your use-case, so I'm going to close this as a duplicate of that feature.

---

I detailed this in one of the comments in that other GH issue, but it is possible to achieve what you're looking for at the moment by implementing a custom type that does handle these conversions properly (my example was a list, but map would be a similar implementation): #973 (comment)

If you implemented a custom type like that, then you could achieve a map(any) with it:

resp.Definition = function.Definition{
    Parameters: []function.Parameter{
        function.MapParameter{
            CustomType: xtypes.DynamicMapType{},
        },
    },
    Return: function.MapReturn{CustomType: xtypes.DynamicMapType{}},
}

[mschuchard]

Ok thanks for the explanation, because it implies the reason I always observed expected behavior is because I developed code to crudely handle situations where the elements in the collection were null or unknown, and therefore never encountered failures when those were present in my unit test matrix.

I was planning on implementing a custom type to deal with a specific situation with a DynamicType in the functions, but not to this degree, or more than one, and so that route does not feel like currently of good cost/benefit. Thank you for the path forward though. I understand my path forward only really works for me, and therefore cannot be supported by tftypes without the possibility of opening the door to issues elsewhere.

I also assume this is not on the roadmap for various reasons like backwards compatibility with the SDKv2 etc. causing it to become poor cost/benefit for you all also.

Thanks again.