FR and/or Doc: support for variadic parameters in unit testing and/or updates to corresponding documentation

[mschuchard]

Version: 1.11.0 <-- refers to terraform-plugin-testing

At the moment I have a simple function with a variadic parameter where the parameter value retrieval occurs according to documentation:

VariadicParameter: function.Int32Parameter{
	Name:        "number_of_characters",
	Description: "Optional: The number of terminating characters at the end of the string to return (default: 1). This must be fewer than the number of characters in the input string.",
},
...
var numCharsVar []int32

resp.Error = function.ConcatFuncErrors(resp.Error, req.Arguments.Get(ctx, &inputString, &numCharsVar))

During unit testing omitting the variadic parameter succeeds completely as expected:

"optional-param-absent": {
	request: function.RunRequest{
		Arguments: function.NewArgumentsData([]attr.Value{types.StringValue("hello"), types.Int32Null()}),
	},
	expected: function.RunResponse{
		Result: function.NewResultData(types.StringValue("o")),
	},
},

but specifying a value errors:

"three-terminating-chars": {
	request: function.RunRequest{
		Arguments: function.NewArgumentsData([]attr.Value{types.StringValue("hello"), types.Int32Value(3)}),
	},
	expected: function.RunResponse{
		Result: function.NewResultData(types.StringValue("llo")),
	},
},

actual value: Value Conversion Error: An unexpected error was encountered trying to convert tftypes.Value into []int32. This is always an error in the provider. Please report the following to the provider developer:
can't unmarshal tftypes.Number into *[]tftypes.Value expected []tftypes.Value

Ok so even though types.Int32Null() is valid in the function signature for the first unit test request, and it would also be a tuple tftype in the actual Terraform config, the expectation here in the unit testing framework is for the variadic parameters to be capable of marshalling to a slice?
Well the function.NewArgumentsData signature only accepts a single slice of attr.Value, and the second argument cannot be []types.Int32{types.Int32Value(3)} because the interface does not implement what is required for the attr.Value type, and so then is the variadic parameter argument supposed to be an enumerable tftype i.e. set or list?

So I am kind of at a coin flip between whether this is unsupported or undocumented.

Corollary: complex type return testing for e.g. tuples is supported although undocumented, and I did miraculously figure it out via:

"normal": {
	request: function.RunRequest{
		Arguments: function.NewArgumentsData([]attr.Value{types.StringValue("foobarbaz"), types.StringValue("bar")}),
	},
	expected: function.RunResponse{
		Result: function.NewResultData(types.TupleValueMust(
			[]attr.Type{types.StringType, types.StringType, types.BoolType},
			[]attr.Value{types.StringValue("foo"), types.StringValue("baz"), types.BoolValue(true)})),
	},
},

but this would also be benficial to add to documentation since e.g. currently in the AWS provider there is a function with an object tftype return and a "TODO: figure out how to do this" which implies both a documentation gap if they cannot figure it out, and also that probably some element of luck was involved with me actually figuring it out.

Thank you.

[austinvalle]

Hey there @mschuchard 👋🏻 , I think our Framework documentation could be updated to give more details on unit testing functions. I'm going to transfer this issue over to terraform-plugin-framework 👍🏻

Confirming what you mentioned, variadic parameters are a little awkward to unit test as they are represented by a types.Tuple under the scenes (tuple is being used instead of a list to allow dynamic parameters of different types), which our reflection logic in req.Arguments.Get actually supports reflecting a tuple into a slice of a single type, as long as the tuple itself contains elements of a single type. We have some documentation about retrieving data for variadic parameters, but no docs about setting up unit tests for it.

If you wanted your unit test to match exactly how the data will come from Terraform to your function, you'd want something like:

"three-terminating-chars": {
    request: function.RunRequest{
        Arguments: function.NewArgumentsData(
            []attr.Value{
                types.StringValue("hello"),
                types.TupleValueMust(
                    []attr.Type{
                        types.Int32,
                    },
                    []attr.Value{
                        types.Int32Value(3),
                    },
                ),
            },
        ),
    },
    expected: function.RunResponse{
        Result: function.NewResultData(types.StringValue("llo")),
    },
},

As for the types.Int32Null() successfully reflecting into a []int32, I think that's just because our underlying reflection logic supports that (null anything to a slice). If you actually executed your logic, the underlying attr.Value should always be a tuple like above.

[mschuchard]

I never received a GH notification for the reply for some reason, and so am noticing this for the first time now as I revisit unit testing variadics.

I did assume that the tftype for the actual Terraform config would be a tuple, but somehow even though I assumed that, and discovered how to unit test with tuples when I tested the DynamicReturn type struct, I failed to make the final connection to your implementation because I incorrectly interpreted the success with the Int32Null method as that I was inputting the correct type (although I did suspect something was wrong with my type in the unit tests). However:

As for the types.Int32Null() successfully reflecting into a []int32, I think that's just because our underlying reflection logic supports that (null anything to a slice).

explains where I went wrong for sure. Thank you for that.

I have converted the variadic parameter types in the unit tests to tuple, and now everything is flowing smoothly. Thanks for the tip.

Whatever action item you want to take on this issue is cool with me as I am good with closing, but an example of unit testing with a tuple and mentioning its relationship to the variadic sounds super good to me too.

[mschuchard]

If you actually executed your logic, the underlying attr.Value should always be a tuple like above.

Does that imply unit tests for functions where no arguments are specified for the variadic parameters should also be of type tuple for the absent arguments by best practices? Would you therefore recommend constructing an empty tuple for the purposes of unit testing via types.TupleValueMust([]attr.Type{}, []attr.Value{}), or is there a better way?

[austinvalle]

Does that imply unit tests for functions where no arguments are specified for the variadic parameters should also be of type tuple for the absent arguments by best practices? Would you therefore recommend constructing an empty tuple for the purposes of unit testing via types.TupleValueMust([]attr.Type{}, []attr.Value{}), or is there a better way?

Yeah that's exactly how it would be coming from Terraform, so using types.TupleValueMust([]attr.Type{}, []attr.Value{}) is the way to go 👍🏻

[mschuchard]

If you think it would be a good idea for me to contribute examples to the function unit tests documentation then please let me know, and I can PR some wherever they would be located.

[austinvalle]

Yeah our documentation has been moved out of this repo so it's a little more of a headache to contribute (especially for future versions of docs to be released)

If you're interested (always appreciated), all of our docs live in the hashicorp/web-unified-docs repo. All of the documentation is versioned into folders, so if you raise a PR you can make the change to the latest 1.14.x folder.