Complicate type system.

The old type system was simple, but insufficient. We erased some
necessary complexity by ignoring unknown values. You could represent
them as tftypes.RawValues, but the intended usage of a tftypes.RawValue
is to call its Unmarshal method as close to the boundary of your
provider as possible. Unfortunately, in that situation, we never really
thought through what would happen with unknown values, which can't be
represented in a strongly-typed Go type, they would need to be an
interface or struct of some sort. So that approach wouldn't work, in
practice.

This new approach is to expose tfprotov5.DynamicValues from the protocol
methods, and they have an Unmarshal method that gives you a
tftypes.Value. The DynamicValue is just the msgpack/json bytes, the
Unmarshal method takes the tftypes.Type you want them decoded into, and
returns a tftypes.Value of that type if possible.

tftypes.Value gained some new helper methods, most notably As(), which
lets you convert from the canonical types that JSON and msgpack will
store in the Value into a more friendly type, if that conversion is
possible. So far, the only conversions that are possible is calling the
UnmarshalTerraform5Type method if the destination has one. A future
improvement would be to build in some default conversions for the
builtin Go types and maybe some collection types for tftypes.Values,
like []tftypes.Value or map[string]tftypes.Value. This approach gives us
a more explicit interface that's decoupled from the type we store in the
Value, and it allows us to massage things if we change details or
introduce new types, so there's not a backwards compatibility concern.

Finally, we added a tftypes.Path type, which is used to keep track of
pathing information into values so we can return more useful, specific
errors. It was integrated into the JSON unmarshaling and msgpack
marshaling, but probably could stand to be integrated into the msgpack
unmarshaling and to have an easy, handy conversion into a
tfprotov5.AttributePath. That may require some minor reworking.

Author: Paddy Carver

tfprotov5/data_source.go

@@ -2,8 +2,6 @@ package tfprotov5
 
 import (
 	"context"
-
-	"github.com/hashicorp/terraform-plugin-go/tfprotov5/tftypes"
 )
 
 type DataSourceServer interface {
@@ -13,7 +11,7 @@ type DataSourceServer interface {
 
 type ValidateDataSourceConfigRequest struct {
 	TypeName string
-	Config   *tftypes.RawValue
+	Config   *DynamicValue
 }
 
 type ValidateDataSourceConfigResponse struct {
@@ -22,11 +20,11 @@ type ValidateDataSourceConfigResponse struct {
 
 type ReadDataSourceRequest struct {
 	TypeName     string
-	Config       *tftypes.RawValue
-	ProviderMeta *tftypes.RawValue
+	Config       *DynamicValue
+	ProviderMeta *DynamicValue
 }
 
 type ReadDataSourceResponse struct {
-	State       *tftypes.RawValue
+	State       *DynamicValue
 	Diagnostics []*Diagnostic
 }

tfprotov5/dynamic_value.go

@@ -0,0 +1,65 @@
+package tfprotov5
+
+import (
+	"bytes"
+	"errors"
+
+	"github.com/hashicorp/terraform-plugin-go/tfprotov5/tftypes"
+	"github.com/vmihailenco/msgpack"
+)
+
+var ErrUnknownDynamicValueType = errors.New("DynamicValue had no JSON or msgpack data set")
+
+// DynamicValue represents a nested encoding value that came from the protocol.
+// The only way providers should ever interact with it is by calling its
+// `Unmarshal` method to retrive a `tftypes.Value`. Although the type system
+// allows for other interactions, they are explicitly not supported, and will
+// not be considered when evaluation for breaking changes. Treat this type as
+// an opaque value, and *only* call its `Unmarshal` method.
+type DynamicValue struct {
+	MsgPack []byte
+	JSON    []byte
+}
+
+// Unmarshal returns a tftypes.Value that represents the information contained
+// in the DynamicValue in an easy-to-interact-with way. It is the main purpose
+// of the DynamicValue type, and is how provider developers should obtain
+// config, state, and other values from the protocol.
+//
+// Pass in the type you want the value to be interpreted as. Terraform's type
+// system encodes in a lossy manner, meaning the type information is not
+// preserved losslessly when going over the wire. Sets, lists, and tuples all
+// look the same, as do user-specified values when the provider has a
+// DynamicPseudoType in its schema. Objects and maps all look the same, as
+// well, as do PseudoDynamicType values sometimes. Fortunately, the provider
+// should already know the type; it should be the type of the schema, or
+// PseudoDynamicType if that's what's in the schema. `Unmarshal` will then
+// parse the value as though it belongs to that type, if possible, and return a
+// tftypes.Value with the appropriate information. If the data can't be
+// interpreted as that type, an error will be returned saying so. In these
+// cases, double check to make sure the schema is declaring the same type being
+// passed into Unmarshal.
+//
+// In the event an ErrUnknownDynamicValueType is returned, one of three things
+// has happened:
+//
+// 1. terraform-plugin-go is out of date and out of sync with the protocol, and
+// an issue should be opened on its repo to get it updated.
+//
+// 2. terraform-plugin-go has a bug somewhere, and an issue should be opened on
+// its repo to get it fixed.
+//
+// 3. The provider or a dependency has modified the DynamicValue in an
+// unsupported way, and should treat it as opaque and not modify it, only call
+// Unmarshal.
+func (d DynamicValue) Unmarshal(typ tftypes.Type) (tftypes.Value, error) {
+	if d.JSON != nil {
+		return jsonUnmarshal(d.JSON, typ, nil)
+	}
+	if d.MsgPack != nil {
+		r := bytes.NewReader(d.MsgPack)
+		dec := msgpack.NewDecoder(r)
+		return msgpackUnmarshal(dec, typ, nil)
+	}
+	return tftypes.Value{}, ErrUnknownDynamicValueType
+}