Document the architecture for dynamic providers (#2137)

Fixes pulumi/home#3471

As part of documenting what each package does, I moved `loader*.go` from
`shim` to `shim/run`, which I think reads clearer. There are no runtime
changes in this commit.

---------

Co-authored-by: VenelinMartinov <[email protected]>

Author: iwahbe

dynamic/README.md

@@ -95,3 +95,158 @@ sequenceDiagram
     destroy B
     B-->>P: Cancel done
 ```
+
+Diving deeper into how the repo is laid out, we see:
+
+``` console
+$ eza -T --classify=always -I 'test*' --git-ignore
+./
+├── go.mod
+├── go.sum
+├── info.go
+├── internal/
+│  └── shim/
+│     ├── go.mod
+│     ├── go.sum
+│     ├── protov5/
+│     │  ├── provider.go
+│     │  └── translate/
+│     │     └── tfplugin5.go
+│     ├── protov6/
+│     │  ├── provider.go
+│     │  └── translate/
+│     │     └── tfplugin6.go
+│     └── run/
+│        ├── loader.go
+│        └── loader_test.go
+├── main.go
+├── Makefile
+├── provider_test.go
+├── README.md
+└── version/
+   └── version.go
+```
+
+The dynamic provider layer is by-design as simple and straight-forward as possible. Each package does one
+thing only and there isn't that much code. As of time of writing, the entire `dynamic` folder is only 2288
+lines of go code[^1]. I'll go through each package in turn.
+
+[^1]: `loc --exclude '*._test.go'`
+
+### `package main`
+
+`package main` is responsible for launching a Pulumi provider and setting up the parameterize call. It does
+this by calling [`pf/tfbridge.Main`](https://pkg.go.dev/github.com/pulumi/pulumi-terraform-bridge/[email protected]/tfbridge#Main), passing in an empty PF provider (from
+[`pf/proto.Empty()`](https://pkg.go.dev/github.com/pulumi/pulumi-terraform-bridge/[email protected]/proto#Empty)). [`pf/tfbridge.ProviderMetadata`](https://pkg.go.dev/github.com/pulumi/pulumi-terraform-bridge/[email protected]/tfbridge#ProviderMetadata) allows overriding the `Parameterize` and
+`GetSchema` call (and we override both).
+
+When `Parameterize` is called, we launch the underlying Terraform provider via
+`internal/shim/run.LocalProvider` or `internal/shim/run.NamedProvider` (downloading as necessary). Both
+functions return a [`tfprotov6.ProviderServer`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-go/tfprotov6#ProviderServer) which is used to re-initialize the running provider via
+[`pf/tfbridge.XParameterizeResetProvider`](https://pkg.go.dev/github.com/pulumi/pulumi-terraform-bridge/[email protected]/tfbridge#XParameterizeResetProvider).
+
+When `GetSchema` is called, it generates a schema from the currently equipped provider with
+[`pkg/tfgen.GenerateSchemaWithOptions`](https://pkg.go.dev/github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen#GenerateSchemaWithOptions) and returns is. All type translation, documentation generation, etc
+are done with standard bridge based functionality.
+
+All other gRPC calls (`Create`, `Read`, `Update`, `Delete`, etc.) are handled normally by `pf`'s existing
+server.
+
+### `package version`
+
+`version.version` is used as a link-time target to bake in the release version to the provider binary. This is
+the same mechanism that Pulumi uses to embed versions in all of our binaries.
+
+### `package run`
+
+`run` defines a running provider for the purposes of `dynamic`.
+
+``` go
+type Provider interface {
+	tfprotov6.ProviderServer
+	io.Closer
+
+	Name() string
+	Version() string
+}
+```
+
+`run` also defines functions to "run" the underlying TF provider:
+
+- `run.NamedProvider` takes a provider definition like `("cloudfront/cloudfront", ">= 3.2.0")` and loads the
+  provider (downloading it if necessary). Named Terraform providers are cached in
+  `PULUMI_DYNAMIC_TF_PLUGIN_CACHE_DIR` (defaulting to `$PULUMI_HOME/dynamic_tf_plugins`).
+
+- `run.LocalProvider` takes a path to a Terraform provider and runs it.
+
+When `run` launches a Terraform provider, the provider may implement either the
+[`tfplugin5.ProviderClient`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tfplugin5#ProviderClient) or [`tfplugin6.ProviderClient`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tfplugin6#ProviderClient) interface. `run` must return a
+[`tfprotov6.ProviderServer`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-go/tfprotov6#ProviderServer). The Terraform ecosystem helps with [translating from v5 to v6](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-mux/tf5to6server#UpgradeServer):
+
+``` go
+func tf5to6server.UpgradeServer(context.Context, func() tfprotov5.ProviderServer) (tfprotov6.ProviderServer, error)
+```
+
+We still need to be able to translate from [`tfplugin5.ProviderClient`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tfplugin5#ProviderClient) and [`tfplugin6.ProviderClient`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tfplugin6#ProviderClient)
+to [`tfprotov5.ProviderServer`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-go/tfprotov5#ProviderServer) and [`tfprotov6.ProviderServer`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-go/tfprotov6#ProviderServer) respectively. For that, see the next
+section.
+
+### `package protov5` & `package protov6`
+
+`package protov5` and `package protov6` are nearly identical packages that translate between gRPC level client
+types to just above gRPC level server types. Both packages are identical in structure, exposing one end point:
+
+``` go
+func New(tfplugin5.ProviderClient) tfprotov5.ProviderServer
+
+func New(tfplugin6.ProviderClient) tfprotov6.ProviderServer
+```
+
+Both packages delegate type conversions to a `translate` sub-package, restricting themselves to fielding gRPC
+calls.
+
+A representative gRPC handler looks like this:
+
+``` go
+// tfprotov6/provider.go
+import (
+	"github.com/opentofu/opentofu/internal/tfplugin6"
+	"github.com/opentofu/opentofu/shim/protov6/translate"
+)
+
+...
+
+func (p shimProvider) ReadResource(
+	ctx context.Context, req *tfprotov6.ReadResourceRequest,
+) (*tfprotov6.ReadResourceResponse, error) {
+	return translateGRPC(ctx,
+		p.remote.ReadResource,
+		translate.ReadResourceRequest(req),
+		translate.ReadResourceResponse)
+}
+```
+
+The `translate.ReadResourceRequest` call looks like this:
+
+``` go
+// tfprotov6/translate/tfplugin6.go
+import (
+	"github.com/hashicorp/terraform-plugin-go/tfprotov6"
+	"github.com/opentofu/opentofu/internal/tfplugin6"
+)
+
+...
+
+func ReadResourceRequest(i *tfprotov6.ReadResourceRequest) *tfplugin6.ReadResource_Request {
+	if i == nil {
+		return nil
+	}
+
+	return &tfplugin6.ReadResource_Request{
+		TypeName:     i.TypeName,
+		CurrentState: dynamicValueRequest(i.CurrentState),
+		Private:      i.Private,
+		ProviderMeta: dynamicValueRequest(i.ProviderMeta),
+	}
+}
+```

dynamic/info.go

@@ -20,15 +20,15 @@ import (
 	"path"
 	"strings"
 
-	"github.com/opentofu/opentofu/shim"
+	"github.com/opentofu/opentofu/shim/run"
 
 	"github.com/pulumi/pulumi-terraform-bridge/pf/proto"
 	"github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge"
 	"github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge/tokens"
 	"github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin"
 )
 
-func providerInfo(ctx context.Context, p shim.Provider) tfbridge.ProviderInfo {
+func providerInfo(ctx context.Context, p run.Provider) tfbridge.ProviderInfo {
 	prov := tfbridge.ProviderInfo{
 		P:           proto.New(ctx, p),
 		Name:        p.Name(),

dynamic/internal/shim/loader.go renamed to dynamic/internal/shim/run/loader.go

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package shim
+package run
 
 import (
 	"context"
@@ -61,13 +61,13 @@ type Provider interface {
 	Version() string
 }
 
-// Load a TF provider with key and version specified.
+// NamedProvider loads a TF provider with key and version specified.
 //
 // If version is "", then whatever version is currently installed will be used. If no
 // version is installed then the latest version can be used.
 //
 // `=`, `<=`, `>=` sigils can be used just like in TF.
-func LoadProvider(ctx context.Context, key, version string) (Provider, error) {
+func NamedProvider(ctx context.Context, key, version string) (Provider, error) {
 
 	p, err := tfaddr.ParseProviderSource(key)
 	if err != nil {
@@ -82,8 +82,8 @@ func LoadProvider(ctx context.Context, key, version string) (Provider, error) {
 	return getProviderServer(ctx, p, v, disco.New())
 }
 
-// RunLocalProvider runs a provider by it's path.
-func RunLocalProvider(ctx context.Context, path string) (Provider, error) {
+// LocalProvider runs a provider by it's path.
+func LocalProvider(ctx context.Context, path string) (Provider, error) {
 	dir, name, ok := cutLast(path, "terraform-provider-")
 	if !ok {
 		return nil, fmt.Errorf("expected path to end with %q", "terraform-provider-${NAME}")

dynamic/internal/shim/loader_test.go renamed to dynamic/internal/shim/run/loader_test.go

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package shim
+package run
 
 import (
 	"context"
@@ -39,7 +39,7 @@ func TestLoadProvider(t *testing.T) {
 		Integration(t)
 		ctx := context.Background()
 
-		p, err := LoadProvider(ctx, "hashicorp/tls", "<4.0.5,>4.0.3")
+		p, err := NamedProvider(ctx, "hashicorp/tls", "<4.0.5,>4.0.3")
 		require.NoError(t, err)
 
 		require.Equal(t, "4.0.4", p.Version())

dynamic/main.go

@@ -21,21 +21,22 @@ import (
 	"os"
 
 	"github.com/blang/semver"
-	"github.com/opentofu/opentofu/shim"
+	"github.com/opentofu/opentofu/shim/run"
+	"github.com/pulumi/pulumi/sdk/v3/go/common/diag"
+	"github.com/pulumi/pulumi/sdk/v3/go/common/diag/colors"
+	"github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin"
+
 	"github.com/pulumi/pulumi-terraform-bridge/pf/proto"
 	pfbridge "github.com/pulumi/pulumi-terraform-bridge/pf/tfbridge"
 	"github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge"
 	"github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen"
-	"github.com/pulumi/pulumi/sdk/v3/go/common/diag"
-	"github.com/pulumi/pulumi/sdk/v3/go/common/diag/colors"
-	"github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin"
 
 	"github.com/pulumi/pulumi-terraform-bridge/dynamic/version"
 )
 
 func initialSetup() (tfbridge.ProviderInfo, pfbridge.ProviderMetadata, func() error) {
 
-	var tfServer shim.Provider
+	var tfServer run.Provider
 	info := tfbridge.ProviderInfo{
 		DisplayName:  "Any Terraform Provider",
 		P:            proto.Empty(),
@@ -139,10 +140,10 @@ func main() {
 	pfbridge.Main(ctx, "terraform-bridge", defaultInfo, metadata)
 }
 
-func getProvider(ctx context.Context, args paramaterizeArgs) (shim.Provider, error) {
+func getProvider(ctx context.Context, args paramaterizeArgs) (run.Provider, error) {
 	if args.path != "" {
-		return shim.RunLocalProvider(ctx, args.path)
+		return run.LocalProvider(ctx, args.path)
 	}
 
-	return shim.LoadProvider(ctx, args.name, args.version)
+	return run.NamedProvider(ctx, args.name, args.version)
 }