Skip to content

Commit da1f7e4

Browse files
[Internal] Rollout Plugin Framework (#4134)
## Changes <!-- Summary of your changes that are easy to understand --> - Made it possible to use environment variable USE_SDK_V2 to control rollout - Bumped resource and data source names from staging names to production names (can debate on whether we want to do that now or in a more gradual fashion) - Proposed mechanism: https://docs.google.com/document/d/1zfTp8YesMe4GxkbIt9Sbwd5nQL7Y4mXed8dPDGmAoxU/edit?tab=t.0 - Added infra for switching providers from sdkv2 to plugin framework in integration tests and added test cases for those cases ## Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] `make test` run locally - [x] relevant change in `docs/` folder - [x] covered with integration tests in `internal/acceptance` - [x] relevant acceptance tests are passing - [x] using Go SDK
1 parent 17641de commit da1f7e4

File tree

14 files changed

+617
-246
lines changed

14 files changed

+617
-246
lines changed

docs/data-sources/volumes.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ subcategory: "Unity Catalog"
77

88
Retrieves a list of [databricks_volume](../resources/volume.md) ids (full names), that were created by Terraform or manually.
99

10+
## Plugin Framework Migration
11+
The volumes data source has been migrated from sdkv2 to plugin framework in version 1.57。 If you encounter any problem with this data source and suspect it is due to the migration, you can fallback to sdkv2 by setting the environment variable in the following way `export USE_SDK_V2_DATA_SOURCES="databricks_volumes"`.
12+
1013
## Example Usage
1114

1215
Listing all volumes in a _things_ [databricks_schema](../resources/schema.md) of a _sandbox_ [databricks_catalog](../resources/catalog.md):

docs/guides/troubleshooting.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,18 @@ TF_LOG=DEBUG DATABRICKS_DEBUG_TRUNCATE_BYTES=250000 terraform apply -no-color 2>
1717

1818
* Open a [new GitHub issue](https://github.com/databricks/terraform-provider-databricks/issues/new/choose) providing all information described in the issue template - debug logs, your Terraform code, Terraform & plugin versions, etc.
1919

20+
## Plugin Framework Migration Problems
21+
The following resources and data sources have been migrated from sdkv2 to plugin framework。 If you encounter any problem with those, you can fallback to sdkv2 by setting the `USE_SDK_V2_RESOURCES` and `USE_SDK_V2_DATA_SOURCES` environment variables.
22+
23+
Example: `export USE_SDK_V2_RESOURCES="databricks_library,databricks_quality_monitor"`
24+
25+
### Resources migrated
26+
- databricks_quality_monitor
27+
- databricks_library
28+
### Data sources migrated
29+
- databricks_volumes
30+
31+
2032
## Typical problems
2133

2234
### Data resources and Authentication is not configured errors

docs/resources/library.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ Installs a [library](https://docs.databricks.com/libraries/index.html) on [datab
77

88
-> `databricks_library` resource would always start the associated cluster if it's not running, so make sure to have auto-termination configured. It's not possible to atomically change the version of the same library without cluster restart. Libraries are fully removed from the cluster only after restart.
99

10+
## Plugin Framework Migration
11+
The library resource has been migrated from sdkv2 to plugin framework。 If you encounter any problem with this resource and suspect it is due to the migration, you can fallback to sdkv2 by setting the environment variable in the following way `export USE_SDK_V2_RESOURCES="databricks_library"`.
12+
1013
## Installing library on all clusters
1114

1215
You can install libraries on all clusters with the help of [databricks_clusters](../data-sources/clusters.md) data resource:

docs/resources/quality_monitor.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@ This resource allows you to manage [Lakehouse Monitors](https://docs.databricks.
77

88
A `databricks_quality_monitor` is attached to a [databricks_sql_table](sql_table.md) and can be of type timeseries, snapshot or inference.
99

10+
## Plugin Framework Migration
11+
The quality monitor resource has been migrated from sdkv2 to plugin framework。 If you encounter any problem with this resource and suspect it is due to the migration, you can fallback to sdkv2 by setting the environment variable in the following way `export USE_SDK_V2_RESOURCES="databricks_quality_monitor"`.
12+
1013
## Example Usage
1114

1215
```hcl

internal/providers/pluginfw/pluginfw.go

Lines changed: 5 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -16,15 +16,6 @@ import (
1616
"github.com/databricks/terraform-provider-databricks/commands"
1717
"github.com/databricks/terraform-provider-databricks/common"
1818
providercommon "github.com/databricks/terraform-provider-databricks/internal/providers/common"
19-
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/catalog"
20-
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/cluster"
21-
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/library"
22-
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/notificationdestinations"
23-
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/qualitymonitor"
24-
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/registered_model"
25-
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/sharing"
26-
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/volume"
27-
2819
"github.com/hashicorp/terraform-plugin-framework/datasource"
2920
"github.com/hashicorp/terraform-plugin-framework/diag"
3021
"github.com/hashicorp/terraform-plugin-framework/path"
@@ -35,34 +26,23 @@ import (
3526
"github.com/hashicorp/terraform-plugin-log/tflog"
3627
)
3728

38-
func GetDatabricksProviderPluginFramework() provider.Provider {
39-
p := &DatabricksProviderPluginFramework{}
29+
func GetDatabricksProviderPluginFramework(sdkV2FallbackOptions ...SdkV2FallbackOption) provider.Provider {
30+
p := &DatabricksProviderPluginFramework{sdkV2Fallbacks: sdkV2FallbackOptions}
4031
return p
4132
}
4233

4334
type DatabricksProviderPluginFramework struct {
35+
sdkV2Fallbacks []SdkV2FallbackOption
4436
}
4537

4638
var _ provider.Provider = (*DatabricksProviderPluginFramework)(nil)
4739

4840
func (p *DatabricksProviderPluginFramework) Resources(ctx context.Context) []func() resource.Resource {
49-
return []func() resource.Resource{
50-
qualitymonitor.ResourceQualityMonitor,
51-
library.ResourceLibrary,
52-
sharing.ResourceShare,
53-
}
41+
return getPluginFrameworkResourcesToRegister(p.sdkV2Fallbacks...)
5442
}
5543

5644
func (p *DatabricksProviderPluginFramework) DataSources(ctx context.Context) []func() datasource.DataSource {
57-
return []func() datasource.DataSource{
58-
cluster.DataSourceCluster,
59-
volume.DataSourceVolumes,
60-
registered_model.DataSourceRegisteredModel,
61-
notificationdestinations.DataSourceNotificationDestinations,
62-
sharing.DataSourceShare,
63-
sharing.DataSourceShares,
64-
catalog.DataSourceFunctions,
65-
}
45+
return getPluginFrameworkDataSourcesToRegister(p.sdkV2Fallbacks...)
6646
}
6747

6848
func (p *DatabricksProviderPluginFramework) Schema(ctx context.Context, req provider.SchemaRequest, resp *provider.SchemaResponse) {
Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
package pluginfw
2+
3+
// This file contains all of the utils for controlling the plugin framework rollout.
4+
// For migrated resources and data sources, we can add them to the two maps below to have them registered with the plugin framework.
5+
// Users can manually specify resources and data sources to use SDK V2 instead of the plugin framework by setting the USE_SDK_V2_RESOURCES and USE_SDK_V2_DATA_SOURCES environment variables.
6+
//
7+
// Example: USE_SDK_V2_RESOURCES="databricks_library" would force the library resource to use SDK V2 instead of the plugin framework.
8+
9+
import (
10+
"context"
11+
"os"
12+
"slices"
13+
"strings"
14+
15+
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/catalog"
16+
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/cluster"
17+
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/library"
18+
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/notificationdestinations"
19+
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/qualitymonitor"
20+
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/registered_model"
21+
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/sharing"
22+
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/resources/volume"
23+
"github.com/hashicorp/terraform-plugin-framework/datasource"
24+
"github.com/hashicorp/terraform-plugin-framework/resource"
25+
)
26+
27+
// List of resources that have been migrated from SDK V2 to plugin framework
28+
var migratedResources = []func() resource.Resource{
29+
qualitymonitor.ResourceQualityMonitor,
30+
library.ResourceLibrary,
31+
}
32+
33+
// List of data sources that have been migrated from SDK V2 to plugin framework
34+
var migratedDataSources = []func() datasource.DataSource{
35+
volume.DataSourceVolumes,
36+
}
37+
38+
// List of resources that have been onboarded to the plugin framework - not migrated from sdkv2.
39+
var pluginFwOnlyResources = []func() resource.Resource{
40+
// TODO Add resources here
41+
sharing.ResourceShare, // Using the staging name (with pluginframework suffix)
42+
}
43+
44+
// List of data sources that have been onboarded to the plugin framework - not migrated from sdkv2.
45+
var pluginFwOnlyDataSources = []func() datasource.DataSource{
46+
registered_model.DataSourceRegisteredModel,
47+
notificationdestinations.DataSourceNotificationDestinations,
48+
catalog.DataSourceFunctions,
49+
// TODO: Add DataSourceCluster into migratedDataSources after fixing unit tests.
50+
cluster.DataSourceCluster, // Using the staging name (with pluginframework suffix)
51+
sharing.DataSourceShare, // Using the staging name (with pluginframework suffix)
52+
sharing.DataSourceShares, // Using the staging name (with pluginframework suffix)
53+
}
54+
55+
type sdkV2FallbackOptions struct {
56+
resourceFallbacks []string
57+
dataSourceFallbacks []string
58+
}
59+
60+
// SdkV2FallbackOption is an interface for acceptance tests to specify resources / data sources to fallback to SDK V2
61+
type SdkV2FallbackOption interface {
62+
Apply(*sdkV2FallbackOptions)
63+
}
64+
65+
type sdkV2ResourceFallback struct {
66+
resourceFallbacks []string
67+
}
68+
69+
func (o *sdkV2ResourceFallback) Apply(options *sdkV2FallbackOptions) {
70+
options.resourceFallbacks = o.resourceFallbacks
71+
}
72+
73+
// WithSdkV2ResourceFallbacks is a helper function to specify resources to fallback to SDK V2
74+
func WithSdkV2ResourceFallbacks(fallbacks ...string) SdkV2FallbackOption {
75+
return &sdkV2ResourceFallback{resourceFallbacks: fallbacks}
76+
}
77+
78+
type sdkv2DataSourceFallback struct {
79+
dataSourceFallbacks []string
80+
}
81+
82+
func (o *sdkv2DataSourceFallback) Apply(options *sdkV2FallbackOptions) {
83+
options.dataSourceFallbacks = o.dataSourceFallbacks
84+
}
85+
86+
// WithSdkV2DataSourceFallbacks is a helper function to specify data sources to fallback to SDK V2
87+
func WithSdkV2DataSourceFallbacks(fallbacks []string) SdkV2FallbackOption {
88+
return &sdkv2DataSourceFallback{dataSourceFallbacks: fallbacks}
89+
}
90+
91+
// GetUseSdkV2DataSources is a helper function to get name of resources that should use SDK V2 instead of plugin framework
92+
func getUseSdkV2Resources() []string {
93+
useSdkV2 := os.Getenv("USE_SDK_V2_RESOURCES")
94+
if useSdkV2 == "" {
95+
return []string{}
96+
}
97+
return strings.Split(useSdkV2, ",")
98+
}
99+
100+
// GetUseSdkV2DataSources is a helper function to get name of data sources that should use SDK V2 instead of plugin framework
101+
func getUseSdkV2DataSources() []string {
102+
useSdkV2 := os.Getenv("USE_SDK_V2_DATA_SOURCES")
103+
if useSdkV2 == "" {
104+
return []string{}
105+
}
106+
return strings.Split(useSdkV2, ",")
107+
}
108+
109+
// Helper function to check if a resource should use be in SDK V2 instead of plugin framework
110+
func shouldUseSdkV2Resource(resourceName string) bool {
111+
useSdkV2Resources := getUseSdkV2Resources()
112+
return slices.Contains(useSdkV2Resources, resourceName)
113+
}
114+
115+
// Helper function to check if a data source should use be in SDK V2 instead of plugin framework
116+
func shouldUseSdkV2DataSource(dataSourceName string) bool {
117+
sdkV2DataSources := getUseSdkV2DataSources()
118+
return slices.Contains(sdkV2DataSources, dataSourceName)
119+
}
120+
121+
// getPluginFrameworkResourcesToRegister is a helper function to get the list of resources that are migrated away from sdkv2 to plugin framework
122+
func getPluginFrameworkResourcesToRegister(sdkV2Fallbacks ...SdkV2FallbackOption) []func() resource.Resource {
123+
fallbackOption := sdkV2FallbackOptions{}
124+
for _, o := range sdkV2Fallbacks {
125+
o.Apply(&fallbackOption)
126+
}
127+
128+
var resources []func() resource.Resource
129+
130+
// Loop through the map and add resources if they're not specifically marked to use the SDK V2
131+
for _, resourceFunc := range migratedResources {
132+
name := getResourceName(resourceFunc)
133+
if !shouldUseSdkV2Resource(name) && !slices.Contains(fallbackOption.resourceFallbacks, name) {
134+
resources = append(resources, resourceFunc)
135+
}
136+
}
137+
138+
return append(resources, pluginFwOnlyResources...)
139+
}
140+
141+
// getPluginFrameworkDataSourcesToRegister is a helper function to get the list of data sources that are migrated away from sdkv2 to plugin framework
142+
func getPluginFrameworkDataSourcesToRegister(sdkV2Fallbacks ...SdkV2FallbackOption) []func() datasource.DataSource {
143+
fallbackOption := sdkV2FallbackOptions{}
144+
for _, o := range sdkV2Fallbacks {
145+
o.Apply(&fallbackOption)
146+
}
147+
148+
var dataSources []func() datasource.DataSource
149+
150+
// Loop through the map and add data sources if they're not specifically marked to use the SDK V2
151+
for _, dataSourceFunc := range migratedDataSources {
152+
name := getDataSourceName(dataSourceFunc)
153+
if !shouldUseSdkV2DataSource(name) && !slices.Contains(fallbackOption.dataSourceFallbacks, name) {
154+
dataSources = append(dataSources, dataSourceFunc)
155+
}
156+
}
157+
158+
return append(dataSources, pluginFwOnlyDataSources...)
159+
}
160+
161+
func getResourceName(resourceFunc func() resource.Resource) string {
162+
resp := resource.MetadataResponse{}
163+
resourceFunc().Metadata(context.Background(), resource.MetadataRequest{ProviderTypeName: "databricks"}, &resp)
164+
return resp.TypeName
165+
}
166+
167+
func getDataSourceName(dataSourceFunc func() datasource.DataSource) string {
168+
resp := datasource.MetadataResponse{}
169+
dataSourceFunc().Metadata(context.Background(), datasource.MetadataRequest{ProviderTypeName: "databricks"}, &resp)
170+
return resp.TypeName
171+
}
172+
173+
// GetSdkV2ResourcesToRemove is a helper function to get the list of resources that are migrated away from sdkv2 to plugin framework
174+
func GetSdkV2ResourcesToRemove(sdkV2Fallbacks ...SdkV2FallbackOption) []string {
175+
fallbackOption := sdkV2FallbackOptions{}
176+
for _, o := range sdkV2Fallbacks {
177+
o.Apply(&fallbackOption)
178+
}
179+
180+
resourcesToRemove := []string{}
181+
for _, resourceFunc := range migratedResources {
182+
name := getResourceName(resourceFunc)
183+
if !shouldUseSdkV2Resource(name) && !slices.Contains(fallbackOption.resourceFallbacks, name) {
184+
resourcesToRemove = append(resourcesToRemove, name)
185+
}
186+
}
187+
return resourcesToRemove
188+
}
189+
190+
// GetSdkV2DataSourcesToRemove is a helper function to get the list of data sources that are migrated away from sdkv2 to plugin framework
191+
func GetSdkV2DataSourcesToRemove(sdkV2Fallbacks ...SdkV2FallbackOption) []string {
192+
fallbackOption := sdkV2FallbackOptions{}
193+
for _, o := range sdkV2Fallbacks {
194+
o.Apply(&fallbackOption)
195+
}
196+
197+
dataSourcesToRemove := []string{}
198+
for _, dataSourceFunc := range migratedDataSources {
199+
name := getDataSourceName(dataSourceFunc)
200+
if !shouldUseSdkV2DataSource(name) && !slices.Contains(fallbackOption.dataSourceFallbacks, name) {
201+
dataSourcesToRemove = append(dataSourcesToRemove, name)
202+
}
203+
}
204+
return dataSourcesToRemove
205+
}

internal/providers/pluginfw/resources/library/resource_library.go

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,14 +62,15 @@ func readLibrary(ctx context.Context, w *databricks.WorkspaceClient, waitParams
6262
type LibraryExtended struct {
6363
compute_tf.Library
6464
ClusterId types.String `tfsdk:"cluster_id"`
65+
ID types.String `tfsdk:"id" tf:"optional,computed"` // Adding ID field to stay compatible with SDKv2
6566
}
6667

6768
type LibraryResource struct {
6869
Client *common.DatabricksClient
6970
}
7071

7172
func (r *LibraryResource) Metadata(ctx context.Context, req resource.MetadataRequest, resp *resource.MetadataResponse) {
72-
resp.TypeName = pluginfwcommon.GetDatabricksStagingName(resourceName)
73+
resp.TypeName = pluginfwcommon.GetDatabricksProductionName(resourceName)
7374
}
7475

7576
func (r *LibraryResource) Schema(ctx context.Context, req resource.SchemaRequest, resp *resource.SchemaResponse) {
@@ -139,6 +140,8 @@ func (r *LibraryResource) Create(ctx context.Context, req resource.CreateRequest
139140

140141
resp.Diagnostics.Append(readLibrary(ctx, w, waitParams, libraryRep, &installedLib)...)
141142

143+
installedLib.ID = types.StringValue(libGoSDK.String())
144+
142145
if resp.Diagnostics.HasError() {
143146
return
144147
}

0 commit comments

Comments
 (0)