Skip to content

Commit 69f7e78

Browse files
trodgezli82016melinath
authored
tgc-revival: new resource doc (#15333)
Co-authored-by: Zhenhua Li <[email protected]> Co-authored-by: Stephen Lewis (Burrows) <[email protected]>
1 parent 84b5239 commit 69f7e78

File tree

2 files changed

+353
-0
lines changed

2 files changed

+353
-0
lines changed

‎docs/content/convert/_index.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
title: "Convert"
3+
weight: 30
4+
params:
5+
bookCollapseSection: true
6+
---

‎docs/content/convert/add-new-resource-tgc.md‎

Lines changed: 347 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,347 @@
1+
---
2+
title: "Add TGC conversion"
3+
weight: 20
4+
---
5+
6+
# Add TGC conversion
7+
8+
{{< hint warning >}}
9+
This method of adding TGC support is currently experimental and not officially supported. We recommended this path only for core contributors.
10+
{{< /hint >}}
11+
12+
terraform-google-conversion (TGC) consumes a Terraform plan and uses it to build Cloud Asset Inventory (CAI) Assets. These built assets only exist in memory locally.
13+
14+
TGC supports only those GCP resources that are available in both the Terraform provider and Cloud Asset Inventory.
15+
16+
## Before you begin
17+
18+
### Getting a Terraform resource name from a GCP resource name
19+
20+
The first step in determining if a GCP resource is supported is to identify the corresponding Terraform resource. You can often do this by searching for the GCP resource name in the [Terraform google provider documentation](https://registry.terraform.io/providers/hashicorp/google/latest/docs).
21+
22+
### Getting the CAI asset type of a GCP resource
23+
24+
The second step in determining if a GCP resource is supported is to verify its inclusion in the Cloud Asset Inventory (CAI). You can complete this by searching for the resource's asset type within CAI's list of [supported asset types](https://cloud.google.com/asset-inventory/docs/asset-types).
25+
26+
## Adding support
27+
28+
Adding support for a resource has 4 steps:
29+
30+
1. Make changes to [Magic Modules](https://github.com/GoogleCloudPlatform/magic-modules) to add resource conversion code.
31+
2. Generate terraform-google-conversion.
32+
3. Run tests locally.
33+
4. Make PRs for Magic Modules with your changes.
34+
35+
Each of these is discussed in more detail below.
36+
37+
### 1. Adding a resource to TGC
38+
39+
{{% tabs "resource" %}}
40+
{{< tab "MMv1" >}}
41+
42+
Magic Modules uses a shared code base to generate terraform-google-conversion and the [google](https://github.com/hashicorp/terraform-provider-google) and [google-beta](https://github.com/hashicorp/terraform-provider-google-beta) Terraform providers.
43+
Most Terraform resources are represented as [yaml files which are grouped by product](https://github.com/GoogleCloudPlatform/magic-modules/tree/master/mmv1/products).
44+
Each product has a `product.yaml` file (which defines the basic product information) and Resource.yaml files (which defines any resource-specific information).
45+
A Resource.yaml file can specify `include_in_tgc_next_DO_NOT_USE: true` to enable converters autogeneration, or `exclude_resource: true` to skip autogeneration for both converters and the providers.
46+
47+
Auto-generating converters code based on yaml files is strongly preferred.
48+
49+
{{< /tab >}}
50+
{{< tab "Handwritten" >}}
51+
52+
#### Handwritten converters
53+
If autogenerated converter are not possible, you can instead place three handwritten files in the specific Product folder in the [`mmv1/third_party/tgc_next/pkg/services` folder](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/tgc_next/pkg/services).
54+
55+
##### Resource.go file
56+
The Resource.go file defines the resource, providing its CAI asset type, Terraform schema name, resource schema, and supporting utility functions.
57+
58+
```golang
59+
// The type comes from https://cloud.google.com/asset-inventory/docs/supported-asset-types
60+
const ProductResourceAssetType string = "whatever.googleapis.com/asset-type"
61+
62+
// ProductResourceSchemaName is the TF resource schema name for the Resource resource within Product.
63+
const ProductResourceSchemaName string = "google_product_resource"
64+
65+
// Copy the resource schema from [google-beta](https://github.com/hashicorp/terraform-provider-google-beta) Terraform provider
66+
func ResourceGoogleProject() *schema.Resource {
67+
return &schema.Resource{
68+
SchemaVersion: 1,
69+
70+
Schema: map[string]*schema.Schema{
71+
...
72+
}
73+
}
74+
}
75+
```
76+
77+
You will also need to add an entry to [`tgc_next/provider/provider_mmv1_resources.go.tmpl`](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/templates/tgc_next/provider/provider_mmv1_resources.go.tmpl), which is used to generate [`pkg/provider/provider_mmv1_resources.go`](https://github.com/GoogleCloudPlatform/terraform-google-conversion/blob/main/pkg/provider/provider_mmv1_resources.go). Each entry in `provider_mmv1_resources.go.tmpl` maps a terraform resource name to a function that returns the resource schema - in this case:
78+
79+
```golang
80+
// ...
81+
"google_product_resource": product.ResourceName(),
82+
// ...
83+
```
84+
85+
##### Resource_tfplan2cai.go file
86+
Most resources will only need a resource converter with a conversion func. For example, Resource resource within Product, this might look like:
87+
88+
```golang
89+
func resourceConverterProductResource() ResourceConverter {
90+
return ResourceConverter{
91+
Convert: GetProductResourceCaiObject,
92+
}
93+
}
94+
95+
func GetProductResourceCaiObject(d TerraformResourceData, config *Config) ([]Asset, error) {
96+
// This function does basic conversion of a Terraform resource to a CAI Asset.
97+
// The asset path (name) will substitute in variables from the Terraform resource.
98+
// The format should match what is specified at https://cloud.google.com/asset-inventory/docs/supported-asset-types
99+
name, err := assetName(d, config, "//whatever.googleapis.com/projects/{{project}}/whatevers/{{name}}")
100+
if err != nil {
101+
return []Asset{}, err
102+
}
103+
if obj, err := GetProductResourceApiObject(d, config); err == nil {
104+
return []Asset{{
105+
Name: name,
106+
Type: ProductResourceAssetType,
107+
Resource: &AssetResource{
108+
Version: "v1", // or whatever the correct version is
109+
DiscoveryDocumentURI: "https://www.googleapis.com/path/to/rest/api/docs",
110+
DiscoveryName: "Whatever", // The term used to refer to this resource by the official documentation
111+
Data: obj,
112+
},
113+
}}, nil
114+
} else {
115+
return []Asset{}, err
116+
}
117+
}
118+
119+
func GetProductResourceApiObject(d TerraformResourceData, config *Config) (map[string]interface{}, error) {
120+
obj := make(map[string]interface{})
121+
122+
// copy values from the terraform resource to obj
123+
// return any errors encountered
124+
// ...
125+
126+
return obj, nil
127+
}
128+
129+
```
130+
131+
You will also need to add an entry to [`tgc_next/tfplan2cai/resource_converters.go.tmpl`](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/templates/tgc_next/tfplan2cai/resource_converters.go.tmpl), which is used to generate [`pkg/tfplan2cai/converters/resource_converters.go`](https://github.com/GoogleCloudPlatform/terraform-google-conversion/blob/main/pkg/tfplan2cai/converters/resource_converters.go). Each entry in `resource_converters.go.tmpl` maps a terraform resource name to a function that returns a ResourceConverter - in this case:
132+
133+
```golang
134+
// ...
135+
"google_product_resource": product.ResourceTfplan2caiConverter(),
136+
// ...
137+
```
138+
139+
##### Resource_cai2hcl.go file
140+
Most resources will only need a resource converter with a conversion func in the handwritten Resource_cai2hcl.go file. For the Resource resource within Product, this might look like:
141+
142+
```golang
143+
// ProductResourceConverter for the Resource resource within Product.
144+
type ProductResourceConverter struct {
145+
name string
146+
schema map[string]*schema.Schema
147+
}
148+
149+
// NewProductResourceConverter returns an HCL converter for compute instance.
150+
func NewProductResourceConverter(provider *schema.Provider) common.Converter {
151+
schema := provider.ResourcesMap[ProductResourceSchemaName].Schema
152+
153+
return &ProductResourceConverter{
154+
name: ProductResourceSchemaName,
155+
schema: schema,
156+
}
157+
}
158+
159+
// Convert converts asset to HCL resource blocks.
160+
func (c *ProductResourceConverter) Convert(assets []*caiasset.Asset) ([]*common.HCLResourceBlock, error) {
161+
var blocks []*common.HCLResourceBlock
162+
for _, asset := range assets {
163+
if asset == nil {
164+
continue
165+
}
166+
167+
if asset.Resource != nil && asset.Resource.Data != nil {
168+
block, err := c.convertResourceData(asset)
169+
if err != nil {
170+
return nil, err
171+
}
172+
blocks = append(blocks, block)
173+
}
174+
}
175+
return blocks, nil
176+
}
177+
178+
func (c *ProductResourceConverter) convertResourceData(asset *caiasset.Asset) (*common.HCLResourceBlock, error) {
179+
if asset == nil || asset.Resource == nil || asset.Resource.Data == nil {
180+
return nil, fmt.Errorf("asset resource data is nil")
181+
}
182+
183+
hclData := make(map[string]interface{})
184+
// copy values from the CAI asset to hclData
185+
// return any errors encountered
186+
// ...
187+
188+
ctyVal, err := common.MapToCtyValWithSchema(hclData, c.schema)
189+
if err != nil {
190+
return nil, err
191+
}
192+
return &common.HCLResourceBlock{
193+
Labels: []string{c.name, "Whatever"},
194+
Value: ctyVal,
195+
}, nil
196+
}
197+
198+
```
199+
200+
You will also need to add an entry to [`tgc_next/cai2hcl/resource_converters.go.tmpl`](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/templates/tgc_next/cai2hcl/resource_converters.go.tmpl), which is used to generate [`pkg/cai2hcl/converters/resource_converters.go`](https://github.com/GoogleCloudPlatform/terraform-google-conversion/blob/main/pkg/cai2hcl/converters/resource_converters.go). Each entry in `resource_converters.go.tmpl` maps a CAI asset type to a function that returns a ResourceConverter - in this case:
201+
202+
```golang
203+
// ...
204+
"alloydb.googleapis.com/Backup": {
205+
"Default": alloydb.NewAlloydbBackupCai2hclConverter(provider),
206+
},
207+
// ...
208+
```
209+
210+
{{< /tab >}}
211+
{{% /tabs %}}
212+
213+
### 2. Generate terraform-google-conversion
214+
To generate terraform-google-conversion code locally, run the following from the root of the `magic-modules` repository:
215+
216+
```
217+
make tgc OUTPUT_PATH="/path/to/your/terraform-google-conversion"
218+
```
219+
220+
### 3. Run tests locally
221+
#### Before you begin
222+
1. Set the following environment variable:
223+
```
224+
export WRITE_FILES=true
225+
```
226+
227+
2. Please request the necessary permission for the GCS bucket [cai_assets_metadata](https://pantheon.corp.google.com/storage/browser/cai_assets_metadata;tab=permissions?forceOnBucketsSortingFiltering=true&e=13802955&inv=1&invt=Ab0TPA&mods=ml_engine_alpha_ng2&project=tgc-test-infra&prefix=&forceOnObjectsSortingFiltering=false), which stores the nightly test metadata required to run the integration tests.
228+
229+
Ensure you have the correct Go version installed. Follow [Before you begin](https://googlecloudplatform.github.io/magic-modules/get-started/generate-providers/#before-you-begin) and [Setup your development environment](https://googlecloudplatform.github.io/magic-modules/develop/set-up-dev-environment/) from the [Magic Modules documentation](https://googlecloudplatform.github.io/magic-modules/get-started/generate-providers/).
230+
231+
#### Run unit tests
232+
To run the unit tests locally, run the following from the root of the `terraform-google-conversion` repository:
233+
234+
```
235+
make test
236+
```
237+
238+
#### Run integration tests
239+
240+
In the following examples, the resource being tested is `google_alloydb_backup`.
241+
242+
To run the integration tests for the added resource locally, run the following from the root of the `terraform-google-conversion` repository:
243+
```
244+
make test-integration-local TESTPATH=./test/services/alloydb TESTARGS='-run=TestAccAlloydbBackup' > alloydbBackup.log
245+
```
246+
247+
To run one integration test for the added resource locally, run the following from the root of the `terraform-google-conversion` repository:
248+
```
249+
make test-integration-local TESTPATH=./test/services/alloydb TESTARGS='-run=TestAccAlloydbBackup_alloydbBackupBasicTestExample' > alloydbBackup.log
250+
```
251+
252+
The core integration tests in `terraform-google-conversion` mirror the naming of the corresponding acceptance tests in the Terraform provider. This testing process uses a crucial round-trip validation method:
253+
254+
1. Test Input Data
255+
256+
The input for each equivalent integration test combines two elements:
257+
258+
* The original Terraform resource configuration from the acceptance test step.
259+
260+
* The Cloud Asset Inventory (CAI) data exported from the actual resources provisioned during the nightly execution of the Terraform provider tests.
261+
262+
2. Test File Generation (WRITE_FILES Flow)
263+
264+
By setting the environment variable WRITE_FILES, running the integration tests locally generates a series of files in the service folder, detailing the entire conversion cycle.
265+
266+
For example, running `TestAccAlloydbBackup_alloydbBackupBasicTestExample` successfully creates five files for each step:
267+
268+
| File | Content | Conversion Step |
269+
| :---------------------- | :------------------------------------------------------------ | :------------------- |
270+
| File 1 (.tf) | Original Terraform configuration (raw_config) | Start |
271+
| File 2 (.json) | CAI Assets exported from the resource (export_assets) | Input for CAI to HCL |
272+
| File 3 (export.tf) | Converted Terraform configuration (export_config) from File 2 | cai2hcl |
273+
| File 4 (roundtrip.json) | CAI Assets (roundtrip_assets) converted from File 3 | tfplan2cai |
274+
| File 5 (roundtrip.tf) | Final Terraform configuration (roundtrip_config) from File 4 | cai2hcl |
275+
276+
The integration tests pass only when two conditions are met:
277+
1. Every field in the original `raw_config` must exist within the generated `export_config`.
278+
2. The `export_config` and the final `roundtrip_config` must contain the same sets of fields.
279+
280+
#### Address integration test failures
281+
To resolve integration test failures, you need to apply the correct override configuration to the Resource.yaml file, based on the root cause of the data mismatch or failure.
282+
283+
* Test Specific Overrides
284+
285+
These rules are applied when a specific test cannot be reliably executed or fails due to expected differences in zero values.
286+
287+
* Ignore Zero Values (in raw_config)
288+
289+
Add `tgc_test_ignore_extra` to the resource example in Resource.yaml. This tells the test to ignore fields that have a zero value in the original Terraform configuration (raw_config) but retain a value in the exported CAI assets (export_assets).
290+
291+
* Skip the Entire Test
292+
293+
Add `tgc_skip_test:REASON` to the resource example in Resource.yaml, providing a brief explanation for why the test is being skipped.
294+
295+
* Resource-specific Overrides
296+
297+
These rules address mismatches in how the resource is identified or where it is converted.
298+
299+
* Fix CAI Resource Kind Mismatch
300+
301+
If the CAI asset type differs from the Terraform API resource kind (`api_resource_type_kind`), add the correct `cai_resource_kind` to the Resource.yaml file.
302+
303+
* Ignore Default Terraform Encoder
304+
305+
If the default Terraform encoding applied during conversion is causing issues, disable it by adding `tgc_ignore_terraform_encoder: true` to the Resource.yaml file.
306+
307+
* Custom Decoder (CAI → GET API object Mismatch)
308+
309+
This is used when the value of a field in a CAI asset is different from the value required in the final API object during cai2hcl.
310+
311+
1. Add the `tgc_decoder` file path to the field in Resource.yaml.
312+
313+
2. Implement the custom Go code in the corresponding file under `mmv1/templates/tgc_next/decoders`.
314+
315+
* Custom Encoder (CREATE API object -> CAI Mismatch)
316+
317+
This is used when the value of a field in a CAI asset is different from the value needed in the API object when the resource is first created during tfplan2cai.
318+
319+
1. Add the `tgc_encoder` file path to the field in Resource.yaml.
320+
321+
2. Implement the custom Go code in the corresponding file under `mmv1/templates/tgc_next/encoders`.
322+
323+
* Field-Specific Overrides
324+
325+
These rules are used when custom code is required because the field values or structure change during the tfplan2cai or cai2hcl conversion process.
326+
327+
* Ignore Field Missing in CAI
328+
329+
If a field is present in the Terraform resource but not supported or exported by Cloud Asset Inventory (CAI), add `is_missing_in_cai: true` to that specific field's definition in the Resource.yaml file.
330+
331+
* Ignore Default Terraform Custom Flatten
332+
333+
If the default flattening logic is incorrect for TGC, disable it by adding `tgc_ignore_terraform_custom_flatten: true` to the field definition.
334+
335+
* TGC Specific Expander / Flattener
336+
337+
For custom tfplan2cai conversion logic, add `custom_tgc_expand` to the field.
338+
339+
For custom cai2hcl conversion logic, add `custom_tgc_flatten` to the field.
340+
341+
### 4. Make PRs
342+
343+
Now that you have your code working locally, open a PR for [Magic Modules](https://github.com/GoogleCloudPlatform/magic-modules).
344+
345+
For the Magic Modules PR, check the build results within `presubmit-generate-diffs`. Click through the build links, and find the results of `tgc-test`, `tgc-test-integration-*` - the other checks only matter if you're also making changes to the terraform provider.
346+
347+
If `tgc-test` fails, make sure you can run the unit tests successfully locally. If any of `tgc-test-integration-*` fails, make sure you can run the integration tests successfully locally.

0 commit comments

Comments
 (0)