feat: google_datastream_private_connection resource network_attachement support via psc_interface_config

[ramonvermeulen]

Closes hashicorp/terraform-provider-google#22759

Release Note Template for Downstream PRs (will be copied)

See Write release notes for guidance.

datastream: `google_datastream_private_connection` resource `network_attachement` support via `psc_interface_config` attribute

[github-actions]

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@BBBmau, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

[ramonvermeulen]

@shuyama1 forgot to open as draft so downgraded the PR to draft again but it already assigned a reviewer.

Will put on Ready for review later this week, after I implemented and ran proper acceptance tests.

[ramonvermeulen]

@shuyama1 FYI

When running the acceptance test TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterface locally, after polling for about ~15 minutes, I run into the following error:

2025/05/16 08:44:57 [DEBUG] Got done: false while polling for operation projects/XXX/locations/us-central1/operations/operation-1747377065177-6353aed42309e-37036802-278e22bb's status
2025/05/16 08:45:07 [DEBUG] Waiting for state to become: [success]
2025/05/16 08:45:07 [DEBUG] Retry Transport: starting RoundTrip retry loop
2025/05/16 08:45:07 [DEBUG] Retry Transport: request attempt 0
2025/05/16 08:45:07 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
GET /v1/projects/XXX/locations/us-central1/operations/operation-1747377065177-6353aed42309e-37036802-278e22bb?alt=json HTTP/1.1
Host: datastream.googleapis.com
User-Agent: Terraform/1.11.3 (+https://www.terraform.io) Terraform-Plugin-SDK/2.36.0 terraform-provider-google/acc
Content-Type: application/json
Accept-Encoding: gzip


-----------------------------------------------------
2025/05/16 08:45:07 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: application/json; charset=UTF-8
Date: Fri, 16 May 2025 06:45:07 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "name": "projects/XXX/locations/us-central1/operations/operation-1747377065177-6353aed42309e-37036802-278e22bb",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.datastream.v1.OperationMetadata",
    "createTime": "2025-05-16T06:31:05.369507791Z",
    "target": "projects/XXX/locations/us-central1/privateConnections/tf-test-my-connectionpctvro2doo",
    "verb": "create",
    "requestedCancellation": false,
    "apiVersion": "v1"
  },
  "done": false
}

-----------------------------------------------------
2025/05/16 08:45:07 [DEBUG] Retry Transport: Stopping retries, last request was successful
2025/05/16 08:45:07 [DEBUG] Retry Transport: Returning after 1 attempts
2025/05/16 08:45:07 [DEBUG] Got done: false while polling for operation projects/XXX/locations/us-central1/operations/operation-1747377065177-6353aed42309e-37036802-278e22bb's status
2025/05/16 08:45:17 [DEBUG] Waiting for state to become: [success]
2025/05/16 08:45:17 [DEBUG] Retry Transport: starting RoundTrip retry loop
2025/05/16 08:45:17 [DEBUG] Retry Transport: request attempt 0
2025/05/16 08:45:17 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
GET /v1/projects/XXX/locations/us-central1/operations/operation-1747377065177-6353aed42309e-37036802-278e22bb?alt=json HTTP/1.1
Host: datastream.googleapis.com
User-Agent: Terraform/1.11.3 (+https://www.terraform.io) Terraform-Plugin-SDK/2.36.0 terraform-provider-google/acc
Content-Type: application/json
Accept-Encoding: gzip


-----------------------------------------------------
2025/05/16 08:45:18 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: application/json; charset=UTF-8
Date: Fri, 16 May 2025 06:45:18 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "name": "projects/XXX/locations/us-central1/operations/operation-1747377065177-6353aed42309e-37036802-278e22bb",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.datastream.v1.OperationMetadata",
    "createTime": "2025-05-16T06:31:05.369507791Z",
    "endTime": "2025-05-16T06:45:13.221555471Z",
    "target": "projects/XXX/locations/us-central1/privateConnections/tf-test-my-connectionpctvro2doo",
    "verb": "create",
    "requestedCancellation": false,
    "apiVersion": "v1"
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.datastream.v1.PrivateConnection",
    "name": "projects/XXX/locations/us-central1/privateConnections/tf-test-my-connectionpctvro2doo",
    "createTime": "2025-05-16T06:31:05.365495471Z",
    "updateTime": "2025-05-16T06:31:05.365495471Z",
    "labels": {
      "goog-terraform-provisioned": "true",
      "key": "value"
    },
    "displayName": "Connection profile",
    "state": "FAILED",
    "error": {
      "reason": "UNKNOWN",
      "message": "An unknown error occurred. Please try again. If the error persists, contact Google support.",
      "errorTime": "2025-05-16T06:45:12.951474Z",
      "details": {
        "originalMessage": ""
      }
    },
    "pscInterfaceConfig": {
      "networkAttachment": "projects/XXX/regions/us-central1/networkAttachments/tf-test-my-network-attachmentpctvro2doo"
    }
  }
}

When looking in the UI just before the error (when it is polling the Operation), at least the network attachment seems to be created succesful:

It is a bit hard to understand what the underlaying issue is, since the API only returns An unknown error occurred. Please try again. If the error persists, contact Google support.

For the network attachement, I took the google_compute_network_attachment as example, see example here.

Upgrading to Ready for review to see if the same issue occurs in the CI.

Maybe the network attachment should only have accepted producers?

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 310 insertions(+), 40 deletions(-))
google-beta provider: Diff ( 4 files changed, 310 insertions(+), 40 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 29 insertions(+))

[modular-magician]

Tests analytics

Total tests: 15
Passed tests: 6
Skipped tests: 8
Affected tests: 1

Click here to see the affected service packages

• datastream

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterfaceExample

Get to know how VCR tests work

[modular-magician]

🔴 Tests failed during RECORDING mode:
TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterfaceExample [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

[shuyama1]

Okay, test failed with the following error

Error: Error waiting for PrivateConnection "projects/ci-test-project-188019/locations/us-central1/privateConnections/tf-test-my-connectionpbg7rdhitb" to be CREATED. "Failed to create PrivateConnection. An unknown error occurred. Please try again. If the error persists, contact Google support. details = {\"originalMessage\":\"\"}"
        
          with google_datastream_private_connection.default,
          on terraform_plugin_test.tf line 2, in resource "google_datastream_private_connection" "default":
           2: resource "google_datastream_private_connection" "default" {

This appears to match the error you saw locally.

[shuyama1]

I'll look into the issue today

[ramonvermeulen]

I'll look into the issue today

Thanks! One thing that I tried is to use a google_compute_network_attachment with only projects in the producer_accept_lists configured, but that resulted in the same error.

Might go over the documentation on psc interfaces for datastream later this week, to see if I miss something.

[ramonvermeulen]

@shuyama1

Test is now passing locally with 33dde3a, had to do with the connection_preference:

~/go/src/github.com/hashicorp/terraform-provider-google  main [$!⇣]                                                                                                                                            v1.23.4 󱇶 default took 1m41s
❯ make testacc TEST=./google/services/datastream TESTARGS='-run=TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterface'

TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google/services/datastream -v -run=TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterface -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google/version.ProviderVersion=acc"
=== RUN   TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterfaceExample
=== PAUSE TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterfaceExample
=== CONT  TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterfaceExample
--- PASS: TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterfaceExample (1222.48s)
PASS
ok      github.com/hashicorp/terraform-provider-google/google/services/datastream       1223.143s

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 238 insertions(+), 40 deletions(-))
google-beta provider: Diff ( 4 files changed, 238 insertions(+), 40 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 29 insertions(+))

[modular-magician]

Tests analytics

Total tests: 15
Passed tests: 6
Skipped tests: 8
Affected tests: 1

Click here to see the affected service packages

• datastream

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccDatastreamPrivateConnection_datastreamPrivateConnectionPscInterfaceExample

Get to know how VCR tests work

[ramonvermeulen]

@ramonvermeulen are we going to submit this PR ? I also have similar PR which is open. please LMK.

I am waiting for a review of @shuyama1, unless there are still things required to be changed before it is ready in your opinion?

[yair-harel]

@shuyama1 can you please review this CL ?

[BharathPrabh]

@shuyama1 can we close this more quickly please

[ramonvermeulen]

@BBBmau @rileykarson @melinath @ScottSuarez

Tagging for visibility, since there seems quite some demand for this one from the Datastream team.

[rileykarson]

@modular-magician reassign-reviewer

[mrcarter7]

Is there an updated version of the docs here https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/datastream_private_connection?

[ramonvermeulen]

Is there an updated version of the docs here https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/datastream_private_connection?

I think this will be updated with the v6.38.0 release, which will probably be somewhere in the upcoming week(s).

See documentation diff at the merged downstream PR to the provider:
https://github.com/hashicorp/terraform-provider-google/pull/23091/files#diff-b20b8afcf3517e9229b7d83214bd6ed6f8ac4c2023de9d8014c4a1e06cdfe08c