docs: Add MKC Samples for Creating Connectors #878

salmany · 2025-07-29T17:51:41Z

Adds Terraform samples for Managed Kafka Connect
to create connectors for BigQuery, Cloud Storage, MirrorMaker,
Pub/Sub Sink, and Pub/Sub Source.

Builds upon #876 and requires a fix for subnet resource (see here).

Description

Fixes https://b.corp.google.com/issues/430087669

Note: If you are not associated with Google, open an issue for discussion before submitting a pull request.

Checklist

Readiness

Yes, merge this PR after it is approved
No, don't merge this PR after it is approved

Style

My sample follows the rules described for Terraform in the Effective Samples style guide
My sample follows the other requirements and best practices in the Contributing
guide

Testing

I have performed tests described in the Contributing guide:
- Tests pass: terraform apply
- Lint pass: terraform fmt check

Intended location

Yes, this sample will be (or already is) included on cloud.google.com
Location(s):

No, this sample won't be included on cloud.google.com
Reason:

API enablement

If the sample needs an API enabled to pass testing, I have added the service to the Test setup file

Review

If this sample adds a new directory, I have added codeowners to the CODEOWNERS file

for Clusters * Add sample code for creating a Kafka Connect cluster

Corrected tags.

for Connectors Adds Terraform samples for Managed Kafka Connect to create connectors for BigQuery, Cloud Storage, MirrorMaker, Pub/Sub Sink, and Pub/Sub Source.

snippet-bot · 2025-07-29T19:34:32Z

Here is the summary of changes.

You are about to add 15 region tags.

managedkafka/managedkafka_create_connector_bigquery_sink/main.tf:17, tag managedkafka_create_connector_bigquery_sink_parent
managedkafka/managedkafka_create_connector_bigquery_sink/main.tf:60, tag managedkafka_subnetwork
managedkafka/managedkafka_create_connector_bigquery_sink/main.tf:74, tag managedkafka_create_connector_bigquery_sink
managedkafka/managedkafka_create_connector_cloud_storage_sink/main.tf:17, tag managedkafka_create_connector_pubsub_source_parent
managedkafka/managedkafka_create_connector_cloud_storage_sink/main.tf:60, tag managedkafka_subnetwork
managedkafka/managedkafka_create_connector_cloud_storage_sink/main.tf:74, tag managedkafka_create_connector_cloud_storage_sink
managedkafka/managedkafka_create_connector_mirrormaker2_source/main.tf:17, tag managedkafka_create_connector_mirrormaker2_source_parent
managedkafka/managedkafka_create_connector_mirrormaker2_source/main.tf:90, tag managedkafka_subnetwork
managedkafka/managedkafka_create_connector_mirrormaker2_source/main.tf:104, tag managedkafka_create_connector_mirrormaker2_source
managedkafka/managedkafka_create_connector_pubsub_sink/main.tf:17, tag managedkafka_create_connector_pubsub_sink_parent
managedkafka/managedkafka_create_connector_pubsub_sink/main.tf:60, tag managedkafka_subnetwork
managedkafka/managedkafka_create_connector_pubsub_sink/main.tf:74, tag managedkafka_create_connector_pubsub_sink
managedkafka/managedkafka_create_connector_pubsub_source/main.tf:17, tag managedkafka_create_connector_pubsub_source_parent
managedkafka/managedkafka_create_connector_pubsub_source/main.tf:60, tag managedkafka_subnetwork
managedkafka/managedkafka_create_connector_pubsub_source/main.tf:74, tag managedkafka_create_connector_pubsub_source

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

glasnt

Is this a replacement for #876? It conflicts with that sample.

managedkafka/managedkafka_create_connector_bigquery_sink/main.tf

managedkafka/managedkafka_create_connect_cluster/main.tf

glasnt · 2025-08-07T00:24:24Z

#876 has been merged, which conflicts with this branch. Please rebase, and apply suggested comments around local-exec and test skips ✨

salmany · 2025-08-07T17:13:15Z

Rebased and applied mitigation for subnet resource destruction issue.

managedkafka/managedkafka_create_connector_mirrormaker_source/main.tf

managedkafka/managedkafka_create_connector_mirrormaker2_source/main.tf

glasnt · 2025-08-14T22:13:42Z

managedkafka/managedkafka_create_connector_bigquery_sink/main.tf

+    "name"                           = "my-bigquery-sink-connector"
+    "project"                        = "GCP_PROJECT_ID"


These look like repeated values from the main resource args. If that's the case, you can use the dynamic project ID rather than hardcoding it here.

Suggested change

"name" = "my-bigquery-sink-connector"

"project" = "GCP_PROJECT_ID"

"name" = "my-bigquery-sink-connector"

"project" = data.google_project.default.project_id

glasnt · 2025-08-14T22:14:04Z

managedkafka/managedkafka_create_connector_pubsub_source/main.tf

+    "tasks.max"        = "3"
+    "kafka.topic"      = "GMK_TOPIC_ID"
+    "cps.subscription" = "CPS_SUBSCRIPTION_ID"
+    "cps.project"      = "GCP_PROJECT_ID"


Same again here

Suggested change

"cps.project" = "GCP_PROJECT_ID"

"cps.project" = data.google_project.default.project_id

glasnt

One small best practice addition, and these are okay to merge.

Follow up to #878

Follow up to #878, minor fix for hardcoded project IDs.

…es#878) * docs: Add Managed Kafka Connect terraform sample for Clusters * Add sample code for creating a Kafka Connect cluster * Update tags in main.tf Corrected tags. * Addressed PR comments. * Add comment to clarify memory_bytes value. * Fix incorrect Kafka cluster reference. * docs: Add Managed Kafka Connect terraform sample for Connectors Adds Terraform samples for Managed Kafka Connect to create connectors for BigQuery, Cloud Storage, MirrorMaker, Pub/Sub Sink, and Pub/Sub Source. * Fix typo in memory_bytes for connectors * Fixed formatting. * Fix formatting * Fix snippet tags. * Fix tags * Adding workaround for subnet resource deletion. * Fixed errors and formatting. * Disable tests and modify subnet resources. * Updated connector sample configs. * Fix formatting. * Add comment for MM connector target cluster. * Change MM2 example to include 2 GMK clusters. * Fix connector name to match id. * Fix connector bootstrap server addresses. * Fix bootstrap server addresses and add comment. * Fix whitespaces and formatting. * Add newline. * Add comments for access to MM2 clusters and other fixes. * Formatting fix. * Applying suggestions. --------- Co-authored-by: Katie McLaughlin <[email protected]>

…le-modules#887) Follow up to terraform-google-modules#878, minor fix for hardcoded project IDs.

salmany and others added 7 commits July 23, 2025 18:10

docs: Add Managed Kafka Connect terraform sample

16d7b2f

for Clusters * Add sample code for creating a Kafka Connect cluster

Update tags in main.tf

163059e

Corrected tags.

Merge branch 'main' into salman-mkc-code-examples

0c1763e

Addressed PR comments.

dec3c5a

Add comment to clarify memory_bytes value.

0886c4e

Fix incorrect Kafka cluster reference.

315b503

docs: Add Managed Kafka Connect terraform sample

be446e6

for Connectors Adds Terraform samples for Managed Kafka Connect to create connectors for BigQuery, Cloud Storage, MirrorMaker, Pub/Sub Sink, and Pub/Sub Source.

salmany changed the base branch from salman-mkc-code-examples to main July 29, 2025 18:00

salmany added 3 commits July 29, 2025 18:08

Fix typo in memory_bytes for connectors

ad846d7

Fixed formatting.

9599c11

Fix formatting

7188e27

salmany marked this pull request as ready for review July 29, 2025 19:34

salmany requested review from a team as code owners July 29, 2025 19:34

salmany and others added 5 commits July 29, 2025 15:34

Merge branch 'main' into mkc-connector-examples

4fdbd4c

Fix snippet tags.

30dd507

Fix tags

274f08f

Adding workaround for subnet resource deletion.

8a162d8

Fixed errors and formatting.

5a99b76

glasnt added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Aug 1, 2025

glasnt requested changes Aug 1, 2025

View reviewed changes

managedkafka/managedkafka_create_connector_bigquery_sink/main.tf Outdated Show resolved Hide resolved

managedkafka/managedkafka_create_connect_cluster/main.tf Outdated Show resolved Hide resolved

salmany requested a review from glasnt August 5, 2025 00:09

salmany added 4 commits August 7, 2025 15:27

Merge branch 'main' into mkc-connector-examples

79d5fc0

Disable tests and modify subnet resources.

34292bd

Updated connector sample configs.

bf50178

Fix formatting.

4217bbe

Add comment for MM connector target cluster.

7887a02

rQ-Qrr reviewed Aug 8, 2025

View reviewed changes

managedkafka/managedkafka_create_connector_mirrormaker_source/main.tf Outdated Show resolved Hide resolved

managedkafka/managedkafka_create_connector_mirrormaker_source/main.tf Outdated Show resolved Hide resolved

salmany added 2 commits August 11, 2025 18:27

Change MM2 example to include 2 GMK clusters.

2094920

Fix connector name to match id.

9b249fe

salmany requested a review from rQ-Qrr August 11, 2025 18:31

salmany added 4 commits August 11, 2025 20:18

Fix connector bootstrap server addresses.

7970c79

Fix bootstrap server addresses and add comment.

080f2b0

Fix whitespaces and formatting.

4571dd6

Add newline.

91d48c7

glasnt removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Aug 11, 2025

rQ-Qrr reviewed Aug 12, 2025

View reviewed changes

salmany added 2 commits August 13, 2025 19:41

Add comments for access to MM2 clusters and other fixes.

ea14a5a

Formatting fix.

4aa8823

rQ-Qrr approved these changes Aug 13, 2025

View reviewed changes

Applying suggestions.

f28c331

salmany enabled auto-merge (squash) August 14, 2025 14:22

glasnt reviewed Aug 14, 2025

View reviewed changes

glasnt approved these changes Aug 14, 2025

View reviewed changes

salmany merged commit 58da4e9 into main Aug 14, 2025
10 checks passed

glasnt added a commit that referenced this pull request Aug 14, 2025

fix(managedkafka): use data resources for project IDs

82732ff

Follow up to #878

glasnt mentioned this pull request Aug 14, 2025

fix(managedkafka): use data resources for project IDs #887

Merged

1 task

salmany pushed a commit that referenced this pull request Aug 14, 2025

fix(managedkafka): use data resources for project IDs (#887)

acfbcb1

Follow up to #878, minor fix for hardcoded project IDs.

niharika-98 pushed a commit to niharika-98/terraform-docs-samples that referenced this pull request Sep 7, 2025

fix(managedkafka): use data resources for project IDs (terraform-goog…

514ee6b

…le-modules#887) Follow up to terraform-google-modules#878, minor fix for hardcoded project IDs.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs: Add MKC Samples for Creating Connectors #878

docs: Add MKC Samples for Creating Connectors #878

Uh oh!

salmany commented Jul 29, 2025 •

edited

Loading

Uh oh!

snippet-bot bot commented Jul 29, 2025 •

edited

Loading

Uh oh!

glasnt left a comment

Uh oh!

Uh oh!

Uh oh!

glasnt commented Aug 7, 2025

Uh oh!

salmany commented Aug 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

glasnt Aug 14, 2025

Uh oh!

glasnt Aug 14, 2025

Uh oh!

glasnt left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		"name" = "my-bigquery-sink-connector"
		"project" = "GCP_PROJECT_ID"

	"cps.project" = "GCP_PROJECT_ID"
	"cps.project" = data.google_project.default.project_id

docs: Add MKC Samples for Creating Connectors #878

docs: Add MKC Samples for Creating Connectors #878

Uh oh!

Conversation

salmany commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

snippet-bot bot commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

glasnt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

glasnt commented Aug 7, 2025

Uh oh!

salmany commented Aug 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

glasnt Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

glasnt Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

glasnt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

salmany commented Jul 29, 2025 •

edited

Loading

snippet-bot bot commented Jul 29, 2025 •

edited

Loading