Skip to content

feat(managedkafka): Connect code samples #10143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

feat(managedkafka): Connect code samples #10143

wants to merge 6 commits into from

Conversation

salmany
Copy link

@salmany salmany commented Aug 4, 2025 •
edited
Loading

Description

This adds code samples for Managed Kafka Connect (MKC).

  • Also updated managedkafka/examples/pom.xml file to update library version and enable tests.

Fixes https://buganizer.corp.google.com/issues/430087669

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

  • I have followed Sample Format Guide
  • pom.xml parent set to latest shared-configuration
  • Appropriate changes to README are included in PR
  • These samples need a new API enabled in testing projects to pass (let us know which ones)
  • These samples need a new/updated env vars in testing projects set to pass (let us know which ones)
  • Tests pass: mvn clean verify required
  • Lint passes: mvn -P lint checkstyle:check required
  • Static Analysis: mvn -P lint clean compile pmd:cpd-check spotbugs:check advisory only
  • This sample adds a new sample directory, and I updated the CODEOWNERS file with the codeowners for this sample
  • This sample adds a new Product API, and I updated the Blunderbuss issue/PR auto-assigner with the codeowners for this sample
  • Please merge this PR for me once it is approved

@product-auto-label product-auto-label bot added api: managedkafka Issues related to the Google Cloud Managed Service for Apache Kafka API. samples Issues that are directly related to samples. labels Aug 4, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Aug 4, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @salmany, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

I've added a comprehensive set of Java code samples for Google Cloud's Managed Kafka Connect service. These samples demonstrate how to programmatically manage Kafka Connect clusters, including creation, retrieval, listing, updating, and deletion. Furthermore, I've included examples for creating and managing various types of connectors, such as BigQuery Sink, Cloud Storage Sink, MirrorMaker2, Pub/Sub Sink, and Pub/Sub Source, along with operations to pause, resume, restart, and stop them.

Highlights

  • New Managed Kafka Connect Samples: Introduced new Java code samples for Google Cloud Managed Kafka Connect, covering both cluster and connector management.
  • Connect Cluster Management: Provided samples for full lifecycle management of Managed Kafka Connect clusters, including creation, retrieval, listing, updating, and deletion.
  • Connector Management: Added diverse connector samples, demonstrating how to create BigQuery Sink, Cloud Storage Sink, MirrorMaker2, Pub/Sub Sink, and Pub/Sub Source connectors.
  • Connector Operations: Included samples for common connector operations like pausing, resuming, restarting, and stopping.
  • Dependency Update: Updated the pom.xml to change packaging from pom to jar and upgraded the libraries-bom dependency to version 26.64.0.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot reviewed Aug 4, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a comprehensive set of code samples for Managed Kafka Connect, covering cluster and connector lifecycle operations. The samples are well-structured and provide good coverage of the API. My review includes a high-severity comment about a potential bug in the BigQuery sink connector sample where a parameter is unused, and several medium-severity comments to address an unused import and improve logging output consistency across the new sample files.

@salmany salmany marked this pull request as ready for review August 4, 2025 17:53
@salmany salmany requested review from yoshi-approver and a team as code owners August 4, 2025 17:53
@snippet-bot Snippet Bot
Copy link

snippet-bot bot commented Aug 4, 2025 •
edited
Loading

Here is the summary of changes.

You are about to add 18 region tags.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

rQ-Qrr
rQ-Qrr reviewed Aug 6, 2025
View reviewed changes
managedkafka/examples/src/main/java/examples/CreateMirrorMaker2Connector.java Outdated
import java.util.HashMap;
import java.util.Map;

public class CreateMirrorMaker2Connector {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Please remove security related configs.
  2. offsetSyncsReplicationFactor is not a default param on the UI
  3. tasks.max=3 is a default param on the UI.
  4. It's worth mentioning that the target cluster is usually the primary cluster
  5. we need to add two files about MM2 checkpoint and MM2 heartbeat connecters.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the detailed feedback. I have implemented the suggested changes for items 1-4.

Regarding point 5, I agree that examples for the MM2 checkpoint and heartbeat connectors would be beneficial. For this pull request, I've prioritized the MM2 Source connector, as it is the only MM2 example featured in the public documentation.

Since the checkpoint and heartbeat connectors are optional and less commonly used, I will address the addition of these examples in a subsequent PR. I discussed with Prad who acknowledged this isn't high priority for now.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds good. Then could you change the name to CreateMirrorMaker2SourceConnector? Please also change it in python, go and tf code samples. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rQ-Qrr rQ-Qrr rQ-Qrr left review comments

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

@yoshi-approver yoshi-approver Awaiting requested review from yoshi-approver yoshi-approver is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
api: managedkafka Issues related to the Google Cloud Managed Service for Apache Kafka API. samples Issues that are directly related to samples.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@salmany @rQ-Qrr