Kafka Consumer Group Properties #34334

talatuyarer · 2025-03-18T17:59:08Z

BeamSQl's Create Table Statement does not support consumer properties. This PR aims to add that support on KafkaTableProvider for all KafkaTable

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

ahmedabu98

LGTM
Just need to check this works for Java and Python

ahmedabu98 · 2025-03-18T18:46:08Z

...src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java

    }
+
+    if (!configUpdates.isEmpty()) {
+      kafkaTable.updateConsumerProperties(configUpdates);


Interestingly this previously was an unused method

github-actions · 2025-03-18T20:36:09Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

* Expose add-modules JVM option in SDKHarnessOptions * add changes.md * Update CHANGES.md Co-authored-by: tvalentyn <[email protected]> * Fix whitespace * fix whitespace --------- Co-authored-by: tvalentyn <[email protected]>

* Refactor and improve Specifiable. - Better typehinting in specifiable with overloads - Move non-method attributes and staticmethod outside of Specifiable Protocol - Make spec_type a function which returns a class-specific spec_type variable - Add tests for some uncommon use cases. * Fix static function names and use pass for empty functions per review. * Add a warning for unsupported types in spec.

Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.26.0 to 0.28.0. - [Commits](golang/oauth2@v0.26.0...v0.28.0) --- updated-dependencies: - dependency-name: golang.org/x/oauth2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Support directory separator for Python local filesystem * fixed the tests

…e#34029) Bumps [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) from 1.39.0 to 1.39.1. - [Release notes](https://github.com/nats-io/nats.go/releases) - [Commits](nats-io/nats.go@v1.39.0...v1.39.1) --- updated-dependencies: - dependency-name: github.com/nats-io/nats.go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add offline detctors. * Fix typo.

Co-authored-by: Naireen <[email protected]>

…nality.

…e timer data + liveness (apache#34292)

…pache#34335) * Remove cancelled tasks from ReadOperation queue when shutting down * Try shutdowNow * spotless

…combinatorics (apache#34132) * [Prism] Refactor stageState to a behavior interface to reduce branch combinatorics * Revert on demand pending change. * revert attempt at lock avoidance, causes hangs. --------- Co-authored-by: lostluck <[email protected]>

* More precise binary operation inference. Handle numeric promotion, sequence multiplication, and fix division for Python 3.11+. * Fix older python versions. They have the opcode module, but no _nb_ops. * Restore fix for 3.9 and 3.10. * yapf

* Fix Precommit Flink Container * Fix image tag in the cleanup step * Add beam-sdk-pr to public repos

…apache#34094) Bumps [github.com/fsouza/fake-gcs-server](https://github.com/fsouza/fake-gcs-server) from 1.52.1 to 1.52.2. - [Release notes](https://github.com/fsouza/fake-gcs-server/releases) - [Commits](fsouza/fake-gcs-server@v1.52.1...v1.52.2) --- updated-dependencies: - dependency-name: github.com/fsouza/fake-gcs-server dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ache#34267) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.23.0 to 0.36.0. - [Commits](golang/net@v0.23.0...v0.36.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…che#34336)

…4293) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.221.0 to 0.226.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.221.0...v0.226.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…3086) This is accomplished by changing to iterators which throw away elements that have been passed as well as just referring to the inputstream of the element instead of the entire stream. If the ByteString is based upon a ByteBuffer this will allow blocks that have been advanced past to be gc'd.

…spaces (apache#34359)

…4356) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.226.0 to 0.227.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.226.0...v0.227.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

apache#34358) Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.10.25 to 2.11.0. - [Release notes](https://github.com/nats-io/nats-server/releases) - [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml) - [Commits](nats-io/nats-server@v2.10.25...v2.11.0) --- updated-dependencies: - dependency-name: github.com/nats-io/nats-server/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…pache#34264) * remove already committed files * changes * simplify validation * validate without loading the whole collection into memory

* file staging * add tests * help message * format * typo * yapf * docs * support for multiple files * changes.md * fix test

talatuyarer · 2025-03-20T18:35:39Z

@ahmedabu98 I want to close this. Somehow It gets all master commits too. I created new pr. #34366

Kafka Consumer Group Properties

6bbe0bd

github-actions bot added java extensions sql labels Mar 18, 2025

Need to remove properies prefix from properties to apply on Kafka

4241b31

ahmedabu98 approved these changes Mar 18, 2025

View reviewed changes

Abacn and others added 23 commits March 20, 2025 11:22

Support directory separator for Python local filesystem (apache#34318)

c3c10d2

* Support directory separator for Python local filesystem * fixed the tests

Make yaml provider an abstract class.

09e9993

Add support for python package dependencies on UDFs.

1c0a362

Add dependency support for delegating providers.

7126636

Allow dependency declarations for java.

a47fa9b

Make yapf, lint, and mypy happy, fix test.

56b3ff6

[AnomalyDetection] Support offline detectors (apache#34311)

c088abe

* Add offline detctors. * Fix typo.

add option to disable metrics (apache#34303)

00d9f00

Co-authored-by: Naireen <[email protected]>

initial schema transform provider implementation for SQS read functio…

b7ec790

…nality.

fixing checkstyle errors.

e0df796

removing region configuration since its not needed

ea7980f

adding package info for the new providers

db4c87d

added tests to the provider

2392a7d

[Dataflow Streaming] WindmillTimerInternals: Use a single map to stor…

76a1374

…e timer data + liveness (apache#34292)

add equals hashCode to BoundedToUnboundedSourceAdapter (apache#34057)

16fb2fb

Remove cancelled tasks from ReadOperation queue when shutting down (a…

fbead7f

…pache#34335) * Remove cancelled tasks from ReadOperation queue when shutting down * Try shutdowNow * spotless

Fix precommit flink container (apache#34250)

d24cbf6

* Fix Precommit Flink Container * Fix image tag in the cleanup step * Add beam-sdk-pr to public repos

dependabot bot and others added 6 commits March 20, 2025 11:22

Fix pandas doctests sensitive to NumpyExtensionArray formatting. (apa…

6c156c5

…che#34336)

Fix relative path yaml provider error when no base provided.

65fab8c

Fixed infinitive loop

ded696c

github-actions bot added python go build infra docker io runners learning dataflow aws kafka typescript yaml prism labels Mar 20, 2025

scwhittle and others added 7 commits March 20, 2025 11:24

Remove finalizers from sdb-cluster before deleting singlestoreio name…

3bd8e02

…spaces (apache#34359)

[IcebergIO] Filter out data files that have already been committed (a…

a90e411

…pache#34264) * remove already committed files * changes * simplify validation * validate without loading the whole collection into memory

[Python] File staging to user worker support (apache#34208)

b7d6e88

* file staging * add tests * help message * format * typo * yapf * docs * support for multiple files * changes.md * fix test

Fix dockerfile version (apache#34352)

e96dff4

github-actions bot added the examples label Mar 20, 2025

talatuyarer closed this Mar 20, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Kafka Consumer Group Properties #34334

Kafka Consumer Group Properties #34334

Uh oh!

talatuyarer commented Mar 18, 2025

Uh oh!

ahmedabu98 left a comment

Uh oh!

ahmedabu98 Mar 18, 2025

Uh oh!

github-actions bot commented Mar 18, 2025

Uh oh!

talatuyarer commented Mar 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

Kafka Consumer Group Properties #34334

Kafka Consumer Group Properties #34334

Uh oh!

Conversation

talatuyarer commented Mar 18, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 18, 2025

Uh oh!

talatuyarer commented Mar 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants