Skip to content

Conversation

@xiangfu0
Copy link
Contributor

This PR removes the Kafka 2.0 stream ingestion plugin and migrates remaining Kafka realtime configs and references to Kafka 3.x. It also adds Kafka 3.x server/producer support and preserves backward compatibility by remapping old kafka20 class names to kafka30.

  • Remove pinot-kafka-2.0 module and packaging references
  • Switch default/quickstart configs, Helm/Docker examples, and UI templates to kafka30 factory
  • Add kafka30 KafkaDataServerStartable and KafkaDataProducer plus ServiceLoader entry
  • Map legacy kafka2/kafka20 classes to kafka30 in PluginManager with tests
  • Align Kafka client versions to Kafka 3

@xiangfu0 xiangfu0 force-pushed the kafka2-deprecation branch 3 times, most recently from 0ba9fc2 to 6bc674b Compare January 30, 2026 13:52
@apache apache deleted a comment from codecov-commenter Jan 30, 2026
@codecov-commenter
Copy link

codecov-commenter commented Jan 30, 2026

Codecov Report

❌ Patch coverage is 50.00000% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.14%. Comparing base (0a951fa) to head (c6c4c5c).
⚠️ Report is 5 commits behind head on master.

Files with missing lines Patch % Lines
...tream/kafka30/server/KafkaDataServerStartable.java 0.00% 6 Missing ⚠️
...lugin/stream/kafka30/server/KafkaDataProducer.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17602      +/-   ##
============================================
+ Coverage     63.11%   63.14%   +0.03%     
+ Complexity     1478     1474       -4     
============================================
  Files          3173     3168       -5     
  Lines        189915   189640     -275     
  Branches      29064    29024      -40     
============================================
- Hits         119872   119756     -116     
+ Misses        60711    60572     -139     
+ Partials       9332     9312      -20     
Flag Coverage Δ
custom-integration1 ?
integration 0.00% <ø> (-100.00%) ⬇️
integration1 ?
integration2 0.00% <ø> (ø)
java-11 63.13% <50.00%> (-36.87%) ⬇️
java-21 63.11% <50.00%> (+<0.01%) ⬆️
temurin 63.14% <50.00%> (+0.03%) ⬆️
unittests 63.14% <50.00%> (+0.03%) ⬆️
unittests1 55.51% <100.00%> (+0.02%) ⬆️
unittests2 33.98% <50.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Jackie-Jiang Jackie-Jiang added incompatible Indicate PR that introduces backward incompatibility kafka ingestion needs-attention Used for sensitive changes - allows searching PRs post release to narrow down causes for regression. labels Jan 30, 2026
@Jackie-Jiang
Copy link
Contributor

Should we consider keeping Kafka 2.0 support, but treat it as deprecated, i.e. no more support/fix for it?
Is there a strong reason why we want to remove the code?

@xiangfu0
Copy link
Contributor Author

Should we consider keeping Kafka 2.0 support, but treat it as deprecated, i.e. no more support/fix for it? Is there a strong reason why we want to remove the code?

Mainly increased binary size for the distribution and potential future vulnerabilities

@xiangfu0 xiangfu0 force-pushed the kafka2-deprecation branch 2 times, most recently from e857699 to 55c0ca7 Compare January 31, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

incompatible Indicate PR that introduces backward incompatibility ingestion kafka needs-attention Used for sensitive changes - allows searching PRs post release to narrow down causes for regression.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants