Skip to content

Releases: zio/zio-kafka

v3.3.0

24 Feb 23:01
205dd7a

Choose a tag to compare

πŸš€ Changes

This release upgrades the kafka library to 4.2.0.

🌱 Dependency updates

πŸ”§ Maintenance

v3.2.0

19 Nov 15:07
881813e

Choose a tag to compare

This is mostly a maintenance release with small bug fixes. It also upgrades the kafka client to 4.1.1 (compatible with most kafka brokers). However, if you care about partition assignors, please read on.

Many thanks for the bug reports and PRs ❀️

Note: version 3.1.1 never made it due to small binary compatibility problems. Use 3.2.0 instead.

πŸ‘¨πŸ½β€πŸ’» Looking for contributors

For the current maintainers zio-kafka is just for fun; we no longer use this project in our day jobs. If you find it challenging to work on this very interesting code base, please join us on Discord and we will help you where we can.

🚧 Partition assignor changes

If your application uses rebalance-safe-commits, AND it explicitly sets a partition assignor, please read this section. Otherwise, there should be no issues, and you can safely upgrade to 3.2.0.

After some bug reports we discovered that not all kafka's partition assignors can be used in all circumstances. Therefore, zio-kafka can now change the default partition assignor(s), and also disallow invalid partition assignor settings.

First of all, we discovered that kafka's CooperativeStickyAssignor should only be used in combination with rebalance-safe-commits, when the consumer is very very fast β€” it should be fast enough to process all pre-fetched records within the maximum rebalance time. Determining whether this is the case requires measuring your system and careful consideration of all settings. When this cannot be guaranteed, it is better to use the RangeAssignor. Therefore, when rebalance-safe-commits is enabled, zio-kafka changes the default partition assignors to [RangeAssignor]. If you wish, you can still set the partition assignors to something else, for example kafka's original default: [CooperativeStickyAssignor, RangeAssignor].

Secondly, we discovered that kafka's CooperativeStickyAssignor is incompatible with transactional producing. If you use transactional, zio-kafka now enforces that the CooperativeStickyAssignor is not used when the consumer is constructed.

For more information see #1597.

πŸš€ Changes

🌱 Dependency updates

πŸ”§ Maintenance

Full Changelog: v3.1.0...v3.2.0

v3.1.0

24 Sep 09:58
bcf78ec

Choose a tag to compare

A brand new release targeting Kafka java client 4.1 (which is compatible with almost all older Kafka brokers). In this release we also increase support for slow consumers in combination with rebalance-safe-commits mode. See the documentation or PR #1576 for more details. Special thanks to @Awethon for analyzing slow consumers in detail ❀️! These changes also benefit fast consumers that use rebalance-safe-commits as rebalances now take less time.

🚧 Breaking changes

  • Despite the minor upgrade of the Kafka java client, zio-kafka 3.1.0 is not binary compatible with the 4.0 Kafka java client.
  • The Kafka java client changed the Admin Client's API. These changes are mirrored in zio-kafka's wrapper and are not fully source (nor binary) compatible.

πŸš€ Changes

🌱 Dependency updates

  • Update zio, zio-streams, zio-test, ... to 2.1.20 by @zio-scala-steward[bot] in #1556
  • Update zio, zio-streams, zio-test, ... to 2.1.21 by @zio-scala-steward[bot] in #1565

πŸ”§ Maintenance

  • Update README.md by @zio-assistant[bot] in #1549
  • Update sbt, scripted-plugin to 1.11.3 by @zio-scala-steward[bot] in #1551
  • Update zio-logging-slf4j, ... to 2.5.1 by @zio-scala-steward[bot] in #1552
  • Test improvements by @erikvanoosten in #1554
  • Test improvements (follow up) by @erikvanoosten in #1555
  • Update sbt, scripted-plugin to 1.11.4 by @zio-scala-steward[bot] in #1557
  • Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.33 by @zio-scala-steward[bot] in #1559
  • Update sbt-native-packager to 1.11.3 by @zio-scala-steward[bot] in #1560
  • Update sbt, scripted-plugin to 1.11.5 by @zio-scala-steward[bot] in #1561
  • Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.34 by @zio-scala-steward[bot] in #1563
  • Update sbt, scripted-plugin to 1.11.6 by @zio-scala-steward[bot] in #1568
  • Reduce change of getting a port above 65535 by @erikvanoosten in #1581
  • Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.35 by @zio-scala-steward[bot] in #1582
  • Update README.md by @zio-assistant[bot] in #1583

Full Changelog: v3.0.0...v3.1.0

3.1.0 RC1 - Better pulling

20 Sep 14:04
1c93cdc

Choose a tag to compare

Pre-release

This is a sneak preview to zio-kafka 3.1.0 (requiring Kafka 4.1) with the better pulling #1576 included.

v3.0.0

08 Jun 09:01
d96d7b0

Choose a tag to compare

A brand new major release for zio-kafka: 3.0.0! πŸ₯³

Zio-kafka 3.0.0 brings new features:

  • a much easier to use transactional producer
  • producer diagnostics
  • producer metrics
  • graceful shutdown of a stream (experimental)
  • improved documentation

Zio-kafka 3.0.0 also brings a number of backwards incompatible changes:

  • Removal of all deprecated methods, including accessor methods
  • Kafka 4.x
  • The new transactional producer
  • Consumer diagnostics set via consumer settings
  • restartStreamOnRebalancing mode is no longer supported

See the migration guide. In short: you should always first migrate to zio-kafka 2.12.0 and solve all deprecation issues.

πŸš€ Changes since 2.12.0

πŸ”§ Maintenance since 2.12.0

  • Simplify ProducerSpec by @erikvanoosten in #1480
  • Simplify ConsumerSpec by @erikvanoosten in #1481
  • Update README.md by @zio-assistant in #1475
  • Update sbt-scalafix to 0.14.2 by @zio-scala-steward in #1478
  • Update scalafmt-core to 3.9.1 by @zio-scala-steward in #1479
  • Fix rebalance coordinator spec by @erikvanoosten in #1485
  • Update zio-logging-slf4j, ... to 2.5.0 by @zio-scala-steward in #1486
  • Update logback-classic to 1.5.17 by @zio-scala-steward in #1488
  • Update zio, zio-streams, zio-test, ... to 2.1.16 by @zio-scala-steward in #1490
  • Update scalafmt-core to 3.9.2 by @zio-scala-steward in #1493
  • Update sbt, scripted-plugin to 1.10.9 by @zio-scala-steward in #1495
  • Update sbt, scripted-plugin to 1.10.10 by @zio-scala-steward in #1496
  • Update scalafmt-core to 3.9.3 by @zio-scala-steward in #1497
  • Remove flaky test by @erikvanoosten in #1499
  • Update scalafmt-core to 3.9.4 by @zio-scala-steward in #1500
  • Update sbt, scripted-plugin to 1.10.11 by @zio-scala-steward in #1502
  • Update logback-classic to 1.5.18 by @zio-scala-steward in #1503
  • Update README.md by @zio-assistant in #1510
  • Update migration docs by @erikvanoosten in #1512
  • Only run benchmark on code changes by @erikvanoosten in #1511
  • Only run benchmark on code changes (2) by @erikvanoosten in #1514
  • Update zio, zio-streams, zio-test, ... to 2.1.17 by @erikvanoosten in #1517
  • Add low level unit test for produceAsync by @erikvanoosten in #1519
  • Followup of #1201 - make tests parallel again and remove non-flaky by @svroonland in #1521
  • Add comment motivating use of lazy in publisher by @ytalashko in #1527
  • Make all benchmarks report average time by @erikvanoosten in #1524
  • Update README.md by @zio-assistant in #1525
  • Update sbt-scalafix to 0.14.3 by @zio-scala-steward in #1532
  • Update scala3-compiler, scala3-library, ... to 3.3.6 by @zio-scala-steward in #1533
  • Update zio, zio-streams, zio-test, ... to 2.1.18 by @zio-scala-steward in #1534
  • More and better producer benchmarks by @erikvanoosten in #1531
  • Update sbt, scripted-plugin to 1.11.0 by @zio-scala-steward in #1535
  • Add new zio-kafka course by Alvin Alexander by @erikvanoosten in #1537
  • Update README.md by @zio-assistant in #1538
  • Update zio, zio-streams, zio-test, ... to 2.1.19 by @zio-scala-steward in #1539
  • Update embedded-kafka to 4.0.1.1 by @zio-scala-steward in #1540
  • Update sbt, scripted-plugin to 1.11.1 by @zio-scala-steward in #1541
  • Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.32 by @zio-scala-steward in #1544
  • Update sbt, scripted-plugin to 1.11.2 by @zio-scala-steward in #1547
  • Await acknowledgements in async benchmarks by @erikvanoosten in #1545

Changelog: v2.12.0...v3.0.0

πŸš€ Changes since 3.0.0-RC2

πŸ”§ Maintenance since 3.0.0-RC2

  • Add new zio-kafka course by Alvin Alexander by @erikvanoosten in #1537
  • Update README.md by @zio-assistant in #1538
  • Update zio, zio-streams, zio-test, ... to 2.1.19 by @zio-scala-steward in #1539
  • Update embedded-kafka to 4.0.1.1 by @zio-scala-steward in #1540
  • Update sbt, scripted-plugin to 1.11.1 by @zio-scala-steward in #1541
  • Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.4.0-alpha.32 by @zio-scala-steward in #1544
  • Update sbt, scripted-plugin to 1.11.2 by @zio-scala-steward in #1547
  • Await acknowledgements in async benchmarks by @erikvanoosten in #1545

Changelog: v3.0.0-rc2...v3.0.0

v3.0.0-rc2

25 May 09:17
6a06397

Choose a tag to compare

v3.0.0-rc2 Pre-release
Pre-release

This is the second and probably the final release candidate for zio-kafka 3.0.0! πŸ₯³

We expect this to be the final release before releasing 3.0.0. Please try it, your feedback is essential! Please find us on Discord, or create an issue on GitHub.

Zio-kafka 3.0.0 brings new features:

  • a much easier to use transactional producer
  • producer diagnostics
  • producer metrics
  • graceful shutdown of a stream (experimental)

Zio-kafka 3.0.0 also brings a number of backwards incompatible changes:

  • Removal of all deprecated methods, including accessor methods
  • Kafka 4.x
  • The new transactional producer
  • Diagnostics set via consumer settings
  • restartStreamOnRebalancing mode is no longer supported

Please also see the migration guide. In short: you should always first migrate to zio-kafka 2.12.0 and solve all deprecation issues.

πŸš€ Changes since 3.0.0-RC1

πŸ”§ Maintenance

  • Add comment motivating use of lazy in publisher by @ytalashko in #1527
  • Make all benchmarks report average time by @erikvanoosten in #1524
  • Update README.md by @zio-assistant in #1525
  • Update sbt-scalafix to 0.14.3 by @zio-scala-steward in #1532
  • Update scala3-compiler, scala3-library, ... to 3.3.6 by @zio-scala-steward in #1533
  • Update zio, zio-streams, zio-test, ... to 2.1.18 by @zio-scala-steward in #1534
  • More and better producer benchmarks by @erikvanoosten in #1531
  • Update sbt, scripted-plugin to 1.11.0 by @zio-scala-steward in #1535

Full Changelog: v3.0.0-rc1...v3.0.0-rc2

v3.0.0-rc1

21 Apr 11:13
e5fda14

Choose a tag to compare

v3.0.0-rc1 Pre-release
Pre-release

This is the first release candidate for zio-kafka 3.0.0! πŸ₯³

We expect this release to be production ready, but just to be sure we created a release candidate to allow everyone to try it out first. Your feedback is essential! Please find us on Discord, or create an issue on GitHub.

Zio-kafka 3.0.0 brings new features:

  • a much easier to use transactional producer
  • producer diagnostics
  • graceful shutdown of a stream (experimental)

Zio-kafka 3.0.0 also brings a number of backwards incompatible changes:

  • Removal of all deprecated methods, including accessor methods
  • Kafka 4.x
  • The new transactional producer
  • Diagnostics set via consumer settings
  • restartStreamOnRebalancing mode is no longer supported

Please also see the migration guide. In short: you should always first migrate to zio-kafka 2.12.0 and solve all deprecation issues.

What's Changed

Full Changelog: v2.11.0...v3.0.0-rc1

v2.12.0

23 Mar 11:07
93ac5f3

Choose a tag to compare

This release does some renaming in the consumer diagnostics API. The old names are deprecated and will be removed in zio-kafka 3.0.

ℹ️ This release is source compatible, but not binary compatible with zio-kafka 2.11.x.

πŸš€ Changes

πŸ”§ Maintenance

Full Changelog: v2.11.0...v2.12.0

v2.11.0

19 Feb 16:19
9353d70

Choose a tag to compare

This release follows ZIO's deprecation of accessor methods. If you are already using the ZIO service pattern, this should not affect you. If you are affected, you can follow our new migration-to-zio-kafka-3 guide to resolve all deprecation warnings.

This release also brings a small performance improvement by fetching group metadata only once after a rebalance (thanks @vermas7988!).

πŸš€ Changes

🌱 Dependency Updates

  • Update zio, zio-streams, zio-test, ... to 2.1.15 by @erikvanoosten in #1453
  • Update scala3-compiler, scala3-library, ... to 3.3.5 by @zio-scala-steward in #1447

πŸ”§ Maintenance

πŸ₯³ New Contributors

Full Changelog: v2.10.0...v2.11.0

v2.10.0

19 Jan 16:41
7318cc9

Choose a tag to compare

The rebalance-safe-commits mode is now also possible in combination with external commits, paving the way to making this mode enabled by default in zio-kafka 3.0.

πŸš€ Changes

🌱 Dependency Updates

  • Update README.md by @zio-assistant in #1436
  • Update scalafmt-core to 3.8.4 by @zio-scala-steward in #1435
  • Update sbt-scalafix to 0.14.0 by @zio-scala-steward in #1438
  • Update scala-library to 2.13.16 by @zio-scala-steward in #1440
  • Update scalafmt-core to 3.8.5 by @zio-scala-steward in #1441

Full Changelog: v2.9.1...v2.10.0