[FLINK-36821] Update to Kafka 3.9.0

[tomncooper]

Currently, the Flink Kafka Connector uses Kafka client version 3.4.0.

This has a medium severity vulnerability (CVE-2024-31141) and needs to be upgraded to >= 3.8.0 to address it.

3.9.0 is the most recently released version and between that and 3.4.0 there have been numerous bug and performance fixes.

This PR:

• Updates the Kafka client version
• Updates the other dependencies, that are shared with Kafka, to be in line with the version used in Kafka 3.9.0 (if the update of the other deps should be done in a separate PR, let me know).
• Updates the Confluent Kafka and Zookeeper images to be in-line with the 3.9.0 Kafka versions.

I am not familiar with ArchUnit, but it appears the violation file changes have been checked into previous commits so I have included them here aswell.

[boring-cyborg]

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

[loserwang1024]

Updates the Confluent Kafka and Zookeeper images to be in-line with the 3.9.0 Kafka versions.

Whether the 3.9.0 version Kafka client is compability with the low version Kafka and zookeeper image?

[showuon]

Whether the 3.9.0 version Kafka client is compability with the low version Kafka and zookeeper image?

Yes, Kafka client 3.9.0 supports old version of Kafka and zookeeper image.

[tomncooper]

This PR might be superseded by #161 if we want to move to Kafka 4.0.0 for the connector's 4.0 release.

We could do a 3.5 release that has Flink 2.0 and Kafka 3.9 for those that need to retain access to very old kafka clusters (2.0 or earlier)?

[AHeise]

I'd prefer to have a release with 3.9 first. DataStream users are usually free to bump independent of this release anyways.

[AHeise]

CI is green. LGTM. Going to merge now.

[boring-cyborg]

Awesome work, congrats on your first merged pull request!