BE: Chore: Upgrade kafka to confluent 7.8.0

[yeikel]

Description

Before this change, we were using both Confluent Artifacts (via confluent.version) as well as the OSS artifacts (via kafka-clients.version and kafka.version ).

With this change, we are upgrading and aligning the versions to use the Confluent Kafka Distribution. This helps us not only to reduce the attack surface but also to simplify the dependency upgrades

As a side effect of this upgrade, we are also extending the retry logic to retry when there is a re-balance in progress as that was previously failing silently as described in https://issues.apache.org/jira/browse/KAFKA-14455

Closes #693
Closes #692

How Has This Been Tested? (put an "x" (case-sensitive!) next to an item)

• Unit checks
• Integration checks
• Covered by existing automation

Checklist (put an "x" (case-sensitive!) next to all the items, otherwise the build will fail)

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (e.g. ENVIRONMENT VARIABLES)
• My changes generate no new warnings (e.g. Sonar is happy)
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged

A picture of a cute animal (not mandatory but encouraged)

[Haarolean]

@yeikel this e2e test seem to be failing, could you take a look?
TopicsTest.retentionBytesAccordingToMaxSizeOnDiskCheck:413 » NoSuchElement No value present

[yeikel]

@yeikel this e2e test seem to be failing, could you take a look? TopicsTest.retentionBytesAccordingToMaxSizeOnDiskCheck:413 » NoSuchElement No value present

@Haarolean I updated the test.

The reason it was failing is because the page was returning 2 pages and the test assumed that all the properties are visible within the first page

[Haarolean]

@yeikel thank you very much for this PR!