Skip to content

Conversation

@geso02
Copy link
Contributor

@geso02 geso02 commented Nov 11, 2024

Closes #522

Description:

This pull request introduces an Apache Kafka event interface, enhancing message processing reliability and scalability, and making the system more suitable for larger and more demanding environments.

Key Features

  • Kafka Integration: Adds Apache Kafka as a message broker, allowing both repositories and registries to utilize Kafka for event processing.
  • Commit-Based Acknowledgments: Implements Kafka’s commit-based message acknowledgment to ensure reliable and confirmed message processing.
  • Documentation: Updated README files with detailed instructions on setting up and configuring the Kafka event interface.
  • Unit Tests: Comprehensive unit tests are included to verify the functionality and reliability of the Kafka integration, ensuring robustness under various conditions.

Thank you for reviewing this addition!

@aaronzi
Copy link
Member

aaronzi commented Nov 12, 2024

Hi @geso02,

thank you very much for your PR and the nice addition to BaSyx. Can you please check the AAS Environment. There is a test that seems to be failing.

- Corrected automatic upload of files by ensuring paths are annotated with @value, so the correct files are uploaded as expected.
- Removed the `basyx.environment` property from tests, as it is unrelated to the test scope and streamlines the configuration.
@geso02
Copy link
Contributor Author

geso02 commented Nov 14, 2024

I found some incorrect configurations in the upload within the AASEnvironment. The pathsToLoad used during bean creation was missing the @value annotation in all places. I've now added it. In the authorization test case, we then encountered an issue because basyx.environment was now being referenced, but this test was not concerned with it and did not provide mocks for it, causing it to fail. I've now pushed the changes again and will check if there are issues in other areas as well.

This commit resolves an issue where a Submodel, which should only be used in submodel-repository tests, was mistakenly registered in environment-related tests.

The Submodel is now conditionally provided via profiles, ensuring it is only available in the appropriate tests and no longer causes failures in the environment tests.
@geso02
Copy link
Contributor Author

geso02 commented Nov 14, 2024

I’ve fixed the issue with the build by adjusting the registration of a Submodel that was intended for submodel-related tests but was mistakenly included in environment tests due to test dependencies. I’ve now ensured that the Submodel is only registered in the submodel tests using profiles, and it’s no longer active in the environment tests, which resolves the issue and allows the build to pass again.

@geso02 geso02 changed the title Add Apache Kafka Event Interface For Repositories [DRAFT] Add Apache Kafka Event Interface For Repositories Nov 15, 2024
@geso02 geso02 marked this pull request as draft November 15, 2024 10:43
basyx environment is not assigned in auth preloader otherwise
@geso02 geso02 changed the title [DRAFT] Add Apache Kafka Event Interface For Repositories Add Apache Kafka Event Interface For Repositories Nov 15, 2024
@geso02 geso02 marked this pull request as ready for review November 15, 2024 12:54
@aaronzi
Copy link
Member

aaronzi commented Feb 11, 2025

Hi @geso02, sorry for not finding the time to review this yet. This is a very big PR and I am missing the relevant background knowledge for Kafka to do the review. Would it be possible to schedule a meeting so that you can give me a walkthrough of your PR?

In the meantime, could you please have a look at the merge conflict that occurred because of the recent changes that happened?

@geso02
Copy link
Contributor Author

geso02 commented Feb 12, 2025

Hi @geso02, sorry for not finding the time to review this yet. This is a very big PR and I am missing the relevant background knowledge for Kafka to do the review. Would it be possible to schedule a meeting so that you can give me a walkthrough of your PR?

yes we can have a meeting in teams, for example Friday afternoon

In the meantime, could you please have a look at the merge conflict that occurred because of the recent changes that happened?

I can look at it again Friday morning

geso02 and others added 9 commits February 18, 2025 18:50
group could already be handled by another listener
- Add default topic names for Kafka setup
- Update and improve README files for better clarity and structure
- Adjust tests and code to resolve merge conflicts
- Fix failing test case that was breaking the CI process
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this unrelated change

@aaronzi aaronzi merged commit f530915 into eclipse-basyx:main Apr 7, 2025
28 checks passed
@geso02 geso02 deleted the kafka-event branch May 6, 2025 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Add Apache Kafka as an Event Interface alongside MQTT for Enhanced Message Processing and Reliability

2 participants