Conversation
|
I like the implementation so far. In general kaskade is not fast already (too much requests), so I wonder how this can affect even more the performance 🤔 |
…t to confluent-kafka objects (which will fail due to unsupported property)
|
So I found an issue with my initial implementation, where passing the |
| kaskade admin -b my-kafka:9092 \ | ||
| -c security.protocol=SASL_SSL \ | ||
| -c sasl.mechanism=OAUTHBEARER \ | ||
| -c aws.region=eu-west-1 |
| ), | ||
| ) | ||
| @cloup.option_group( | ||
| "Cloud configuration options", |
There was a problem hiding this comment.
I would prefer AWS, and instead of --cloud-config we pass --aws, so the very optional options will be:
--registry
--avro
--protobuf
--aws
| kaskade admin -b localhost:9092 | ||
| kaskade admin -b localhost:9092 --config security.protocol=SSL | ||
| kaskade admin -b localhost:9092 --config-file kafka.properties | ||
| kaskade admin -b localhost:9092 -c security.protocol=SASL_SSL -c sasl.mechanism=OAUTHBEARER --cloud-config aws.region=eu-west-1 |
| bootstrap_servers: str, | ||
| kafka_config_file: str | None, | ||
| kafka_config: dict[str, str], | ||
| cloud_config: dict[str, str], |
There was a problem hiding this comment.
It's also possible to validate the configs and example: https://github.com/sauljabin/kaskade/blob/main/kaskade/main.py#L327
We can list all the aws available configurations https://github.com/sauljabin/kaskade/blob/main/kaskade/configs.py#L6
| ) -> tuple[int, int]: | ||
| low, high = 0, 0 | ||
|
|
||
| consumer = Consumer(self.config | {GROUP_ID: f"kaskade-{uuid.uuid4()}"}) |
There was a problem hiding this comment.
I remember there was a reason 🤔 , but I need to test it again, then I will document it. But so far looks not necessary
2 participants
This adds support for AWS IAM Authentication (also see #44) to be able to connect to an AWS MSK cluster.
Note that due to the necessity of having to call
.poll()several times, there will be some delays in retrieving data from Kafka.Usage is described in the README section.