Kafka / Event Hub Source

[zpriddy]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When trying to read from EventHub with Kafka I keep getting errors like the following:

Is there any suggestions or configuration I should try?

-- Subject: Unit vector.service has finished start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit vector.service has finished starting up.
--
-- The start-up result is done.
Apr 19 22:50:01 log-aggregatorvector[16157]: 2023-04-19T22:50:01.604132Z ERROR source{component_kind="source" component_id=azure_activity_event_hub component_type=kafka component_name=azure_activity_event_hub}: vector::internal_events::kafka: Failed to read message. error=Message consumption error: MessageSizeTooLarge (Broker: Message size too large) error_code="reading_message" error_type="reader_failed" stage="receiving" internal_log_rate_limit=true
Apr 19 22:50:01 log-aggregatorvector[16157]: 2023-04-19T22:50:01.717203Z ERROR source{component_kind="source" component_id=azure_activity_event_hub component_type=kafka component_name=azure_activity_event_hub}: vector::internal_events::kafka: Internal log [Failed to read message.] is being rate limited.
Apr 19 22:50:11 log-aggregatorvector[16157]: 2023-04-19T22:50:11.650908Z ERROR source{component_kind="source" component_id=azure_activity_event_hub component_type=kafka component_name=azure_activity_event_hub}: vector::internal_events::kafka: Internal log [Failed to read message.] has been rate limited 29 times.

Configuration

type = "kafka"
bootstrap_servers = "${AZURE_ACTIVITY_EVENT_HUB_ENDPOINT}:9093"
group_id = "vector"
topics = ["^${AZURE_ACTIVITY_EVENT_HUB_TOPIC_PREFIX}.+"]

  [sasl]
  enabled = true
  mechanism = "PLAIN"
  password = "${AZURE_ACTIVITY_EVENT_HUB_CONNECTION_STRING}"
  username = "$$ConnectionString"

  [tls]
  enabled = true

Version

0.28.2

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[bruceg]

This appears to be an error reported by the underlying librdkafka library used by this source. You could try setting message.max.bytes to a larger value than the default of 1000000 in the librdkafka_options map.