Kafka Sink missing Keys

[themkarimi]

Hi there,

I’m attempting to replicate Kafka messages from a topic in Cluster A to another topic in Cluster B. The messages are being sent without any issues, but unfortunately, they lack the Key fields (all of them are set to Null). I’ve been searching for an option to force the replication of the keys as well, but I haven’t been able to find one. Is there a way to achieve this in Vector?
Here is my configuration:

  sources:
    topic:
      type: kafka
      bootstrap_servers: "kafkaA:9092"
      topics: ["topic"]
      group_id: "vector-group"
      auto_offset_reset: "latest"

    internal_metrics:
      type: internal_metrics
 
  transforms:
    retain_key:
      type: remap
      inputs: ["topic"]
      source: |
        # Preserve the Kafka key by copying it explicitly
        .key = .key

  sinks:
    vector:
      type: kafka
      bootstrap_servers: "KafkaB:32040"
      inputs:
        - retain_key
      topic: "test-vector"
      encoding:
        codec: "raw_message"
        include_keys: true
      sasl:
        enabled: true
        mechanism: "SCRAM-SHA-512"
        username: "user"
        password: "asda"
      librdkafka_options:
        security.protocol: "SASL_PLAINTEXT"
    prometheus_exporter:
      type: prometheus_exporter
      inputs: ["internal_metrics"]
      address: 0.0.0.0:9090