Default characer encoding for KafkaConnect built images

[akomisarek]

Hello,

I am wondering what's expected default character encoding for images built by KafkaConnect operator. It's not specified in the documentation (or I can't find it), I would assume it's UTF-8, but it doesn't seem to be the case.

We encountered an issue with IBM RabbitMQ Source Connector (but I guess it doesn't matter what source connector is used, it could break whenever the processing is encoding dependant), which I think managed to distill to KafkaConnect Operator and automatically built images.

I am using basic config for KafkaConnect yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  namespace: my-kafka-project
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap:9092
  config:
    group.id: my-connect-cluster
    offset.storage.topic: my-connect-cluster-offsets
    config.storage.topic: my-connect-cluster-configs
    status.storage.topic: my-connect-cluster-status
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: true
    value.converter.schemas.enable: true
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
  build:
    output:
      type: docker
      image: adamkomisarek/testing-rabbitmq-strimzi:latest
      pushSecret: my-registry-credentials
    plugins:
      - name: ibm-kafka-connect-rabbitmq
        artifacts:
          - type: jar
            url: https://github.com/akomisarek/kafka-connect-rabbitmq-source/raw/main/kafka-connect-rabbitmq-source-1.0-20220211.125431-1-jar-with-dependencies.jar
  logging:
    type: inline
    loggers:
      log4j.rootLogger: "INFO"
  readinessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  jvmOptions:
    "-Xmx": "1g"
    "-Xms": "1g"
  template:
    connectContainer:
      env:
        - name: JAVA_TOOL_OPTIONS
          value: '-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005'

And when the KafkaConnect starts it gets, the US-ASCII encoding, which is something I wouldn't expect. What is more confusing if you exec into the Docker image, you see UTF-8 as expected. I guess it has got something with the fact that outside the shell locale is not initialised? Shouldn't UTF-8 be defualt/expected?

Couple of commands to validate:

Running locale outside the shell:

docker run  adamkomisarek/testing-rabbitmq-strimzi locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
➜  ~ docker run  adamkomisarek/testing-rabbitmq-strimzi java -XshowSettings
VM settings:
    Max. Heap Size (Estimated): 1.95G
    Using VM: OpenJDK 64-Bit Server VM
Property settings:
    awt.toolkit = sun.awt.X11.XToolkit
    file.encoding = ANSI_X3.4-1968
➜  ~ docker run  adamkomisarek/testing-rabbitmq-strimzi java -XshowSettings
kafka

Running locale inside the shell:

docker run -it  adamkomisarek/testing-rabbitmq-strimzi  /bin/bash
[kafka@7436e97d5931 kafka]$ whoami
kafka
[kafka@7436e97d5931 kafka]$ locale
LANG=C.utf8
LC_CTYPE="C.utf8"
LC_NUMERIC="C.utf8"
LC_TIME="C.utf8"
LC_COLLATE="C.utf8"
LC_MONETARY="C.utf8"
LC_MESSAGES="C.utf8"
LC_PAPER="C.utf8"
LC_NAME="C.utf8"
LC_ADDRESS="C.utf8"
LC_TELEPHONE="C.utf8"
LC_MEASUREMENT="C.utf8"
LC_IDENTIFICATION="C.utf8"
LC_ALL=
[kafka@7436e97d5931 kafka]$ java -XshowSettings
VM settings:
    Max. Heap Size (Estimated): 1.95G
    Using VM: OpenJDK 64-Bit Server VM
Property settings:
    awt.toolkit = sun.awt.X11.XToolkit
    file.encoding = UTF-8

Is such difference expected/desirable? Wouldn't we benefit of using UTF8 as default, by enforcing this while image is built?
Thanks for the input!

[scholzj]

I do not think we set anything with regards to encoding. So I guess I would expect it to be the default inherited from the Red Hat UBI8 iamegs or from OpenJDK 11.

Out of curiosity, why does the encoding matter? Nobody ever raised it before.

[akomisarek]

Yeah I could imagine it's not common people rely on the encoding. We use the IBM RabbitMQ source connector and it does the following:
https://github.com/ibm-messaging/kafka-connect-rabbitmq-source/blob/f25726d5a70593252c537278fd5723b229de407d/src/main/java/com/ibm/eventstreams/connect/rabbitmqsource/schema/ValueSchema.java#L25

 public static Struct toStruct(String consumerTag, Envelope envelope, AMQP.BasicProperties basicProperties, byte[] body) {
        String bodyString = new String(body);

And the above breaks for special characters outside the US-ASCII encoding! It wouldn't be an issue if they simply passed the bytes around...

I think that's if it used default encoding it would be fine, but because the profile is not loaded outside the shell (similar question: https://stackoverflow.com/questions/57266806/the-result-of-locale-command-differs-between-inside-and-outside-docker-contain). Maybe the build process should set the expected locale explicitly? Happy to help with further investigation/PR if required.

[akomisarek]

But I wanted to open this as a discussion as I am not sure what is the best fix, we could try to change our KafkaConnect pod properties to enforce the file.encoding, but they it would need to happen for every kafkaconnect "cluster" and could happen for others? Thanks for thoughts and input!