What is the ordering priority for loading configurations in Kafka-ui?

[SamsadSajid]

Hi. I am deploying the latest version of Kafka-ui as an ECS app. I set the environment variables of the app as follows:

"environment": [
  {
     "name": "KAFKA_CLUSTERS_0_NAME",
     "value": "cluster1"
  },
 {
     "name": "KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS",
     "value": "..."
 },
 ...
],

This works well for me! Now, I want to add a proto-sede to browse my messages. To do that I added new configurations to the above environment variables. After adding them it looks like this:

"environment": [
  {
     "name": "KAFKA_CLUSTERS_0_NAME",
     "value": "cluster1"
  },
 {
     "name": "KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS",
     "value": "..."
 },
{
     "name": "kafka.clusters.0.serde.0.name",
     "value": "ProtobufFile"
 },
{
     "name": "kafka.clusters.0.serde.0.properties.protobufFilesDir",
     "value": "/proto/"
 },
{
     "name": "kafka.clusters.0.serde.0.properties.protobufMessageNameByTopic.topic1",
     "value": "package.protobuf.messages.MyMessage"
 },
 ...
],

After this deployment, I expected the Kafka-ui to show the ProtobufFile option in the value serde dropdown options. But it did not!

So I changed the configurations to use the uppercase format as follows:

"environment": [
  {
     "name": "KAFKA_CLUSTERS_0_NAME",
     "value": "cluster1"
  },
 {
     "name": "KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS",
     "value": "..."
 },
{
     "name": "KAFKA_CLUSTERS_0_SERDE_NAME",
     "value": "ProtobufFile"
 },
{
     "name": "KAFKA_CLUSTERS_0_SERDE_0_PROPERTIES_PROTOBUFFILESDIR",
     "value": "/proto/"
 },
{
     "name": "KAFKA_CLUSTERS_0_SERDE_0_PROPERTIES_PROTOBUFMESSAGENAMEBYTOPIC_TOPIC1",
     "value": "package.protobuf.messages.MyMessage"
 },
 ...
],

After this deployment, I can see the ProtobufFile option in the value serde dropdown options. And I could browse the topic1 messages. But I faced a new problem.

I have topic names with the following formats:

topic1
topic_2
topic.3

With my last configurations, I could browse messages for topics that did not have _ in their name. But for topics with _ in their name, Kafka-ui did not show the ProtobufFile option in the value serde dropdown options and did not deserialize those messages.

At this point, it seems like _ is char based on which Kafka-ui translates the key name. So when I added KAFKA_CLUSTERS_0_SERDE_0_PROPERTIES_PROTOBUFMESSAGENAMEBYTOPIC_TOPIC_2, Kafka-ui translated it to

kafka:
  clusters:
     - name: cluster1
        serde:
          properties:
              protobufMessageNameByTopic:
                topic1: "package.protobuf.messages.MyMessage"
                topic:
                 2: "package.protobuf.messages.MyMessage2"
                

Of course, it will not deserialize messages because I don't have a topic called 2. So the problem here is Kafka-ui is splitting the key by _ and splitting topic_2 to 2. So here are my questions:

1. On my machine, I can have both dot notation and uppercase notation env vars i.e., KAFKA_CLUSTERS_0_NAME and kafka.clusters.0.serde.0.properties.protobufFilesDir mixed in docker-compose and it works! But the same does not work in ECS env vars. I don't seem to get why. Can you please share the reasons why?
2. If I cannot use the mix name and I have to use only the uppercase format, how can I support topics that has _ in its name?
3. What is the priority ordering for the configs? If I provide a config.yml file as explained in the kafka-ui doc, and uppercase format env vars which will be used by kafka-ui at runtime?

[SamsadSajid]

This discord thread asks a question

Hi there!
I want to use by default yaml config file (it works fine), but when I want to combine this with only one environment variable (as jaas config to keep secret hidden), config file seems to be ignored.

@Haarolean you replied

I don't remember exactly but they might get overridden completely :(

I think I am also facing the same issue. By any chance, Kafka-ui can only work with one type of config? i.e., either config file or env variables but not both of them together? If so, doesn't it contradict the kafka-ui doc

You can use env vars to override the default config on some different environments.