Failing to template annotations in kafka services

[itayvolo]

Hey Everyone,

Im trying to create nlb for each kafka-broker, so I've set the kafka service type to load-balancer which actually creates service for each broker.. but the problem is that its not templating the annotations I've set in the new services so basically its not creating the load-balancers I want.. (nlb and not classic)
Please help me figure this out

Here's my configuration:

I found the next example in your documentation which basically tells me to set the same configuration as I did but its not working..

https://strimzi.io/blog/2020/01/02/using-strimzi-with-amazon-nlb-loadbalancers/
https://strimzi.io/docs/operators/0.29.0/configuring.html#type-KafkaClusterTemplate-reference

[scholzj]

Can you share the full custom resource including the status section? Also, can you share the YAMLs of the services?

[itayvolo]

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
annotations:
creationTimestamp: '2022-08-08T14:24:27Z'
generation: 2
labels:
app.kubernetes.io/instance: sgt2-kafka
name: kafka
namespace: datalayer
resourceVersion: '469695591'
uid: cc1ddacb-58ae-411f-a6e5-38b9269b766a
spec:
entityOperator:
template:
pod:
metadata:
labels:
app: kafka-entity-operator
securityContext:
fsGroup: 1201
runAsGroup: 1201
runAsUser: 1201
topicOperator: {}
userOperator: {}
kafka:
config:
auto.create.topics.enable: 'true'
default.replication.factor: 5
log.message.format.version: '2.5'
log.retention.bytes: 2147483648
log.retention.hours: 72
num.partitions: 10
offsets.topic.replication.factor: 5
transaction.state.log.min.isr: 1
transaction.state.log.replication.factor: 5
listeners:
- configuration:
useServiceDnsDomain: true
name: plain
port: 9092
tls: false
type: internal
- authentication:
type: tls
name: tls
port: 9093
tls: true
type: internal
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
key: kafka
name: kafka-metrics-config
replicas: 5
resources:
limits:
cpu: 2
memory: 2G
requests:
cpu: 2
memory: 2G
storage:
class: ebs-sc-gp3
deleteClaim: true
size: 400Gi
type: persistent-claim
template:
pod:
metadata:
labels:
app: kafka-kafka
securityContext:
fsGroup: 1201
runAsGroup: 1201
runAsUser: 1201
version: 3.1.1
kafkaExporter:
groupRegex: .*
resources:
limits:
cpu: 500m
memory: 200M
requests:
cpu: 250m
memory: 100M
template:
pod:
metadata:
labels:
app: kafka-kafka-exporter
securityContext:
fsGroup: 1201
runAsGroup: 1201
runAsUser: 1201
service:
metadata:
labels:
app: kafka-kafka-exporter
topicRegex: .*
zookeeper:
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
key: zookeeper
name: kafka-metrics-config
replicas: 5
resources:
limits:
cpu: 1
memory: 1G
requests:
cpu: 1
memory: 1G
storage:
class: ebs-sc-gp3
deleteClaim: true
size: 200Gi
type: persistent-claim
template:
pod:
metadata:
labels:
app: kafka-zookeeper
securityContext:
fsGroup: 1201
runAsGroup: 1201
runAsUser: 1201
status:
clusterId: d5rBqnBDQeCZKnQSCMXiQQ
conditions:
- lastTransitionTime: '2022-08-08T14:59:56.669340Z'
message: >-
log.message.format.version does not match the Kafka cluster version,
which suggests that an upgrade is incomplete.
reason: KafkaLogMessageFormatVersion
status: 'True'
type: Warning
- lastTransitionTime: '2022-08-08T14:59:56.669350Z'
message: >-
min.insync.replicas option is not configured. It defaults to 1 which
does not guarantee reliability and availability. You should configure
this option in .spec.kafka.config.
reason: KafkaMinInsyncReplicas
status: 'True'
type: Warning
- lastTransitionTime: '2022-08-08T15:03:05.313Z'
status: 'True'
type: Ready
- lastTransitionTime: '2022-08-08T14:59:54.238426Z'
message: >-
In API version kafka.strimzi.io/v1beta2 the service property at path
spec.kafkaExporter.template.service has been deprecated. The Kafka
Exporter service has been removed.
reason: DeprecatedFields
status: 'True'
type: Warning
listeners:
- addresses:
- host: kafka-kafka-bootstrap.datalayer.svc.cluster.local
port: 9092
bootstrapServers: 'kafka-kafka-bootstrap.datalayer.svc.cluster.local:9092'
name: plain
type: plain
- addresses:
- host: kafka-kafka-bootstrap.datalayer.svc
port: 9093
bootstrapServers: 'kafka-kafka-bootstrap.datalayer.svc:9093'
certificates:
- |
-----BEGIN CERTIFICATE-----

---

      -----END CERTIFICATE-----
  name: tls
  type: tls

observedGeneration: 2

Service example

apiVersion: v1
kind: Service
metadata:
creationTimestamp: '2022-08-09T07:48:16Z'
finalizers:
- service.kubernetes.io/load-balancer-cleanup
labels:
app.kubernetes.io/instance: kafka
app.kubernetes.io/managed-by: strimzi-cluster-operator
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: strimzi-kafka
strimzi.io/cluster: kafka
strimzi.io/kind: Kafka
strimzi.io/name: kafka-kafka
name: kafka-kafka-nlb-0
namespace: datalayer
ownerReferences:
- apiVersion: kafka.strimzi.io/v1beta2
blockOwnerDeletion: false
controller: false
kind: Kafka
name: kafka
uid: cc1ddacb-58ae-411f-a6e5-38b9269b766a
resourceVersion: '470437223'
uid: 48cfe48a-6351-404d-9e2d-81d5b18d1ce2
spec:
clusterIP: ****
clusterIPs:
- *****
externalTrafficPolicy: Cluster
ports:
- name: tcp-nlb
nodePort:*****
port: 9092
protocol: TCP
targetPort: 9092
selector:
statefulset.kubernetes.io/pod-name: kafka-kafka-0
strimzi.io/cluster: kafka
strimzi.io/kind: Kafka
strimzi.io/name: kafka-kafka
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- hostname: >-
******.ap-southeast-1.elb.amazonaws.com

[scholzj]

Can you please format it as code so that it can be used to reproduce the issue?

[scholzj]

BTW: Regardless the formatting, you do not seem to have the annotations in the resource in Kubernetes. So maybe you never applied your changes?

[itayvolo]

Yes thats exactly the problem.. its not templating it, attaching here the original yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: kafka
namespace: datalayer
spec:
kafkaExporter:
# image: bitnami/kafka-exporter:1.2.0-debian-10-r170
groupRegex: "."
topicRegex: "."
resources:
requests:
cpu: 250m
memory: 100M
limits:
cpu: 500m
memory: 200M
template:
service:
metadata:
labels:
app: kafka-kafka-exporter
pod:
metadata:
labels:
app: kafka-kafka-exporter
securityContext:
runAsUser: 1201
runAsGroup: 1201
fsGroup: 1201
kafka:
version: 3.1.1
replicas: 5
listeners:
- name: nlb
port: 9092
type: loadbalancer
tls: false
# - name: plain
# port: 9092
# type: internal
# tls: false
# configuration:
# useServiceDnsDomain: true
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: tls
template:
externalBootstrapService:
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
perPodService:
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
config:
# on how many partitions topic will spread
num.partitions: 10
# ensures topic will be replicated on multiple brokers
offsets.topic.replication.factor: 5
#replication factor defines the number of copies of a topic in a Kafka cluster
default.replication.factor: 5
#internal topic creation will fail until the cluster size meets this replication factor requirement
transaction.state.log.replication.factor: 5
#number of replicas that must acknowledge a write for the write to be considered successful
transaction.state.log.min.isr: 1
log.message.format.version: "2.5"
auto.create.topics.enable: "true"
log.retention.hours: 72
log.retention.bytes: 2147483648
storage:
deleteClaim: true
size: 400Gi
type: persistent-claim
class: ebs-sc-gp3
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kafka-metrics-config
key: kafka
resources:
{{- $myenv := .Values.global.type }}
{{- range $key, $val := .Values.env_type_kafka }}
{{- if eq $key $myenv }}
requests:
cpu: {{ $val.requests.cpu }}
memory: {{ $val.requests.memory }}
limits:
cpu: {{ $val.limits.cpu }}
memory: {{ $val.limits.memory }}
{{- end }}
{{- end }}
template:
pod:
metadata:
labels:
app: kafka-kafka
securityContext:
runAsUser: 1201
runAsGroup: 1201
fsGroup: 1201
zookeeper:
replicas: 5
storage:
deleteClaim: true
size: 200Gi
type: persistent-claim
class: ebs-sc-gp3
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kafka-metrics-config
key: zookeeper
resources:
{{- if eq .Values.cluster_name "***" }}
requests:
cpu: 3500m
memory: 3500M
limits:
cpu: 3500m
memory: 3500M
{{- else }}
{{- $myenv := .Values.global.type }}
{{- range $key, $val := .Values.env_type_zookeeper }}
{{- if eq $key $myenv }}
requests:
cpu: {{ $val.requests.cpu }}
memory: {{ $val.requests.memory }}
limits:
cpu: {{ $val.limits.cpu }}
memory: {{ $val.limits.memory }}
{{- end }}
{{- end }}
{{- end }}
template:
pod:
metadata:
labels:
app: kafka-zookeeper
securityContext:
runAsUser: 1201
runAsGroup: 1201
fsGroup: 1201
entityOperator:
topicOperator: {}
userOperator: {}
template:
pod:
metadata:
labels:
app: kafka-entity-operator
securityContext:
runAsUser: 1201
runAsGroup: 1201
fsGroup: 1201

[scholzj]

As I said:

• Please make sure to share the YAMLs as code. Without that they are not readable!
• In the previous comment you share the Kafka CR as it exists in Kubernetes => and there are no annotations configured there. So you did not applied your changes to Kubernetes or you did it incorrectly (hard to say without the proper formatting which removes all whitespacing).