Strimzi standalone user operator and Java heap space errors

[jslusher]

I'm trying to deploy a standalone user operator and I'm running into an issue with java heap space.

I currently have the following allocated to my pods:

          resources:
            limits:
              memory: 2Gi
              cpu: 500m
            requests:
              memory: 2Gi
              cpu: 500m

If I omit STRIMZI_JAVA_OPTS from my Deployment, the operator starts up but then fails:

Terminating due to java.lang.OutOfMemoryError: Java heap space

If I try setting STRIMZI_JAVA_OPTS to any variation of the following according to the user operator docs:

          - name: STRIMZI_JAVA_OPTS
            value: "-Xmx=512M -Xms=256M"

I get the following error:

Invalid maximum heap size: -Xmx=512M
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

It seems to be a problem with whichever I set first, so if I set Xmx=512M first I get the error above, but if I set Xms=256M first it complains about that setting instead. I wonder if it's an issue of ordering in the java command. This is what it looks like with the settings:

java -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Xmx=512M -Xms=256M -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom '-Xlog:gc*:stdout:time' -XX:NativeMemoryTracking=summary -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20 -classpath .....

[scholzj]

I don't think you can set -Xmx to 512M with memory request 512Mi. That will not fit. The -Xmx has to be a lot smaller because you need to fit many other parts into the memory then just the Java heap.

[scholzj]

I don't know as I never tried that. But that is not used by the Kafka clients so that would definitely not trigger this issue in the Kafka clients. If that would be the case, it would be in your Kafka configuration.

[jslusher]

The MSK cluster to which I'm trying to connect doesn't have mTLS configured. It's only set up for SASL/SCRAM.

STRIMZI_KAFKA_BOOTSTRAP_SERVERS is set up for the brokers listening on the SASL endpoint (port 9096). Would that classify as misconfigured TLS for the Kafka cluster? In other words, does the operator need to have TLS configured and connect to the TLS endpoint, even if it eventually uses a different method of authentication?

[scholzj]

TBH, I'm not sure I understand how did you configure the broker and the operator side. I think this is the Kafka issue I'm referring to here: https://issues.apache.org/jira/browse/KAFKA-4090

[jslusher]

At the moment, there are no SSL settings configured (other than SASL/SCRAM) on the MSK brokers, so I'm not sure the issue you linked would apply to this case.

[jslusher]

Also, we have other connectors/consumers configured to connect to the same endpoint I'm trying to use for the operator and they're not having the same OOM problem.

[cknowles]

@jslusher I had a similar issue, try setting the env var without the equals signs -Xmx512M -Xms256M.