Pod Restart Policy

[ranajoy1998]

Currently I'm using Artemis Cloud Operator to deploy broker Pods in EKS. Upon stress testing the broker, sometimes the broker container is OOM killed and restarted in the same Pod. After restart the container remains in an unresponsive state, maybe because it is trying to read data from the old heap(not sure). Only when I manually delete the pod, a new pod starts and broker behaves normally. I was checking the restartPolicy in the Pod yaml, it is set to Always but CRD does not support setting it. Is there any way to set it to Never so that a new Pod is created always? Thanks in advance!

[brusdev]

Changing the restart policies only works for standalone Pods. Pods managed via StatefulSets only support a restart policy of Always and the operator creates StatefulSets.

Anyway, after restart the container should not remain in an unresponsive state. Could you share the broker log?

[ranajoy1998]

Ohh right that's a valid point.

Context of my test: 2 broker pods are running in EKS cluster with Artemis Cloud operator. I'm doing stress test through a Java app having HiveMQ Client (fire ~4k connections per min in batches -> disconnect each connection after 40 seconds). As per my use case, I'm using MQTT as the messaging protocol. After OOM when the container restarted, I could see all connections are being dropped instantly. I have added a log as well for every new client connected. But if I manually delete the pod, after restart the pod is able to take connections again. I'm attaching the debug log. Thanks again for the help!

debug-log.txt

[brusdev]

But if I manually delete the pod, after restart the pod is able to take connections again.
This is expected behavior. You could reduce the deployment size if you need to delete a broker pod.