Skip to content

NiFi 1.23.2 caught in Restart Loop due to LivenessProbe timeouts (NiFiKop) #638

@udaykumar36

Description

@udaykumar36

What steps will reproduce the bug?

I am deploying a NiFi cluster using the NiFiKop operator on [AWS/Azure/Local] Kubernetes. The NiFi pods are entering a continuous restart loop. Based on the logs, NiFi is still in the process of unpacking NAR files and initializing the Jetty server when Kubernetes/NiFiKop triggers a shutdown because the health checks (LivenessProbes) fail.

Environment:

NiFi Version: 1.23.2
Storage Type: [e.g., EBS gp3]

The Problem: NiFi 1.23.2 takes approximately 5-8 minutes to fully start in my environment. The default livenessProbe settings in the operator (or the inability to override them in my CRD version) cause Kubernetes to kill the pod before it reaches a "Ready" state.

Logs Observed:

[Standard NiFi logs showing NAR extraction...]
INFO [main] o.a.n.controller.StandardFlowService...
INFO [Thread-1] org.apache.nifi.NiFi NiFi PID [1] shutdown started...
Received SIGTERM, shutting down gracefullly.
What I have tried:

Adding livenessProbe overrides in nodeConfigGroups (Rejected by CRD or ignored).

Manually patching the pod (Operator reverts the change).

Increasing resources (CPU/Memory), but the unpacking time still exceeds the probe timeout.

Question: How can I correctly override the initialDelaySeconds for liveness and readiness probes in this version of the operator to allow for a 600-900 second startup time? Is there a specific CRD path I should be using for NiFi 1.2x?

I am stuck here , Please help me out of this cycle

What is the expected behavior?

Pods should be up and running and I should able to open UI

What do you see instead?

Pods are restarting for every 10 - 15 seconds and its noy memory issue I am using t3.Xlarge aws instance

Possible solution

No response

NiFiKop version

v1.16.0

Golang version

1.19

Kubernetes version

Client Version: v1.35.0
Kustomize Version: v5.7.1
Server Version: v1.34.3+k3s1

NiFi version

1.23

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions