Ephemeral local storage usage exceeds the total limit, Apache hop Pod Crashing

[swaroopchenchu]

We are running Apache Hop 2.7.0, deployed via Helm chart on Google Kubernetes Engine (GKE).

To prevent pod eviction (since Apache Hop consumes significant resources during pipeline execution), we configured the following resource requests and limits:

limits:
ephemeral-storage: 70Gi
memory: 16Gi
requests:
cpu: 1
ephemeral-storage: 60Gi
memory: 8Gi

However, the pod still crashes while the pipeline is running.

📌 Pod Events
Normal Scheduled 57m default-scheduler Successfully assigned etl/hop-server-7b88f876c8-ld8bv to gke-exito-lab-1-obp-dabc0b17-8cjc
Normal Pulling 57m kubelet Pulling image "us-central1-docker.pkg.dev/product-obp/obp/apache-hop:2.7.0"
Normal Pulled 56m kubelet Successfully pulled image in 27.7s (image size ~940MB)
Normal Created 56m kubelet Created container: hop
Normal Started 56m kubelet Started container hop
⚠️ Warning Evicted 4m kubelet Pod ephemeral local storage usage exceeds the total limit of containers (70Gi)
Normal Killing 4m kubelet Stopping container hop
⚠️ Warning FailedKillPod 3m57s kubelet failed to kill container (ttrpc: closed)

🤔 Observation / Question

The eviction reason clearly indicates ephemeral local storage exhaustion

The pipeline performs a large number of database lookups

Question:
Could the high number of database lookups (temporary files, caching, sorting, or transform spill-to-disk) be causing excessive ephemeral storage usage, leading to pod eviction?

📷 Reference

Could anyone please guide here.

Thanks

[hansva]

Unless your failed cases are writing to the local storage, and have a massive amount of those, there is nothing in this pipeline that is writing to disk. We only have a couple of transforms that will use disk space during processing the main culprit usually is the "Sort Rows" transform.

I would also take a look at the failed cases field definition. If you have string fields with max length (2GB) you could be writing unessesary white spacing to disk causing it to fill up.

[swaroopchenchu]

Unless your failed cases are writing to the local storage, and have a massive amount of those, there is nothing in this pipeline that is writing to disk. We only have a couple of transforms that will use disk space during processing the main culprit usually is the "Sort Rows" transform.

I would also take a look at the failed cases field definition. If you have string fields with max length (2GB) you could be writing unessesary white spacing to disk causing it to fill up.

I do see three fields have bigger size, but failed cases are small in number while pipeline running before pod got evicted.

[hansva]

With those field sizes each line in the file will take 3gb of space. Best practice would be to press the minimal width button in the bottom

[swaroopchenchu]

With those field sizes each line in the file will take 3gb of space. Best practice would be to press the minimal width button in the bottom

🙏 Many thanks, it worked.