Ray temp directory memory leak

[kipraveen]

Describe the bug

Ray temp directory memory leak when running tasks multiple times.

Steps/Code to reproduce bug

Run any nemo curator pipeline which uses the ray cluster multiple times. Observe the size of the directory at /tmp/ray

Expected behavior

Size of the temp directory does not keep on increasing.

[ayushdg]

Thanks a couple of questions:

1. Are you running this in a container or bare metal?

Run any nemo curator pipeline which uses the ray cluster multiple times.

To clarify some information here is it the same cluster running multiple pipelines or are you starting up different clusters for different jobs? Does the memory clear up when the cluster is stopped?

[kipraveen]

Sorry for missing the details of my run.

I am running nemo curator inside a docker container with pip install.

For the second question, even after the python script exits, the temp directory is not cleaned up. It just keeps on adding to the same temp directory in subsequent runs.

[ayushdg]

Thanks for that information. The Ray object store uses shared memory for allocations. In containers where the default shared memory is super small Ray defaults to using /tmp instead. For container runs I would recommend setting a larger shm size when starting the container.

With docker you can use something like this --shm-size 500000000000 to ensure /dev/shm is used for the object store instead of /tmp.

I would assume that ray would clean up the object store/tmp memory even after exiting the python program but I'll need to look into that