specifying the tmp dir in `optimize` function

[Phimos]

🚀 Feature

Support specifying the tmp dir in optimize function via environment variables or arguments.

Motivation

I'm trying to use ld.optimize function to optimize my dataset and facing No space error, it seems like it use /tmp/chunks as the tmp dir.

[bhimrazy]

Hi @Phimos, just in case you're still looking into it, here are a few environment variables that might be helpful.

litData/src/litdata/processing/data_processor.py

Lines 80 to 100 in 0770595

	def _get_default_cache() -> str:
	return "/cache" if _IS_IN_STUDIO else tempfile.gettempdir()
	def _get_cache_dir(name: Optional[str] = None) -> str:
	"""Returns the cache directory used by the Cache to store the chunks."""
	cache_dir = os.getenv("DATA_OPTIMIZER_CACHE_FOLDER", f"{_get_default_cache()}/chunks")
	if name is None:
	return cache_dir
	return os.path.join(cache_dir, name.lstrip("/"))
	def _get_cache_data_dir(name: Optional[str] = None) -> str:
	"""Returns the cache data directory used by the DataProcessor workers to download the files."""
	cache_dir = os.getenv("DATA_OPTIMIZER_DATA_CACHE_FOLDER", f"{_get_default_cache()}/data")
	if name is None:
	return cache_dir
	return os.path.join(cache_dir, name.lstrip("/"))