spatial_autocorr uses all the cores

[GloriaLiu28]

Hi,
when I run squidpy.gr.spatial_autocorr to calculate moran, it used all the 112 cores of my server, even the n_jobs were set to n_jobs=8, how to limit the number of cores it uses ? I find the old issues about occurence also talked about this. Could you please also set numba_parallell = False to squidpy.gr.spatial_autocorr function?

[maltekuehl]

For a quick fix, you should be able to set the environment variable NUMBA_NUM_THREADS.

[selmanozleyen]

hi @GloriaLiu28 can you check if this still exists with the most recent version of scanpy?

@flying-sheep I noticed we use gearys stats in this spatial_autocorr and some of the gearys related metrics use a decorator from scanpy which might set the parallel=True https://github.com/scverse/scanpy/blob/b058b1792b91b1baaf2e5caf395960005098c2ac/src/scanpy/metrics/_gearys_c.py#L233-L234. Here is the decorator https://github.com/scverse/scanpy/blob/b058b1792b91b1baaf2e5caf395960005098c2ac/src/scanpy/_compat.py#L116. The function is compiled parallel if not _is_in_unsafe_thread_pool() but I am not sure if this means spatial_autocorr will run a parallel numba kernel and I think we need to make sure it doesn't.

[flying-sheep]

It will not be parallelized when the current thread’s name starts with ThreadPoolExecutor, which is the case in dask.

joblib seems to use this: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.ThreadPool

so scanpy’s workaround will probably not work there. we should augment it so it does.

I also found the way joblib adapts to manually reduced thread numbers, which seems like a nice starting point for us: https://github.com/joblib/joblib/blob/ed0806a497268005ad7dad30f79e1d563927d7c6/joblib/_parallel_backends.py#L65

[ilan-gold]

@selmanozleyen Ensure that numba only uses one core here going forward.