Inquiry Regarding Memory Issues with 10X HD Data Analysis

[Wanrh-mimi]

Dear SpiceMix Team,

I hope this email finds you well!

I am currently using the SpiceMix package to analyze 10X HD data, which is quite large—each sample contains approximately 400,000 spots. However, I am encountering memory issues when attempting to run the analysis on a server with 80GB of GPU memory and 1000GB of CPU memory, causing the program to run out of memory.

I have tried reducing batch sizes and optimizing data processing, but the out-of-memory issue persists. I would like to ask if there are any strategies or suggestions for managing memory effectively when working with large-scale 10X HD datasets, especially when GPU and CPU memory are limited.

Here are my current hardware specifications and the issues I am facing:

GPU: 80GB

CPU: 1000GB

Dataset size: Approximately 400,000 spots per sample

Specifically, I would like to know:

1. Are there any memory management optimizations for large datasets that you recommend?

2. Are there specific parameters or settings within SpiceMix that could be adjusted to avoid memory overflow?

3. Are there alternative approaches to reducing memory usage while maintaining the analysis accuracy?

If possible, I would greatly appreciate any advice or solutions you may have. Thank you for your work on SpiceMix, and I look forward to your response.

Best regards,

mimi

[Wanrh-mimi]

Here is the error message，

RuntimeError Traceback (most recent call last)
/tmp/ipykernel_1034078/3335021423.py in
24
25 # -- 初始化模型（Louvain 聚类）
---> 26 obj.initialize(
27 method='louvain',
28 kwargs=dict(num_pcs=50, n_neighbors=20, resolution_boundaries=(.5, 2.), num_rs=10)

/data/han_group/wanrh/Tools/SpiceMix/SpiceMix-master/SpiceMix/model.py in initialize(self, method, random_state, kwargs, precomputed_clusters)
165 raise NotImplementedError()
166
--> 167 self.estimate_sigma_yx()
168 self.initialize_Sigma_x_inv()
169

/data/han_group/wanrh/Tools/SpiceMix/SpiceMix-master/SpiceMix/model.py in estimate_sigma_yx(self)
182
183 def estimate_sigma_yx(self):
--> 184 d = np.array([
185 torch.linalg.norm(torch.addmm(Y.to(X.device), X, self.M.T, alpha=-1), ord='fro').item() ** 2
186 for Y, X in zip(self.Ys, self.Xs)

/data/han_group/wanrh/Tools/SpiceMix/SpiceMix-master/SpiceMix/model.py in (.0)
183 def estimate_sigma_yx(self):
184 d = np.array([
--> 185 torch.linalg.norm(torch.addmm(Y.to(X.device), X, self.M.T, alpha=-1), ord='fro').item() ** 2
186 for Y, X in zip(self.Ys, self.Xs)
187 ])

RuntimeError: CUDA out of memory. Tried to allocate 59.63 GiB (GPU 3; 79.10 GiB total capacity; 59.70 GiB already allocated; 18.30 GiB free; 59.72 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF