Error: std::bad_alloc

[ConYel]

I am using a SingleCellExperiment object to create a reference.
The object is quite large and I have imported it using zellkonverter using out of memory representation.
The data is public and can be downloaded using these steps from here: https://github.com/AllenInstitute/abc_atlas_access/blob/main/notebooks/WHB_cluster_annotation_tutorial.ipynb

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from pathlib import Path

from abc_atlas_access.abc_atlas_cache.abc_project_cache import AbcProjectCache
abc_cache.get_directory_metadata('WHB-taxonomy')
abc_cache.get_directory_metadata('WHB-10Xv3')
abc_cache.get_directory_data('WHB-10Xv3') # ~70 Gb

Then after some preprocessing to find the classes I wanted a subset the dataset I tried to make a reference using:

h5ad_file <- "./vol_data/_data/expression_matrices/WHB-10Xv3/20240330/WHB-10Xv3-Nonneurons-log2.h5ad"
sce <- zellkonverter::readH5AD(h5ad_file, use_hdf5 = TRUE)
identical(colnames(sce), cell_extended$cell_label)
colData(sce)$supercluster <- cell_extended$supercluster
names(assays(sce)) <- "logcounts"
trained <- trainSingleR(sce, labels = sce$supercluster, test.genes = rownames(sce), aggr.ref = TRUE)
Error: std::bad_alloc

How could I trace what could cause the error?

Thank you in advance!

[LTLA]

traceback() would help to identify the problem.

Also note that for single-cell references, one usually sets de.method= to something other than classic.

[ConYel]

Thank you, I am not sure exactly what the issue is.
here is part of the traceback()

traceback()
8: stop(structure(list(message = "std::bad_alloc", call = NULL, 
       cppstack = NULL), class = c("std::bad_alloc", "C++Error", 
   "error", "condition")))
7: (function (x, groups, num_groups, block, block_weight_policy, 
       variable_block_weight, threshold, num_threads, compute_delta_mean, 
       compute_delta_detected, compute_cohens_d, compute_auc) 
   {
       .Call("_scrapper_score_markers_pairwise", PACKAGE = "scrapper", 
           x, groups, num_groups, block, block_weight_policy, variable_block_weight, 
           threshold, num_threads, compute_delta_mean, compute_delta_detected, 
           compute_cohens_d, compute_auc)
   })(<pointer: 0x7fcea3d6a860>, groups = c(7L, 7L, 7L, 7L, 7L, 
   7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 
   7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 
   7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 
   7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 
.
.
.
.


   "Microglia", "Microglia", "Microglia", "Microglia", "Microglia", 
   "Microglia", "Microglia", "Microglia", "Microglia", "Microglia", 
   "Microglia", "Microglia", "Microglia", "Microglia", "Microglia", 
   "Microglia", "Microglia", "Microglia", "Microglia", "Microglia"
   ), num.threads = 1L, all.pairwise = TRUE, compute.delta.detected = FALSE, 
       compute.delta.mean = FALSE, compute.auc = TRUE, compute.cohens.d = FALSE)
4: do.call(scrapper::scoreMarkers, c(list(ref, groups = labels, 
       num.threads = num.threads, all.pairwise = TRUE, compute.delta.detected = FALSE, 
       compute.delta.mean = FALSE, compute.auc = compute.auc, compute.cohens.d = compute.cohens.d), 
       de.args))
3: .get_genes_by_de(ref, labels, de.n = de.n, de.method = de.method, 
       de.args = de.args, num.threads = num.threads)
2: .identify_genes(ref = curref, labels = curlabels, genes = genes[[l]], 
       de.method = de.method, de.n = de.n, de.args = de.args, restrict = restrict, 
       test.genes = test.genes, num.threads = num.threads)
1: trainSingleR(sce, labels = sce$supercluster, de.method = "wilcox", 
       test.genes = rownames(sce), aggr.ref = TRUE)

I will retry with part of the dataset to see if it is possible to get something at first.

[ConYel]

With this it run:

cell_s <- cell_extended %>%
    group_by(supercluster) %>%
    sample_n(1000)
sce_s <- sce[1:10000, cell_s$cell_label]

trained <- trainSingleR(sce_s, labels = sce_s$supercluster, de.method = "wilcox", test.genes = rownames(sce_s), aggr.ref = TRUE)

I am not sure how to proceed on this terms.
When the big dataset run didn't seem to spike RAM (as expected I guess?)

Could it be an issue of Disk Space?
Do you have any thoughts on how I could create that reference with less cells but more chance to have good representation of the dataset?

EDIT:
Is it possible to use slices of the dataset (all genes(~50k), but ~100k cells per run, or something like that), make them into references and then combine them? Would that be a solution in some way?
2nd EDIT:
I managed to run that dataset with a sample of ~4000 cells and all genes (4k cells as the lowest cluster had ~4k cells) and afterwards I tried to run another (larger dataset, ~2m cells), subsetting it to ~10k cells (again all genes, and the clusters this time were 31),
Unfortunately it threw another error, so I sampled-down even more to 4k cells per cluster and (without using multi-cores) it worked.
RAMhas arrived around 70+ Gb of usage.

Is there a way to combine those two? Seem that they are S4Vectors.

[LTLA]

Hm. The memory usage of scoreMarkers is linear with respect to the number of cells, so the raw size shouldn't matter.

The most likely cause is that you have lots of groups, i.e., unique labels. The memory usage of scoreMarkers with compute.auc=TRUE and all.pairwise=TRUE is quadratic with respect to the number of groups as it computes AUCs for all all pairwise comparisons. So if you had 100 groups and 50000 genes, it would create a 3D array of 100 * 100 * 50000 entries, which would end up being about 4 GB. This gets out of hand once you have many more groups.

If you have too many groups, I'd suggest using some broader labels for the time being. There are some opportunities to reduce memory usage for scoreMarkers() in this case, but I probably won't be able to get to it before the next release.