Unfinished execution probably due to memory issue

[eltonjrv]

Dear Federico,
Following up on a similar memory issue as the one already posted by other user, I followed your recommendations on creating a DelayedArray class for my SCE assay (14k genes x 227k cells):

My SCE object

A1Bmtx_sce
class: SingleCellExperiment
dim: 13966 271111
metadata(0):
assays(1): counts
rownames(13966): LINC00115 NOC2L ... MT-ND6 MT-CYB
rowData names(0):
colnames(271111): SRR7666705 SRR7666706 ... TTTGTCAAGCTGAACG.1
TTTGTCACAATCGGTT.1
colData names(3): cells batch Biological_Condition

Transforming the "batch" field from colData to a factor

colData(A1Bmtx_sce)$batch <- as.factor(colData(A1Bmtx_sce)$batch)

Converting the assay (counts) to a DelayedArray

library(DelayedArray)
assay(A1Bmtx_sce) = DelayedArray(assay(A1Bmtx_sce))
class(assay(A1Bmtx_sce))
[1] "DelayedMatrix"
attr(,"package")
[1] "DelayedArray"

Running newWave

A1Bzinb <- newWave(A1Bmtx_sce, K=9, X="~batch", children=24, n_gene_par=1500, n_cell_par=30000, verbose=FALSE, commondispersion=FALSE)

I submitted this code as job script to a large-memory node (768 Gb) from my cluster, and it stopped after ~30 min with the SGE exit status #37: "failed 37 : qmaster enforced h_rt, h_cpu, or h_vmem limit", reaching a maxvmem of 487 Gb, while I've asked for 720Gb.
I've also got info from my IT regarding intermediate files ( 'sharedObjectCounter', 'SO_X64_1', 'SO_X64_2', ... 'SO_X64_24') left in the /dev/shm due to the unfinished status of my newWave run.

I wonder if you would have any tip to circumvent this issue, and please let me know if you need more info.

Many thanks in advance.
Elton