Cholmod error 'problem too large' while exporting peak matrix

[romain-lopez]

Hi,

I am trying to export my peak matrix from archR (300k cells, 150k peaks, one large arrow file). Unfortunately, I am having this error.

peak <- getMatrixFromProject(proj, useMatrix='PeakMatrix')
ArchR logging to : ArchRLogs/ArchR-getMatrixFromProject-106023160ae35-Date-2022-11-19_Time-21-56-46.log
If there is an issue, please report to github with logFile!
Error in .rbind2Csp(x, y) :
Cholmod error 'problem too large' at file ../Core/cholmod_sparse.c, line 89

Is there a chance that this comes from all cells being in one arrow file? Or is it just a limitation of R, or of my memory (300Gb RM + 500Gb SWAP)? Thanks so much for your help, looking forward to reading you!

EDIT: I looked on this forum, and made sure it was not a multi-threading problem by also running the command with a single thread. It still gives me the same error.

Log ArchR-getMatrixFromProject-106023160ae35-Date-2022-11-19_Time-21-56-46.log

[rcorces]

Please search your topic prior to posting:
https://github.com/GreenleafLab/ArchR/search?q=cholmod&type=issues

The upper limit in vector size in R is 2*10^9. 300,000 x 150000 = 45 * 10^9

[romain-lopez]

Thanks Ryan for your answer. I understand your point, but I conceptually disagree with your calculations. ArchR should be able to deal with sparse matrices all the way to exporting them into h5 / MM format? In that case, I expect around 1% non-zero values, which would make it a "valid" vector size.

Is the error related to this upper limit while considering sparsity, or because it densifies the matrix?

Anyway, feel free to close this issue, I have just split my samples in two and it works. But it could be nice to consider adding this to the docs, or adding a flag in the code that gives a warning before the error.

[rcorces]

ArchR doesnt store this information as a single sparse matrix normally. It is spread out across many individual arrow files in hdf5.
Hopefully soon, ArchR will have a more transparent way of working with the underlying data which will address your point but I dont expect that to be available until later next year.

Is the error related to this upper limit while considering sparsity, or because it densifies the matrix?

To my knowledge, it does not densify the matrix during export.

[romain-lopez]

Understood! Thanks for your help!