Understanding the impact of sampleCells

[parashardhapola]

Hi @rcorces and @jgranja24

Great job with the package! Even more with all the maintenance and support. Very much appreciated.

I have a question about the sampleCells parameter in addIterativeLst, 'addUMAP' and addClustering functions.
From what I understand, only a fraction of cells (10k by default ) are used to 'learn the manifold' and these cells are selected at random. This helps reduce memory requirement and overall helps speed up Archr's workflow.

By using only a fraction of cells, do I risk that a rare population of cells will not be visualized as a separate cluster in the UMAP and that such a population will not make a cluster partition of its own?

Or is there a step that I'm missing? Are there any workarounds to avoid this kind of problem?

Thanks!

[rcorces]

addIterativeLSI() - there is sampleCellsPre and sampleCellsFinal these do different things. In my understanding, it is really sampleCellsFinal that will cause the problem that you mention and this is why it defaults to NULL

addUMAP() - Yes - sampling cells here could lead to poor visualization of rare populations of cells

addClusters() - Yes - sampling cells here could lead to a rare population of cells not being called as a cluster.

[parashardhapola]

Thanks for your prompt answer!

So is this a workaround or 'best practices' for such a scenario where there are more than 500K cells and also rare cell population with, say, <100 cells?

Is there a way to know how much memory would be required if I increase the value of sampleCells to, say 100K?
Would I still be able to process it under 32GB RAM?

[rcorces]

RAM usage will depend on how many threads you give ArchR to play with. For 32 GB, I would limit to 8 threads. But these things are so environment specific it is hard to give recommendations. See Sup Fig 8F below for our benchmarking. I think you're going to need to try a few different parameter sets to find what works for you. In my mind, anything less than 200 cells is dubious and you wont have enough data from those 100 cells to do anything with downstream.

[parashardhapola]

Hi again,

You are right, having just 100 cells might not be enough for much of downstream analysis, but simply identifying them might still be a breakthrough in itself in some studies. Obviously the cells will then need to be enriched and assayed again for improved characterization.

You are quite right that suggesting the exact RAM requirements might be hard due to environment specificities.

Figure 8F is interesting, but from our discussion, I think that it is not relevant for my question because all this benchmarking was done using default sampleCells of 10K. Am I wrong in this assumption? Shall I not expect the memory requirement to jump drastically if I increase sampleCells for these benchmarks?

[rcorces]

I think if you really want to capture groups of 100 cells out of a lot of cells, then you're going to need to look for a computational setup that can handle the analysis without sub-sampling the cells. Even so, I doubt they will show up unless they are extremely distinct.