addCluster spend oneday without result

[chen-zhan]

Hi, thanks for the great analysis software.
I try to run "addCluster", but this step have spend so long. I have no idea why it need so long.
Could maybe it was caused by my huge data? I have 990,000 cells. But my colleague just use less than 1min with 440,000 cell.
Or it cause by that I add the parametre sampleCellsFinal=500000 when run "addIterativeLSI".
Thanks for you advise again.

` proj <- addIterativeLSI(
ArchRProj = proj,
useMatrix = "TileMatrix",
name = "IterativeLSI",
iterations = 4, #up
clusterParams = list( #See Seurat::FindClusters
resolution = c(0.2),
sampleCells = 10000,
n.start = 10
),
force = TRUE,
varFeatures = 25000, #down
dimsToUse = 1:30,
sampleCellsPre = 100000,
sampleCellsFinal=500000
)

#Remove Batch
proj <- addHarmony(
    ArchRProj = proj,
    reducedDims = "IterativeLSI",
    name = "Harmony",
    force = TRUE,
    lambda = 0.6,
    epsilon.harmony = -Inf,
    corCutOff = 0.7,
    max.iter.harmony = 10,
    groupBy = "Sample"
)
saveRDS(proj,"proj_3.1_before.rds")

#Cluster
proj <- addClusters(
    input = proj,
    reducedDims = "Harmony",
    method = "Seurat",
    force = TRUE,
    corCutOff = 0.7,
    name = "Clusters",
    resolution = 1,
    maxClusters = 40
)

`
ArchR-addIterativeLSI-1f22f7bcd0953-Date-2022-11-15_Time-09-24-07.log
ArchR-addClusters-1f22f2b6b030b-Date-2022-11-16_Time-19-20-18.log

[rcorces]

1 min for 500,000 cells sounds unlikely to me. I think this depends most on the number of threads and the overall computational configuration. Youve given a lot of threads which will also take up a lot of memory. If you have an inbalance in threads and memory, then you would use up all of your available memory and potentially start using swap memory which is terribly slow. Beyond that, I dont have much insight here. addClusters is really just a wrapper around Seurat::FindClusters

[chen-zhan]

Thanks for you reply. I set num_proc=10,virtual_free=500g when I submitted the work. I think it is enough for me to run this work. Do you have any idea about the second assumption. This is my first time try to sampleCells in the last LSI, which is the another difference with my colleague. And it is easy for my to run this step when I use the default script with the a fewer data.

[chen-zhan]

I try to download the proj.RDS, and run in the personnal laptop. Now 3 hours escaped, it is still running. Would appreciate it if you could give me some advice about how to accelerate this step.

[rcorces]

Sorry, I dont have any advice for you. First off, ArchR is not supported on Windows. Second, the aspects of your workflow that are taking long amounts of time are calls to Seurat. Lastly, downloading a project RDS file and loading it with readRDS is not the supported way of working with an ArchRProject (saveArchRProject() and loadArchRProject() are the only correct ways to do this). I wont have any additional insight to this.

[chen-zhan]

Hi, dear rcorces. I had confused again when I run addCluster. I try to run addCluster with "Harmony" and "IterativeLSI" as the same time sampleCells=500,000.
proj <- addClusters( input = proj, reducedDims = "IterativeLSI", method = "Seurat", name = "Clusters", resolution = 0.8, sampleCells = 500000, )
At the first time, using "IterativeLSI", it complete spent 1 hours.
proj <- addClusters( input = proj, reducedDims = "Harmony", method = "Seurat", name = "Clusters", resolution = 0.8, sampleCells = 500000, )
Then, I try to run with "Harmony". The problem come out, which is same to this discssions. There is no result.

So I was confused. When I deal with the scRNA, I will use reducedDims after Harmony. But the ArchR's tuition is use reducedDims before Harmony(https://www.archrproject.com/bookdown/clustering-using-seurats-findclusters-function.html), despite have run "Batch Effect Correction wtih Harmony“ in step4.4. So why we need to choose "IterativeLSI". Last but no least, I can not get the result that is because reducedDims = "Harmony"? And why cause this?

Thanks for your patience.
ArchR-addClusters-277351cabaed-Date-2022-11-23_Time-10-10-08.log
ArchR-addClusters-a1ac4e7f25a5-Date-2022-11-23_Time-09-00-54.log

[rcorces]

I'm not sure I understand your questions. In the manual, we dont use the Harmony reducedDims for downstream assays because in the tutorial data Harmony doesnt really do anything.