mcmapply memory error

[asifzubair]

When doing the 3vs8 analysis with default settings, getting the following error:

.Quitting from lines 458-473 (atacseeker.Rmd) 
��Error in mcfork() : 
  unable to fork, possible reason: Cannot allocate memory
Calls: <Anonymous> ... eval -> mcmapply -> mclapply -> lapply -> FUN -> mcfork
��
��Execution halted
�=Warning message:
system call failed: Cannot allocate memory 

[ttriche]

probably a nested loop of some sort -- you may need to replace mcmapply
and/or farm the analysis out over multiple instances if the child processes
are eating up too much RAM on one AMI.

--t

On Sun, May 15, 2016 at 3:34 PM, Asif Zubair notifications@github.com
wrote:

When doing the 3vs8 analysis with default settings, getting the following
error:

.Quitting from lines 458-473 (atacseeker.Rmd)
��Error in mcfork() :
unable to fork, possible reason: Cannot allocate memory
Calls: ... eval -> mcmapply -> mclapply -> lapply -> FUN -> mcfork
��
��Execution halted
�=Warning message:
system call failed: Cannot allocate memory

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#9

[asifzubair]

this is the exact invocation:

command = paste("bash bamToBigWig_dedup.sh", bam.files)
tmp <- mcmapply(system, command = command)

I was trying to call a script in parallel for converting bam to BigWigs. However, now i've changed to mapply. Of course, now it takes longer :/

[ttriche]

better to take longer and complete successfully than to quickly fail :-)

--t

On Sun, May 15, 2016 at 10:36 PM, Asif Zubair notifications@github.com
wrote:

this is the exact invocation:

command = paste("bash bamToBigWig_dedup.sh", bam.files)tmp <- mcmapply(system, command = command)

I was trying to call a script in parallel for converting bam to BigWigs.
However, now i've changed to mapply. However, of course now it takes
longer :/

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9 (comment)

[asifzubair]

also, this is happening when i have dedup = FALSE. In fact, this is causing PWMEnirch to fail as well. Please see this issue. I think the simpler solution to our problems will be to clear memory once we are done with csaw as indicated in this SO. In any case, once we have got the differentially accessible regions and have exported the SummarizedExperiment produced by csaw, we shouldn't need it for further downstream analysis - LOLA, BigWigs and PWMEnrich

[ttriche]

true. gc(,T) sometimes helps (run it 10 times if you're going to run it
once; R uses a generational garbage collector and this legitimately can
help clear RAM)

--t

On Mon, May 16, 2016 at 1:20 AM, Asif Zubair notifications@github.com
wrote:

also, this is happening when i have dedup = FALSE. In fact, this is
causing PWMEnirch to fail as well. Please see this issue
#10. I think the
simpler solution to our problems will be to clear memory once we are done
with csaw as indicated in this SO
http://stackoverflow.com/questions/8813753/what-is-the-difference-between-gc-and-rm.
In any case, once we have got the differentially accessible regions and
have exported the SummarizedExperiment produced by csaw, we shouldn't
need it for further downstream analysis - LOLA, BigWigs and PWMEnrich

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9 (comment)

[asifzubair]

For now, I'm going to do the following:

rm(list=setdiff(ls(),
    c("test", "dump_dir", "output_dir", "dedup", "bam.files", 
    "compute_library_complexity", "generate_BigWigs", "do_motif_analysis", 
    "is.sig", "diffAccRanges")))
gc()

Once the significant regions have been called, I really only need the bed file for motif enrichment analysis. Also, apropos your suggestion on speed up, PWMEnrich does have a parallel processing mode and that might help with speedup.

[ttriche]

re: PWM:

just make sure it's using openMP and not forking a zillion processes :-)

--t

On Mon, May 16, 2016 at 1:14 PM, Asif Zubair notifications@github.com
wrote:

For now, I'm going to do the following:

rm(list=ls())
gc()diffAccRanges = import(file.path(output_dir, "diffAccRanges.NS.fdrLt005.hg19.bed"), genome="hg19")

Once the significant regions have been called, I really only need the bed
file for motif enrichment analysis. Also, apropos your suggestion on speed
up, PWMEnrich does have a parallel processing mode and that might help
with speedup.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9 (comment)

[asifzubair]

I think this might be the case:

Motif scanning is the most time consuming operation. Because of this, the package has a support for parallel motif scanning using the R parallel core package.

[ttriche]

gah, might want to skip it. People can wait longer if the bottleneck is on
someone else's computer (e.g. Amazon's or Illumina's) instaed of locally

--t

On Mon, May 16, 2016 at 5:04 PM, Asif Zubair notifications@github.com
wrote:

I think this might be the case:

Motif scanning is the most time consuming operation. Because of this, the
package has a support for parallel motif scanning using the R parallel
core package.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9 (comment)

[asifzubair]

this is still failing, despite memory clean up, when dedup = FALSE:

�.Quitting from lines 470-485 (atacseeker.Rmd) 
���Error in mcfork() : 
  unable to fork, possible reason: Cannot allocate memory

[ttriche]

it looks like you are still parallelizing it at process level -- maybe just
use lapply or mapply as needed? It will take longer but it should not seg
fault.

--t

On Tue, May 17, 2016 at 12:39 AM, Asif Zubair notifications@github.com
wrote:

this is still failing, despite memory clean up, when dedup = FALSE:

�.Quitting from lines 470-485 (atacseeker.Rmd)
���Error in mcfork() :
unable to fork, possible reason: Cannot allocate memory

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9 (comment)

[asifzubair]

I did try that, but even systems calls are failing:

## Warning in system(cmd5): system call failed: Cannot allocate memory

[ttriche]

You may need to print out memory usage ( gc(TRUE, TRUE) will do it) along
the way in order to debug this. How much RAM is in a standard BaseSpace
EC2 AMI?

--t

On Tue, May 17, 2016 at 9:23 AM, Asif Zubair notifications@github.com
wrote:

I did try that, but even systems calls are failing:

Warning in system(cmd5): system call failed: Cannot allocate memory

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9 (comment)

[asifzubair]

60gb. Yes, I read somewhere that it is always advisable to call gc(T,T) before running system. That's what I am testing out. Also, when dedup = TRUE, I don't find any significant regions. However, for completeness, I want this option to be available.

[asifzubair]

this is what gc(T,T) spits out

�|Garbage collection 489 = 337+76+76 (level 2) ... 
433.0 Mbytes of cons cells used (17%)
7174.9 Mbytes of vectors used (29%)