Memory usage for concatenating outputs

[Jay-Leung]

Hello @chenweng1991 ,

Thanks for the great work! I am trying to run the tutorial and my dataset. However, when I was trying to run the final concatenation, it keeps running into OOM issues. I have attached the error output here. I have also increased memory to 600G on my slurm job and the problem still persists. Can I check what the memory requirements are for the tutorial dataset or if I am possibly doing something wrongly? I have also pasted the job script here.

Error output:

/home/users/astar/gis/leungjy1/Github/REDEEM-V/mitoConsensus//Finalize.sh: line 19: 1910984 Killed          
python3 $mitoConsensus/AddQualifiedTotalCts.py $WD

Attaching package: ‘dplyr’

The following objects are masked from ‘package:plyr’:

    arrange, count, desc, failwith, id, mutate, rename, summarise,
    summarize

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Error in type.convert.default(data[[i]], as.is = as.is[i], dec = dec,  :
  long vectors not supported yet: ../../../include/Rinlinedfuns.h:537
Calls: read.table -> type.convert -> type.convert.default
Execution halted
Traceback (most recent call last):
  File "/home/users/leungjy1/Github/REDEEM-V/mitoConsensus//RemoveStrandBias.py", line 16, in <module>
    with open(StrandBiaseBlackList_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/users/leungjy1/scratch/testing/ReDeeM_tutorial/Out_mitoConsensus/final/StrandBiaseBlackList'
slurmstepd: error: Detected 1 oom_kill event in StepId=204302.batch. Some of the step tasks have been OOM Killed.

Script:

#!/bin/bash

#SBATCH --job-name=concat_mito
#SBATCH --partition=cpu
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=600G
#SBATCH --time=1-00:00:00
#SBATCH --output=./log/%x-%j.out
#SBATCH --error=./log/%x-%j.err

# Export environment variables
export REDEEM_V=$HOME/Github/REDEEM-V
export mitoConsensus=$REDEEM_V/mitoConsensus/
export WORKING_DIR=$HOME/scratch/testing/ReDeeM_tutorial/Out_mitoConsensus

# State no. of bam files/threads from previous steps
Threads=24

# Load modules
module load mambaforge
mamba activate atac-seq
module load samtools

# Change directory and create necessary directories
cd ${WORKING_DIR} || { echo "Failed to change directory to ${WORKING_DIR}"; exit 1; }

# Print environment variables for debugging
echo "REDEEM_V: $REDEEM_V"
echo "mitoConsensus: $mitoConsensus"
echo "WORKING_DIR: $WORKING_DIR"
echo "Current directory: $(pwd)"

# Concat var calling outputs
$mitoConsensus/Finalize.sh ${WORKING_DIR} ${Threads} $mitoConsensus

Thank you!