Segmentation fault

[kokyriakidis]

I get a segmentation fault when I try to run the RAFT stage using the following command:

COVERAGE=$(grep "homozygous" "${OUTPUT_PATH}/1_HIFIASM_ASSEMBLIES/${SAMPLE_NAME}/${SAMPLE_NAME}.errorcorrect.log" | tail -1 | awk '{print $6}')

/var/lib/slurm/slurmd/job1460416/slurm_script: line 96: 3795738 Segmentation fault      (core dumped) 
raft -e ${COVERAGE} -o "${OUTPUT_PATH}/1_HIFIASM_ASSEMBLIES/${SAMPLE_NAME}/${SAMPLE_NAME}.fragmented" "${OUTPUT_PATH}/1_HIFIASM_ASSEMBLIES/${SAMPLE_NAME}/${SAMPLE_NAME}.errorcorrect.ec.fa" "${OUTPUT_PATH}/1_HIFIASM_ASSEMBLIES/${SAMPLE_NAME}/${SAMPLE_NAME}.overlaps.paf"

Tried to rerun it and got the same error again. I am using a node with 128 CPUs and 1500GB RAM.

I am running the pipeline with the new High Accuracy ONT Ultra long dataset that was released recently.

[cjain7]

Hi @kokyriakidis

Can you share the complete logs of both Hifiasm and RAFT runs?
There is also a possibility that your Hifiasm run did not complete properly.
Also, do you mind sharing the link to the new dataset? We would also try to run at our end.

Chirag

[kokyriakidis]

I rerun it from the start and got the same error.

This is the log from RAFT:
HACUL_RAFT.10122023.1467406.log

And these are the logs from Hifiasm:

HACUL_RAFT.errorcorrect.log
HACUL_RAFT.finalasm.log
HACUL_RAFT.getOverlaps.log

The HAC UL dataset is released at the following address:

https://labs.epi2me.io/gm24385_ncm23_preview/

[cjain7]

Thanks for sharing the information.

Unfortunately I am unable to reproduce the segmentation fault issue. Hifiasm-RAFT workflow on the new ONT UL simplex dataset ran fine at our end. I am copy-pasting the commands here if you wish to verify your scripts again.

Hardware: AMD EPYC 7763 (128 cores, 512 GB RAM)
Software: gcc 11.2.0, hifiasm v0.19.8-r603, RAFT (commit: fb8dcc0)

Downloading the data:

aws s3 sync --no-sign-request s3://ont-open-data/gm24385_2023.12/ HG002_UL_highacc
samtools fasta -@4 -0 reads.fasta all_pass.vhg002v1.bam

Read length statistics:

N50 read length: 91670, count of reads=5552000, max read length: 1397771, total bases: 126.9 Gbp

Commands (Hifiasm-RAFT):

$HIFIASM -o errorcorrect -t 128 --write-ec reads.fasta 2> errorcorrect.log
COVERAGE=$(grep "homozygous" errorcorrect.log | tail -1 | awk '{print $6}')
$HIFIASM -o getOverlaps -t 128 --dbg-ovec errorcorrect.ec.fa 2> getOverlaps.log
cat getOverlaps.0.ovlp.paf getOverlaps.1.ovlp.paf > overlaps.paf
$RAFT -e ${COVERAGE} -o fragmented errorcorrect.ec.fa overlaps.paf
$HIFIASM -o finalasm -t 128 -r1 fragmented.reads.fasta 2> finalasm.log

Command (Hifiasm alone):

$HIFIASM -o finalasm -t 128 reads.fasta 2> finalasm.log

Here are the statistics of contigs obtained by using Hifiasm-RAFT :

[Hap1] Length: 3.047 Gbp, N50: 81.3 Mbp
[Hap2] Length: 3.072 Gbp, N50: 50.7 Mbp
Runtime: ~12 hours

Using Hifiasm only:

[Hap1] Length: 3.439 Gbp, N50: 15.0 Mbp
[Hap2] Length: 3.435 Gbp, N50: 18.4 Mbp
Runtime: ~6 hours

Log files from Hifiasm-RAFT run:

• hifiasm_errorcorrect.log
• hifiasm_getOverlaps.log
• raft.log
• hifiasm_finalasm.log

One thing which I did not expect is the unusual assembly length with Hifiasm. I am also rerunning the above analysis with the scaffolding --dual-scaf option of Hifiasm; will post the results here once that finishes.

If you are able to identify the cause of the segmentation faults at your end, let us know.

Chirag

[kokyriakidis]

I keep getting a double free or corruption (!prev) error.

A hifiasm only job completes successfully and I get more or less the same stats as yours.

Is it possible to share the 2 haplotypes generated with RAFT?

Konstantinos

[cjain7]

Since the files are too big to share here, I have emailed you a OneDrive link.

To help with reproducing the issue, can you convey the GCC version and the operating system you are using? Do you observe the same error when you run RAFT on the toy dataset listed in the README?

[kokyriakidis]

When I run the toy example, everything works fine.

In the real dataset I keep getting a Segmentation fault (core dumped)

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0