AWS Fusion integration is less efficient than NO Fusion when concatenating a large number of files

[wikiselev]

Bug report

AWS Fusion integration is less efficient than NO Fusion when concatenating a large number of files (~20,000).

Expected behavior and actual behavior

Using Fusion is supposed to make file operations faster, but I observe the opposite.

Steps to reproduce the problem

My pipeline has a large number of the same processes that generate a large number of csv files. These files contain only a single row with 10 columns each (200-300 bytes in size). The files are then concatenated together using a single process with the following script:

find . -name "*.csv" -exec cat {} \\; > combined_raw.txt

Program output

I've tested the pipeline for different number of csv files and with/without Fusion and got the following results:

# of csv files	With Fusion	Without Fusion
32	OK (12GB RAM, 1 min)	Didn't run
1,000	OK (12GB RAM, 36 min)	Didn't run
10,000	OK (24GB RAM, 5.5 hours)	Didn't run
20,000	Failed with both 12GB and 24GB of RAM (maxRetries = 5), cancelled after waiting too long with 36GB of RAM	OK (24GB RAM, 3 hours)

Environment

• Tower + AWS Batch Spot instances

[pditommaso]

Look @jordeu new challenge for easter time! 😆

@wikiselev should cat *.csv > combined_raw.txt do the same?

[jordeu]

How do you setup the Compute Environment? Are you using Tower? With "fast instance storage" enabled?

[wikiselev]

Look @jordeu new challenge for easter time! 😆

@wikiselev should cat *.csv > combined_raw.txt do the same?

Yes, that was my original code, but then I found some discussions about a too large number of arguments for cat and decided to use find to be on a safe side.

[wikiselev]

How do you setup the Compute Environment? Are you using Tower? With "fast instance storage" enabled?

Yes, this is exactly the setup that I call With Fusion.

[jordeu]

These 20.000 files are generated by 20.000 different processes or and then declared as input of the "concatenate" process?

[wikiselev]

These 20.000 files are generated by 20.000 different processes or and then declared as input of the "concatenate" process?

They are generated by 20,000 processes. In the workflow I pass generate_files.out.collect() to the process that concatenate the files.

[wikiselev]

Actually, after answering to @pditommaso I realised that the long failed run was using just cat without find. Though I think (hope!) it does not make a difference... So the table describing scripts used in the concatenation process looks like this:

# of csv files	With Fusion	Without Fusion
32	find + cat	Didn't run
1,000	find + cat	Didn't run
10,000	find + cat	Didn't run
20,000	cat	find + cat

[jordeu]

I'll test this use case. It's a difficult one and I don't expect to get better results, but it should be possible to at least give similar results.

Fusion performance improvements come from the fact that it can download/upload files on the background while the process is running. Also is design to improve the performance when dealing with big files.

And this use case is a bit of the opposite, just a pure download and upload of small files and nearly nothing is done by the process.

The best solution that you can build now (with or without Fusion) is to add an intermediate step that in parallel concatenates batches of 1000 files and then a final process that collects all of them into a single file.

[pditommaso]

I wonder if could just be handled via Nextflow collectFile?

[wikiselev]

I wonder if could just be handled via Nextflow collectFile?

Thanks, Paolo! This approach seems to work. In this case the concatenation is performed by Nextflow itself in the NF main process/job. It takes some time (probably more than an hour) and during that time Tower does not do anything (does not start the next process until collectFile is finished). So, may be confusing for someone as it is not clear what is happening during that time...

It would also be good to add a use case of sending multiple files to colectFile to the documentation on collectFile. Currently there are only examples with multiple items and a single file.

Thanks again, for you quick response, @pditommaso , the issues seems to be resolved for me.

[pditommaso]

Moving this to the discussion, because it's not a Nextflow issue

[jordeu]

We found a bug that was making this use case highly inefficient. We've fixed it on latest Fusion version automatically available when running Nextflow 23.04

With the new version when we run this pipeline

nextflow run jordeu/nf-tests -r filecollect -profile fusion --files 20000

with an AWS Batch + Fusion v2 + fast storage Tower compute environment the collector process that concatenates 20k files it takes 37 minutes.

[wikiselev]

Amazing, thanks for the update! I will need some time to test this. Will update once done.

[pditommaso]

Go Vlad! break the system! 😆