map and filter operators not working correctly when using Google Batch

[saberiato]

Hi,
Recently, I realized that when using google-batch as executor, and applying map or filter operators on a Channel, the result is not as expected.

Consider this scenario:

samples = Channel.of(['sample1', 'sample2', 'sample3'])

samples
       .map { sample ->
             fasta = sample + '.fasta'
             tuple(sample, fasta)
       }
       .filter { sample_fasta ->
              sample = sample_fasta[0]
              fasta = sample_fasta[1]
              file(fasta).isEmpty()
        }
        .set { new_samples }

Let's assume that sample2.fasta exists, and the others are not.
So, it is expected to have new_samples like this:

new_samples = [ ['sample1', 'sample1.fasta'], ['sample3', 'sample3.fasta'] ]

However, it is not always the case... sometimes I get results like this:

new_samples = [ ['sample1', 'sample3.fasta'], ['sample3', 'sample1.fasta'] ]

Which means that the channel elements get mixed up... and make the results unrealiable.

This was only a simple example... I used splitCsv, map, and filter in different settings, and again the results are not reliable and the channel tuples get mixed up.

Is this a bug, or I'm missing something?

[mribeirodantas]

Could you please share a minimal reproducible example? One in which locally it works fine, but on Google Batch it doesn't?

By the way, you mean samples = Channel.of('sample1', 'sample2', 'sample3') , right? (No brackets)