Potential issue with fastq compression

[Hoohm]

Hello!

We are trying out scan_fastq and read_fastq atm and the problem is it crashes on files coming from basespace.

It does work when we use our own compressed files.

8192rows [00:00, 13066.77rows/s]
Traceback (most recent call last):
  File "/home/ubuntu/working_dir/notebooks/pipeline_dev/COMB-243/test.py", line 7, in <module>
    print(df.collect())
          ~~~~~~~~~~^^
  File "/home/ubuntu/miniforge3/envs/dev/lib/python3.13/site-packages/polars/_utils/deprecation.py", line 97, in wrapper
    return function(*args, **kwargs)
  File "/home/ubuntu/miniforge3/envs/dev/lib/python3.13/site-packages/polars/lazyframe/opt_flags.py", line 330, in wrapper
    return function(*args, **kwargs)
  File "/home/ubuntu/miniforge3/envs/dev/lib/python3.13/site-packages/polars/lazyframe/frame.py", line 2407, in collect
    return wrap_df(ldf.collect(engine, callback))
                   ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
polars.exceptions.ComputeError: caught exception during execution of a Python source, exception: Exception: DataFusion error: IoError(Kind(UnexpectedEof))

The script basically reads a file (tested s3 and locally).

Any idea of how to confirm this suspicion?

[Hoohm]

My current solution is a bit ugly but works, I decompress the file, read the decompressed one and then delete it.

[mwiewior]

@Hoohm could you pls share some more details on the the compression used (GZip or BGZF )+ and a small code snippet you use?

[Hoohm]

I used pigz with defaults. (GZip)

I don't know what BCLconvert uses :(
Here are the params that we can see were used:

[BCLConvert_Settings]
SoftwareVersion,4.3.16
FastqCompressionFormat,gzip

[Hoohm]

code is simply: zcat path.fastq.gz | pigz > new_path.fastq.gz
the original file does not work, the new one works fine.

[mwiewior]

Thanks for reporting this issue. I'll try to reproduce it first.

[Hoohm]

I have an idea, I might be able to reproduce it by running bclconvert myself

…

On Sat, 6 Dec 2025 at 19:00, Marek Wiewiórka ***@***.***> wrote: *mwiewior* left a comment (biodatageeks/polars-bio#258) <#258 (comment)> Thanks for reporting this issue. I'll try to reproduce it first. — Reply to this email directly, view it on GitHub <#258 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJVO2BUMY7ABCPOMU3MAOL4AMKVHAVCNFSM6AAAAACNYZFCU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMMRQHAZTEMBRGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Hoohm]

moved down

[Hoohm]

Number of members

Could not recreate the issue yet, but I have some interesting findings.

using grep -aob $'\x1f\x8b\x08' original.fastq.gz I get 473 members but only 4 on the recompressed by pigz.

I checked that the recompressed version did actually get all reads through in read_fastq and it's the case.

Concatenated Gzipped files

using a concatenated fastq seems to break on read_fastq

# Split into chunks of 4 lines (1 read per file)
split -l 4 fastq_100.fastq fastq_100_read_

# Concatenate 100 gzipped members
: > fastq_100_100members.fastq.gz
for f in fastq_100_read_*; do
    gzip -c "$f" >> fastq_100_100members.fastq.gz
done

If you read this with plb.read_fastq you only get 1 read back

At least there is an issue if you concatenate multiple gzip files as read_fastq will only catch the first "member".

[mwiewior]

hey @Hoohm is there anything to be fixed here?

[Hoohm]

Technically I have not been able to recreate the problem by creating the files myself. I might just ask if we can share some data directly with you.

…

On Sat, 14 Feb 2026 at 13:18, Marek Wiewiórka ***@***.***> wrote: *mwiewior* left a comment (biodatageeks/polars-bio#258) <#258 (comment)> hey @Hoohm <https://github.com/Hoohm> is there anything to be fixed here? — Reply to this email directly, view it on GitHub <#258 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJVO2FHJMRPMKFADJRUXTD4L4HBHAVCNFSM6AAAAACNYZFCU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSMBRHAYTCMRSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>