Seek-point spacing

[leanhdung1994]

I have a Python file:

from concurrent.futures import ProcessPoolExecutor
import tarfile, rapidgzip

def processNdjson(ndjsonName):
    with rapidgzip.open(inTarDir) as myZip:
        myZip.import_index(rapidgzipDir)
        with tarfile.open(fileobj=myZip, mode="r:*") as f:
            member = f.getmember(ndjsonName)
            dataFile = f.extractfile(member)
            for oneLine in dataFile:
                # process oneLine here

if __name__ == "__main__":
    rapidgzipDir = ...
    inTarDir = ...
    nCore = 5
    ndjsonNames = ["name1.ndjson", "name2.ndjson"]

    with ProcessPoolExecutor(nCore) as pool:
        results = pool.map(worker, ndjsonNames)

Above,

• inTarDir is the directory to a .tar.gz file that contains multiple (less than 1000) .ndjson files of approximately equal sizes. My use case is the HTML dump of Wikipedia.
• rapidgzipDir is the pre-index file to be used by rapidgzip. This allows fast random access and is a drop-in replacement for the built-in Python gzip.GzipFile.
• Each .ndjson file will be processed sequentially. It seems reading the NDJSON file is pure sequential decompression, not affected by checkpoint spacing.

As such, the index file for my .tar.gz file does not need to have many checkpoints. I would like to ask if rapidgzip allows configure the checkpoint spacing of the index file. This is possible in indexed_gzip.

Thank you for your elaboration.

[mxmlnkn]

I would like to ask if rapidgzip allows configure the checkpoint spacing of the index file.

Yes, the --chunk-size <kiB> argument controls the parallelization chunk size as well as the seek point spacing for the index. This double meaning is not ideal, but should not be much of a problem for this use case. If you have index file size problems, I also pointed you to --index-format gztool in your other issue, which should reduce the index size by ~10x and more. Shaving off the index with something like rapidgzip --chunk-size $(( 32 * 1024 )) --import-index mimi --index-format gztool --export-index mimi2 archive.gz should also work.

[leanhdung1994]

Thank you very much for your prompt respons. The following methods work for me:

1. Run in CMD:

C:\Users\Akira\anaconda3\Scripts\rapidgzip.exe --chunk-size 32768 --index-format gztool --export-index "E:\Personal Projects\tmp\tarFiles\test.gz.gzindex" "E:\Personal Projects\tmp\tarFiles\enwiktionary_namespace_0.tar.gz"

2. Run CLI in Python:

import subprocess

inTarDir = r"E:\Personal Projects\tmp\tarFiles\enwiktionary_namespace_0.tar.gz"
indexGzipDir = r"E:\Personal Projects\tmp\tarFiles\test.gz.gzindex"
rapidgzipDir = r"C:\Users\Akira\anaconda3\Scripts\rapidgzip.exe"

subprocess.run([rapidgzipDir, "--chunk-size", "32768", "--index-format", "gztool", "--export-index", indexGzipDir, inTarDir], check=True)

I have two questions:

• With option --index-format gztool, is the correct file extension is .gz.gzindex?
• Can we do it with Python API? I tried

import rapidgzip

inTarDir = r"E:\Personal Projects\tmp\tarFiles\enwiktionary_namespace_0.tar.gz"
indexGzipDir = r"E:\Personal Projects\tmp\tarFiles\entest.gz.gzindex"

with rapidgzip.open(inTarDir, chunk_size=32768) as myZip:
    myZip.export_index(indexGzipDir)

and got error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[34], line 6
      3 inTarDir = r"E:\Personal Projects\tmp\tarFiles\enwiktionary_namespace_0.tar.gz"
      4 indexGzipDir = r"E:\Personal Projects\tmp\tarFiles\entest.gz.gzindex"
----> 6 with rapidgzip.open(inTarDir, chunk_size=32768) as myZip:
      7     myZip.export_index(indexGzipDir)

File rapidgzip.pyx:648, in rapidgzip.open()

TypeError: open() got an unexpected keyword argument 'chunk_size'

[leanhdung1994]

Yes, the --chunk-size <kiB> argument controls the parallelization chunk size as well as the seek point spacing for the index.

I observe that increasing chunk-size leads to

• decrease in the index file size.
• increase in the RAM usage. I guess this is due to the increase of parallelization chunk size.

1. With chunk-size equal to 4096,

• the index file size is 1.24GB.
• RAM usage is 230MB

2. With chunk-size equal to 32768,

• the index file size is 158MB.
• RAM usage is 600MB

In my experiments, you default parallelization chunk size is already very good. I only need to adjust seek-point spacing to suit my particular .tar.gz file.

[mxmlnkn]

I observe that increasing chunk-size leads to

• decrease in the index file size.
• increase in the RAM usage. I guess this is due to the increase of parallelization chunk size.

Yes. That is the trade-off. This probably cannot be changed. You can use a larger chunk size than the index was created with by simply discarding some seek points during index import, but the other way around simply does not work as new seek points would have to be created on-the-fly after importing index, which defeats the purpose of the index.

• With option --index-format gztool, is the correct file extension is .gz.gzindex?

There is no standard extension. You can name it however you want. gztool uses .gzi in the documentation and examples.

• Can we do it with Python API? I tried

The convenience open method does not support it. You'll have to use RapidgzipFile directly, e.g.,

with rapidgzip.RapidgzipFile(inTarDir, chunk_size=32768 * 1024) as myZip:
    myZip.export_index(indexGzipDir, rapidgzip.IndexFormat.GZTOOL)

and as already shown above, export_index has an argument for the index format.

Edit: The chunk size in the Python API is in bytes as opposed to the rapidgzip CLI where the chunk size is in KiB!

[leanhdung1994]

Thank you very much for your dedicated support.

[mxmlnkn]

In my experiments, you default parallelization chunk size is already very good.

In my benchmarks from some years ago, it was close to ideal:

https://github.com/mxmlnkn/indexed_bzip2/blob/master/results/paper/plots/decompression-chunk-size-bandwidths-number-of-threads.pdf

[leanhdung1994]

In my workflow,

with rapidgzip.open(inTarDir) as myZip:
    myZip.import_index(rapidgzipDir)
    with tarfile.open(fileobj=myZip, mode="r:*") as f:
        member = f.getmember(ndjsonName)
        dataFile = f.extractfile(member)
        for oneLine in dataFile:
            # process oneLine here

With myZip.import_index(rapidgzipDir), I almost immediately seek to the beginning of a ndjson file. After that, I process it sequentially line-by-line where oneLine is a small json object (usually less than 1KB). I guess parallel decompression does not help much here. I wonder if disabling parallel decompression (something like parallelization=None) would help reduce the RAM usage by chunk buffer (required for parallel decompression). In this setting of single-thread, the parameters seek_point_spacing and readbuf_size can be set independently. Actually, indexed_gzip implemented these options.

[mxmlnkn]

With myZip.import_index(rapidgzipDir), I almost immediately seek to the beginning of a ndjson file. After that, I process it sequentially line-by-line where oneLine is a small json object (usually less than 1KB). I guess parallel decompression does not help much here. I wonder if disabling parallel decompression (something like parallelization=None) would help reduce the RAM usage by chunk buffer (required for parallel decompression). In this setting of single-thread, the parameters spacing and readbuf_size can be set independently. Actually, indexed_gzip implemented these options.

indexed_gzip is not multi-threaded, that's why configuring spacing and readbuf_size separately makes sense.

• Yes, parallelization=1 should decrease your RAM usage quite a bit, will not start any threads, and should not prefetch anything.
• If you are not seeking and only reading sequentially, then you don't even need to export/import any index.
• At this point, you might also be interested in using the Python ISA-l wrapper because this is basically what rapidgzip without any parallelization and index usage devolves into.
• These requirements also have similarities to this issue, but currently, the legacy non-parallel GzipReader is not exposed to Python anymore. I guess, memory usage could be a good argument to add it to the Python API again, and would save me the pointer to Python ISA-l wrapper. For this legacy GzipReader, the chunk size only refers to the index seek points, not any read buffering, so basically what you want.
• The rapidgzip command line tool also tries to optimize memory usage by not creating the index seek points at all if it detects simple sequential decompression. Unfortunately, the setKeepIndex method is not exposed to the Python API yet.

[leanhdung1994]

My use case is the following:

1. I would like to build index with parallelization, which is fast. I use multiprocessing, so fast seeking to an arbitrary ndjson file is crucial. Because the .tar.gz files contain a few hundred ndjson files of approximately equal size. Then I can set spacing of the index file large to reduce the index file size.
2. Next I process each ndjson file sequentially line-by-line. Each json file in the ndjson file is usually less than 1MB. So I can set readbuf_size to be small, around 4MB.

For above use case, a good combination seems

• Step 1: using rapidgzip with option chunk_size=... and parallelization=0 to create the index file.
• Step 2: using the non-parallel rapidgzip ( I guess you ment that version by GzipReader) with option readbuf_size to process ndjson files.

For Step 2, the non-parallel rapidgzip is better suited than indexed_gzip. This is because the RAM usage (for the index file) by rapidgzip is much smaller than that by indexed_gzip. For example, for an index file of 1.2GB, the RAM usage by rapidgzip is around 200MB, while that by indexed_gzip is around 1.2GB.

[mxmlnkn]

For Step 2, the non-parallel rapidgzip is better suited than indexed_gzip. This is because the RAM usage (for the index file) by rapidgzip is much smaller than that by indexed_gzip. For example, for an index file of 1.2GB, the RAM usage by rapidgzip is around 200MB, while that by indexed_gzip is around 1.2GB.

But you don't need any index file for sequential decompression and therefore could avoid the index memory usage completely ... This is what setKeepIndex would help with. GzipReader could help with reducing memory usage caused by the chunk size even further, but for parallelization=1 and only sequential access, the memory usage for the chunk size should only be roughly the same as the chunk size. There is a special case that keeps the chunk cache empty in case of sequential access.

I see two actionable items:

• Add setKeepIndex to Python interface. Possibly add something similar to GzipReader if necessary.
• Add GzipReader to the Python interface.

You can then try out what best fits your purpose.

[mxmlnkn]

Btw, creating RapidgzipFile with verbose=True should print some helpful performance metrics when the object is destructed.

[leanhdung1994]

But you don't need any index file for sequential decompression and therefore could avoid the index memory usage completely

Exactly, I only need the index to navigate to the the beginning of each ndjson file. I have a few hundreds such ndjson files, so I only need a few hundreds seek points.

You mentioned in this comment that

...since I added the no-additional-thread version of ParallelGzipReader...

I wonder if it may help in my use case. On the other hand, I will test your solutions when they are available and report the performance.

[leanhdung1994]

Btw, creating RapidgzipFile with verbose=True should print some helpful performance metrics when the object is destructed.

I process two ndjson files in parallel. My configuration is

1. The index file is generated with

with rapidgzip.RapidgzipFile(self.tarDir, chunk_size=32*1024*1024) as myGzip:
    myGzip.export_index(self.indexGzipDir)

2. The index file size is 158MB.

3. I open each ndjson file with

with rapidgzip.open(self.tarDir, parallelization=1, verbose=True) as myGzip:
    myGzip.import_index(self.indexGzipDir)
    with tarfile.open(fileobj=myGzip, mode="r:*") as f:
        member = f.getmember(ndjsonName)
        dataFile = f.extractfile(member)
        for i, oneLine in enumerate(dataFile):
            # process oneLine here

4. The RAM usage per thread is around 600MB.

5. The output from CMD is

P:\Personal Projects\Dicts\English Wiktionary - October 2025>C:\\Users\\Akira\\anaconda3\\python.exe "P:\Personal Projects\Dicts\English Wiktionary - October 2025\process.py" --lang en
Generate index of the .tar.gz file...
Finish generating the index.
the indexing time is 62.267749799997546
[ParallelGzipReader::importIndex] Took 1.53848 s
[ParallelGzipReader::importIndex] Took 1.53655 s
[GzipChunkFetcher::GzipChunkFetcher] First block access statistics:
    Number of false positives                : 0
    Time spent in block finder               : 0 s
    Time spent decoding with custom inflate  : 0 s
    Time spent decoding with inflate wrapper : 0 s
    Time spent decoding with ISA-L           : 0 s
    Time spent allocating and copying        : 0 s
    Time spent applying the last window      : 0 s
    Time spent computing the checksum        : 0 s
    Time spent compressing seek points       : 0 s
    Time spent queuing post-processing       : 0 s
    Total decompressed bytes                 : 0
    Non-marker symbols                       : 0
    Replaced marker symbol buffers           : 0
    Chunks exceeding max. compression ratio  : 0
    Thread Pool Utilization:
        Total Real Decode Duration    : 827.396 s
        Theoretical Optimal Duration  : 4.15122 s
        Pool Efficiency (Fill Factor) : 0.501722 %
    BGZF file          : false
[15:41:57.279][0x2336] [BlockFetcher::~BlockFetcher]
    Parallelization                   : 1
    Cache
        Hits                          : 262223
        Misses                        : 145
        Unused Entries                : 0
        Maximum Fill Size             : 16
        Capacity                      : 16
    Prefetch Cache
        Hits                          : 0
        Misses                        : 145
        Unused Entries                : 0
        Prefetch Queue Hit            : 0
        Maximum Fill Size             : 0
        Capacity                      : 2
    Cache Hit Rate                    : 99.9447 %
    Useless Prefetches                : 0 %
    Access Patterns
        Total Accesses                : 262368
        Duplicate Block Accesses      : 262224
        Sequential Block Accesses     : 65
        Block Seeks Back              : 1
        Block Seeks Forward           : 78
    Blocks
        Total Existing                : 5079
        Total Fetched                 : 145
        Prefetched                    : 0
        Fetched On-demand             : 145
    Prefetch Stall by BlockFinder     : 0
    Time spent in:
        decodeBlock                   : 4.15122 s
        std::future::get              : 4.15222 s
        get                           : 4.55187 s
    Thread Pool Utilization:
        Total Real Decode Duration    : 827.396 s
        Theoretical Optimal Duration  : 4.15122 s
        Pool Efficiency (Fill Factor) : 0.501722 %
[15:41:57.279][0x2336] [SharedFileReader::~SharedFileReader]
    seeks back    : ( 0 <= 300000000 +- 2400000000 <= 18300000000  ) B ( 60 calls )
    seeks forward : ( 100000000 <= 234000000 +- 27000000 <= 273000000  ) B ( 78 calls )
    reads         : ( 0 <= 125000 +- 25000 <= 131000  ) B ( 4357 calls )
    locks         : 796198
    read in total 545335237 B out of 18518286031 B, i.e., read the file 0.0294485 times
    time spent seeking and reading: 0.187338 s

enwiktionary_namespace_0_1.ndjson has been processed in 14 minutes.

[GzipChunkFetcher::GzipChunkFetcher] First block access statistics:
    Number of false positives                : 0
    Time spent in block finder               : 0 s
    Time spent decoding with custom inflate  : 0 s
    Time spent decoding with inflate wrapper : 0 s
    Time spent decoding with ISA-L           : 0 s
    Time spent allocating and copying        : 0 s
    Time spent applying the last window      : 0 s
    Time spent computing the checksum        : 0 s
    Time spent compressing seek points       : 0 s
    Time spent queuing post-processing       : 0 s
    Total decompressed bytes                 : 0
    Non-marker symbols                       : 0
    Replaced marker symbol buffers           : 0
    Chunks exceeding max. compression ratio  : 0
    Thread Pool Utilization:
        Total Real Decode Duration    : 860.574 s
        Theoretical Optimal Duration  : 4.20614 s
        Pool Efficiency (Fill Factor) : 0.48876 %
    BGZF file          : false
[15:42:25.284][0x17756] [BlockFetcher::~BlockFetcher]
    Parallelization                   : 1
    Cache
        Hits                          : 262224
        Misses                        : 145
        Unused Entries                : 0
        Maximum Fill Size             : 16
        Capacity                      : 16
    Prefetch Cache
        Hits                          : 0
        Misses                        : 145
        Unused Entries                : 0
        Prefetch Queue Hit            : 0
        Maximum Fill Size             : 0
        Capacity                      : 2
    Cache Hit Rate                    : 99.9447 %
    Useless Prefetches                : 0 %
    Access Patterns
        Total Accesses                : 262369
        Duplicate Block Accesses      : 262225
        Sequential Block Accesses     : 65
        Block Seeks Back              : 1
        Block Seeks Forward           : 78
    Blocks
        Total Existing                : 5079
        Total Fetched                 : 145
        Prefetched                    : 0
        Fetched On-demand             : 145
    Prefetch Stall by BlockFinder     : 0
    Time spent in:
        decodeBlock                   : 4.20614 s
        std::future::get              : 4.20714 s
        get                           : 4.59449 s
    Thread Pool Utilization:
        Total Real Decode Duration    : 860.574 s
        Theoretical Optimal Duration  : 4.20614 s
        Pool Efficiency (Fill Factor) : 0.48876 %
[15:42:25.285][0x17756] [SharedFileReader::~SharedFileReader]
    seeks back    : ( 0 <= 300000000 +- 2500000000 <= 18500000000  ) B ( 53 calls )
    seeks forward : ( 100000000 <= 234000000 +- 27000000 <= 273000000  ) B ( 78 calls )
    reads         : ( 0 <= 126000 +- 24000 <= 131000  ) B ( 4415 calls )
    locks         : 796317
    read in total 554296057 B out of 18518286031 B, i.e., read the file 0.0299324 times
    time spent seeking and reading: 0.183216 s

enwiktionary_namespace_0_0.ndjson has been processed in 14 minutes.

This mission is completed.
The running time is 14 minutes.
Scan "E:\Personal Projects\tmp\tarFiles\result\enWiktionary.txt": 81467

Pack to "E:\Personal Projects\tmp\tarFiles\result\English Wiktionary.mdx"
100%|██████████████████████████████████████████████████████████████████████████| 81467/81467 [00:03<00:00, 25267.00rec/s]
                    --- Elapsed time: 10.351099 seconds ---

P:\Personal Projects\Dicts\English Wiktionary - October 2025>pause
Press any key to continue . . .