Performance options for a large number of small thumbnails

[davxiao]

I have a large number (>10,000) of small size thumbnails (256x256x16 bit) to process.

I'm looking at two options and need your advice on which is better from pure compression and decompression perspective.

To simplify for the sake of the discussion, let's say in either option, all data is loaded into memory so there's no disk I/O to accounted for, and assume it's a single thread process and no concurrency.

option 1 is to handle each thumbnail is an individual qoi object, so each thumbnail gets compressed/decompressed individually. For the sake of benchmark purposes let's say this process times 100, so 100 thumbnails are processed.

option 2 is to create a large canvas for every, say 100 thumbnails, so that every 100 thumbnails are ordered in a 10x10 and filled up the canvas, then the canvas gets compressed/decompressed as one qoi object.

Which option will have less processing time?

[markg85]

This is not a QOI specific question. This is benchmarking and batching that you can apply to any library you want to test.

I actually am deep into measuring loading a file and decoding it into QOI, my hardware is a mid-range android phone from a couple years ago and i'm loading images with 500x500 resolution (RGB). Guess what, loading one such image takes about 0.2ms. Processing it with the QOI reference implementation (decoding) takes about 3ms.

With that in mind, i too made a threadpool. Making a large pool isn't useful as your IO will become a problem. Despite it loading fast it's still random read which is the slowest part of IO. So moral of the story here, just create a threadpool of the number of CPU cores you have and load+process that many images in parallel. For 10k images like your description with that small resolution. That's probably done within a second. And if it's not then you're doing it wrong ;) Consider it a challenge.