Insert_pdf/save Performance Question

[NilesRath]

I had an extremely large pdf, 343,652 pages. I set a loop for 14 new docs pulling 25k pages per doc using the insert_pdf method. The doc2 is created and saved 14 times, once for each loop. The amount of time to insert and save grew with each iteration of the loop. So for example, the first 25k pages took ~25min, the next took ~40min, the next took ~50min, and so on with the last few taking close to 2 hours each. If they are all 25k pages and the output PDFs are all roughly the same size ~140MB, can you help me understand why the amount of time grows with each loop? Is it because the range being pulled is further and further into the source PDF? I'm trying to understand how I can improve the speed/performance of this process as we are going to get more of these large PDFs.

This library is great btw, thank you so much for developing!

[JorjMcKie]

Hard to tell the reason for your duration time growth - without seeing the code.
But as a general recommendation I would consider the following:

• Close and re-open the input between each segment of extracted pages. This should free up internally kept caches of MuPDF.
• Generally make sure to explicitely close every file you are done with (or equivalently use context managers for file handling).
• This use case also suggests using Python multithreading (PyMuPDF supports no threading). PyMuPDF can be used with multiprocessing - examples are in the documentation.
• A lot also depends on the internal making of the input. Every page copied over is recursively inspected for referenced objects, which often are more than one would expect: I have seen examples where extracting 1 page literally copied over all pages, because they were cross-referencing each other. The target thus had the same size as the source ...

Opening a PDF and jumping to an arbitrary page is very fast, so that should not be the reason for the problem. Confirm this by measuring only extracting the last 25k pages.