performance of "mutool -sg " and "page.clean_contents()"

[HinTak]

This is sort of a follow-up to my discussion item. While Page.clean_contents() does the job, its addition makes the script many times slower. Fortunately it is essentially a one-off operation, so I'll just let it run for how every long it takes.

I think mutool clean -sg ... is roughly equivalent to what I intend to do in one simple command line. But I found that it is even slower, and also takes up much more memory on the way, that I have not even let it work on the complete doc as is. Since I only want the front and back of the original as they are, so I only do mutool clean -sg original.pdf new.pdf 1,last. ( #3160 (comment) - trying to use front and back of original as they are, but only the mask of page 2 to N -1 . ).

So here are 3 bugs/questions/feature requests:

• the speed of page.clean_contents()
• the memory consumption of mutool clean -sg ... seems to be a lot worse then iterating page.clean_contents() over individual pages?
• Give item two above, A doc.insert_page(page_object) (with a cleaned page_object) would be nice. I see most of the doc.*page(pno) methods takes page numbers (insert, copy, move) - wouldn't it be nice to have an doc.insert_page(page_object) method?

[JorjMcKie]

All your problems should go away with the following recipe(s):

1. Use page.wrap_contents() instead of page.clean_contents(). There is a lengthy reasoning here. This method is extremely fast - shouldn't have any noticeable effect on your run times. Also generally recommended if you want or need to use incremental saves.
2. Somewhat similar, but still a little more effort: use page.clean_contents(sanitize=False).

[HinTak]

Neither of your suggestion, improves the situation:
If I replace the page.clean_contents() call in my code with page.wrap_contents() or page.clean_contents(sanitize=False), len(page.get_images()) afterwards still evaluates to 616 (instead of 2, it is a 308-page pdf). It is with page.clean_contents() only so far, that I get len(page.get_images()) =2.

[HinTak]

If it helps, the problem with this pdf, as far as I see, is something like this:

4668 0 obj
<<
/Type /Page
...
/Parent ...
/Resources <</ProcSet ...
                         /Font <<...>>
               /XObject <</I1 4673 0 R
                                    /I2 4674 0 R
    ...>>
...

The /Resources dictionary of every page indeed has an /XObject key which is a dictionary of every image.

The pdf literally have the above /XObject entry 308 times, in each <</Type /Page ...>>.

[JorjMcKie]

I finally understood your problem:
You want to access only those images that are actually displayed by the page?
Your PDF creator made his job simple by providing one object that lists all images displayed by whatever page in the document.
Cleaning up this situation with clean_contents(sanitize=True) (the default) does more than what you really need.

Page method get_image_info() walks through the page's appearance source(s) and records every image display command. The result is a list of dictionaries containing metadata of the images.

[HinTak]

Yes, page.get_image_info() evaluates to 2, but it doesn't seem to reference either the data directly or xref? So that's doesn't help much.

I have since looked at about 100 of pdf from the same-ish source. There is only one with this problem (so far), and it is also by far the oldest according to its creation/modification date, too.

[JorjMcKie]

Have you read the documentation of the method? You can specify xrefs=True for identification purposes. But this may take too long as well b/o the peculiar making of this PDF.
If you indeed need the image itself, then use text extraction with option "dict" and skip the text blocks (block type 0). The image blocks contain the full image.

[HinTak]

I looked at the output of page.get_image_info() I think - I'll give it a try again. I also looked at page.get_text('rawdict'). Yes, this particular pdf (and the oldest of the ~100, spanning over a decade according to the creation/modification date) seems to have a particularly lazy/simple way of attaching every image to every page and just displaying two out of that list, as far as I understand it.

[HinTak]

I think page.get_image_info(xrefs=True) can eventually get the same result as page.clean_contents() plus iterate over page.get_images() - slight correction, in this case there seems to be 3 images per page, a lower (1/3) res image , and a full one with a mask - page.get_image_info(xrefs=True) returns the main image of the latter one needs to visit that xref of the latter to read its mask.

However, page.get_image_info(xrefs=True) just for the first page takes about 2 minutes, while the whole run with the latter takes 3 to 4 minutes. If that scales with number of pages, the former is about 150 to 200 times slower, for this document.

I'll tidy up the code and post it as a repo at some point - it is not pretty, it dumps all the masks, wrap them into a new pdf, run mtool to get at the front and back covers, join these two with qpdf. Much of it could be neater, but 3 to 4 minutes for a one-off conversion is not bad.

[HinTak]

The code is posted at
https://github.com/HinTak/pymupdf-jbig2-extract
, FYI. It is not pretty, but get the work done in a few minutes per pdf. It is much slower than it could be, mainly because of the problem with every-page-referencing-every-image, which in fact affect only one file (of about 400).

The code is mainly just for your amusement :-), to see what some wants to use pymupdf for - definitely the lossless-extraction/conversion part.

It is doing a fair number of subprocess.call at the moment. I think I can eventually convert it to a "pure" pymupdf script if I try. I could probably figure out these question eventually, but if you have a quick answer to them, much appreciated:

• one file seems to have a number of (blank) pages, with a mask-only page according to "mutool info . ". The usual way of telling that it is a mask (xref of the background image) fails... I suppose I can read the mutool code to see how it is done...
• it may be a non-question: what is the mutool/mupdf equivalent of qpdf --deterministic-id? Ie. Use a deterministic (reproducible) way of updating the 2nd entry of the /ID key?
• there are probably better heuristics to determine the resolution and invertedness of the mask. It is failing for about 40/400 resolution-wise, and 5/400 invertedness-wise. I am sure if I look at the 5 cases it, the invertedness should be do-able. The resolution seems more tricky, maybe just historical, (documents scanned at different dpi at different times), but the 642dpi/ 643dpi number isn't a rounded one like 300/350/360/500, and I don't know where that comes from.