How to fix cannot find page in page tree error?

[Danielfordd]

I have a project that was working well for the last 6 months and recently I've started to get some cannot find page x in page tree where x is the page number.

The project takes a PDF from emails and merges it into a master PDF for a range of different healthcare clients.

The same issue happens if I try to key into the doc at the relevant page number (for example doc[10] might throw the error "cannot find page 11 in page tree".

I tried googling the questions and referencing the docs but have been unable to find relevant information.

[JorjMcKie]

These PDFs contain internal errors. The page tree can be a tree-like structure, but often also is just an array of cross reference numbers. That object is referenced by the /Pages key in the PDF catalog. The structure it is pointing to is corrupt for whatever reason.
There is no way to heal this except saving it with the maximum of garbage collection option garbage=4 and hope that this will rebuild the document.

[JorjMcKie]

No. Would require a detail investigation of the file.
You could enclose page loading in a try/except to prevent your whole script from breaking down b/o rare errors.

[Danielfordd]

Could different versions of PDF cause the issue? If I isolate the file and open in vscode, the broken page starts with "%PDF-1.3" and the rest start with "%PDF-1.7"

[Danielfordd]

It seems like this page isn't "broken" when considered in isolation. If I separate it from the rest of the doc and open it with fitz I can access the page

(Pdb) broken_doc[0]
page 0 of scripts/broken_page.pdf

[JorjMcKie]

The message did not say the page is broken, but the page tree did not contain a reference to it.
No more, no less.
I don't understand what you are doing: you cannot "open a page". And the PDF version identifier is always at the start of file. You cannot / should not have 2 of them.
The version as such plays no role at all.

[Danielfordd]

My goal is to merge the two pdfs together and I'm trying to understand the underlying reason why the page tree is not working.

If I isolate the "missing" page and merge the "isolated" page and (original doc - "missing page") then no exception happens.

In this below sandbox example, I show the exception being thrown and then the success with the "isolated"

import fitz

destination_pdf = fitz.open('scripts/destination_file.pdf')
original_doc = fitz.open('scripts/original_doc.pdf')

try:
    destination_pdf.insert_pdf(original_doc)
except Exception as e:
    print(e) # *** RuntimeError: cannot find page 10 in page tree

try:
    print([x for x in original_doc.pages()])
except Exception as e:
    print(e) # *** RuntimeError: cannot find page 10 in page tree

# I deleted every page except the missing page manually
doc_with_miss_page_removed = fitz.open('scripts/example1.pdf')
destination_pdf.insert_pdf(doc_with_miss_page_removed)

# I deleted the "missing" page manually
missing_page_ref_doc1 = fitz.open('scripts/broken_page1.pdf')
destination_pdf.insert_pdf(missing_page_ref_doc1)

# no exception thrown
print([x for x in destination_pdf.pages()])

[JorjMcKie]

For background reading look at this.

Your catalog / page tree has an invalid entry /Types. Should be /Type!

[Danielfordd]

That link 404s for me, what should I search to find the info?

[JorjMcKie]

seen it, best google for it, title "Document management — Portable document format — Part 1:
PDF 1.7"

[JorjMcKie]

A /MediaBox spec in the catalog just specifies a default value for all pages not explicitely specifying their own - so mediabox is one of the inheritable values.

[Danielfordd]

I read through the suggested materials and did fairly extensive googling but can't find anything about why my PDF has a /Types.

I suspect that this is the issue causing the page tree to break. Is there low level code or a part of this repos codebase that I can look at to understand how the PDF is being built and interacted with? I was wondering if there is a way to identify why the /Types is happening.

If not, totally understand and I appreciate the answers to my questions.

[JorjMcKie]

why the /Types is happening.

People don't need reasons for making errors, do they?
You can edit your PDF with PyMuPDF's lowlevel code and correct that error and see if things work better then.

cat_string = doc.xref_object(doc.pdf_catalog()).replace("/Types","/Type")
doc.update_object(doc.pdf:catalog(), cat_string)

[Danielfordd]

unfortunately, no luck, thank you though! I also tried to programmatically remove the page but can't since it is not in the page tree.

[JorjMcKie]

can you send me the file and let me try it?

[Danielfordd]

Unfortunately, these are medical documents and protected by HIPAA + which violates my company's BAA. I tried to reduce the pages and sanitize the private health information but interestingly if I delete ANY page, the document is "fixed".

I tried the above with many iterations, and each time it fixes the page tree:

1. Delete all pages except the "missing" page and page after
2. Delete all pages except the "missing" page and page before
3. Delete only the first page
4. Delete only the last page
5. Delete random page(s) (I Did 5 variations of this)

Not sure that means anything, maybe when the page is deleted my mac is "fixing" the issue? That got me thinking, is there a combination of "save" variables that can resolve my issue?

==== Solution =====
If I use .save and pass linear=True the new document can be merged in.

I'll use this workaround for my project. Any ideas why this "fixed" the original document? Is there an underlying function called on the original document

Below is the sandbox code I used to "fix" the merge

destination_pdf = fitz.open('scripts/destination_file.pdf')
original_doc = fitz.open('sanitized_name.pdf')

try:
    destination_pdf.insert_pdf(original_doc)
except Exception as e:
    print(e) # *** RuntimeError: cannot find page 3 in page tree

try:
    print([x for x in original_doc.pages()])
except Exception as e:
    print(e) # *** RuntimeError: cannot find page 3 in page tree

original_doc.save('scripts/throwaway.pdf', linear=True)
new_doc = fitz.open('scripts/throwaway.pdf')

print([x for x in original_doc.pages()]) # works
print([x for x in new_doc.pages()]) # works

print(new_doc.page_count) # 81
print(destination_pdf.page_count) #53
destination_pdf.insert_pdf(new_doc) #134

print(destination_pdf.page_count)
destination_pdf.save('scripts/final_output.pdf')

scripts/final_output.pdf has all pages merged in

[JorjMcKie]

As mentioned a few times in the previous posts, this document contains structural problems. As always: a wrong information may lead to unpredictable behavior - potentially in seemingly unrelated aspects.

The only promising / worthwhile strategy is rebuilding the document - and be it only temporarily, for the purpose of your application.
So the question is how to achieve this with (Py-) MuPDF, if MuPDF did not detect issues during open.

We must force MuPDF to rescan the whole document. When doing this, all object definitions are being inspected and important structure components are being rebuilt based on information gathered.

You apparently have detected that a temporary save with option linear=True can trigger this to happen.
I know that MuPDF also rebuilds if it cannot find the PDF trailer. So some hack that removes the end of the file may also be an idea.

Did we mention the possibility to convert PDF-to-PDF? This build a new PDF obviously. It can be invoked via mutool draw -o output.pdf input.pdf or doc.convert_to_pdf(). But chances are that this way also fails because of pagetree issues.

[Danielfordd]

Just wanted to close this out and say thank you, doing the save + linear=True resolved my issue

[JorjMcKie]

You are welcome.
I would be interesting to see what MuPDF did with that illegal /Types parameter in the end.
If you save that PDF again - this time without linear, how does the page tree look then?