doc.new_page() unexpected side effect

[abu-irrational]

After adding a new page to a document, all the previously loaded pages become invalid

import fitz
doc = fitz.open("/tmp/test.pdf") # -- any existing document (with at least 1 page!)
p0 = doc.load_page(0)
p0.number # --> 0
px = doc.new_page(0,333,444)
px # --> OK: page 0 of /tmp/test.pdf
p0 #--> ERROR .... ValueError: orphaned object: parent is None

The expected result should be to 'shift' all pagenumbers of the opened pages.
p0 # -> page 1 of /tmp/test.pdf

[JorjMcKie]

This effect cannot be avoided:
There is no way of knowing what other zillions of objects in your code may contain document page numbers. Not even theoretically. You might have an old table of contents sitting around, just to name one example.

This is the reason, why on every page creation, deletion, move, copy, doc.select(), ... all potentially existing pages will be deleted / orphaned.

[abu-irrational]

Hi Jorj

I really appreciate PyMuPDF but honestly I don't use it very often since I'm the developer
of the Tcl language-binding (tclMuPDF).
For my work I'm directly integrating the core MuPDF and it is for this reason
I also submitted a bug (?) to MuPDF
https://bugs.ghostscript.com/show_bug.cgi?id=706859

So, I agree with you this is not a PyMuPDF bug - maybe just a lack of documentation if this is the intended side-effect -
but I think this should be fixed or at least clarified at MuPDF core level.

For tclMuPDF it's very important that all the opened pages be consistent with the related document.
Just to say ..
when a document is closed, all the opened pages are destroyed,
when a page is added/deleted, all the already opened pages are 'shifted'.
This worked up to now, until I discovered that ..
.. when I try to insert a new page K and the page K is already loaded, I cannot load the (new) page K.

So, the problem is not with pdf_insert_page(), but it's with pdf_load_page()
because, this latter function behaves differently if I ask for a page that is already loaded or not.

MAYBE I've found a solution for my code (and as far as I tested is OK)

With a big surprise I've found that a fz_document keeps a list of the opened pages
( I don't know why, I guess just for caching ... (??)),
so the following fragment can scan all the opened pages and renumber them correctly.

// to be called after each addpage/deletepage
// increment is +1 or -1
void PATCH_renumber_openedpages(fz_context* ctx, pdf_document* doc, int pageNum, int increment) {
fz_page* page;
fz_document* fzdoc = (fz_document*)doc;
for (page=fzdoc->open; page; page=page->next) {
if (page->number >= pageNum) {
page->number += increment;
}
}
}

In short:
When inserting a page K (provided 0<=K<=N)
pdf_insert_page(ctx,doc,K, ..);
then I call
PATCH_renumber_openedpages(ctx, doc, K, +1);
newPage = pdf_load_page(ctx,K);

and
when deleting a page K (provided 0<=K<=N-1)
.. close it if it's loaded ..
pdf_delete_page(ctx,doc,K);
then I call
PATCH_renumber_openedpages(ctx, doc, K, -1);

---

As far As I know this works for PDF, but since it uses the core fz_document*, I fear
there are some worst side-effects (crash?) for other use cases.
Do you think it can work, in general ?

[JorjMcKie]

Let me convert this to a "Discussions" item. This is where the post belongs to.

[JorjMcKie]

Page numbers are not a property of MuPDF page objects. Only PyMuPDF assigns page.number as a convenience to the shadowing Python Page object.
In contrast to the number, a page's xref is fixed. You can reorder the page tree of a PDF, insert new pages or delete some, and so on: all this changes the sequence number of a potentially unforeseeable set of other pages. But never their cross reference numbers.
The fact that MuPDF keeps a list of loaded pages has nothing to do with this.
So your bug submission to MuPDF is probably unfounded - I can't see your bug's content because of lack of permissions. You may want to authorize me ...

In PyMuPDF, you can load the same document page to different Page objects. If you do this, there is no place anywhere else outside your script to detect this. And therefore, there is no way to adjust their respective Page.number properties.
As I wrote, the same is true for the list doc.get_toc(). If you assign this list to some Python variable, it is out of reach for potential page number update algorithms.
These things are the motivation behind deleting all loaded pages whenever the PDF page tree is changed.

But I promise to give this more consideration.

[abu-irrational]

Now the MuPDF BUG I cited
https://bugs.ghostscript.com/show_bug.cgi?id=706859
is public.
IMHO this is a serious MuPDF bug, or a limitation of use if they prefer ...
As I wrote before, I also found a "quick and dirty" solution, but I don't know how much dirty it is in general .....
Up to now, I can say "It works on my computer!" :-P

[abu-irrational]

Addendum: 18-jul-2023
The above-cited MuPDFBug I submitted
https://bugs.ghostscript.com/show_bug.cgi?id=706859
has been closed (FIXED).