Replacing images - refresh issue

[foranuj]

I have a GUI tool based on GTK 3.0, where I load a PDF page in the UI, I want to allow users to drag and drop images from a different panel onto a particular image on the PDF displayed, and we will replace the image that is underlying in the pdf page.

So here is what I do

1. Find the sub image with the PDF page where the new image is dropped
2. Replace the image using the following code

def img_replace(doc, page, img_xref, mask_xref, filename):
    """Replace image identified by xref.

    Args:
        page:
        xref: cross-reference number of image to replace
        filename, stream, pixmap: must be given as for
        page.insert_image().
        :param doc: a fitz.Document object
        :param page: a fitz.Page object
        :param img_xref: XRef for the image that will be replaced
        :param mask_xref: XRef for the mask if any around that image
        :param filename: The name of the file to use as replacement

    """

    if bool(filename) != 1:
        raise ValueError("Exactly one of filename must be given")

    # Get the details of the original image
    orig_img = fitz.Pixmap(doc.extract_image(img_xref)["image"])

    # This step will resize the new image to fit into the old image rectangle
    resized_pixmap = get_pixmap_with_resized_img(filename, orig_img.width, orig_img.height)

    # Account for masks if any are present
    if mask_xref > 0:
        mask_img = fitz.Pixmap(doc.extract_image(mask_xref)["image"])
        resized_pixmap = fitz.Pixmap(resized_pixmap, mask_img)

    # insert new image anywhere in page
    new_xref = page.insert_image(
        page.rect, pixmap=resized_pixmap
    )

    doc.xref_copy(new_xref, img_xref)  # copy over new to old
    last_contents_xref = page.get_contents()[-1]
    # new image insertion has created a new /Contents source,
    # which we will set to spaces now
    doc.update_stream(last_contents_xref, b" ")

3. Refresh the UI, using something like this

fitz_page = doc.reload_page(fitz_page)

self.image = pil_image_to_cairo_surface(get_img_from_pdf_page(fitz_page, width, height))

The idea of pil_image_to_cairo_surface is to do this

def pil_image_to_cairo_surface(src):
    # Python 3, so we use PNG as an intermediate.
    buf = BytesIO()
    src.save(buf, "png")
    buf.seek(0)
    surface = cairo.ImageSurface.create_from_png(buf)
    buf.close()
    return surface

However, I can't get the correct image. if I save this image to disk, using pil_save, I end up getting only the original. I am not sure what I need to get the correct image? Also if I save the full pdf using ez_save, I can see that the correct image is replaced as expected.

Wondering what am I missing when it comes to refreshing the image on the GUI?

[foranuj]

And this is the definition of get_img_from_pdf_page

    print("getting img_from pdf page..")
    zoom: fitz.Matrix = get_page_zoom(page, max_width, max_height)
    pix_map = page.get_pixmap(matrix=zoom, alpha=False)
    mode = "RGBA" if pix_map.alpha else "RGB"
    img_from_bytes = Image.frombytes(mode, [pix_map.width, pix_map.height], pix_map.samples)
    return img_from_bytes```

[JorjMcKie]

Hi, a few comments.
You only need the xref of the to-be-replaced image. Whether or not that image has an SMask xref is of no importance - because the original image will be removed with everything associated with it, including the SMask.

You mentioned that the new image seems to be successfully inserted - because the PDF looks correct when saved.

So we only need to focus on refreshing the page in such a way that your GUI takes note of it.
While having no experience with your GUI manager, I conclude that it displays a page using a PNG representation of it.
You can directly make a PNG image from the page Pixmap without using Pillow or dealing with io.BytesIO():

png = pixmap.tobytes()
surface = cairo.ImageSurface.create_from_png(png)

For debugging purposes you can also store that image to disk to confirm the page looks alright with the new image inserted, via pixmap.save("debug.png").

You already are controlling how pixmap is being created: you are doing this without alpha, therefore you need not later ask / check for alpha.

I don't understand your second post about get_img_from_pdf_page. This seems to contradict pil_image_to_cairo_surface.
In addition it refers to Image. which looks like a code piece dealing with Pillow images - which we do not need here at all.

[foranuj]

Thanks for the reply. Unfortunately this does not help resolve the issue.

The main issue is the refresh, however the underlying pixmap that I am retrieving is not reflecting the change.

Here's a small setup
Test input file --> sample3.pdf
Correct pdf output -->
sample3_replaced.pdf

Incorrect png created from pixmap -->

This is the image that I use to replace one subimage in the PDF

import fitz
from io import BytesIO
fitz_doc = fitz.open("sample3.pdf")
fitz_page_no = 0
fitz_page = fitz_doc.load_page(fitz_page_no)
fitz_page.get_images(True)
[(6, 0, 1985, 1600, 8, 'ICCBased', '', 'Im1', 'DCTDecode', 0), (7, 0, 1800, 1200, 8, 'ICCBased', '', 'Im2', 'DCTDecode', 0), (8, 0, 2100, 1402, 8, 'ICCBased', '', 'Im3', 'DCTDecode', 0), (9, 0, 808, 1436, 8, 'ICCBased', '', 'Im4', 'DCTDecode', 0), (11, 0, 1800, 1200, 8, 'ICCBased', '', 'Im5', 'DCTDecode', 0), (12, 0, 1200, 1800, 8, 'ICCBased', '', 'Im6', 'DCTDecode', 0), (13, 20, 1767, 1144, 8, 'ICCBased', '', 'Im7', 'FlateDecode', 0), (14, 0, 1200, 1800, 8, 'ICCBased', '', 'Im8', 'DCTDecode', 0), (15, 21, 1365, 1365, 8, 'ICCBased', '', 'Im9', 'FlateDecode', 0), (16, 22, 924, 1159, 8, 'ICCBased', '', 'Im10', 'FlateDecode', 0)]
from util.fitz.pdfParser import get_img_from_pdf_page, get_page_zoom, img_replace, refresh_image, \
...     get_xref_for_img_and_mask_at_pos, pdf_show_page, get_img_from_pdf_page_num

#This is my window size 378, 500
zoom = get_page_zoom(fitz_page, 378, 500)

#This is my custom method that finds where in the page the user clicked and returns a sub-image that contains the point
img_xref, mask_xref, img_key = get_xref_for_img_and_mask_at_pos(fitz_page, 120, 90, zoom)

#In this case, the img_xref is 11
old_xref = img_xref

#This does some resizing, and copies this new image by replacing underneath, using the code pasted for this method.
img_replace(doc, fitz_page, old_xref, mask_xref, filename="modifiedimg.png")

#Now this is where the correct PDF gets generated
fitz_doc.ez_save("sample3_replaced.pdf")

#But this is where my PNG doesn't come out right
png = fitz_doc.get_page_pixmap(fitz_page_no).tobytes()
img = Image.open(BytesIO(png))
img.save("sample3_replaced.png")

Am I missing the use of any reload, clean_contents?

[JorjMcKie]

Ah, interesting: I found the reason I think.
MuPDF caches images as much as it can and in the course of this remembers page pixmaps. So to force making a new pixmap, the cache must be emptied.
Then MuPDF will actually re-render the page. The following snippet produces correct before and after pixmaps:

import fitz

doc = fitz.open("sample3.pdf")
page = doc[0]
pix = page.get_pixmap()
pix.save("0before.png")
page.replace_image(11, filename="newimg.png")
fitz.Tools().store_shrink(100)  # empty 100% of the cache
pix = page.get_pixmap()
pix.save("1after.png")

[JorjMcKie]

Please note that there is a typo-bug in the published method .replace_image(). I should have used .xref_is_image() and not .is_image().
So continue using the script as you did instead.

[foranuj]

That was it. Thanks a bunch!!

[foranuj]

As a follow up, I was wondering if reload_page should empty the cache?

[JorjMcKie]

As a follow up, I was wondering if reload_page should empty the cache?

No, it does not. I also had the idea that that would resolve the hickup. But it only applies any pending updates to the page. The same would be achieved with loading another page.
Then I checked everything and found all good - and finally remembered the MuPDF caching mechanism.
Caching images is very important for performance, because these are the largest object.
Caching even keeps some stuff beyond closing all documents.