Isolate PyMuPDF from other libraries; issues when PyMuPDF is loaded with other libraries like GdkPixbuf

[arun-mani-j]

Hey all.
I'm facing some really odd issues with page rendering. Sometimes the pages render just blank (white or black). After a deep analysis, I found that PyMuPDF undergoes this behavior when GdkPixbuf is loaded.

A small example.

In [1]: import fitz

In [2]: d = fitz.open("sample/alice-in-wonderland.epub")

In [3]: d[0].get_pixmap().save("without-library.png")

Result:

Now load GdkPixbuf.

In [1]: from gi.repository import GdkPixbuf
<ipython-input-1-e18be571bbf2>:1: PyGIWarning: GdkPixbuf was imported without specifying a version first. Use gi.require_version('GdkPixbuf', '2.0') before import to ensure that the right version gets loaded.
  from gi.repository import GdkPixbuf

In [2]: import fitz

In [3]: d = fitz.open("sample/alice-in-wonderland.epub")

In [4]: d[0].get_pixmap().save("with-library.png")

Result:

I know this is not a problem of PyMuPDF actually but more or less about how libraries conflict or something. I tried doing lsof for both the processes but can't find anything clashing except that GdkPixbuf loading some GLib libraries.

My primary question would be if anybody has an idea why this issue occurs? Am I looking at it in the wrong direction?
Then, if it is indeed a library clashing, then how do you prevent that? Any way to isolate the behavior?

Thanks in advance! Sorry if it has been already asked.

[arun-mani-j]

Looks like this is something that has already been explored a lot like in #85, #1396, #133 etc.
I upgraded to version 1.24.4 and now I'm getting this error:

MuPDF error: library error: jpeg error: Wrong JPEG library version: library is 62, caller expects 90

[arun-mani-j]

I have found python3-fitz in Debian repositories. It works perfectly fine. Is it because it uses Debian's MuPDF builds (say with patches to ensure compatibility with other libraries that need JPEG etc.)? However the version is also 1.23, not the latest 1.24 series. So is the issue not present in old version or what?

However, installing from system package manager is a temporary solution as, if I want to keep my app cross-platform, then I would have to get the official PyMuPDF builds anyway.

One way I thought of is to make the PyMuPDF logic a separate process and use a shared memory to communicate with the GUI. However I'm not sure how the latency will be in this case.

Other solution would be to build PyMuPDF (and in turn MuPDF) from source with patches to make it use the JPEG libraries GdkPixbuf is using.

@JorjMcKie @rk700 any thoughts or help on how it can be taken forward? (Extremely sorry for tagging, but I feel like you both have more expertise in this issue than me, apologies again!)

[JorjMcKie]

The user rk700 is no longer part of the PyMuPDF team since many years now.

I am adding @cgdae to this thread instead. I am sure he has an explanation for what is happening here.

[arun-mani-j]

Thanks!
I'm also thinking of forking MuPDF and make it use the the same libjpeg version, GdkPixbuf is using.
I also found that MuPDF is using libjpeg (https://www.ijg.org/) and GdkPixbuf is using libjpeg-turbo (https://libjpeg-turbo.org/) which claims to be faster than libjpeg by a significant factor.
So I thought I might fork MuPDF and replace libjpeg with libjpeg-turbo. But I'm interested in knowing why the MuPDF developers didn't do it, like if there is a reason they did not go with libjpeg-turbo. I hope I get an answer for that too :).

[I will be releasing the fork under GPL 3 license, so I think that is confirms with MuPDF licensing terms, or the best, the changes are done in the upstream MuPDF itself]

[julian-smith-artifex-com]

I would have hoped that MuPDF would see the library code that it was built with, before any system-wide library, so i'm not sure what's going on here.

How are you installing/building PyMuPDF? Are you using a pre-built wheel from pypi.org?

[arun-mani-j]

I would have hoped that MuPDF would see the library code that it was built with, before any system-wide library, so i'm not sure what's going on here.

From my understanding so far, both MuPDF and GdkPixbuf depend on libjpeg API compliant implementation. But since both of them use different versions, they get overloaded or something? I might try running gdb through MuPDF this weekend and try to get the exact location this error pops up and ideally use strace or something to understand the flow.

How are you installing/building PyMuPDF? Are you using a pre-built wheel from pypi.org?

PyMuPDF I'm simply installing via pip install PyMuPDF, which seems to download the pre-built SO file I think? (I can see a bunch of .so files inside the installation folder).

[arun-mani-j]

Regarding testing PyMuPDF with a custom MuPDF build, how does one do that?

I was able to build MuPDF with:

 $ make HAVE_X11=no HAVE_GLUT=no

Then build PyMuPDF with:

$ PYMUPDF_LIB=../mupdf/build/release/ pyproject-build --sdist .

But I don't think this is right. My goal is to make is a PyMuPDF build that uses my custom MuPDF version and then run a small Python script to test PyMuPDF from a user point-of-view.

[julian-smith-artifex-com]

Ok that's good to know, thanks.

I'm so far unable to reproduce the problem.

• I'm using PyMuPDF's tests/resources/Bezier.epub which is the only .epub i have access to.
• I see the PyGIWarning: GdkPixbuf was imported without specifying a version first warning.
• I do not see the MuPDF error: library error: jpeg error: Wrong JPEG library version: library is 62, caller expects 90 diagnostic from MuPDF.

A couple if things come to mind:

• Could you upload alice-in-wonderland.epub so i can try something closer to what you're doing?

• Could you modify your script to look like this:

  from gi.repository import GdkPixbuf
  import pymupdf
  
  print(pymupdf.version)
  d = pymupdf.open("sample/alice-in-wonderland.epub")
  d[0].get_pixmap().save("with-library.png")
  
  import gi
  print(gi.__version__)
  

I'm rather clutching at straws here, but that might at least verify some things that we need to be certain about.

[arun-mani-j]

I do not see the MuPDF error: library error: jpeg error: Wrong JPEG library version: library is 62, caller expects 90 diagnostic from MuPDF.

Can you share your distribution details?
I'm on Debian Testing.

$ apt show libgdk-pixbuf-2.0-0
Package: libgdk-pixbuf-2.0-0
Version: 2.42.12+dfsg-1
Priority: optional
Section: libs
Source: gdk-pixbuf
Maintainer: Debian GNOME Maintainers <[email protected]>
Installed-Size: 488 kB
Depends: libgdk-pixbuf2.0-common (>= 2.42.12+dfsg-1), shared-mime-info, libc6 (>= 2.34), libglib2.0-0t64 (>= 2.79.0), libjpeg62-turbo (>= 1.3.1), libpng16-16t64 (>= 1.6.2), libtiff6 (>= 4.0.3)
Recommends: libgdk-pixbuf2.0-bin
Homepage: https://www.gtk.org/

Oh, if you look at the Depends section we have: libjpeg62-turbo (>= 1.3.1), I think this is where the 62 thing comes from??

I'm using the EPub from Gutenberg - https://www.gutenberg.org/ebooks/11.epub3.images

Yea, so we got this:

$ cat modified.py 
from gi.repository import GdkPixbuf
import pymupdf

print(pymupdf.version)
d = pymupdf.open("sample/alice-in-wonderland.epub")
d[0].get_pixmap().save("with-library.png")

import gi

print(gi.__version__)

$ python3 modified.py 
/home/arun-mani-j/modified.py:1: PyGIWarning: GdkPixbuf was imported without specifying a version first. Use gi.require_version('GdkPixbuf', '2.0') before import to ensure that the right version gets loaded.
  from gi.repository import GdkPixbuf
('1.24.4', '1.24.2', '20240516000001')
MuPDF error: library error: jpeg error: Wrong JPEG library version: library is 62, caller expects 90

3.48.2

[julian-smith-artifex-com]

One other thing to try is to swap the import of GdkPixbuf and pymupdf:

import pymupdf
from gi.repository import GdkPixbuf

print(pymupdf.version)
d = pymupdf.open("sample/alice-in-wonderland.epub")
d[0].get_pixmap().save("with-library.png")

import gi
print(gi.__version__)

This may well avoid the problem, but even if does i wouldn't be surprised if attempts to use GdkPixbuf then go wrong. And it wouldn't be a general solution.

[arun-mani-j]

Changing the import order actually... works!

$ cat another.py
import pymupdf
from gi.repository import GdkPixbuf, GLib

print(pymupdf.version)
d = pymupdf.open("sample/alice-in-wonderland.epub")
d[0].get_pixmap().save("with-library.png")

pix = d[0].get_pixmap()
pbf = GdkPixbuf.Pixbuf.new_from_bytes(
    GLib.Bytes(pix.samples),
    GdkPixbuf.Colorspace.RGB,
    False,
    pow(2, pix.n),
    pix.width,
    pix.height,
    pix.stride,
)
pbf.savev("with-pixbuf.png", "png")

import gi

print(gi.__version__)

$ python3 another.py
/home/arun-mani-j/another.py:2: PyGIWarning: GdkPixbuf was imported without specifying a version first. Use gi.require_version('GdkPixbuf', '2.0') before import to ensure that the right version gets loaded.
  from gi.repository import GdkPixbuf, GLib
('1.24.4', '1.24.2', '20240516000001')
3.48.2

Assuming that this is a golden bullet-proof solution, why does GdkPixbuf is immune to library overloads but not MuPDF?

[julian-smith-artifex-com]

I've managed to reproduce the problem.

It looks like linking MuPDF with -Bsymbolic fixes things. I'm not sure this is the best solution, but it might be worth doing this in a future release.

BTW to answer to your earlier question about building PyMuPDF with a local MuPDF, see: https://pymupdf.readthedocs.io/en/latest/installation.html#build-and-install-from-a-local-pymupdf-source-tree. Note that we always build MuPDF in a way that is correct for PyMuPDF (for example always build as a shared-library), one cannot currently simply specify pre-built MuPDF libraries.

Also see comments at top of PyMuPDF:setup.py for information about configuration with environment variables.

[arun-mani-j]

Thanks, hope that is the best solution and gets upstreamed!

Thanks for the build instructions, I crashed GCC trying to build it 0_0. Hopefully, me tweaking MuPDF should not be required as we have found a (temporary) work around ^^.

[julian-smith-artifex-com]

This will be fixed in the next release - we will be building MuPDF with -bSymbolic.

[arun-mani-j]

Thank you so much @julian-smith-artifex-com and @JorjMcKie !

[julian-smith-artifex-com]

Fixed in 1.24.6.