imageio-ext-turbojpeg depends on an obsolete version of libjpeg-turbo (v1.2.1)

[miceg]

Originally reported in: kartoza/docker-geoserver#653

When using the GeoServer libjpeg-turbo-plugin with libjpeg-turbo v3.0.3 (the current version) to render a layer as JPEG over WMS, it fails with:

16 May 14:36:59 WARN   [geoserver.servlets] - OutputStream could not be aborted in time. An error has occurred and could not be sent to the user.
16 May 14:36:59 ERROR  [geoserver.ows] - 
java.lang.UnsatisfiedLinkError: 'int org.libjpegturbo.turbojpeg.TJCompressor.compress(byte[], int, int, int, int, byte[], int, int, int)'
	at org.libjpegturbo.turbojpeg.TJCompressor.compress(Native Method)
	at org.libjpegturbo.turbojpeg.TJCompressor.compress(TJCompressor.java:146)
	at org.libjpegturbo.turbojpeg.TJCompressor.compress(TJCompressor.java:164)
	at it.geosolutions.imageio.plugins.turbojpeg.TurboJpegImageWriter.write(TurboJpegImageWriter.java:201)
	at org.geoserver.map.turbojpeg.TurboJpegImageWorker.writeTurboJPEG(TurboJpegImageWorker.java:126)

There are similar issues when rendering a raster layer compressed with JPEG.

The root cause of the issue is the it.geosolutions.imageio-ext.imageio-ext-turbojpeg package.

Detail

it.geosolutions.imageio-ext.imageio-ext-turbojpeg (as used in the GeoServer TurboJPEG plugin) currently depends on libjpegturbo "v1.2.1.5" Java wrappers.

Despite the package being org.libjpegturbo.turbojpeg-wrapper v1.2.1.5, it appears that GeoSolutions IT publishes its own fork of the library with the same class names, and it appears to be forked from a pre-1.0 version of libjpeg-turbo from 2013 – not v1.2.1 (released in 2015) EDIT: GitHub's timestamps are wrong, v1.2.1 was actually released in 2012; so this is not a pre-1.0 version.

Upstream libjpeg-turbo still maintains their own libraries, but don't appear to publish them in Maven.

It doesn't look like this repository includes any of the JNI C code (turbojpeg-jni.c), so this fork depends on upstream's libturbojpeg.so.

Around September 2022, upstream removed a bunch of methods from the Java and JNI wrappers, breaking ABI compatibility. These landed in libjpeg-turbo v2.1.90/3.0 beta1 (released in February 2023), and are now in stable v3.0.x releases (from July 2023) – so probably been broken since at least then... but it could have been anything in the last 11 years.

What I think needs to be done here

• Rename org.libjpegturbo.turbojpeg-wrapper to something like it.geosolutions.turbojpeg-wrapper, so it's very clear that it's GeoSolutions IT's fork of the package, and not upstream.

  While upstream not publishing their own packages is annoying, GeoSolutions stomping their package namespaces isn't the right answer.

• Update the libjpeg-turbo Java bindings to the current upstream version (currently v3.0.3).

In any case, expecting someone to use an 11+ year old pre-1.0 version of libjpeg-turbo is totally unreasonable. 😄

[aaime]

The direction you propose is more than reasonable. We don't "expect" anyone to use the turbo-jpeg plugin, but would gladly accept a pull request. The existing code is free, development time is not.

[NyakudyaA]

If the expectation is that no one should use the plugin I guess the documentation also needs to be updated to reflect such i.e https://docs.geoserver.org/main/en/user/extensions/libjpeg-turbo/index.html

[aaime]

No, the expectation is that anyone is free to do what they want. We have no complaints from customers about the 11 years old library, so it stays that way. If anyone steps up to upgrade the library, or is willing to fund such activity, then it will happen. Complaints won't achieve anything.

In the GeoServer wiki we have this useful guide on how to get things done, which is factual and applies to all upstream GeoServer projects managed by the same community of developers (GeoTools, GeoWebCache, jai-ext, imageio-ext): read warmly recommended.

[miceg]

Understood re: development effort.

For what it's worth, I don't think the original author intended or anticipated this outcome. I think they were just trying to implement a feature, and took the path of least resistance which then unintentionally introduced this dependency.

As for my comment about "expecting" someone to run an old version of libjpeg-turbo:

• The plugin docs reference libjpeg-turbo v1.2.90 (aka 1.3beta1, released in 2013) and their old SourceForge site.

  It looks like the last major update to the docs was to promote it from a community plugin in 2014.

• Assuming semantic versioning rules and that the JNI interface is supposed to retain ABI compatibility, libturbojpeg.so had an opportunity to break compatibility in 2018 (with v2.0.0).

  I've gone back in the history, and noticed that the timestamps in GitHub are inaccurate for old releases – the accurate ones are on SourceForge, and v1.2 was released in 2012. So this code was actually v1.2.1-based, not pre-v1.0 based.

  If it was pre-1.0, then there would be opportunities for compatibility breakage pre-v1.0 and at v1.0.

It's lucky that it did not happen sooner. 😅

Assuming that there are no issues with using v2.1.x (< v2.1.90) ABIs, at least v2.1.x still gets maintenance support, and many Linux distros haven't migrated to v3.0.x yet, so users of that GeoServer plugin can still apply (security) updates. As for how long that will last, time will tell, but it will certainly not last forever.

That all said, many Linux distros have switched from libjpeg to libjpeg-turbo, and it is now an official reference implementation. libjpeg-turbo can be built to be ABI and API compatible with various versions of libjpeg, so it may not be necessary to use the TurboJPEG-specific Java APIs to take advantage of libjpeg-turbo's performance improvements.

It might not be worth the effort to "fix" this, and it may be a signal to retire the plugin.

I'll have to see what I can do about employer funding / support for that work (to potentially test performance assertions, make a formal proposal and then ship it), as I'm evaluating GeoServer among other choices for a couple of different projects.

[miceg]

I've posted an issue on GeoServer's issue tracker, asking for opinions on the different options: https://osgeo-org.atlassian.net/browse/GEOS-11417

Once I have a better idea of what people want to do and what the bar is, I can figure out what to ask for from my employer.

[aaime]

I guess you're not realizing that the same group of people work on the entire stack of projects. I'll answer you there. (surprise surprise!)

[miceg]

I understand it to be a similar group of people, and I saw your involvement on GeoServer PRs for that plugin. 😄

I don't know the scopes of all of the various issue trackers, and who "owns" what. I apologise if this came across as hostile.

My impression was that as the root cause of the issue is in this repository owned by GeoSolutions IT, that it wasn't GeoServer's problem to solve – even if GeoServer is the only one to see the impact – which is why I posted the initial issue here.

I want to find the most efficient way to solve the issue, which at that point requires a choice about code in GeoServer's repositories... and so I felt it belonged in that issue tracker. I don't expect the wider group to be across what is happening in every one of their library dependencies. 😄

[dcommander]

libjpeg-turbo maintainer here. I wanted to clarify something, which is that the JNI functions are not part of any ABI. ABI stands for "Application Binary Interface", i.e. the (presumably public) binary interface between an application and a library. The TurboJPEG JNI functions are a project-private binary interface between the public TurboJPEG Java API and the underlying C library. They are not documented and cannot be used except by the Java front end, so it should be understood that you cannot mix and match a JAR from one version of libjpeg-turbo with a JNI library from another version. That may have worked accidentally with prior versions of libjpeg-turbo, but it was never a project policy or guarantee. (I doubt that very many other Java projects provide that guarantee, either.)

With the TurboJPEG C API, we maintain backward ABI compatibility by implementing wrappers that emulate previous versions of the API using the new version. With the Java API, we manage changes in a more typical manner for Java APIs: deprecation. The TurboJPEG 3 API overhaul (introduced in libjpeg-turbo 3.0 beta) replaced stateless arguments with stateful parameters. At that point, the Java methods that used stateless arguments were deprecated (and will be removed in libjpeg-turbo 3.1.) The deprecated methods were reimplemented as wrappers for the newer methods, so it was no longer necessary to maintain JNI functions that corresponded to the deprecated methods. In fact, it would have been ill-advised to do so, since those JNI functions would have effectively been orphaned and thus untested. (The only way to test them would have been to use a prior version of the Java front end, and that's just not a sane project management strategy.) Java applications that are written for the TurboJPEG 2.1.x Java API still work with TurboJPEG 3, because we have a backward compatibility policy (deprecation) between the Java application and the Java API. But no such policy/guarantee is provided between the Java interface and the JNI functions [*]. The private JNI functions are, from the point of view of Java applications, hidden and subject to change without notice.

[*] A few of the public documented Java API methods are implemented entire in JNI, so those JNI functions are subject to the same deprecation policy as other public documented Java API methods. However, if a JNI function corresponds only to a private Java method, then it is project-private and cannot be relied upon.

To address some other comments:

• The GitHub asset timestamps for libjpeg-turbo 3.0.2 and later are accurate. For prior releases (which were all posted to GitHub last year when we migrated away from SourceForge), you can click on the Git commit corresponding to the GitHub release to obtain the approximate release date (within a day or so.) If packages were subsequently replaced after the release date, then the packages and replacement date are documented under "Packaging Changes" in the release notes.
• Most Linux distros don't even include the TurboJPEG Java API, which reinforces my point that the JNI functions are not part of the TurboJPEG ABI.
• You are correct that the TurboJPEG Java API is not necessary in order to take advantage of libjpeg-turbo's performance improvements. You can use JNA to call either of libjpeg-turbo's C APIs, and it's possible that you can even get similar performance through ImageIO if you are using a version of OpenJDK built with libjpeg-turbo. (I haven't tried that, though.) My own downstream project, which previously used the TurboJPEG Java API, has moved away from that and is now using its own JNI wrapper that calls the TurboJPEG C API, in part because (as mentioned above) few Linux distros ship the TurboJPEG Java API. I am even questioning whether it makes sense to keep maintaining the TurboJPEG Java API. I was paid to implement it originally, but continuing to maintain it is frankly a royal PITA, particularly when I don't personally use it anymore. (When the TurboJPEG Java API was introduced, JNA didn't exist. Now both JNA and jextract provides ways of calling the C API without using JNI.)