Reading PBF based maps depends on old Protobuf library

[jurmous]

Describe the bug

Osmosis dependency used by Graphhopper depends on protobuf-java version 3.25.5 max.

My project depends on protobuf-java: 4.28.2 and overwrites dependency version of the one defined by osmosis.

Project fails because makeExtensionsImmutable method is not present anymore in GeneratedMessage in any version of ProtoBuf higher than 3.25.5.

To Reproduce

I have a higher protobuf-java dependency in my project.

I read PBF formatted OSM maps from
http://download.geofabrik.de/europe/netherlands-latest.osm.pbf

I import it with GraphHopper.

It fails with java.lang.NoSuchMethodError for makeExtensionsImmutable.

Caused by: java.lang.RuntimeException: Unable to read PBF file.
	at com.graphhopper.reader.osm.pbf.PbfReader.close(PbfReader.java:66)
	at com.graphhopper.reader.osm.OSMInputFile.close(OSMInputFile.java:241)
	at com.graphhopper.reader.osm.WaySegmentParser.readOSM(WaySegmentParser.java:402)
	... 14 more
Caused by: java.lang.NoSuchMethodError: 'void org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader.makeExtensionsImmutable()'
	at org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader.<init>(Fileformat.java:1235)
	at org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader.<init>(Fileformat.java:1158)
	at org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader$1.parsePartialFrom(Fileformat.java:1914)
	at org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader$1.parsePartialFrom(Fileformat.java:1908

Expected behavior
No errors with latest protobuf version in project.

System Information
JVM 17 on macOS

[jalfonsocu]

Having the same problem, Linux, OpenJDK 21

[mprins]

I think that should be fixed in the dependency (which still have 3.25.0) eg. https://search.maven.org/artifact/org.openstreetmap.osmosis/osmosis-pbf2/0.49.2/jar and https://search.maven.org/artifact/org.openstreetmap.osmosis/osmosis-pbf/0.49.2/jar

even updating to the latest release in

graphhopper/core/pom.xml

Lines 108 to 112 in 4e232ac

	<dependency>
	<groupId>org.openstreetmap.osmosis</groupId>
	<artifactId>osmosis-osm-binary</artifactId>
	<version>0.48.3</version>
	</dependency>

will not bring what you want, also note: https://github.com/openstreetmap/osmosis?tab=readme-ov-file#status
Perhaps if you opened a PR with osmosis it will be considered and an updated release put forward

[karussell]

I've asked a related question at the osmosis repo.

[otbutz]

https://github.com/openstreetmap/OSM-binary as mentioned in openstreetmap/osmosis#97 ?

https://github.com/openstreetmap/OSM-binary/blob/a2e364e5e7baa7da9bb24024eb50cdc29bbd4d8c/pom.xml#L34

For us:

<dependency>
  <groupId>org.openstreetmap.pbf</groupId>
  <artifactId>osmpbf</artifactId>
  <version>1.5.0</version>
</dependency>

[otbutz]

Looks like a 1.6.0 release with updated dependencies is imminent: openstreetmap/OSM-binary#87

Edit: already available https://repo1.maven.org/maven2/org/openstreetmap/pbf/osmpbf/1.6.0/

[thecoden]

We are experiencing the same issue. Has anyone found a workaround?

I have tried all combinations of excluding the osm-binary dependency and including osmpbf 1.6.0 with no effect.

Thanks all for the work on this issue!

[thecoden]

Small update that will hopefully help at least one person / team:

I added the following to my gradle file to prove that overriding the protobuf dependency allowed us to move forward:

	implementation("com.google.protobuf:protobuf-java") {
		version {
			strictly("3.25.5")
		}
	}

As @jurmous points out, that is the latest version that can work with the osm dependency that graphhopper currently uses.

Reason

For our project, the Tink encryption library and MySQL Connector/J were also in the project, depending on 4.x versions of Protobuf. These were causing the later protobuf version to load and causing graphhopper's osm reading to fail.

Solution

Since we only need the OpenStreetMap functionality to build a graph from the downloaded maps, we made a process separated from our application that simple initializes GraphHopper and builds the graph, with no encryption or database connections required, allowing us to use the above dependency only.

Thank You

All the previous comments connected the dots for us once we got deeper in it, so thanks to everyone for sharing. Even though we are late to this problem, it helped a lot for finding a workaround in our project. Cheers.

[otbutz]

You can follow the discussion in #3089. The remaining problem is how we want to deal with autogenerated protobuf classes.