Error in Osmosis: Java BufferOverflowException for tiles with many POIs

[ems299]

Hey,

thanks a lot for this great piece of software! I've successfully used it to create custom maps for my Wahoo Elemnt Roam.
For tiles with many POIs (e.g. 129/88; Paris) I, however, ran into a Java BufferOverflowException. I couldn't find a solution anywhere, so I decided to dig deeper and finally managed to solve it. This involved changing some variables in Osmosis and re-compiling it.
I would like to share the fix, so others can benefit as well. I'm just not sure, would I report it here or in an Osmosis bug tracker? I'll file the issue and reply tomyself. If this is not the right place, feel free to move it.

Expected Behavior

No Java BufferOverflowException ;-)

Current Behavior

Java BufferOverflowException

Steps to Reproduce the Issue

1. Add the following POIs in your tag-wahoo.xml file:
   fuel, cafe, toilets, drinking_water, shelter, cementery, church, supermarket, bakery (I added a few more, but these should be enough). If it helps I could also share my file.
2. python -m wahoomc cli -xy 129/88 -tag tag-wahoo_test.xml

Context

• OS/Platform: Windows 10
• Other information regarding your environment (optional): nothing

Log Output / Stack Trace

INFO:--------------------------------------------------------------------------------
INFO:# Creating .map files for tiles
INFO:+ (tile 1 of 1) Coordinates: 129,88
ERROR:subprocess error output:
ERROR:Jan. 17, 2025 12:19:20 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Osmosis Version 0.49.2
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Jan. 17, 2025 12:19:20 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Preparing pipeline.
Jan. 17, 2025 12:19:20 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>
INFORMATION: mapfile-writer version: mapsforge-map-writer-0.21.0
Jan. 17, 2025 12:19:20 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>
INFORMATION: mapfile format specification version: 3
Jan. 17, 2025 12:19:20 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Launching pipeline execution.
Jan. 17, 2025 12:19:20 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Pipeline executing, waiting for completion.
Jan. 17, 2025 12:19:20 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask process
INFORMATION: start reading data...
Jan. 17, 2025 12:19:26 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete
INFORMATION: completing read...
Jan. 17, 2025 12:19:26 PM org.mapsforge.map.writer.RAMTileBasedDataProcessor complete
INFORMATION: handle relations...
Jan. 17, 2025 12:19:28 PM org.mapsforge.map.writer.RAMTileBasedDataProcessor complete
INFORMATION: handle ways...
Jan. 17, 2025 12:19:32 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete
INFORMATION: start writing file...
Jan. 17, 2025 12:19:32 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete
INFORMATION: overwriting file D:\tmp\wahooMapsCreator\wahooMapsCreatorData\_tiles\129\88.map
Jan. 17, 2025 12:19:37 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFORMATION: written 10% of sub file for zoom interval 1 (10,0,17)
Jan. 17, 2025 12:19:44 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SCHWERWIEGEND: Thread for task 1-rbf failed
java.nio.BufferOverflowException
        at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:238)
        at org.mapsforge.map.writer.MapFileWriter.processTile(MapFileWriter.java:919)
        at org.mapsforge.map.writer.MapFileWriter.writeSubfile(MapFileWriter.java:977)
        at org.mapsforge.map.writer.MapFileWriter.writeFile(MapFileWriter.java:375)
        at org.mapsforge.map.writer.osmosis.MapFileWriterTask.complete(MapFileWriterTask.java:113)
        at org.openstreetmap.osmosis.pbf2.v0_6.PbfReader.run(PbfReader.java:117)
        at java.base/java.lang.Thread.run(Thread.java:1575)

Jan. 17, 2025 12:19:44 PM org.openstreetmap.osmosis.core.Osmosis main
SCHWERWIEGEND: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)


ERROR:Error in creating map file via Osmosis with tile: 129,88. mapwriter plugin installed?```

[ems299]

This is that tag-wahoo.xml file to reproduce the Exception
tag-wahoo_test.zip

Just in case also the corresponding tags-to-keep.json file
tags-to-keep.zip

[ems299]

So here's my solution:

The reason seems to be that the buffers for POI_DATA_BUFFER_SIZE and WAY_DATA_BUFFER_SIZE are too small, each only 10MB. They are hardcoded into the MapFileWriter class in Osmosis
The values for these variables need to be changed, Osmosis then needs to be re-compiled. In the following you can find the steps to do this. Alternatively you can also simply use the attached file where this is corrected already and copy it to the folder described in step 10 (or whichever is the respective folder on your system; it is platform independent). Enjoy!

1. The WahooMapCreator uses Osmosis with version 0.21 of the Mapsforge MapWriter Plugin. With standard settings the WMC stores it here:
   C:\Users\Username\AppData\Roaming\Openstreetmap\Osmosis\Plugins
2. Download the compiled *.jar file from here
   https://repo.maven.apache.org/maven2/org/mapsforge/mapsforge-map-writer/0.21.0/
   or get it from the folder named in 1.
3. Download source code from here
   https://github.com/mapsforge/mapsforge/releases/tag/0.21.0
4. The *.jar is just a zip file. In this file we are gooing to replace the following file:
   ".\org\mapsforge\map\writer\MapFileWriter.class"
5. Now we need the source code of that file. It was downloaded in step 3. Unzip this file and find the MapFileWriter.java file in .\mapsforge-map-writer\src\main\java\org\mapsforge\map\writer
6. Open this file and change the values for POI_DATA_BUFFER_SIZE, TILE_BUFFER_SIZE and WAY_DATA_BUFFER_SIZE from 0xA00000 (10MB) to 0x1400000 (20MB)
   Also change WAY_BUFFER_SIZE which in the comment is said to be 10MB but in the code is acutally set to 1MB. Set it to 10MB (0xA00000).
7. Now we need to compile the file again. Download OpenJDK version 21 (Later versions won’t work!)
   Make sure to set environment variables JAVA_HOME and ANDROID_HOME to the JDK folder.
8. Run the following command from the main folder where the source code is located
   gradlew -p .\mapsforge-map-writer build
9. Get the file MapFileWriter.class from the folder .\mapsforge-map-writer\build\classes\java\main\org\mapsforge\map\writer and replace it in the *.jar file mapsforge-map-writer-0.21.0-jar-with-dependencies.jar (which, remember, is just a ZIP file)
10. Copy the modified mapsforge-map-writer-0.21.0-jar-with-dependencies.jar to C:\Users\Username\AppData\Roaming\Openstreetmap\Osmosis\Plugins

mapsforge-map-writer-0.21.0-jar-with-dependencies.jar.zip

[treee111]

Hi @ems299, thanks for digging so deep into that topic and posting this workaround! 👍
I'm also not sure how to get this trick/hack into "the masses". I never had this kind of error (because I never generated maps for Paris ;-)). Kind of testing would be needed and also a "compliant" way of doing the hack and delivering it out to the users.

Hi @Ebe66, did you know about that trick/hack? Looks similar to #269. what do you thing would be best way to deliver this to the users?

1. filing a ticket to mapsforge
2. trying to hack the .class file into the downloaded map-writer .jar file
3. host a modified map-writer .jar file (e.g. in this repo) and deliver it during install of wahooMapsCreater or download it from this repo

[Ebe66]

Hi @treee111, as you know I'm, also, no expert on any of this, just trying to make things work with the information we have ;-)

The "Paris" thing has been a good while ago, but as far as I remember I never encountered the issue mentioned here and I do include A LOT of POI's in my maps and just finished creating new maps of Europe and Asia without encountering it.
As I remember the Paris thingy came up as people in the Telegram group (https://t.me/+Mor244hPZw85YTVk) reported blank/not loading tiles in Paris. After a lot of searching and trying, the solution I ended up with was using another map interval as indeed mentioned in issue #269. However this was a solution for the devices running out of memory and subsequently not rendering the tile, there was no problem generating it. Other than me having a more than usual amount of ram, I should have encountered the same issue, but didn't.
Maybe using the other map interval of (12,0,17) has something to do with this?
In short I can not directly confirm this is a problem for others too.
Not sure what the right thing to do here would be but number 3 sounds like the easiest thing to do but would be a patch, not a solution. Maybe @ems299 can open an issue at mapsforge? 10MB for those buffers may not sound like a lot nowadays but that are a lot of points. Even the French would not have that many bakeries I thought ;-)

[Ebe66]

@treee111 @ems299 Decided to give the provided config files a try and for me it finished without an error. (!)

Screenshot of the resulting map loaded in Cruiser. It may look different because it's using my own theme.

When zooming in it also displays toilets for example so pretty sure the config files where used.

This is on a recent Anaconda/WMC install on windows 10 with release 4.3.0 and the latest commits of WMC. That might explain why it works on my very old 16GB ram test laptop . It actually finished (excluding the downloads) in 202 seconds which quite surprised me :-)

While I hardly have any real knowledge about Java and different versions etc I more or less contribute this issue not happening for me to the only other difference, besides map versions, I can think of, being the fact that I use Eclipse Temurin Java.
I have been using it on my own version of WMC for the past ~year I think.
Again, no expert on this, but where Oracle Java gave me out of memory errors while I had ram to spare, this (with the right config) happily takes my 128GB of ram machine to 99% ram usage and continues with the swap file if needed.
Which it does for me atm when generating Greenland.

I'll be happy to test other configs but so far not seeing this issue.

Edit: Adding a screenshot showing toilets, drinking water, cafe and a supermarket