Best practice for reconstructing historical MultiPolygons (Boundaries) from full PBF

[BaBoWaGa]

Question

Hello,
I am working on a project to build a historical PostGIS database of world administrative boundaries (admin_level 2-8) using pyosmium. My input is a full history planet file (history-latest.osm.pbf).

My goal is to extract all versions of boundary relations as valid MultiPolygon geometries (essentially "exploding" the history into a timeline where each version of a relation has its own geometry).

The Problem

I have successfully implemented a pipeline for Nodes and Ways using a persistent SQLite cache for node coordinates (handling the timeline logic manually). However, I am struggling to reconstruct Polygons (Relations).

When I use osmium.area.AreaManager on a history file, I consistently get 0 polygons or incomplete geometries.

What I have tried

1. Filtering: I used osmium tags-filter to extract r/boundary and all nodes (n/), but I initially lost untagged member ways. I fixed this by keeping w/ (all ways), which results in a valid PBF containing member ways.
2. AreaManager: I implemented a handler using osmium.area.AreaManager. While this works perfectly for snapshot files (.osm.pbf), it seems to fail on history files (.osh.pbf).

My Assumptions / Confusion

It seems AreaManager might be designed for snapshots (assembling the "current" state). In a history file, a Relation V1 (from 2010) might depend on Way V1, but the file also contains Way V2 (from 2015).

Questions

1. Is AreaManager capable of handling history files? Does it automatically resolve the correct version of member ways based on the relation's timestamp?
2. If not, what is the recommended strategy? Do I need to manually cache all versions of all ways (nodes refs) in a DB and then manually stitch the geometry for every relation version?
3. Handling "Broken" Histories: Often osmium extract (bbox) cuts member ways, resulting in open rings. Is there a robust way in pyosmium to force-close these rings during assembly to salvage at least partial geometries?

Any guidance or pointers to examples for handling Historical Polygon Assembly would be greatly appreciated.

Environment:

• Python 3.10
• Osmium 3.x
• Input: history-latest.osm.pbf (from planet.osm.org)

[joto]

First a warning because there is often confusion about what the OSM history files can be used for: If you want to research how actual boundaries changed over time, OSM data is not a great resource. The reason is that more changes in OSM are due to how OSM is getting this data and improving it over time than with actual changes in the real world.

Generally working with OSM history data is difficult and there are many problems of data quality and changes in the tags OSM uses and there aren't great tools around. Most tools will not work with OSM history data. Osmium/PyOsmium does have some functionality for that but there are a lot of limitations when it comes to history data. Some of them are due to the fact that there just aren't that many people who want to work with the history data, some are due to the technically challenges.

Osmium/PyOsmium can not assemble polygons based on the OSM history file. There is no code in Osmium/PyOsmium that can get the correct versions of the member ways of relations or the nodes in ways needed for assembly. So either you have to implement all of that yourself, which is difficult, or you have to find a different way.

The usual way to handle history data is to pick specific date/times and do the processing for them independently. So you can, say, do your processing for every day (or every month or so) in the time frame you are interested in by using osmium time-filter to get the data for that point in time and then any normal processing for OSM data will work on that.

To reduce the amount of data you need to work with, you should be able to use osmium tags-filter -R r/boundary or something like that on the history file to only get the boundary relations from the history data and then use osmium getid -H -r -I ... to backfill the necessary nodes and ways. See the manual for the details of these commands. After that you can do further processing. Keep in mind that older data doesn't have relations but only ways with boundary tags.

If you actually need to capture all the changes in OSM, you'll need to create the output for every change in the relation itself or any of its way members and their node members. That's left as an exercise for the reader.

You can also have a look at the Ohsome which might work for you, I don't know much about what it is doing.

[BaBoWaGa]

@joto Thank you for the definitive clarification. It confirms what I suspected: AreaManager is strictly for snapshot assembly and cannot handle the temporal dependencies of a full history file out-of-the-box.

Based on your input ("implement all of that yourself"), I decided to tackle the "hard way" and built a custom assembler using pyosmium. Just to share the approach for future readers facing the same "0 polygons" issue:

The Solution Implemented:
Instead of relying on AreaManager, I wrote a custom Python handler that acts as a "Manual Stitcher":

1. Disk-Based Cache: I use a temporary SQLite database to cache all Node coordinates and Way structures (node refs) indexed by ID and Timestamp. This avoids RAM exhaustion on large history files.
2. Time-Travel Lookup: When processing a Relation (vN), the script queries the SQLite cache to fetch the specific version of member Ways (and their Nodes) that were valid at relation.timestamp.
3. Geometry Stitching: I implemented a custom algorithm that takes these raw segments, stitches them into rings, and forces closure if the ring is open (e.g., due to clip errors or incomplete history data).
4. Result: I am now successfully reconstructing valid MultiPolygons for historical boundaries and exporting them to PostGIS for analysis.

Regarding OSM History vs. Real World History:
Your warning is very well taken. My goal is primarily to analyze the evolution of the database (user metrics, edit frequencies), so the "OSM internal history" is exactly what I need. For visual "Real World" context, I am pairing this dataset with OpenHistoricalMap vector tiles as a background layer to avoid anachronisms.

Thanks again for maintaining such a powerful tool suite!