OSError: exception: stack overflow (Appears during clean_network)

[jobelanger]

Received this error when running clean_network on a large graph. In my case I was using it for country-level datasets in Mali and Burkina Faso, running on a Windows 10 machine. My current workaround as suggested by @d3netxer is to skip the cleaning function altogether.

[d3netxer]

I'm doing some work on my horn of africa analysis. I believe that totally skipping the cleaning is why I had trouble getting O-D routes. Missing reflected edges still need to be added. I'm trying to run this separately:

G_full = gn.add_missing_reflected_edges(G)

will provide update...

[gracedoherty]

@d3netxer Any update on this issue? I'm getting the same error. I was going to run clean_network, then salt_long_lines, then proceed with the time-based OD creation. Sounds like we shouldn't be skipping the clean_network function if so?

I'm using the latest Africa osm pbf and clipping it to a buffered border of Senegal & The Gambia.

[d3netxer]

no, you don’t want to skip cleaning the network. But you can try running simplify junctions and adding missing reflected edges separately, and see if this works.

[gracedoherty]

Okay, thanks. simplify_junctions is the point where the stack overflow error prints. Other issues cropped up for me when I removed it from the .py file. I ran an abbreviated version of the function and it completed. I included:

• add_missing_reflected_edges()
• custom_simplify() # the second iteration
• convert_to_MultiDiGraph()
• add_missing_reflected_edges()

And I removed the following:

• simplify_junctions() # Stack overflow error.
• custom_simplify() # the first iteration
• remove_duplicate_edges # I forgot to document what line this was happening, but for either custom simplify or remove duplicate edges, the kernel stalls and never finishes (still "running" even overnight).
  

[jwrap]

I'm working on the OSM network for Cabo Verde (downloaded from Geofabrik) and ran into the same issue. I managed to resolve it by modifying the simplify_junctions function as follows:

1. not sure why but attempting to change the crs of gdfnodes simply changed my geometries to POINT (inf inf)
2. unary_union was causing the stack overflow, so i used the geometry after passing .dissolve() instead

Further down the line I see that the custom_simplify function tries to call "ConvertToMultiDiGraph" instead of "convert_to_MultiDiGraph" so i fixed that too.

Now the clean_network function seems to work well for me. Hope this helps!

[d3netxer]

Thanks @jwrap for your comments. Here are some responses:

• I would like to drill down why the attempting to change the crs didn't work for you. Do you know the which measure_crs you used? Also the projection/coordinates of you graph?
• I do recall isolating the stack_overflow() error to the unary_union code. It tends to pop-up in very large graphs. This seems like a great way to get around it is using the Geopandas dissolve function you found? I didn't know about that.
• The current master branch does have the up-to-date convert_to_MultiDiGraph function. I recommend you uninstall GOSTnets, then pull the master branch and reference it in your code.

[Charlesfox1]

GOSTs in the shell… unary_union sucks for large graphs. I’m sorry I ever used it frankly. One solution I had was to split the big graph into say ten equal parts, unary union them… then join them at the end. Or push whole graph down to something like igraph so it does it in C… Or use a PostgreSQL instance + something like fastapi to send the graph in node/ edge form into a DB, do geom transformation there, then send it back again… “

…

On 1 Mar 2022, at 18:50, @tomgertin ***@***.***> wrote:  Thanks @jwrap for your comments. Here are some responses: I would like to drill down why the attempting to change the crs didn't work for you. Do you know the which measure_crs you used? Also the projection/coordinates of you graph? I do recall isolating the stack_overflow() error to the unary_union code. It tends to pop-up in very large graphs. This seems like a great way to get around it is using the Geopandas dissolve function you found? I didn't know about that. The current master branch does have the up-to-date convert_to_MultiDiGraph function. I recommend you uninstall GOSTnets, then pull the master branch and reference it in your code. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.

[d3netxer]

Interesting, you got me thinking now Charles...maybe Dask can be used. I'm reading there is a dask-geopandas, I never tried that before; but I want to try that now, I'm making a mental note.

[Charlesfox1]

Dask could work computationally, for sure. If you guys don’t have a PostGIS server set up on a box somewhere yet I’d recommend that as a real priority though. You could store every network ever made in a dedicated GOSTnets Schema… even connects to QGIS directly for super fast loading / rendering. Slap a FastAPI on top of that and you could do a load of heavy lifting on the database, even if you were on mission on the other side of the world…

…

On 1 Mar 2022, at 19:49, @tomgertin ***@***.***> wrote:  Interesting, you got me thinking now Charles...maybe Dask can be used. I'm reading there is a dask-geopandas, I never tried that before; but I want to try that now, I'm making a mental note. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.