Refactor: spaghetti v2.0

[jGaboardi]

• Before any more work done on minor refactoring spaghetti, consider implications of a major refactor in terms of methods for storing and accessing attributes.

• Much of the code base written years ago for python2.7 and is not making use of recent improvements, specifically the use of pandas for storing/accessing attributes.

• Many of the class attributes in Network and PointPattern are make use of dict and OrderedDict datatypes. pandas may help with streamlining workflow, efficient processing, and storage.

Thoughts? @sjsrey

[jGaboardi]

This should also serve as a discussion for also implementing pandana under the hood to big performance improvements.

@knaaptime

[jGaboardi]

document: Refactor: spaghetti v2.0

[jGaboardi]

pyrosm? @HTenkanen

[HTenkanen]

Thanks for pinging! It was nice to meet you all today! 🙂 Would it perhaps be a good idea to arange another chat sometime to discuss about these things specifically? @jGaboardi @knaaptime

Would be good to understand how different pieces go together, what is the longer term vision for spaghetti and other graph stuff, etc. 🙂 I think e.g the desired scale for the network analysis influences quite a bit which libraries/data structures / algorithms would be optimal to use. For metropolitan scale, I think pandana is great, but if going to state or country level analysis, some other approaches might be needed as the memory usage is quite often the bottleneck.

[jGaboardi]

Would it perhaps be a good idea to arange another chat sometime to discuss about these things specifically?

Absolutely.

[jGaboardi]

We should loop @martinfleis and @ljwolf in on this, as well.

[martinfleis]

After a bit of thinking, my first ideas:

• pyrosm as a PBF parser should remain a package focusing on a single purpose: get data from OSM into Python ecosystem. That entails GeoDataFrames as well as various graph representations. Maybe a possibility to directly parse PBF to geo-arrow (once ready) would be handy as well.
• spaghetti 2.0 should still focus on networks. It can be tightly connected to pyrosm but it should have a clear single purpose. The potential work on network weights and such should therefore go to spaghetti (and loaded from libpysal I guess).
• since pyrosm is focusing on data retrieval more than analysis, it may be more suitable to give it home under geopandas org, alongside contextily.

[HTenkanen]

Thanks @martinfleis for sharing your thoughts! 👍🏻 I agree, the idea and the main purpose for pyrosm has been to parse the data efficiently and that's about it. I was already thinking quite a bit whether the graph export functions should be part of it, but then decided to include them because being able to construct the graphs anyway required modifying the pyrosm codes quite a bit as well. I think geopandas organization could indeed be quite a nice home for pyrosm as it is anyway closely linked to geopandas. So it might be a logical solution. I guess this should be discussed also with Joris..