Vector Feature Relationships

[annehaley]

Resolves #285.

This PR updates the ingest process for network and region datasets such that vector features have relationships to their counterpart NetworkNode, NetworkEdge and Region representations.

Note: To apply these changes, existing Network and Region datasets should be overwritten. Re-run the ingest task with "action": "replace" added to the JSON specification for each of these datasets.

[BryonLewis]

I think this is fine. I pulled and ran the code and everything seems to work.

I just want to confirm my understanding.

Before:
You would create these NetworkEdge and NetworkNode objects when creating network. This would be followed by taking all the data from the dataset (shapefile) and creating a vector feature for each feature in it.

For Regions before it would do a similar thing where it would only create a Region model object.

After:
The updated version now uses the data for the NetworkEdge and NetworkNode to create the vector features instead of relying on the raw 'features' of the source. It's technically the same thing because you are using the source_data.get('features').

The only thing that I see that could be different between before and after for the create_network change is that if you had a dataset that had a mix of Points, Lines, and Polygons the before method would add the polygons as a single feature. This method only adds the Points and Lines. It may be that the geopandas splits a polygon into it's points and lines and so while you see the same thing it may be split among multiple VectorFeature objects. This may be a benefit also because you could upload a polygon and it would split to edges and nodes.

Edit: I missed that the bottom of create_network had a modification to rewrite the vector_data geojson_data, so it would have broken the polygon apart in create_network, rewritten it and then loaded the broken apart nodes/edges in create_vector_features before this change.

[annehaley]

Yes, that's correct. Before, we created network and region objects first, referencing the GeoJSON as the source of truth (the input data has already been converted to GeoJSON by this point). Once those specialized objects were created, we modified the GeoJSON to include things like node_id and region_id in the properties. Then separately, we referenced the modified GeoJSON to create VectorFeature objects. No database relationship was made between them, only the properties contained a reference to the ID.

And it's true that this change would make it so that for a network dataset, features not included in the network will be dropped. Due to the way that we process network topology, polygons are excluded entirely, not broken into constituent points and lines. For region datasets, non-region features would be dropped, including all points and lines. I think this is acceptable behavior for these specialized datasets. We were already operating under the assumption that these datasets would not contain extra features. Datasets can only use the special Network or Region behavior via the ingest script; the dataset upload UI does not have that option. So we have control over which datasets will undergo this transformation.

[BryonLewis]

And it's true that this change would make it so that for a network dataset, features not included in the network will be dropped. Due to the way that we process network topology, polygons are excluded entirely, not broken into constituent points and lines. For region datasets, non-region features would be dropped, including all points and lines. I think this is acceptable behavior for these specialized datasets. We were already operating under the assumption that these datasets would not contain extra features. Datasets can only use the special Network or Region behavior via the ingest script; the dataset upload UI does not have that option. So we have control over which datasets will undergo this transformation.

Thanks, I was just trying to confirm my understanding seeing as I haven't really dealt with the network stuff in this repo before.