Merge pull request #71 from VIDA-NYU/feat/line_polygon_geometries

feat(loader): Add Support for geometry_column in UrbanMapper Loaders

Author: soniacq

docs/api/filters.md

@@ -2,7 +2,7 @@
 
 !!! tip "What is the Filter module?"
     The `filter` module is responsible for filtering geospatial datasets based on specific criteria or conditions out 
-    of your `urban layer`.
+    of your `urban layer` and based on information of latitude-longitude data columns or geometry specified in [WKT format](https://libgeos.org/specifications/wkt/).
 
     Meanwhile, we recommend to look through the [`Example`'s Filter](../copy_of_examples/1-Per-Module/4-filter/) for a more hands-on introduction about
     the Filter module and its usage.

docs/api/imputers.md

@@ -1,7 +1,7 @@
 # Imputers
 
 !!! tip "What is the Imputer module?"
-    The `imputer` module is responsible for handling missing data in geospatial datasets.
+    The `imputer` module is responsible for handling missing data in geospatial datasets, dealing with missing information of latitude-longitude data columns or geometry specified in [WKT format](https://libgeos.org/specifications/wkt/).
 
     Meanwhile, we recommend to look through the [`Example`'s Imputer](../copy_of_examples/1-Per-Module/3-imputer/) for a more hands-on introduction about
     the Imputer module and its usage.

docs/api/loaders.md

@@ -4,9 +4,9 @@
     The `loader` module is responsible for loading geospatial data into `UrbanMapper`. 
     It provides a unified interface for loading various data formats, including `shapefiles`, `parquet`, and `CSV` files 
     with geospatial information.
-
     `UrbanMapper` steps support using multiple datasets. The user can create multiple loader instances, one for each dataset, 
     combine them in a single dictionary with suitable keys, and use it in your pipeline.
+    Besides, geolocation can be loaded from latitude-longitude data columns or geometry specified in [WKT format](https://libgeos.org/specifications/wkt/).
 
     Meanwhile, we recommend to look through the [`Example`'s Loader](../copy_of_examples/1-Per-Module/1-loader/) for a more hands-on introduction about
     the Loader module and its usage.

docs/api/urban_layers.md

@@ -4,6 +4,7 @@
     The `urban_layer` module is responsible for the spatial canvases on which your datasets are displayed. These layers
     provide structure for urban insights, such as mapping taxi trips to busy `intersections` or analysing `neighbourhood`
     demographics.
+    A dataset is projected over an `urban_layer` based on latitude-longitude data columns or geometry specified in [WKT format](https://libgeos.org/specifications/wkt/).    
 
     Meanwhile, we recommend to look through the [`Example`'s Urban layer](../copy_of_examples/1-Per-Module/2-urban_layer/) for a more hands-on introduction about
     the Urban layer module and its usage.

docs/copy_of_examples/1-Per-Module/1-loader.ipynb

@@ -43,7 +43,7 @@
    "source": [
     "## Loading CSV Data\n",
     "\n",
-    "First up, let’s load a CSV file with PLUTO data. We’ll tell UrbanMapper where to find the longitude and latitude columns so it knows what’s what and can make sure those colums are well formatted prior any analysis.\n",
+    "First up, let’s load a CSV file with PLUTO data. We’ll tell UrbanMapper where to find the longitude-latitude or geometry columns so it knows what’s what and can make sure those colums are well formatted prior any analysis.\n",
     "\n",
     "Note that below we employ a given csv, but you can put your own path, try it out!"
    ]
@@ -59,6 +59,7 @@
     "    .loader # From the loader module\n",
     "    .from_file(\"<path_to>/pluto.csv\") # To update with your own path\n",
     "    .with_columns(longitude_column=\"longitude\", latitude_column=\"latitude\") # Inform your long and lat columns\n",
+    "#    .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.\n",
     ")\n",
     "\n",
     "gdf = csv_loader.load() # Load the data and create a geodataframe's instance\n",
@@ -87,6 +88,7 @@
     "    loader. # From the loader module\n",
     "    from_file(\"<path_to>/taxisvis5M.parquet\") # To update with your own path\n",
     "    .with_columns(\"pickup_longitude\", \"pickup_latitude\") # Inform your long and lat columns\n",
+    "#    .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.\n",
     ")\n",
     "\n",
     "gdf = parquet_loader.load() # Load the data and create a geodataframe's instance\n",
@@ -163,6 +165,7 @@
     "    .loader # From the loader module\n",
     "    .from_dataframe(df) # To update with your dataframe\n",
     "    .with_columns(longitude_column=\"longitude\", latitude_column=\"latitude\") # Inform your long and lat columns\n",
+    "#    .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.\n",
     ")\n",
     "\n",
     "gdf = df_loader.load() # Load the data and create a geodataframe's instance\n",
@@ -187,7 +190,30 @@
    "outputs": [],
    "source": [
     "# Load a full dataset directly from Hugging Face\n",
-    "loader = mapper.loader.from_huggingface(\"oscur/pluto\", number_of_rows=100).with_columns(longitude_column=\"longitude\", latitude_column=\"latitude\")\n",
+    "loader = (\n",
+    "  mapper\n",
+    "  .loader\n",
+    "  .from_huggingface(\"oscur/pluto\", number_of_rows=100)\n",
+    "  .with_columns(longitude_column=\"longitude\", latitude_column=\"latitude\")\n",
+    "#  .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.  \n",
+    ")  \n",
+    "gdf = loader.load()\n",
+    "gdf  # Next steps: analyze or visualize the data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load a full dataset directly from Hugging Face\n",
+    "loader = (\n",
+    "  mapper\n",
+    "  .loader\n",
+    "  .from_huggingface(\"oscur/NYC_raised_crosswalk\", number_of_rows=100)\n",
+    "  .with_columns(geometry_column=\"WKT Geometry\") # Inform your geometry column instead of longitude and latitude columns. \n",
+    ")  \n",
     "gdf = loader.load()\n",
     "gdf  # Next steps: analyze or visualize the data"
    ]
@@ -224,12 +250,20 @@
    "outputs": [],
    "source": [
     "# Load datasets directly from Hugging Face\n",
-    "pluto_data = mapper.loader.from_huggingface(\"oscur/pluto\", number_of_rows=100).with_columns(longitude_column=\"longitude\", latitude_column=\"latitude\").load()\n",
+    "pluto_data = (\n",
+    "  mapper\n",
+    "  .loader\n",
+    "  .from_huggingface(\"oscur/pluto\", number_of_rows=100)\n",
+    "  .with_columns(longitude_column=\"longitude\", latitude_column=\"latitude\")\n",
+    "#  .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.    \n",
+    "  .load()\n",
+    ")\n",
     "taxi_data =  (\n",
     "  mapper\n",
     "  .loader\n",
     "  .from_huggingface(\"oscur/taxisvis1M\", number_of_rows=100)\n",
     "  .with_columns(longitude_column=\"pickup_longitude\", latitude_column=\"pickup_latitude\")\n",
+    "#  .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.    \n",
     "  .with_map({\"pickup_longitude\": \"longitude\", \"pickup_latitude\": \"latitude\"}) ## Routines like layer.map_nearest_layer needs datasets with the same longitude_column and latitude_column\n",
     "  .load()\n",
     ")\n",

docs/copy_of_examples/1-Per-Module/3-imputer.ipynb

@@ -48,9 +48,12 @@
     "data = (\n",
     "    mapper\n",
     "    .loader\n",
-    "    .from_huggingface(\"oscur/pluto\", number_of_rows=20000, streaming=True).with_columns(\"longitude\", \"latitude\").load()\n",
-    "    # From the loader module, from the following file within the OSCUR HuggingFace datasets hub and with the `longitude` and `latitude`\n",
-    ")"
+    "    .from_huggingface(\"oscur/pluto\", number_of_rows=20000, streaming=True)\n",
+    "    .with_columns(\"longitude\", \"latitude\")\n",
+    "#    .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.\n",
+    "    .load()\n",
+    "    # From the loader module, from the following file within the OSCUR HuggingFace datasets hub and with the `longitude` and `latitude` or only with `geometry`\n",
+    ")    "
    ]
   },
   {
@@ -88,6 +91,7 @@
     "    .imputer # From the imputer module\n",
     "    .with_type(\"SimpleGeoImputer\") # With the type SimpleGeoImputer\n",
     "    .on_columns(longitude_column=\"longitude\", latitude_column=\"latitude\") # On the columns longitude and latitude\n",
+    "#    .on_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.      \n",
     "    .transform(data, layer)  # All imputers require access to the urban layer in case they need to extract information from it.\n",
     ")\n",
     "\n",
@@ -136,8 +140,11 @@
     "data1 = (\n",
     "    mapper\n",
     "    .loader\n",
-    "    .from_huggingface(\"oscur/pluto\", number_of_rows=1000, streaming=True).with_columns(\"longitude\", \"latitude\").load()\n",
-    "    # From the loader module, from the following file and with the `longitude` and `latitude`\n",
+    "    .from_huggingface(\"oscur/pluto\", number_of_rows=1000, streaming=True)\n",
+    "    .with_columns(\"longitude\", \"latitude\")\n",
+    "#    .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.\n",
+    "    .load()\n",
+    "    # From the loader module, from the following file and with the `longitude` and `latitude` or only `geometry`\n",
     ")\n",
     "\n",
     "# Load Parquet data\n",
@@ -146,6 +153,7 @@
     "    .loader\n",
     "    .from_huggingface(\"oscur/taxisvis1M\", number_of_rows=1000, streaming=True) # To update with your own path\n",
     "    .with_columns(\"pickup_longitude\", \"pickup_latitude\").load() # Inform your long and lat columns\n",
+    "#    .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.      \n",
     ")\n",
     "\n",
     "data = {\n",
@@ -161,6 +169,7 @@
     "    .imputer # From the imputer module\n",
     "    .with_type(\"SimpleGeoImputer\") # With the type SimpleGeoImputer\n",
     "    .on_columns(longitude_column=\"longitude\", latitude_column=\"latitude\") # On the columns longitude and latitude\n",
+    "#    .on_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.      \n",
     "    .with_data(data_id=\"pluto_data\") # On a specific data from the dictionary\n",
     "    .transform(data, layer)  # All imputers require access to the urban layer in case they need to extract information from it.\n",
     ")"

docs/copy_of_examples/1-Per-Module/4-filter.ipynb

@@ -50,7 +50,7 @@
     "    mapper\n",
     "    .loader\n",
     "    .from_huggingface(\"oscur/pluto\", number_of_rows=5000, streaming=True).with_columns(\"longitude\", \"latitude\").load()\n",
-    "    # From the loader module, from the following file within the HuggingFace OSCUR datasets hub and with the `longitude` and `latitude`\n",
+    "    # From the loader module, from the following file within the HuggingFace OSCUR datasets hub and with the `longitude` and `latitude` or only `geometry`\n",
     ")\n",
     "\n",
     "# Create urban layer\n",
@@ -127,15 +127,15 @@
     "    mapper\n",
     "    .loader\n",
     "    .from_huggingface(\"oscur/pluto\", number_of_rows=1000, streaming=True).with_columns(\"longitude\", \"latitude\").load()\n",
-    "    # From the loader module, from the following file and with the `longitude` and `latitude`\n",
+    "    # From the loader module, from the following file and with the `longitude` and `latitude` or only `geometry`\n",
     ")\n",
     "\n",
     "# Load Parquet data\n",
     "data2 = (\n",
     "    mapper\n",
     "    .loader\n",
     "    .from_huggingface(\"oscur/taxisvis1M\", number_of_rows=1000, streaming=True) # To update with your own path\n",
-    "    .with_columns(\"pickup_longitude\", \"pickup_latitude\").load() # Inform your long and lat columns\n",
+    "    .with_columns(\"pickup_longitude\", \"pickup_latitude\").load() # Inform your long and lat columns or only geometry\n",
     ")\n",
     "\n",
     "data = {\n",

docs/copy_of_examples/1-Per-Module/5-enricher.ipynb

@@ -54,7 +54,7 @@
     "    mapper\n",
     "    .loader\n",
     "    .from_huggingface(\"oscur/pluto\", number_of_rows=5000, streaming=True).with_columns(\"longitude\", \"latitude\").load()\n",
-    "    # From the loader module, from the following file within the HuggingFace OSCUR datasets hub and with the `longitude` and `latitude`\n",
+    "    # From the loader module, from the following file within the HuggingFace OSCUR datasets hub and with the `longitude` and `latitude` or only `geometry`\n",
     ")\n",
     "\n",
     "# Create urban layer\n",
@@ -98,8 +98,8 @@
     "# so that we can take into account when enriching.\n",
     "_, mapped_data = layer.map_nearest_layer(\n",
     "    data,\n",
-    "    longitude_column=\"longitude\",\n",
-    "    latitude_column=\"latitude\",\n",
+    "    longitude_column=\"longitude\", latitude_column=\"latitude\",\n",
+    "#   geometry_column=<geometry_column_name>\", # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.\n",
     "    output_column=\"nearest_intersection\", # Will create this column in the data, so that we can re-use that throughout the enriching process below.\n",
     ")\n",
     "\n",
@@ -182,8 +182,11 @@
     "data1 = (\n",
     "    mapper\n",
     "    .loader\n",
-    "    .from_huggingface(\"oscur/pluto\", number_of_rows=1000, streaming=True).with_columns(\"longitude\", \"latitude\").load()\n",
-    "    # From the loader module, from the following file and with the `longitude` and `latitude`\n",
+    "    .from_huggingface(\"oscur/pluto\", number_of_rows=1000, streaming=True)\n",
+    "    .with_columns(\"longitude\", \"latitude\")\n",
+    "#    .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.    \n",
+    "    .load()\n",
+    "    # From the loader module, from the following file and with the `longitude` and `latitude` or only `geometry`\n",
     ")\n",
     "\n",
     "# Load Parquet data\n",
@@ -192,6 +195,7 @@
     "    .loader\n",
     "    .from_huggingface(\"oscur/taxisvis1M\", number_of_rows=1000, streaming=True) # To update with your own path\n",
     "    .with_columns(\"pickup_longitude\", \"pickup_latitude\") # Inform your long and lat columns\n",
+    "#    .with_columns(geometry_column=<geometry_column_name>\") # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.    \n",
     "    .with_map({\"pickup_longitude\": \"longitude\", \"pickup_latitude\": \"latitude\"}) ## Routines like layer.map_nearest_layer needs datasets with the same longitude_column and latitude_column\n",
     "    .load() \n",
     ")\n",
@@ -215,8 +219,8 @@
     "# so that we can take into account when enriching.\n",
     "_, mapped_data = layer.map_nearest_layer(\n",
     "    data,\n",
-    "    longitude_column=\"longitude\",\n",
-    "    latitude_column=\"latitude\",\n",
+    "    longitude_column=\"longitude\", latitude_column=\"latitude\",\n",
+    "#    geometry_column=<geometry_column_name>\", # Replace <geometry_column_name> with the actual name of your geometry column instead of latitude and longitude columns.\n",
     "    output_column=\"nearest_intersection\", # Will create this column in the data, so that we can re-use that throughout the enriching process below.\n",
     ")\n",
     "\n",