[Backend] Investigate Architecture - Generate PMTiles

[cka-y]

Suggested Implementation (Based on Issue #909):

• Deploy a Cloud Run Instance:
  Use a Dockerfile that supports Tippecanoe (Tippecanoe Docker Image).

• Set Up a Python Environment:
  Create an environment capable of running a Python script to generate a GeoJSON file.

• Create wrapper API

• Figure out the triggers

Why use a Cloud Run Instance instead of a Cloud Function?
Tippecanoe currently lacks an official Python package, and the PMTiles documentation exclusively references Tippecanoe. Investigating alternative tools would require additional research and time, making a Cloud Run instance the more efficient choice.

Investigate adding watermarks to PMTiles data

** Test generating PMTiles for 1 large feed (ideally biggest)

[davidgamez]

Timebox: 3 days

[emmambd]

@davidgamez Could you add your notes of how you suggested approaching implementing this?

[davidgamez]

@davidgamez Could you add your notes of how you suggested approaching implementing this?

I suggested using Cloud Jobs to avoid creating a wrapper API. This is the official documentation: https://cloud.google.com/run/docs/create-jobs.
We used cloud tasks before for the metrics POC, but they were created from a workflow, details

[jcpitre]

As we have seen with other functions we have (e.g. batch-datasets) the pmtiles generation has to be done for 2 use cases:

• Backfill (or Batchfill), where we go through the existing dataset and create the pmtiles
  • This is triggered by an http call and is done periodically or on-demand.
• New datasets, where we create the pmtiles for a newly downloaded dataset

Components

• Docker image

  • We create a Docker image based on the python:3.11 image on which we build and install tippecanoe
  • We run the functions-framework as the command. This lets us use the @functions_framework decoration and the infrastructure that comes with it.
  • We create a cloud run function that uses the docker image.
  • We found a python package that contains tippecanoe. We can just use a cloud run based on python.

• Trigger

  • The unzipping of GTFS files is done by another process (See Unzip and store latest GTFS feeds #911)
  • After unzipping, this other process publishes to the dataset-unzipped-{env} topic.
    • We might use another trigger instead, like a http trigger.

• Payload includes:

  • The feed stable id e.g. mdb-1
  • The dataset stable_id e.g. mdb-1-202406181928
  • Note: The exact URL of the unzipped files directory and the bucket URL for uploading the results will be built from the stable ids. This is already done with other functions (e.g. reverse_geolocation.py)

• GCP Function

  • Once it has received a message from the topic, the function does this (with calls to subprocess.run):
    • Download the useful unzipped files to the local file system (stops.txt, shapes, txt, etc)
    • Create a routes.geojson and a stops.geojson (with python script)
    • Run tippecanoe on these files.
    • Create routes.json (with python script)
    • Upload the resulting pmtiles and json files to the bucket
    • Maybe publish to another pubsub topic to say the pmtiles were created?
    • Cleanup files locally

• Batch processing

  • We will need another cloud function triggered by an http message. This will:
    • consult the DB to retrieve the latest datasets names
    • Establish if there is an extracted_files folder with the unzipped dataset
    • If there is, publish to the dataset-unzipped-{env} topic.

• Database access

  • We should consider adding a flag to the dataset entity on the DB (or have a separate table) that says if the pmtiles were generated or not
  • This flag would be useful to gather statistics in Metabase
  • To maintain integrity, we could have a GCP function that is notified when a pmtiles file is created in the bucket. It's job would be to set the pmtiles flag in the DB.
  • We might be able to use the task infrastructure put in place by fix: missing validation reports #1102

[jcpitre]

As a first step, on order to have a MVP, we will create the GCP function to do the processing and with an simple HTTP trigger.
See #1296

[jcpitre]

Possible optimisation:

• When calling the function with a dataset, also provide the previous latest dataset for the given feed. We could then check if the unzipped filesof interest (stops.txt, shapes.txt, etc) have changed and regenerate the pmtiles only in case of a change.

See #1313