feat: transitFeedSyncProcessor implementation (#760)

Author: AlfredNwolisa

functions-python/README.md

@@ -32,6 +32,24 @@ The function configuration file contains the following properties:
 - `min_instance_count`: The minimum number of function instances that can be created in response to a load.
 - `available_cpu_count`: The number of CPU cores that are available to the function.
 
+# Local Setup
+
+## Requirements
+The requirements to run the functions locally might differ depending on the Google cloud dependencies. Please refer to each function to make sure all the requirements are met.
+
+## Install the Google Cloud SDK
+To be able to run the functions locally, the Google Cloud SDK should be installed. Please refer to the [Google Cloud SDK documentation](https://cloud.google.com/sdk/docs/install) for more information.
+
+## Install the Google Cloud Emulators
+
+```bash
+gcloud components install cloud-datastore-emulator
+```
+
+- Install the Pub/Sub emulator
+```bash
+gcloud components install pubsub-emulator
+```
 
 # Useful scripts
 - To locally execute a function use the following command:

functions-python/feed_sync_dispatcher_transitland/.coveragerc

@@ -0,0 +1,10 @@
+[run]
+omit =
+    */test*/*
+    */database_gen/*
+    */dataset_service/*
+    */helpers/*
+
+[report]
+exclude_lines =
+    if __name__ == .__main__.:

functions-python/feed_sync_dispatcher_transitland/.env.rename_me

@@ -0,0 +1,5 @@
+# Environment variables for tokens function to run locally. Delete this line after rename the file.
+FEEDS_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/MobilityDatabase
+PROJECT_ID=my-project-id
+PUBSUB_TOPIC_NAME=my-topic
+TRANSITLAND_API_KEY=your-api-key

functions-python/feed_sync_dispatcher_transitland/README.md

@@ -0,0 +1,79 @@
+# Batch Datasets
+This directory contains the GCP serverless function that triggers the sync feeds in transitland. 
+The function publish one Pub/Sub message per transitland feed to be synced.
+```json
+    {
+        "message": {
+            "data": 
+            {
+                external_id="feeds_onestop_id",
+                feed_id="feed_id",
+                execution_id=execution_id,
+                feed_url="feed_url",
+                spec="spec",
+                auth_info_url="auth_info_url",
+                auth_param_name="auth_param_name",
+                type="type",
+                operator_name="operator_name",
+                country="country",
+                state_province="state_province",
+                city_name="city_name",
+                source="TLD",
+                payload_type=payload_type
+            }            
+        }
+    }
+``` 
+
+# Function configuration
+The function is configured using the following environment variables:
+- `PUBSUB_TOPIC`: The Pub/Sub topic to publish the messages to.
+- `PROJECT_ID`: The GCP Project id.
+- `TRANSITLAND_API_KEY`: The Transitland API key(secret).
+
+# Local development
+The local development of this function follows the same steps as the other functions.
+
+Install Google Pub/Sub emulator, please refer to the [README.md](../README.md) file for more information.
+
+## Python requirements
+
+- Install the requirements
+```bash
+    pip install -r ./functions-python/feed_sync_dispatcher_transitland/requirements.txt
+```
+
+## Test locally with Google Cloud Emulators
+
+- Execute the following commands to start the emulators:
+```bash
+    gcloud beta emulators pubsub start --project=test-project --host-port='localhost:8043'
+```
+
+- Create a Pub/Sub topic in the emulator:
+```bash
+    curl -X PUT "http://localhost:8043/v1/projects/test-project/topics/feed-sync-transitland"
+```
+
+- Start function
+```bash
+  export PUBSUB_EMULATOR_HOST=localhost:8043 && ./scripts/function-python-run.sh --function_name feed_sync_dispatcher_transitland
+```
+
+- [Optional]: Create a local subscription to print published messages:
+```bash
+./scripts/pubsub_message_print.sh feed-sync-transitland
+```
+
+- Execute function
+```bash
+   curl http://localhost:8080
+```
+
+- To run/debug from your IDE use the file `main_local_debug.py`
+
+# Test
+- Run the tests
+```bash
+  ./scripts/api-tests.sh --folder functions-python/feed_sync_dispatcher_transitland 
+```

functions-python/feed_sync_dispatcher_transitland/function_config.json

@@ -0,0 +1,19 @@
+{
+  "name": "feed-sync-dispatcher-transitland",
+  "description": "Feed Sync Dispatcher for Transitland",
+  "entry_point": "feed_sync_dispatcher_transitland",
+  "timeout": 540,
+  "memory": "512Mi",
+  "trigger_http": true,
+  "include_folders": ["database_gen", "helpers"],
+  "secret_environment_variables": [
+    {
+      "key": "FEEDS_DATABASE_URL"
+    }
+  ],
+  "ingress_settings": "ALLOW_INTERNAL_AND_GCLB",
+  "max_instance_request_concurrency": 20,
+  "max_instance_count": 10,
+  "min_instance_count": 0,
+  "available_cpu": 1
+}

functions-python/feed_sync_dispatcher_transitland/main_local_debug.py

@@ -0,0 +1,26 @@
+# Code to be able to debug locally without affecting the runtime cloud function
+
+
+# Requirements:
+# - Google Cloud SDK installed
+# - Make sure to have the following environment variables set in your .env.local file
+# - Local database in running state
+# - Follow the instructions in the README.md file
+#
+# Usage:
+# - python feed_sync_dispatcher_transitland/main_local_debug.py
+
+from src.main import feed_sync_dispatcher_transitland
+from dotenv import load_dotenv
+
+# Load environment variables from .env.local
+load_dotenv(dotenv_path=".env.local_test")
+
+if __name__ == "__main__":
+
+    class RequestObject:
+        def __init__(self, headers):
+            self.headers = headers
+
+    request = RequestObject({"X-Cloud-Trace-Context": "1234567890abcdef"})
+    feed_sync_dispatcher_transitland(request)

functions-python/feed_sync_dispatcher_transitland/requirements.txt

@@ -0,0 +1,20 @@
+# Common packages
+functions-framework==3.*
+google-cloud-logging
+psycopg2-binary==2.9.6
+aiohttp~=3.10.5
+asyncio~=3.4.3
+urllib3~=2.2.2
+requests~=2.32.3
+attrs~=23.1.0
+pluggy~=1.3.0
+certifi~=2024.8.30
+pandas
+
+# SQL Alchemy and Geo Alchemy
+SQLAlchemy==2.0.23
+geoalchemy2==0.14.7
+
+# Google specific packages for this function
+google-cloud-pubsub
+cloudevents~=1.10.1

functions-python/feed_sync_dispatcher_transitland/requirements_dev.txt

@@ -0,0 +1,2 @@
+Faker
+pytest~=7.4.3