-
Notifications
You must be signed in to change notification settings - Fork 6
feat: sync transitfeeds data with the mdb #1451
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR refactors the data import infrastructure and adds support for importing TransitFeeds historical data. The changes extract common helper functions for feed and entity type management, reorganize the JBDA import logic into a dedicated subpackage, and implement a new task for syncing TransitFeeds data including feeds, real-time feeds, and historical datasets.
Key Changes:
- Extracted shared utilities (
_get_or_create_feed,_get_or_create_entity_type,get_feed) todata_import_utils.pyfor reuse across importers - Reorganized JBDA import into
jbda/subpackage - Added new
sync_transitfeedstask with comprehensive CSV processing for GTFS and GTFS-RT feeds plus historical datasets - Added pandas dependency for CSV data processing
Reviewed Changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
sync_transitfeeds.py |
New implementation for TransitFeeds data synchronization with support for GTFS/GTFS-RT feeds and historical datasets |
test_transitfeeds_sync.py |
Test suite with mocked CSV data validating feed creation, dataset assignment, and edge cases |
gtfs_feeds.csv |
CSV data containing 270 GTFS feed definitions with metadata |
gtfs_rt_feeds.csv |
CSV data containing 21 GTFS-RT feed definitions with entity types |
import_jbda_feeds.py |
Refactored to use shared utilities from data_import_utils.py |
data_import_utils.py |
New shared utilities extracted from JBDA importer, now used by both JBDA and TransitFeeds |
main.py |
Updated imports and registered new sync_transitfeeds_data task |
requirements.txt |
Added pandas dependency for CSV processing |
functions-python/tasks_executor/src/tasks/data_import/data_import_utils.py
Outdated
Show resolved
Hide resolved
functions-python/tasks_executor/src/tasks/data_import/data_import_utils.py
Outdated
Show resolved
Hide resolved
functions-python/tasks_executor/src/tasks/data_import/data_import_utils.py
Outdated
Show resolved
Hide resolved
functions-python/tasks_executor/src/tasks/data_import/transitfeeds/sync_transitfeeds.py
Show resolved
Hide resolved
functions-python/tasks_executor/src/tasks/data_import/transitfeeds/sync_transitfeeds.py
Outdated
Show resolved
Hide resolved
functions-python/tasks_executor/src/tasks/data_import/transitfeeds/sync_transitfeeds.py
Outdated
Show resolved
Hide resolved
functions-python/tasks_executor/src/tasks/data_import/transitfeeds/sync_transitfeeds.py
Show resolved
Hide resolved
|
This PR implements a "one-time" functionality and adds a few "large" JSONs to the task executor. This extra size will affect all executions of the function, increasing memory usage and startup time. It's preferable to have the JSONs outside the code to avoid impacting other tasks within the executor. However, it's also acceptable to keep the JSON in the function, run the function on prod, and follow up PR to remove it. It might also be a good idea to keep the JSONs around for reference. Any thoughts? |
davidgamez
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice refactoring!
Closes #1017
Summary:
This PR refactors and extends the data import utilities and handlers in the Python tasks executor. The changes include extracting shared helper functions for feed and entity type management, reorganizing the JBDA import logic into a subpackage, and registering a new task for syncing TransitFeeds data. Additionally, the
pandaslibrary is added as a dependency.New features and task registration:
sync_transitfeeds_datatask, enabling synchronization of TransitFeeds data into the database. [1] [2]Please make sure these boxes are checked before submitting your pull request - thanks!
./scripts/api-tests.shto make sure you didn't break anything