Skip to content

Conversation

@cka-y
Copy link
Contributor

@cka-y cka-y commented Nov 11, 2025

Closes #1017

Summary:

This PR refactors and extends the data import utilities and handlers in the Python tasks executor. The changes include extracting shared helper functions for feed and entity type management, reorganizing the JBDA import logic into a subpackage, and registering a new task for syncing TransitFeeds data. Additionally, the pandas library is added as a dependency.

New features and task registration:

  • Added a new handler and registration for the sync_transitfeeds_data task, enabling synchronization of TransitFeeds data into the database. [1] [2]

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y cka-y requested a review from Copilot November 11, 2025 02:20
@cka-y cka-y changed the title Feat/1017 feat: sync transitfeeds data with the mdb Nov 11, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the data import infrastructure and adds support for importing TransitFeeds historical data. The changes extract common helper functions for feed and entity type management, reorganize the JBDA import logic into a dedicated subpackage, and implement a new task for syncing TransitFeeds data including feeds, real-time feeds, and historical datasets.

Key Changes:

  • Extracted shared utilities (_get_or_create_feed, _get_or_create_entity_type, get_feed) to data_import_utils.py for reuse across importers
  • Reorganized JBDA import into jbda/ subpackage
  • Added new sync_transitfeeds task with comprehensive CSV processing for GTFS and GTFS-RT feeds plus historical datasets
  • Added pandas dependency for CSV data processing

Reviewed Changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
sync_transitfeeds.py New implementation for TransitFeeds data synchronization with support for GTFS/GTFS-RT feeds and historical datasets
test_transitfeeds_sync.py Test suite with mocked CSV data validating feed creation, dataset assignment, and edge cases
gtfs_feeds.csv CSV data containing 270 GTFS feed definitions with metadata
gtfs_rt_feeds.csv CSV data containing 21 GTFS-RT feed definitions with entity types
import_jbda_feeds.py Refactored to use shared utilities from data_import_utils.py
data_import_utils.py New shared utilities extracted from JBDA importer, now used by both JBDA and TransitFeeds
main.py Updated imports and registered new sync_transitfeeds_data task
requirements.txt Added pandas dependency for CSV processing

@davidgamez
Copy link
Member

This PR implements a "one-time" functionality and adds a few "large" JSONs to the task executor. This extra size will affect all executions of the function, increasing memory usage and startup time. It's preferable to have the JSONs outside the code to avoid impacting other tasks within the executor. However, it's also acceptable to keep the JSON in the function, run the function on prod, and follow up PR to remove it. It might also be a good idea to keep the JSONs around for reference. Any thoughts?

@cka-y cka-y requested a review from davidgamez November 12, 2025 18:18
Copy link
Member

@davidgamez davidgamez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice refactoring!

@cka-y cka-y merged commit b64cb0c into main Nov 12, 2025
3 checks passed
@cka-y cka-y deleted the feat/1017 branch November 12, 2025 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import datasets based on feed URL matching

3 participants