Skip to content

Conversation

@cka-y
Copy link
Contributor

@cka-y cka-y commented Nov 12, 2024

Summary:
This pull request introduces updates to enhance logging, authentication, and error handling in api/src/scripts/load_dataset_on_create.py and related files.

  • Replaced standard logging with a custom logger in load_dataset_on_create.py for more structured, consistent log messages.
  • Adjusted the logging level for downstream task triggers in populate_db_gtfs.py to ensure critical events are logged appropriately.
  • Ensured the message publishing process in load_dataset_on_create.py now blocks until confirmation is received, preventing potential issues with unacknowledged messages.

Issue Explanation:

Several issues were identified and addressed in this update:

  1. Lack of Confirmation for Message Publishing: Previously, we did not wait for confirmation of message publication to Pub/Sub topic, which led to silent failures. This issue went unnoticed after moving the database instance from DEV to QA, as logs did not indicate the failure caused by a permission error.
  2. Misconfigured Project ID and Pub/Sub Topic: After fixing the permission issue, I discovered that the project_id and pubsub_topic_name were incorrectly set. These variables were initialized at a high level in the script before the environment file was fully loaded, resulting in attempts to publish to the default (dev) topic instead of the QA topic or PROD topic.

Testing Instructions:

To test the updates:

  • Recently added feeds (e.g., stable_id > mdb-2155) should all have a corresponding dataset in QA and PROD environments.
  • For example, you can check mdb-2156 in QA here: QA Feed Example.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y cka-y changed the title Feat/762 fix: trigger downstream tasks when a new feed is added Nov 12, 2024
@cka-y cka-y marked this pull request as ready for review November 12, 2024 20:09
@cka-y cka-y requested a review from davidgamez November 13, 2024 16:20
Copy link
Member

@davidgamez davidgamez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cka-y cka-y merged commit d5c46c2 into main Nov 13, 2024
2 checks passed
@cka-y cka-y deleted the feat/762 branch November 13, 2024 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update pipeline to automatically generate dataset after merge, not wait till Monday and Thursday

3 participants