Fix/dependency resolution robustness #3608
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The variable 's' (sqlalchemy) was undefined in start_tasks.py, causing a NameError in cleanup tasks. Imported 'sqlalchemy as s' to fix. Signed-off-by: atheendre130505 <atheendreramesh@gmail.com>
The process_libyear_dependency_metrics task was failing when encountering dependencies that couldn't be resolved on PyPI or NPM (e.g., the 'python' runtime requirement or versions like '2.9.0.0' not found in release history). This fix adds validation checks in get_deps_libyear_data: - Skips dependencies if the package cannot be found on the registry. - Skips dependencies if the current version requirement cannot be resolved. - Skips dependencies if release date information is missing for the current or latest version. These encountered issues are logged as warnings instead of causing task failures, ensuring the overall collection process remains robust even when some metadata is unavailable. Signed-off-by: atheendre130505 <atheendreramesh@gmail.com>
MoralCode
reviewed
Jan 20, 2026
Contributor
MoralCode
left a comment
MoralCode
left a comment
There was a problem hiding this comment.
Overall I like the shift in patterns used in this PR! i.e. using continue rather than storing the failure and repeatedly checking it later.
Will review in more depth after CHAOSScon (i.e. in early-mid feb).
In the meantime feel free to join the CHAOSS Slack in the #wg-augur-8knot channel if you'd like to interact with other maintainers/contributors
sgoggins
added
add-feature
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR enhances the robustness of the process_libyear_dependency_metrics task by implementing strict validation for dependency metadata.
It adds logic to verify that packages and their specific versions exist in external registries (PyPI/NPM) and contain valid release dates before attempting calculations.
Instead of failing the entire background job when encountering non-standard dependencies (like the python runtime requirement or malformed version strings), the system now logs a warning and skips the problematic entry.
This PR fixes #3606
PR also references previous pr #3607.
Notes for Reviewers
The fix primarily targets
augur/tasks/git/dependency_libyear_tasks/libyear_util/util.py
.
The main challenge was that modern lockfiles (Poetry/Pipfile) often list environment requirements that are not standard PyPI packages. Previously, looking these up would cause the task to fail with a KeyError or return invalid default versions.
I've replaced logger.error calls with logger.warning for these scenarios to prevent task crashes while still providing visibility into why a dependency was skipped.
Manual verification was performed using a mock-registry script to ensure the skipping logic correctly filters out invalid data while still processing valid dependencies (like requests).
Signed commits
atheendre130505-atheendreramesh@gmail.com
Yes, I signed my commits.