feat(ingestion/deps): add upper bounds to dependency versions in setup.py #15813
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
kyungsoo-datahub
commented
Jan 6, 2026
kyungsoo-datahub
commented
- Add upper bounds to all dependencies using next major version (e.g., <3.0.0)
- For 0.x packages, use tight bounds (<=current) or <1.0.0 where no 1.x exists
- Add comments documenting automatic dependency chains:
- ex> numpy<2 -> feast<=0.47 -> pyarrow<18.1 (resolved automatically)
- Keep explicit constraints for deliberate choices (sqlalchemy<2, numpy<2)
Contributor
|
Linear: ING-1334 |
github-actions
bot
added
the
ingestion
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
✅ Meticulous spotted 0 visual differences across 967 screens tested: view results. Meticulous evaluated ~8 hours of user flows against your PR. Expected differences? Click here. Last updated for commit 974146c. This comment will update as new commits are pushed. |
datahub-cyborg
bot
added
the
needs-review
Bundle ReportBundle size has no change ✅ |
askumar27
reviewed
Jan 7, 2026
metadata-ingestion/setup.py
Outdated
| "click>=7.1.2, !=8.2.0", | ||
| "click>=7.1.2,!=8.2.0,<9.0.0", | ||
| "click-default-group", | ||
| "PyYAML", |
Contributor
There was a problem hiding this comment.
[MINOR] There are still unbounded packages, plan to upper-bound all the packages right ?
Contributor
Author
There was a problem hiding this comment.
Done. Thanks.
askumar27
reviewed
Jan 7, 2026
metadata-ingestion/setup.py
Outdated
| # `aws_access_key_id`, `aws_secret_access_key`, and `aws_session_token` were deprecated and removed in version | ||
| # 0.8.0. | ||
| "pyiceberg[glue,hive,dynamodb,snappy,hive,s3fs,adlfs,pyarrow,zstandard]>=0.8.0", | ||
| "pyiceberg[glue,hive,dynamodb,snappy,hive,s3fs,adlfs,pyarrow,zstandard]>=0.8.0,<=0.10.0", |
Contributor
There was a problem hiding this comment.
pyiceberg is still actively developed - should we strict pin to the latest version ? same with databricks-sdk
askumar27
reviewed
Jan 7, 2026
Contributor
askumar27
left a comment
askumar27
left a comment
There was a problem hiding this comment.
Question: Do we need the lower bounds - would upper bound just suffice?
datahub-cyborg
bot
added
pending-submitter-response
…p.py - Add upper bounds to all dependencies using next major version (e.g., <3.0.0) - For 0.x packages, use tight bounds (<=current) or <1.0.0 where no 1.x exists - Add comments documenting automatic dependency chains: - numpy<2 -> feast<=0.47 -> pyarrow<18.1 (resolved automatically) - sqlalchemy<2 -> sqlalchemy-pytds<1.0, sqlalchemy-hana<4.0 (resolved automatically) - protobuf<5 -> grpcio-tools<1.63 (resolved automatically) - urllib3<2 -> tableauserverclient<0.27 (resolved automatically) - Keep explicit constraints for deliberate choices (sqlalchemy<2, numpy<2)
Add version upper bounds to packages that previously had no constraints.
Strategy:
- For 1.x+ packages: <next_major.0.0
- For 0.x packages: <0.{minor+1}.0
kyungsoo-datahub
force-pushed
the
feat/metadata-ingestion-pin-versions
branch
from
9506d43 to
321530f
Compare
askumar27
reviewed
Jan 15, 2026
metadata-ingestion/setup.py
Outdated
| "excel": { | ||
| "openpyxl>=3.1.5", | ||
| "openpyxl>=3.1.5,<4.0.0", | ||
| "pandas", |
Contributor
There was a problem hiding this comment.
Let add a upper bound here as well
Contributor
Author
There was a problem hiding this comment.
Added. Thanks.
datahub-cyborg
bot
added
pending-submitter-response
…kages Add version upper bounds to packages that were missed by the automated script (packages with no existing version specifiers): - google-cloud-bigquery<4.0.0 - google-cloud-resource-manager<2.0.0 - google-cloud-dataplex<3.0.0 - pandas<3.0.0 (excel plugin)
datahub-cyborg
bot
added
needs-review
askumar27
approved these changes
Jan 15, 2026
Contributor
askumar27
left a comment
askumar27
left a comment
There was a problem hiding this comment.
Notes from review meetings:
There are 4 types of py package bounding cases we have:
- With strict upper and lower bounds - these are untouched
- With lower bounds only (mainly the case) - upper bounds are added to the current latest MAJOR version - ensuring not to cause breaking changes
- With no bounds - upper bounds are added to the current latest MAJOR version - ensuring not to cause breaking changes
- With upper bounds only - these are untouched
datahub-cyborg
bot
added
pending-submitter-merge
and removed
needs-review
…ow 3.1.x compatibility
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.