Skip to content

Replace pip3 download with curl for Python deps #1283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
tuhaihe merged 1 commit into from
Aug 6, 2025
Merged

Replace pip3 download with curl for Python deps #1283

tuhaihe merged 1 commit into from
Aug 6, 2025

Conversation

@tuhaihe
Copy link
Member

@tuhaihe tuhaihe commented Aug 1, 2025 •
edited
Loading

Replace pip3 download commands with curl for downloading psutil,
PyYAML and PyGreSQL packages in gpMgmt/bin/Makefile.

This change unifies the download mechanism to use curl consistently
for all Python dependencies, avoiding potential pip3 download
compatibility issues across different environments.

Changes:

  • Remove pip3 download commands for psutil, PyYAML and PyGreSQL
  • Use curl to download packages directly from PyPI
  • Maintain existing wheel and cython installation for PyYAML build
  • Keep the same package versions and download locations

Fixes #ISSUE_Number

What does this PR do?

Type of Change

  • Bug fix (non-breaking change)
  • New feature (non-breaking change)
  • Breaking change (fix or feature with breaking changes)
  • Documentation update

Breaking Changes

Test Plan

  • Unit tests added/updated
  • Integration tests added/updated
  • Passed make installcheck
  • Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:

User-facing changes:

Dependencies:

Checklist

Additional Context

CI Skip Instructions

@tuhaihe
Copy link
Member Author

tuhaihe commented Aug 1, 2025 •
edited
Loading

We will have a failure during the build process if enabling the configure option --with-pythonsrc-ext without pre-installing the python3-pip package in Ubuntu. python3-pip will be installed as a dependency by Python in Rocky Linux, but will not be installed in Ubuntu Docker Image.

The following is how to fix it:

  • Pre-install python3-pip as a required dependency;
  • Check if the python3-pip is installed; if not, will install this package to resolve the failure.

@tuhaihe tuhaihe force-pushed the fix-pip-ubuntu branch 5 times, most recently from 2f15c6b to 09851d4 Compare August 3, 2025 04:00
@tuhaihe tuhaihe changed the title Add automatic pip3 check based on package manager Replace pip3 download with curl for Python deps Aug 3, 2025
@tuhaihe
Copy link
Member Author

tuhaihe commented Aug 4, 2025 •
edited
Loading

Updates: will add the curl, pip3 check in the configure.ac.

Why?

  • curl and pip3 are not default-installed in the Ubuntu 20.04/22.04 Docker image. So for a better user experience, we need to check these dependencies before make install the Cloudberry. When users run the configure with the option --with-pythonsrc-ext, it will check the dependencies earlier.

Why use curl instead of pip3 download?

  • pip3 download has some compatibility issues between different Python versions (3.6 under Rocky8, 3.9 under Rocky9), especially for downloading pyyaml - pyaml 5.7 is a little old and has different behavior across Ubuntu and Rockylinux. So for a simple way, change the way of downloading these Python packages.

@tuhaihe tuhaihe force-pushed the fix-pip-ubuntu branch 2 times, most recently from a5b64d0 to 96f4788 Compare August 4, 2025 10:02
@tuhaihe
Copy link
Member Author

tuhaihe commented Aug 4, 2025

Already tested successfully on Ubuntu 20.04/22.04 + Rocky Linux 8/9.

@tuhaihe
Copy link
Member Author

tuhaihe commented Aug 4, 2025

Note

This commit will also be cherry-picked into the branch REL_2_STABLE.

@tuhaihe tuhaihe requested a review from gfphoenix78 August 4, 2025 10:15
jiaqizho
jiaqizho reviewed Aug 5, 2025
View reviewed changes
jiaqizho
jiaqizho approved these changes Aug 5, 2025
View reviewed changes
Copy link
Contributor

@jiaqizho jiaqizho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tuhaihe
Replace pip3 download with curl for Python deps
b260d9e 
Replace pip3 download commands with curl for downloading psutil,
PyYAML, PyGreSQL and mock packages in gpMgmt/bin/Makefile.

This change unifies the download mechanism to use curl consistently
for all Python dependencies, avoiding potential pip3 download
compatibility issues across different environments.

Changes:
- Remove pip3 download commands for psutil, PyYAML, PyGreSQL & mock
- Use curl to download packages directly from PyPI
- Maintain existing wheel and cython installation for PyYAML build
- Keep the same package versions and download locations

Also, add curl and pip3 dependency detection in configure.ac file for
better user experience. When `--with-pythonsrc-ext` option is enabled,
configure can check the required dependencies early.
xtangcode
xtangcode approved these changes Aug 6, 2025
View reviewed changes
Copy link

@xtangcode xtangcode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Configure updates to support deps.

@tuhaihe tuhaihe merged commit d5967fd into apache:main Aug 6, 2025
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xtangcode xtangcode xtangcode approved these changes

@jiaqizho jiaqizho jiaqizho approved these changes

@yjhjstz yjhjstz Awaiting requested review from yjhjstz

@gongxun0928 gongxun0928 Awaiting requested review from gongxun0928

@reshke reshke Awaiting requested review from reshke

@gfphoenix78 gfphoenix78 Awaiting requested review from gfphoenix78

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tuhaihe @xtangcode @jiaqizho