Skip to content

Avoid pip install --dry-run downloading full wheels #13482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Avoid pip install --dry-run downloading full wheels #13482

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
196f992
Avoid `pip install --dry-run` downloading full wheels
pelson Jul 15, 2025
a844f3b
Merge branch 'main' into feature/no-download-for-dry-run
notatallshaw Sep 13, 2025
fc2eb8d
Refine the news
pelson Sep 24, 2025
11d2a6f
Merge branch 'main' into feature/no-download-for-dry-run
notatallshaw Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions news/12603.feature.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
When PEP-658 metadata is available, full distribution download no longer occurs when using dry-run mode on install.
Copy link
Member

@notatallshaw notatallshaw Sep 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be updated slightly, in-particular all sdists are still downloaded (at least for now), either specifically call out wheels or change to something like:

When PEP-658 metadata is available for a distribution the full file is no longer downloaded

Or

When PEP-658 metadata is available, full wheel download no longer occurs

Further, also add that pip lock no longer downloads distributions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there's no update and no objections in the next few days I will update this news items myself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically pip has no problem fetching metadata for sdists already 😉. So this PR does make --dry-run avoid an sdist download/build 🎉 (this is a pretty substantial benefit as it means you don't need to build the sdist to figure out the dependencies... if only we had any repositories with sdist metadata served ala 658).

To prove this, I did the following:

mkdir -p ./delme3-project; cd ./delme3-project

cat << EOF > pyproject.toml
[project]
name = "delme3"
version = "0.1.0"
requires-python = ">=3.11"
EOF

python -m venv ./venv
source ./venv/bin/activate
python -m pip install build

python -m build --sdist .

mkdir -p dummy-repo/delme3

mv dist/delme3-0.1.0.tar.gz ./dummy-repo/delme3/

tar -xOf ./dummy-repo/delme3/delme3-0.1.0.tar.gz delme3-0.1.0/PKG-INFO > ./dummy-repo/delme3/delme3-0.1.0.tar.gz.metadata

cat << EOF > ./dummy-repo/delme3/index.html
<!DOCTYPE html><html><body>
<a href="/delme3/delme3-0.1.0.tar.gz" data-core-metadata="true">delme3-0.1.0.tar.gz</a><br/>
</body></html>
EOF

cd dummy-repo
python -m http.server

On the pip side, I ran:

$ python -m pip install delme3 -v --no-cache --dry-run --index-url http://localhost:8000/
Using pip 25.2.dev0 from /media/important/github/pypa/pip/src/pip (python 3.11)
Looking in indexes: http://localhost:8000/
Collecting delme3
  Obtaining dependency information for delme3 from http://localhost:8000/delme3/delme3-0.1.0.tar.gz.metadata
  Downloading http://localhost:8000/delme3/delme3-0.1.0.tar.gz.metadata (74 bytes)
Would install delme3-0.1.0

With only the following http.server output:

Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
127.0.0.1 - - [24/Sep/2025 12:43:06] "GET /delme3/ HTTP/1.1" 200 -
127.0.0.1 - - [24/Sep/2025 12:43:06] "GET /delme3/delme3-0.1.0.tar.gz.metadata HTTP/1.1" 200 -

(i.e. the sdist is never downloaded)

Whereas on master (f2b9231):

$ python -m pip install delme3 -v --no-cache --dry-run --index-url http://localhost:8000/
Using pip 25.3.dev0 from /media/important/github/pypa/pip/src/pip (python 3.11)
Looking in indexes: http://localhost:8000/
Collecting delme3
  Obtaining dependency information for delme3 from http://localhost:8000/delme3/delme3-0.1.0.tar.gz.metadata
  Downloading http://localhost:8000/delme3/delme3-0.1.0.tar.gz.metadata (74 bytes)
Downloading http://localhost:8000/delme3/delme3-0.1.0.tar.gz (646 bytes)
Running command pip subprocess to install build dependencies
Using pip 25.3.dev0 from /media/important/github/pypa/pip/src/pip (python 3.11)
Looking in indexes: http://localhost:8000/
ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none)
ERROR: No matching distribution found for setuptools>=40.8.0
Ensuring compatibility with Acc-Py
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /media/important/github/pypa/pip/venv/bin/python /media/important/github/pypa/pip/src/pip/__pip-runner__.py install --ignore-installed --no-user --prefix /tmp/pip-build-env-ohiuyo5e/overlay --no-warn-script-location --disable-pip-version-check --no-compile --target '' -v --no-binary :none: --only-binary :none: -i http://localhost:8000/ --trusted-host acc-py-repo.cern.ch -- 'setuptools>=40.8.0'
cwd: [inherit]
Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

With the http log:

$ python -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
127.0.0.1 - - [24/Sep/2025 12:43:56] "GET /delme3/ HTTP/1.1" 200 -
127.0.0.1 - - [24/Sep/2025 12:43:56] "GET /delme3/delme3-0.1.0.tar.gz.metadata HTTP/1.1" 200 -
127.0.0.1 - - [24/Sep/2025 12:43:56] "GET /delme3/delme3-0.1.0.tar.gz HTTP/1.1" 200 -
127.0.0.1 - - [24/Sep/2025 12:43:56] code 404, message File not found
127.0.0.1 - - [24/Sep/2025 12:43:56] "GET /setuptools/ HTTP/1.1" 404 -

I proceeded to try pip lock out on the above setup, but got a failure:

$ python -m pip lock delme3 -v --no-cache  --index-url http://localhost:8000/
Using pip 25.2.dev0 from /media/important/github/pypa/pip/src/pip (python 3.11)
WARNING: pip lock is currently an experimental command. It may be removed/changed in a future release without prior warning.
Looking in indexes: http://localhost:8000/
Collecting delme3
  Obtaining dependency information for delme3 from http://localhost:8000/delme3/delme3-0.1.0.tar.gz.metadata
  Downloading http://localhost:8000/delme3/delme3-0.1.0.tar.gz.metadata (74 bytes)
ERROR: Exception:
Traceback (most recent call last):
  File "/media/important/github/pypa/pip/src/pip/_internal/cli/base_command.py", line 107, in _run_wrapper
    status = _inner_run()
             ^^^^^^^^^^^^
  File "/media/important/github/pypa/pip/src/pip/_internal/cli/base_command.py", line 98, in _inner_run
    return self.run(options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/important/github/pypa/pip/src/pip/_internal/cli/req_command.py", line 70, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/important/github/pypa/pip/src/pip/_internal/commands/lock.py", line 162, in run
    pylock_toml = Pylock.from_install_requirements(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/important/github/pypa/pip/src/pip/_internal/models/pylock.py", line 179, in from_install_requirements
    packages=sorted(
             ^^^^^^^
  File "/media/important/github/pypa/pip/src/pip/_internal/models/pylock.py", line 181, in <genexpr>
    Package.from_install_requirement(ireq, base_dir)
  File "/media/important/github/pypa/pip/src/pip/_internal/models/pylock.py", line 138, in from_install_requirement
    raise NotImplementedError()
NotImplementedError

I think that is because my package repository didn't include hashes...

This was the changed line in question:

$ cat ./delme3/index.html | grep sha256
<a href="/delme3/delme3-0.1.0.tar.gz#sha256=713524ababddfe0944ccc8bef33535a13e5ddcb92156dddfc37fc838c8e91acc" data-core-metadata="true">delme3-0.1.0.tar.gz foo</a><br/>

When I added them it correctly generated the lock file without downloading:

$ python -m pip lock delme3 -v --no-cache  --index-url http://localhost:8000/
Using pip 25.2.dev0 from /media/important/github/pypa/pip/src/pip (python 3.11)
WARNING: pip lock is currently an experimental command. It may be removed/changed in a future release without prior warning.
Looking in indexes: http://localhost:8000/
Collecting delme3
  Obtaining dependency information for delme3 from http://localhost:8000/delme3/delme3-0.1.0.tar.gz.metadata
  Downloading http://localhost:8000/delme3/delme3-0.1.0.tar.gz.metadata (74 bytes)

$ python -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
127.0.0.1 - - [24/Sep/2025 12:56:44] "GET /delme3/ HTTP/1.1" 200 -
127.0.0.1 - - [24/Sep/2025 12:56:44] "GET /delme3/delme3-0.1.0.tar.gz.metadata HTTP/1.1" 200 -

🎉

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you like to mention anything about performance in the what's new? Whilst pip doesn't quite reach uv performance, it is still very reasonable when the cache is cold! The narrative that pip is much slower might be nice to counter here 😉

$ time uv pip install --dry-run torch --no-cache --index-url https://pypi.org/simple/
Using Python 3.11.11 environment at: venv
Resolved 25 packages in 371ms
Would download 24 packages
Would install 24 packages
 + filelock==3.19.1
 + fsspec==2025.9.0
 + jinja2==3.1.6
 + markupsafe==3.0.2
 + mpmath==1.3.0
 + networkx==3.5
 + nvidia-cublas-cu12==12.8.4.1
 + nvidia-cuda-cupti-cu12==12.8.90
 + nvidia-cuda-nvrtc-cu12==12.8.93
 + nvidia-cuda-runtime-cu12==12.8.90
 + nvidia-cudnn-cu12==9.10.2.21
 + nvidia-cufft-cu12==11.3.3.83
 + nvidia-cufile-cu12==1.13.1.3
 + nvidia-curand-cu12==10.3.9.90
 + nvidia-cusolver-cu12==11.7.3.90
 + nvidia-cusparse-cu12==12.5.8.93
 + nvidia-cusparselt-cu12==0.7.1
 + nvidia-nccl-cu12==2.27.3
 + nvidia-nvjitlink-cu12==12.8.93
 + nvidia-nvtx-cu12==12.8.90
 + sympy==1.14.0
 + torch==2.8.0
 + triton==3.4.0
 + typing-extensions==4.15.0

real	0m0.492s
user	0m0.132s
sys	0m0.039s

$ time python -m pip lock torch --no-cache --index-url https://pypi.org/simple/
WARNING: pip lock is currently an experimental command. It may be removed/changed in a future release without prior warning.
Looking in indexes: https://pypi.org/simple/
Collecting torch
  Downloading torch-2.8.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (30 kB)
Collecting filelock (from torch)
  Downloading filelock-3.19.1-py3-none-any.whl.metadata (2.1 kB)
Collecting typing-extensions>=4.10.0 (from torch)
  Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Collecting sympy>=1.13.3 (from torch)
  Downloading sympy-1.14.0-py3-none-any.whl.metadata (12 kB)
Collecting networkx (from torch)
  Downloading networkx-3.5-py3-none-any.whl.metadata (6.3 kB)
Collecting jinja2 (from torch)
  Downloading jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec (from torch)
  Downloading fsspec-2025.9.0-py3-none-any.whl.metadata (10 kB)
Collecting nvidia-cuda-nvrtc-cu12==12.8.93 (from torch)
  Downloading nvidia_cuda_nvrtc_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cuda-runtime-cu12==12.8.90 (from torch)
  Downloading nvidia_cuda_runtime_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cuda-cupti-cu12==12.8.90 (from torch)
  Downloading nvidia_cuda_cupti_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cudnn-cu12==9.10.2.21 (from torch)
  Downloading nvidia_cudnn_cu12-9.10.2.21-py3-none-manylinux_2_27_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-cublas-cu12==12.8.4.1 (from torch)
  Downloading nvidia_cublas_cu12-12.8.4.1-py3-none-manylinux_2_27_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cufft-cu12==11.3.3.83 (from torch)
  Downloading nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-curand-cu12==10.3.9.90 (from torch)
  Downloading nvidia_curand_cu12-10.3.9.90-py3-none-manylinux_2_27_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cusolver-cu12==11.7.3.90 (from torch)
  Downloading nvidia_cusolver_cu12-11.7.3.90-py3-none-manylinux_2_27_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-cusparse-cu12==12.5.8.93 (from torch)
  Downloading nvidia_cusparse_cu12-12.5.8.93-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-cusparselt-cu12==0.7.1 (from torch)
  Downloading nvidia_cusparselt_cu12-0.7.1-py3-none-manylinux2014_x86_64.whl.metadata (7.0 kB)
Collecting nvidia-nccl-cu12==2.27.3 (from torch)
  Downloading nvidia_nccl_cu12-2.27.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.0 kB)
Collecting nvidia-nvtx-cu12==12.8.90 (from torch)
  Downloading nvidia_nvtx_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvjitlink-cu12==12.8.93 (from torch)
  Downloading nvidia_nvjitlink_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl.metadata (1.7 kB)
Collecting nvidia-cufile-cu12==1.13.1.3 (from torch)
  Downloading nvidia_cufile_cu12-1.13.1.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)
Collecting triton==3.4.0 (from torch)
  Downloading triton-3.4.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (1.7 kB)
Collecting setuptools>=40.8.0 (from triton==3.4.0->torch)
  Downloading setuptools-80.9.0-py3-none-any.whl.metadata (6.6 kB)
Collecting mpmath<1.4,>=1.1.0 (from sympy>=1.13.3->torch)
  Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch)
  Downloading MarkupSafe-3.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)

real	0m1.555s
user	0m0.807s
sys	0m0.104s

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting about the sdists, I wasn't aware of that impact, I'll try and review later this week.

The Windows test failure looks unrelated. I'll try and make, or review, a PR to fix as soon as I can.

4 changes: 2 additions & 2 deletions src/pip/_internal/commands/download.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,15 +130,15 @@ def run(self, options: Values, args: list[str]) -> int:

requirement_set = resolver.resolve(reqs, check_supported_wheels=True)

preparer.prepare_linked_requirements_more(requirement_set.requirements.values())

downloaded: list[str] = []
for req in requirement_set.requirements.values():
if req.satisfied_by is None:
assert req.name is not None
preparer.save_linked_requirement(req)
downloaded.append(req.name)

preparer.prepare_linked_requirements_more(requirement_set.requirements.values())

if downloaded:
write_output("Successfully downloaded %s", " ".join(downloaded))

Expand Down
7 changes: 7 additions & 0 deletions src/pip/_internal/commands/install.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,13 @@ def run(self, options: Values, args: list[str]) -> int:
)
return SUCCESS

# If there is any more preparation to do for the actual installation, do
# so now. This includes actually downloading the files in the case that
# we have been using PEP-658 metadata so far.
preparer.prepare_linked_requirements_more(
requirement_set.requirements.values()
)

try:
pip_req = requirement_set.get_requirement("pip")
except KeyError:
Expand Down
4 changes: 2 additions & 2 deletions src/pip/_internal/commands/wheel.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,15 +145,15 @@ def run(self, options: Values, args: list[str]) -> int:

requirement_set = resolver.resolve(reqs, check_supported_wheels=True)

preparer.prepare_linked_requirements_more(requirement_set.requirements.values())

reqs_to_build: list[InstallRequirement] = []
for req in requirement_set.requirements.values():
if req.is_wheel:
preparer.save_linked_requirement(req)
else:
reqs_to_build.append(req)

preparer.prepare_linked_requirements_more(requirement_set.requirements.values())

# build wheels
build_successes, build_failures = build(
reqs_to_build,
Expand Down
6 changes: 6 additions & 0 deletions src/pip/_internal/operations/prepare.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,12 @@ def prepare_linked_requirement(
metadata_dist = self._fetch_metadata_only(req)
if metadata_dist is not None:
req.needs_more_preparation = True
req.set_dist(metadata_dist)
# Ensure download_info is available even in dry-run mode
if req.download_info is None:
req.download_info = direct_url_from_link(
req.link, req.source_dir
)
return metadata_dist

# None of the optimizations worked, fully prepare the requirement
Expand Down
11 changes: 10 additions & 1 deletion src/pip/_internal/req/req_install.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,10 @@ def __init__(
# details).
self.metadata_directory: str | None = None

# The cached metadata distribution that this requirement represents.
# See get_dist / set_dist.
self._distribution: BaseDistribution | None = None

# The static build requirements (from pyproject.toml)
self.pyproject_requires: list[str] | None = None

Expand Down Expand Up @@ -604,8 +608,13 @@ def metadata(self) -> Any:

return self._metadata

def set_dist(self, distribution: BaseDistribution) -> None:
self._distribution = distribution

def get_dist(self) -> BaseDistribution:
if self.metadata_directory:
if self._distribution is not None:
return self._distribution
elif self.metadata_directory:
return get_directory_distribution(self.metadata_directory)
elif self.local_file_path and self.is_wheel:
assert self.req is not None
Expand Down
5 changes: 0 additions & 5 deletions src/pip/_internal/resolution/resolvelib/resolver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,11 +180,6 @@ def resolve(

req_set.add_named_requirement(ireq)

reqs = req_set.all_requirements
self.factory.preparer.prepare_linked_requirements_more(reqs)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the key change - the resolver no longer triggers additional preparation. Instead, we move this responsibility to consumers of the resolve result.

Technically, I could have made the preparer aware of the fact that we don't want to download anything, but the preparer is already a grab-bag of flags, and it felt cleaner to do it this way.

for req in reqs:
req.prepared = True
req.needs_more_preparation = False
return req_set

def get_installation_order(
Expand Down
Loading