Skip to content

Commit ee0a9eb

Browse files
edmorleyedmorley
authored
Change the editable VCS directory location for pip and Pipenv (#1753)
When a dependency from a version control system (eg Git) is installed in editable mode, the package manager has to clone the repository somewhere long-lived, that is then referenced by the `.pth` file added to `site-packages`. (When installed in normal non-editable mode, the repo checkouts are instead saved to a temporary directory and deleted after the package is installed.) Until now, the buildpack configured pip and Pipenv to store these repos at `/app/.heroku/src/`, then later copied those files into the build directory and build cache. However, this approach isn't needed with the `.pth` rewriting we have now. In addition, the existing implementation didn't actually restore the cached `src/` directory, so the repos stored in the cache were never re-used on subsequent builds anyway. Now, pip and pipenv are configured to store the repositories at `<BUILD_DIR>/.heroku/python/src/`, which means: - The behaviour now matches that when using Poetry. - The repos get cached/restored/invalidated for free, as part of the existing handling of the `.heroku/python/` directory, and we avoid the additional directory copy from `/app` to `/tmp`, both of which help reduce build times. GUS-W-17863838.
1 parent 51193fa commit ee0a9eb

File tree

8 files changed

+48
-87
lines changed

8 files changed

+48
-87
lines changed

‎CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
## [Unreleased]
44

5+
- Changed the location of repositories for editable VCS dependencies when using pip and Pipenv, to improve build performance and match the behaviour when using Poetry. ([#1753](https://github.com/heroku/heroku-buildpack-python/pull/1753))
56

67
## [v277] - 2025-02-17
78

‎bin/compile

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -178,8 +178,6 @@ cache::restore "${BUILD_DIR}" "${CACHE_DIR}" "${STACK}" "${cached_python_full_ve
178178

179179
# The directory for the .profile.d scripts.
180180
mkdir -p "$(dirname "$PROFILE_PATH")"
181-
# The directory for editable VCS dependencies.
182-
mkdir -p /app/.heroku/src
183181

184182
# On Heroku CI, builds happen in `/app`. Otherwise, on the Heroku platform,
185183
# they occur in a temp directory. Because Python is not portable, we must create
@@ -191,7 +189,6 @@ if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then
191189
# python expects to reside in /app, so set up symlinks
192190
# we will not remove these later so subsequent buildpacks can still invoke it
193191
ln -nsf "$BUILD_DIR/.heroku/python" /app/.heroku/python
194-
# Note: .heroku/src is copied in later.
195192
fi
196193

197194
python::install "${BUILD_DIR}" "${STACK}" "${python_full_version}" "${python_major_version}" "${python_version_origin}"
@@ -250,15 +247,6 @@ nltk_downloader_start_time=$(nowms)
250247
sub_env "${BUILDPACK_DIR}/bin/steps/nltk"
251248
meta_time "nltk_downloader_duration" "${nltk_downloader_start_time}"
252249

253-
# Support for editable installations.
254-
# In CI, $BUILD_DIR is /app.
255-
# Realpath is used to support use-cases where one of the locations is a symlink to the other.
256-
# shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value.
257-
if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then
258-
rm -rf "$BUILD_DIR/.heroku/src"
259-
deep-cp /app/.heroku/src "$BUILD_DIR/.heroku/src"
260-
fi
261-
262250
# Django collectstatic support.
263251
# The buildpack automatically runs collectstatic for Django applications.
264252
collectstatic_start_time=$(nowms)

‎bin/utils

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,6 @@ shopt -s nullglob
88

99
source "${BUILDPACK_DIR:?}/vendor/buildpack-stdlib_v8.sh"
1010

11-
# Does some serious copying.
12-
deep-cp() {
13-
declare source="$1" target="$2"
14-
15-
mkdir -p "$target"
16-
17-
# cp doesn't like being called without source params,
18-
# so make sure they expand to something first.
19-
# subshell to avoid surprising caller with shopts.
20-
(
21-
shopt -s nullglob dotglob
22-
set -- "$source"/!(tmp|.|..)
23-
[[ $# == 0 ]] || cp -a "$@" "$target"
24-
)
25-
}
26-
2711
# Measure the size of the Python installation.
2812
measure-size() {
2913
{ du -s .heroku/python 2>/dev/null || echo 0; } | awk '{print $1}'

‎lib/cache.sh

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,6 @@ function cache::restore() {
129129
"${cache_dir}/.heroku/python-poetry" \
130130
"${cache_dir}/.heroku/python-stack" \
131131
"${cache_dir}/.heroku/python-version" \
132-
"${cache_dir}/.heroku/src" \
133132
"${cache_dir}/.heroku/requirements.txt"
134133

135134
meta_set "cache_status" "discarded"
@@ -143,17 +142,13 @@ function cache::restore() {
143142
# TODO: Compare the performance of moving the directory vs copying files.
144143
cp -R "${cache_dir}/.heroku/python" "${build_dir}/.heroku/" &>/dev/null || true
145144

146-
# Editable VCS code repositories, used by pip/pipenv.
147-
if [[ -d "${cache_dir}/.heroku/src" ]]; then
148-
cp -R "${cache_dir}/.heroku/src" "${build_dir}/.heroku/" &>/dev/null || true
149-
fi
150-
151145
meta_set "cache_status" "reused"
152146
fi
153147

154148
# Remove any legacy cache contents written by older buildpack versions.
155149
rm -rf \
156150
"${cache_dir}/.heroku/python-sqlite3-version" \
151+
"${cache_dir}/.heroku/src" \
157152
"${cache_dir}/.heroku/vendor"
158153

159154
meta_time "cache_restore_duration" "${cache_restore_start_time}"
@@ -175,13 +170,6 @@ function cache::save() {
175170
rm -rf "${cache_dir}/.heroku/python"
176171
cp -R "${build_dir}/.heroku/python" "${cache_dir}/.heroku/"
177172

178-
# Editable VCS code repositories, used by pip/pipenv.
179-
rm -rf "${cache_dir}/.heroku/src"
180-
if [[ -d "${build_dir}/.heroku/src" ]]; then
181-
# TODO: Investigate why errors are ignored and ideally stop doing so.
182-
cp -R "${build_dir}/.heroku/src" "${cache_dir}/.heroku/" &>/dev/null || true
183-
fi
184-
185173
# Metadata used by subsequent builds to determine whether the cache can be reused.
186174
# These are written/consumed via separate files and not the metadata store for compatibility
187175
# with buildpack versions prior to the metadata store existing (which was only added in v252).

‎lib/pip.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ function pip::install_dependencies() {
110110
--no-cache-dir \
111111
--no-input \
112112
--progress-bar off \
113-
--src='/app/.heroku/src' \
113+
--src='/app/.heroku/python/src' \
114114
|& tee "${WARNINGS_LOG:?}" \
115115
|& sed --unbuffered --expression '/Requirement already satisfied/d' \
116116
|& output::indent

‎lib/pipenv.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ function pipenv::install_dependencies() {
8181
# shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled.
8282
if ! {
8383
"${pipenv_install_command[@]}" \
84-
--extra-pip-args='--src=/app/.heroku/src' \
84+
--extra-pip-args='--src=/app/.heroku/python/src' \
8585
--system \
8686
|& tee "${WARNINGS_LOG:?}" \
8787
|& output::indent

‎spec/hatchet/pip_spec.rb

Lines changed: 22 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -100,21 +100,21 @@
100100
app.deploy do |app|
101101
expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX))
102102
remote: -----> Running bin/post_compile hook
103-
remote: easy-install.pth:/app/.heroku/src/gunicorn
104-
remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py
105-
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
106-
remote: gunicorn.egg-link:/app/.heroku/src/gunicorn
107-
remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py
103+
remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn
104+
remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py
105+
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
106+
remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn
107+
remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py
108108
remote:
109109
remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
110110
remote: Running entrypoint for the setup.py-based local package: Hello setup.py!
111111
remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\)
112112
remote: -----> Inline app detected
113-
remote: easy-install.pth:/app/.heroku/src/gunicorn
114-
remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py
115-
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
116-
remote: gunicorn.egg-link:/app/.heroku/src/gunicorn
117-
remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py
113+
remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn
114+
remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py
115+
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
116+
remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn
117+
remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py
118118
remote:
119119
remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
120120
remote: Running entrypoint for the setup.py-based local package: Hello setup.py!
@@ -123,10 +123,10 @@
123123

124124
# Test rewritten paths work at runtime.
125125
expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT)
126-
easy-install.pth:/app/.heroku/src/gunicorn
126+
easy-install.pth:/app/.heroku/python/src/gunicorn
127127
easy-install.pth:/app/packages/local_package_setup_py
128128
__editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
129-
gunicorn.egg-link:/app/.heroku/src/gunicorn
129+
gunicorn.egg-link:/app/.heroku/python/src/gunicorn
130130
local-package-setup-py.egg-link:/app/packages/local_package_setup_py
131131
132132
Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
@@ -139,21 +139,21 @@
139139
app.push!
140140
expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX))
141141
remote: -----> Running bin/post_compile hook
142-
remote: easy-install.pth:/app/.heroku/src/gunicorn
143-
remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py
144-
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
145-
remote: gunicorn.egg-link:/app/.heroku/src/gunicorn
146-
remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py
142+
remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn
143+
remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py
144+
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
145+
remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn
146+
remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py
147147
remote:
148148
remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
149149
remote: Running entrypoint for the setup.py-based local package: Hello setup.py!
150150
remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\)
151151
remote: -----> Inline app detected
152-
remote: easy-install.pth:/app/.heroku/src/gunicorn
153-
remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py
154-
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
155-
remote: gunicorn.egg-link:/app/.heroku/src/gunicorn
156-
remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py
152+
remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn
153+
remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py
154+
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
155+
remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn
156+
remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py
157157
remote:
158158
remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
159159
remote: Running entrypoint for the setup.py-based local package: Hello setup.py!

‎spec/hatchet/pipenv_spec.rb

Lines changed: 22 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -356,21 +356,21 @@
356356
app.deploy do |app|
357357
expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX))
358358
remote: -----> Running bin/post_compile hook
359-
remote: easy-install.pth:/app/.heroku/src/gunicorn
360-
remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py
361-
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
362-
remote: gunicorn.egg-link:/app/.heroku/src/gunicorn
363-
remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py
359+
remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn
360+
remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py
361+
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
362+
remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn
363+
remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py
364364
remote:
365365
remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
366366
remote: Running entrypoint for the setup.py-based local package: Hello setup.py!
367367
remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\)
368368
remote: -----> Inline app detected
369-
remote: easy-install.pth:/app/.heroku/src/gunicorn
370-
remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py
371-
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
372-
remote: gunicorn.egg-link:/app/.heroku/src/gunicorn
373-
remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py
369+
remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn
370+
remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py
371+
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
372+
remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn
373+
remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py
374374
remote:
375375
remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
376376
remote: Running entrypoint for the setup.py-based local package: Hello setup.py!
@@ -379,10 +379,10 @@
379379

380380
# Test rewritten paths work at runtime.
381381
expect(app.run('bin/test-entrypoints.sh')).to include(<<~OUTPUT)
382-
easy-install.pth:/app/.heroku/src/gunicorn
382+
easy-install.pth:/app/.heroku/python/src/gunicorn
383383
easy-install.pth:/app/packages/local_package_setup_py
384384
__editable___local_package_pyproject_toml_0_0_1_finder.py:/app/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
385-
gunicorn.egg-link:/app/.heroku/src/gunicorn
385+
gunicorn.egg-link:/app/.heroku/python/src/gunicorn
386386
local-package-setup-py.egg-link:/app/packages/local_package_setup_py
387387
388388
Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
@@ -395,21 +395,21 @@
395395
app.push!
396396
expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX))
397397
remote: -----> Running bin/post_compile hook
398-
remote: easy-install.pth:/app/.heroku/src/gunicorn
399-
remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py
400-
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
401-
remote: gunicorn.egg-link:/app/.heroku/src/gunicorn
402-
remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py
398+
remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn
399+
remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py
400+
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
401+
remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn
402+
remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py
403403
remote:
404404
remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
405405
remote: Running entrypoint for the setup.py-based local package: Hello setup.py!
406406
remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\)
407407
remote: -----> Inline app detected
408-
remote: easy-install.pth:/app/.heroku/src/gunicorn
409-
remote: easy-install.pth:/tmp/build_.*/packages/local_package_setup_py
410-
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.*/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
411-
remote: gunicorn.egg-link:/app/.heroku/src/gunicorn
412-
remote: local-package-setup-py.egg-link:/tmp/build_.*/packages/local_package_setup_py
408+
remote: easy-install.pth:/tmp/build_.+/.heroku/python/src/gunicorn
409+
remote: easy-install.pth:/tmp/build_.+/packages/local_package_setup_py
410+
remote: __editable___local_package_pyproject_toml_0_0_1_finder.py:/tmp/build_.+/packages/local_package_pyproject_toml/local_package_pyproject_toml'}
411+
remote: gunicorn.egg-link:/tmp/build_.+/.heroku/python/src/gunicorn
412+
remote: local-package-setup-py.egg-link:/tmp/build_.+/packages/local_package_setup_py
413413
remote:
414414
remote: Running entrypoint for the pyproject.toml-based local package: Hello pyproject.toml!
415415
remote: Running entrypoint for the setup.py-based local package: Hello setup.py!

0 commit comments

Comments
 (0)