Skip to content

Try uv second attempt #2161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kshtsk wants to merge 21 commits into
base: main
Choose a base branch
from
Open

Try uv second attempt #2161

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4313217
Replace pip with uv
zmc Apr 10, 2025
f3267b9
Drop pip-tools
zmc Apr 10, 2025
fd8a974
tox: Use tox-uv
zmc Apr 10, 2025
f3180ec
Update gevent
zmc Apr 10, 2025
bed71a7
Simplify bootstrap script
zmc Apr 10, 2025
2cdc2a2
containers: Install pipx
zmc Apr 10, 2025
6dbb16a
Use uv's venv path
zmc Apr 10, 2025
d245242
Update install doc to cover uv
zmc Apr 10, 2025
7459c11
bootstrap: Fix shellcheck warnings
zmc Sep 8, 2025
9548c6f
Drop flake8 in favor of ruff
zmc Sep 9, 2025
2725da0
Fix errors reported by ruff
zmc Sep 9, 2025
b6b0427
tox: Use uv to run pytest and sphinx
zmc Sep 9, 2025
90c2101
pyproject.toml: use setuptools-scm for uv
May 19, 2025
a95337b
teuthology/Dockerfile: add env setuptools-scm version
May 28, 2025
469acdd
bootstrap: fix suse family deps
kshtsk Sep 9, 2025
d82c055
bootstrap: Drop lsb_release support
zmc Sep 16, 2025
9af0449
pyproject: add missing oauthlib dep for maas
Mar 25, 2026
a596266
uv.lock: update deps for oauthlib
Mar 25, 2026
e75d042
bootstrap: use "dpkg -s" again
kshtsk Apr 8, 2026
06e3129
bootstrap: make zypper call non-interactive
kshtsk Apr 8, 2026
8136b00
teuthology: load importlib after monkey patching
kshtsk Apr 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,15 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
- name: Install tox
run: pip install tox
- name: Run flake8
run: tox -e flake8
- name: Run unit tests
run: tox -e py3
- name: Install pipx
run: pip install pipx
- name: Install uv
run: pipx install uv
- name: Run pipx ensurepath
run: pipx ensurepath
- name: Lint
run: uv tool run ruff check
- name: Run tests
run: uv run --all-extras pytest teuthology scripts
- name: Run docs build
run: tox -e docs
run: uv run tox -e docs
155 changes: 42 additions & 113 deletions bootstrap
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,109 +1,74 @@
#!/bin/bash
set -e
if [ $# -eq 0 ]; then
install=false
install=false
else
if [ "$1" = "install" ]; then
install=true
else
echo "Invalid command, supported commands are: 'install'"
exit 1
fi
fi

if [[ "$PYTHON" =~ "python2" ]]; then
echo "python2 is not supported." >&2
exit 1
fi

# Use the newest version we find
if [ -z "$PYTHON" ]; then
for i in 12 11 10; do
command -v "python3.$i" && PYTHON="python3.$i" &>/dev/null && break
done
fi
if [ -z "$PYTHON" ]; then
# This would be bizarre, but I suppose possible
PYTHON=${PYTHON:-"python3"}
if [ "$1" = "install" ]; then
install=true
else
echo "Invalid command, supported commands are: 'install'"
exit 1
fi
fi
echo "Using python: $PYTHON"

case "$(uname -s)" in
Linux)
if command -v lsb_release; then
OS=$(lsb_release --id --short)
else
. /etc/os-release
OS=$(echo $NAME | tr -d ' ')
fi
# rpm/dnf is the default, to reduce repetition in the case statement
has_pkg="rpm -q --whatprovides"
install_pkg="sudo dnf install -y"
. /etc/os-release
OS="${NAME// /}"
case "$OS" in
Ubuntu|Debian|LinuxMint)
deps=(qemu-utils python3-dev libssl-dev python3-pip python3-wheel python3-venv libev-dev libvirt-dev libffi-dev libyaml-dev build-essential jq curl)
has_pkg="dpkg -s"
deps=(qemu-utils python3-dev libssl-dev libev-dev libvirt-dev libffi-dev libyaml-dev pipx build-essential jq curl)
has_pkg="dpkg -C"
Copy link
Copy Markdown
Contributor

@deepssin deepssin Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dpkg -C is ->
-C|--audit [...] Check for broken package(s)
in practice dpkg -C still exits 0 even when the package is missing (it prints “not installed” to stderr). I verified this in the environment: both an installed and a bogus package name lead the bootstrap loop to treat the dependency as present.

Copy link
Copy Markdown
Contributor Author

@kshtsk kshtsk Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zmc what did you have in mind when changing to -s to -C?

Copy link
Copy Markdown
Member

@zmc zmc Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I just wanted a quieter -s, but I will amend this to use -s and drop stdout/err. Good catch!!

Copy link
Copy Markdown
Member

@zmc zmc Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or, @kshtsk, if you'd rather just push a small fix to this branch that works as well.

install_pkg="sudo apt install -y"
;;
RedHatEnterpriseWorkstation|RedHatEnterpriseServer|RedHatEnterprise|CentOS)
deps=(python39-pip python39-devel mariadb-devel libev-devel libvirt-devel libffi-devel)
;;
CentOSStream)
PYTHON=python3.12
deps=($PYTHON-pip $PYTHON-devel)
;;
AlmaLinux|RockyLinux)
PYTHON=python3.12
deps=($PYTHON-pip $PYTHON-devel libev-devel libvirt-devel libffi-devel)
;;
Fedora|FedoraLinux)
deps=($PYTHON-pip $PYTHON-devel libev-devel libvirt-devel libffi-devel)
RedHatEnterpriseWorkstation|RedHatEnterpriseServer|RedHatEnterprise|CentOS|CentOSStream|AlmaLinux|RockyLinux|Fedora|FedoraLinux)
# Use the newest version we find
if [ -z "$PYTHON" ]; then
for i in 13 12 11 10; do
command -v "python3.$i" > /dev/null && PYTHON="python3.$i" && break
done
fi
if [ -z "$PYTHON" ]; then
# This would be bizarre, but I suppose possible
PYTHON=${PYTHON:-"python3"}
fi
# rpm/dnf is the default, to reduce repetition in the case statement
has_pkg="rpm -q --whatprovides"
install_pkg="sudo dnf install -y"
deps=(libev-devel libffi-devel libvirt-devel "$PYTHON-devel" pipx)
;;
"openSUSE project"|"SUSE LINUX"|"openSUSE"|"openSUSELeap"|"openSUSETumbleweed")
PYTHON=python3.12
deps=(python312-pip python312-devel python312 libev-devel libvirt-devel libffi-devel)
PYTHON=${PYTHON:-"python3"}
deps=("$PYTHON" "$PYTHON-devel" "$PYTHON-pipx" libev-devel libffi-devel libvirt-devel)
has_pkg="rpm -q --whatprovides"
install_pkg="sudo zypper install"
Copy link
Copy Markdown
Contributor

@deepssin deepssin Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This being non-interactive , I think bootstrap can get stuck here.

Copy link
Copy Markdown
Contributor Author

@kshtsk kshtsk Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand what is the problem here. What is your suggestion change. By design bootstrap should stop immediately when it finds some of the system dependencies are missing.
In this case it just reports users so they can install missing dependencies manually and try again. The "sudo zipper install" is interactive and needs "-y" to auto accept all the packages. Since it is a manual step, guess it is a user choice how to proceed.
Could you please elaborate.
Btw, this works for me on the latest tumbleweed.

Copy link
Copy Markdown
Contributor

@deepssin deepssin Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two paths: plain ./bootstrap already exits when deps are missing—that’s fine. The wrinkle is ./bootstrap install, which runs the package manager for you. On SUSE that’s zypper install without -y, so in non-interactive contexts (like cloud-init, CI or any other automations) it can block on prompts, unlike apt/dnf which already pass -y. Suggest adding non-interactive only for that install path so scripted runs don’t hang; interactive users can still install manually after a plain ./bootstrap.

Copy link
Copy Markdown
Contributor Author

@kshtsk kshtsk Apr 7, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, thanks, I'll add -y then.

;;
esac
;;

Darwin)
deps="python libvirt libev libffi"
deps=(python libvirt libev libffi pipx uv)
has_pkg="brew list"
install_pkg="brew install"
;;
esac
for package in ${deps[@]}; do
if ! $has_pkg $package &>/dev/null; then
for package in "${deps[@]}"; do
if ! $has_pkg "$package" &>/dev/null; then
# add a space after old values
missing="${missing:+$missing }$package"
echo missing=${missing}
fi
done
if [ -n "$missing" ]; then
echo "$0: missing required packages:" 1>&2
echo "$missing"
if [ "$install" = true ]; then
echo "Installing missing packages..."
$install_pkg $missing
$install_pkg "$missing"
else
echo "Please install missing packages or run './bootstrap install'"
echo "$install_pkg $missing"
exit 1
fi
fi

PYTHON_BIN=$(command -v $PYTHON)
if [ -z $PYTHON_BIN -o ! -e $PYTHON_BIN -o ! -x $PYTHON_BIN ]; then
echo "Cannot find $PYTHON!"
exit 1
fi
PYTHON_VER_OUT=$($PYTHON_BIN --version)

VENV=${VENV:-"./virtualenv"}
# If the venv was set to use system site-packages, fix that
if [ -f "$VENV/pyvenv.cfg" ]; then
sed -i'' -e 's/\(include-system-site-packages\s*=\s*\)true/\1false/g' $VENV/pyvenv.cfg
fi

# Attempt to force a UTF-8 locale without forcing English
Expand All @@ -112,50 +77,14 @@ LC_ALL=${LC_ALL:=C}
export LANG="${LANG/.*/}.utf-8"
export LC_ALL="${LC_ALL/.*/}.utf-8"

if [ -z "$NO_CLOBBER" ] && \
[ ! -e "$VENV/bin/pip" -o ! -e "$VENV/bin/$PYTHON" ] || \
[ "${PYTHON_VER_OUT}" != "$($VENV/bin/$PYTHON --version)" ] \
; then
echo "Deleting existing virtual environment"
rm -rf virtualenv
fi

if [ -z "$NO_CLOBBER" ] || [ ! -e $VENV ]; then
echo "Creating new venv at $VENV"
$PYTHON_BIN -m venv $VENV
fi

PY_MAJOR=$($VENV/bin/python -c "import sys; print(sys.version_info[0])")
PY_MINOR=$($VENV/bin/python -c "import sys; print(sys.version_info[1])")

# Python version check
if [[ "$PY_MAJOR" -ne 3 || "$PY_MINOR" -lt 10 ]]; then
echo "Python version should be 3.10 or higher, found $PY_MAJOR.$PY_MINOR"
exit 1
fi

$VENV/bin/pip install packaging

if [ -f "$VENV/bin/ansible" ]; then
uninstall_ansible=$($VENV/bin/python3 -c "import ansible; from packaging.version import parse; print(parse(ansible.__version__) < parse('2.10.0'))")
if [ "$uninstall_ansible" = "True" ]; then
$VENV/bin/pip uninstall -y ansible
fi
fi

# First, upgrade pip
$VENV/bin/pip install --upgrade pip

# See https://github.com/pypa/pip/issues/8559
$VENV/bin/pip install -r requirements.txt --use-pep517

# By default, install teuthology in editable mode
$VENV/bin/pip install ${PIP_INSTALL_FLAGS:---editable '.[test]'}

# Check to make sure requirements are met
$VENV/bin/pip check
[ -z "$NO_CLOBBER" ] && rm -rf virtualenv
command -v uv > /dev/null || pipx install uv
command -v uv > /dev/null || pipx ensurepath
PATH=$PATH:$HOME/.local/bin
# Create the venv if it does not exist, and install teuthology and dependencies
uv sync --frozen --all-extras
# To avoid crashing older dispatchers
ln -sf .venv virtualenv

# Install ansible collections
$VENV/bin/ansible-galaxy install -r requirements.yml

echo "Bootstrap completed successfully!!!"
uv run ansible-galaxy install -r requirements.yml
8 changes: 4 additions & 4 deletions containers/teuthology-dev/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
FROM ubuntu:noble
ENV DEBIAN_FRONTEND=noninteractive
ENV SETUPTOOLS_SCM_PRETEND_VERSION_FOR_TEUTHOLOGY=0.0.0
ENV LANG=C.UTF-8
RUN apt-get update && \
apt-get install -y \
Expand All @@ -8,8 +9,7 @@
python3-dev \
libssl-dev \
ipmitool \
python3-pip \
python3-venv \
pipx \
vim \
jq \
curl \
Expand All @@ -22,13 +22,13 @@
apt-get clean all && \
locale-gen $LC_ALL
WORKDIR /teuthology
COPY requirements.txt requirements.yml ansible.cfg bootstrap /teuthology/
COPY pyproject.toml requirements.txt requirements.yml uv.lock ansible.cfg bootstrap /teuthology/
RUN \
cd /teuthology && \
mkdir ../archive_dir && \
mkdir log && \
chmod +x /teuthology/bootstrap && \
PIP_INSTALL_FLAGS="-r requirements.txt" ./bootstrap
./bootstrap
COPY . /teuthology
RUN \
(git config -f ./.git/config --unset 'http.https://github.com/.extraheader' || true ) && \
Expand All @@ -42,4 +42,4 @@
chmod 600 $HOME/.ssh/id_rsa && \
echo "StrictHostKeyChecking=no" > $HOME/.ssh/config && \
echo "UserKnownHostsFile=/dev/null" >> $HOME/.ssh/config
ENTRYPOINT /teuthology.sh

Check warning on line 45 in containers/teuthology-dev/Dockerfile

View workflow job for this annotation

GitHub Actions / docker (ubuntu-24.04, 3.12)

JSON arguments recommended for ENTRYPOINT/CMD to prevent unintended behavior related to OS signals 

JSONArgsRecommended: JSON arguments recommended for ENTRYPOINT to prevent unintended behavior related to OS signals
More info: https://docs.docker.com/go/dockerfile/rule/json-args-recommended/

Check warning on line 45 in containers/teuthology-dev/Dockerfile

View workflow job for this annotation

GitHub Actions / docker (ubuntu-24.04-arm, 3.12)

JSON arguments recommended for ENTRYPOINT/CMD to prevent unintended behavior related to OS signals 

JSONArgsRecommended: JSON arguments recommended for ENTRYPOINT to prevent unintended behavior related to OS signals
More info: https://docs.docker.com/go/dockerfile/rule/json-args-recommended/
2 changes: 1 addition & 1 deletion containers/teuthology-dev/teuthology.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/bash
set -e
source /teuthology/virtualenv/bin/activate
PATH=$PATH:/teuthology/.venv/bin
set -x
cat /run/secrets/id_rsa > $HOME/.ssh/id_rsa
if [ -n "$TEUTHOLOGY_TESTNODES" ]; then
Expand Down
Loading
Loading