|
| 1 | +--- |
| 2 | +tags: python, bootstrap, venv |
| 3 | +category: review |
| 4 | +date: 2025-11-24 |
| 5 | +title: SPL Bootstrap - Handle Python Version |
| 6 | +--- |
| 7 | + |
| 8 | +## SPL Bootstrap - Handle Python Version |
| 9 | + |
| 10 | +### Introduction |
| 11 | + |
| 12 | +The pipeline implementation for our Software Product Line (SPL) repositories uses the [pypeline](https://github.com/cuinixam/pypeline) Python application. |
| 13 | +In order to start the `pypeline` application, we need to install Python (with a specified version) and create the Python virtual environment. |
| 14 | +This step we call `bootstrap` and it is implemented [here](https://github.com/avengineers/bootstrap). |
| 15 | + |
| 16 | +```{note} |
| 17 | +Currently bootstrap supports only Windows for installing Python. |
| 18 | +``` |
| 19 | + |
| 20 | +### Workflow |
| 21 | + |
| 22 | +The bootstrap process is managed by two main scripts: a PowerShell script (`bootstrap.ps1`) for environment setup and a Python script (`bootstrap.py`) for dependency management. The diagram below shows the high-level workflow. |
| 23 | + |
| 24 | +```{mermaid} |
| 25 | +flowchart TD |
| 26 | + A[bootstrap.ps1] --> B{Python installed?}; |
| 27 | + B -- No --> C[Install Python w/ Scoop 1️⃣]; |
| 28 | + B -- Yes --> D; |
| 29 | + C --> D[Execute bootstrap.py]; |
| 30 | + |
| 31 | + D --> E{Dependencies changed?}; |
| 32 | + E -- No --> F[End]; |
| 33 | + E -- Yes --> G[Create/Update .venv 2️⃣]; |
| 34 | + G --> H[Install Dependencies]; |
| 35 | + H --> F; |
| 36 | +``` |
| 37 | + |
| 38 | +1. This step will first install the `scoop` windows package manager and then call `scoop` to install the configured python version. |
| 39 | +2. The python virtual environment is updated in multiple steps: |
| 40 | + - create an empty virtual environment with python `venv.create` |
| 41 | + - install the package manager (e.g., poetry, uv, etc.) and `pip-system-certs` for using the system SSL certificates |
| 42 | + - run the package manager to install the python dependencies |
| 43 | + |
| 44 | +#### bootstrap.ps1 |
| 45 | + |
| 46 | +- Scoop configuration and installation |
| 47 | +- Python version detection and installation |
| 48 | +- Delegates to `bootstrap.py` for virtual environment setup |
| 49 | + |
| 50 | +#### bootstrap.py |
| 51 | + |
| 52 | +- OS-agnostic virtual environment creation (Windows/Unix) |
| 53 | +- Smart caching with hash-based dependency tracking. Only runs if dependencies have been updated. |
| 54 | +- Package manager support: Poetry, UV, Pipenv |
| 55 | +- Automatic pip configuration for custom PyPI sources |
| 56 | + |
| 57 | +#### bootstrap.json Configuration |
| 58 | + |
| 59 | +```json |
| 60 | +{ |
| 61 | + "python_version": "3.11", |
| 62 | + "python_package_manager": "poetry>=2.1.0", |
| 63 | + "scoop_ignore_scoopfile": true |
| 64 | +} |
| 65 | +``` |
| 66 | + |
| 67 | +### Things to Improve |
| 68 | + |
| 69 | +#### Python Version Management |
| 70 | + |
| 71 | +Currently the bootstrap process installs a fixed Python version as specified in the `bootstrap.json` configuration file. |
| 72 | +When checking if python is installed, it only verifies that the major and minor version match (e.g., `3.11`), |
| 73 | +but does not check for patch versions (e.g., `3.11.4` vs `3.11.5`). |
| 74 | +This can lead to situations that on different machines, slightly different patch versions of Python are installed. |
| 75 | + |
| 76 | +```{note} |
| 77 | +This might not be a big issue in most cases, but it can lead to inconsistencies in our particular case, |
| 78 | +because we use the python `venv` module to create the initial virtual environment, |
| 79 | +which will include the standard library of the installed Python version. |
| 80 | +``` |
| 81 | + |
| 82 | +The solution could be: |
| 83 | + |
| 84 | +- to find all installed python versions in path and check if the exact version is already installed |
| 85 | +- it might be that python versions installed with scoop are not available in path, so we need to query scoop for installed versions |
| 86 | +- if the exact version is not installed, install it with scoop |
| 87 | +- the `bootstrap.py` shall fail if the python interpreter version does not match exactly the configured version. |
| 88 | + This can happen if the user runs the `pypeline` including the `CreateVEnv` step which calls directly the `bootstrap.py` script. |
| 89 | + |
| 90 | +#### Virtual Environment Management |
| 91 | + |
| 92 | +As mention above, the virtual environment is created in three steps: |
| 93 | + |
| 94 | +1. create an empty virtual environment with python `venv.create` |
| 95 | +1. install the package manager (e.g., poetry, uv, etc.) and `pip-system-certs` for using the system SSL certificates |
| 96 | +1. run the package manager to install the python dependencies |
| 97 | + |
| 98 | +There is a problem with the second step: the package manager and pip-system certs are installed with pip. |
| 99 | +These packages have their own dependencies, which are not tracked by pip because pip has no support for `.lock` files. |
| 100 | + |
| 101 | +When step three is executed, the package manager might need to upgrade/downgrade some packages which were implicitly installed in step two |
| 102 | +and this can cause crashes due to dependency conflicts. |
| 103 | + |
| 104 | +#### Solution 1: User-defined Initial Packages |
| 105 | + |
| 106 | +One way to solve this is to let the user define the list of package to install in step two in the `bootstrap.json` configuration file. |
| 107 | +For example: |
| 108 | + |
| 109 | +```{code-block} json |
| 110 | +{ |
| 111 | + "python_version": "3.11.4", |
| 112 | + "venv_initial_packages": [ |
| 113 | + "poetry==2.1.0", |
| 114 | + "pip-system-certs==2.2.1", |
| 115 | + "wrapt==1.14.0" |
| 116 | + ], |
| 117 | +} |
| 118 | +``` |
| 119 | + |
| 120 | +In case there is a conflict between the initial packages and the packages defined in the package manager lock file, |
| 121 | +the user can adjust the versions in the configuration file accordingly. |
| 122 | + |
| 123 | +#### Solution 2: Install Package Manager with Scoop |
| 124 | + |
| 125 | +Another way could be to install both python and the package manager with scoop in the `bootstrap.ps1` script, |
| 126 | +so that pip is not involved at all in step two. This means that package manager will install exactly the version specified in the `.lock` file |
| 127 | +and therefore always have a consistent set of dependencies. |
| 128 | + |
| 129 | +The problem with this approach is that `pip-system-certs` is not available as a scoop package |
| 130 | +and package installation will fail when ran against an on-premise PyPI server with self-signed certificates. |
| 131 | + |
| 132 | +#### Package manager command |
| 133 | + |
| 134 | +Currently the command to install dependencies with the package manager is hardcoded in the `bootstrap.py` script |
| 135 | +with extra arguments being supported via the `bootstrap.json` configuration file. |
| 136 | + |
| 137 | +```{code-block} json |
| 138 | +{ |
| 139 | + "python_package_manager_args": "--clean" |
| 140 | +} |
| 141 | +``` |
| 142 | + |
| 143 | +We added the feature to support extra arguments because we needed for some projects to pass the `--clean` to `pipenv` |
| 144 | +to remove unused packages already existing in the virtual environment. |
| 145 | + |
| 146 | +A better approach would be to let the user define the full command to install dependencies with the package manager. |
| 147 | +For example: |
| 148 | + |
| 149 | +```{code-block} json |
| 150 | +{ |
| 151 | + "venv_install_command": "poetry install --no-dev --clean" |
| 152 | +} |
| 153 | +``` |
| 154 | + |
| 155 | +This would give the user full control over how dependencies are installed. |
| 156 | + |
| 157 | +## Conclusion |
| 158 | + |
| 159 | +If all these improvements are implemented, the bootstrap process will be more reliable and flexible, ensuring consistent Python environments. |
| 160 | + |
| 161 | +The configuration will allow users to tailor the bootstrap process to their specific needs. |
| 162 | + |
| 163 | +```{code-block} json |
| 164 | +{ |
| 165 | + "python_version": "3.11.4", |
| 166 | + "venv_initial_packages": [ |
| 167 | + "poetry==2.1.0", |
| 168 | + "pip-system-certs==2.2.1" |
| 169 | + ], |
| 170 | + "venv_install_command": "poetry install --no-dev --clean" |
| 171 | +} |
| 172 | +``` |
| 173 | + |
| 174 | +This configuration will ensure that: |
| 175 | + |
| 176 | +- Python 3.11.4 is installed |
| 177 | +- The package manager and pip-system-certs are installed with the specified versions |
| 178 | +- Dependencies are installed with the specified command |
| 179 | + |
| 180 | +Have fun bootstrapping! ✌️ |
0 commit comments