Script `make-firmware.bash` for Python >= 3.11 and virtual envs

[EskoDijk]

The script make-firmware.bash has 4 instances of pip3 install --user <other params>. The --user flag causes problems with modern Python (>= 3.11 ?) which need to run in a Python virtual env to be able to install anything with pip3. The pip3 install fails due to this in a venv. And without venv, it fails because installing via an (APT) package is the desired way, if not in a venv.

I tried just removing the --user flags and that works for a Python venv - the build succeeds.

• Should we remove the --user flags in the repo also?
• Or is there a reason to keep the --user flag for those running an older Python3 version?
• If so, should we just detect the python3 version and add the --user flag conditionally on version?
  • For OTNS2 I already have some bash code to check if the python3 version is higher than 3.N.

See for background: https://packaging.python.org/en/latest/specifications/externally-managed-environments/#externally-managed-environments

[EskoDijk]

@suveshpratapa Do you get any issues when trying to build on newer/modern Linux distros?
In any case it should be fairly easy to fix this and support both older/newer Linuxes.

[EskoDijk]

See the few commits I made here: https://github.com/EskoDijk/ot-reference-release/commits/main/

This adds 2 scripts in the root for bootstrap and building the release/BR using a Python virtual env.
For Ubuntu 24.04 for example, this is required otherwise the Python package installs all fail.

[EskoDijk]

@suveshpratapa @jwhui @yangsong-cnyn @JaneFromSilabs FYI I spoke to some people at a recent embedded dev conference, and everyone agreed that nowadays any Python work/installs as part of build processes are done using virtual envs (venvs).
Having the build break on newer Linux OS'es is becoming a pain!

Could we move exclusively to a virtual-env based build for this? Any thoughts? I would be in favor.

[jwhui]

@EskoDijk , I agree that using Python virtual env is generally considered best practice.

[yangsong-cnyn]

@EskoDijk, venvs are also necessary for my Raspbian Upgrade (to Debian 12 bookworm), it's part of my ongoing PR #100

After OS upgrade, we will make Python install all under virtual environment

[EskoDijk]

@yangsong-cnyn This sounds good - we could include this issue with the Debian 12 PR to be merged (as part of an overall effort to support more 'modern' OS versions, for the OTBR and for the build-OS).

[suveshpratapa]

@EskoDijk I am also in favor of this best practice. I don't think it should affect our reference release build process too much.

[EskoDijk]

@suveshpratapa I've been already using this process with venv for a while, seems to work ok!

[EskoDijk]

@yangsong-cnyn Can you let me know when venv is included in the PR?

[EskoDijk]

@yangsong-cnyn It looks like the venv is not (fully) included in the latest Debian 12 PR. I still get an error in the build as below.
Should I try to submit a fix for this, of is there a reason that the venv wasn't fully implemented yet? (Or am I doing something wrong?) In my local modified build setup I had already everything running with venv enabled.

Log:

eskod@localhost:~/ot-reference-release$ REFERENCE_PLATFORM=ncs REFERENCE_RELEASE_TYPE=1.4 ./script/make-reference-release.bash
+ main
+ echo REFERENCE_RELEASE_TYPE=1.4
REFERENCE_RELEASE_TYPE=1.4
+ mkdir -p build
+++ date +%Y%m%d
+++ cd openthread
+++ git rev-parse --short HEAD
++ realpath build/ot-1.4-20260113-4f3892804
+ OUTPUT_ROOT=/home/eskod/ot-reference-release/build/ot-1.4-20260113-4f3892804
+ mkdir -p /home/eskod/ot-reference-release/build/ot-1.4-20260113-4f3892804
+ ./script/bootstrap.bash
+ [[ -n ./script/bootstrap.bash ]]
+ script_path=./script/bootstrap.bash
+++ dirname ./script/bootstrap.bash
++ realpath ./script
+ script_dir=/home/eskod/ot-reference-release/script
++ dirname /home/eskod/ot-reference-release/script
+ repo_dir=/home/eskod/ot-reference-release
+ main
+ '[' 0 == 0 ']'
+ install_packages
+ PM=source
+ command -v apt-get
/usr/bin/apt-get
+ PM=apt
+ install_packages_apt
+ echo 'Installing apt dependencies...'
Installing apt dependencies...
+ sudo apt-get update
Hit:1 http://mirrors.linode.com/ubuntu noble InRelease
Hit:2 http://mirrors.linode.com/ubuntu noble-updates InRelease
Hit:3 http://mirrors.linode.com/ubuntu noble-security InRelease
Hit:4 http://mirrors.linode.com/ubuntu noble-backports InRelease
Hit:5 https://download.docker.com/linux/ubuntu noble InRelease
Reading package lists... Done
+ sudo apt-get --no-install-recommends install -y parted fdisk git wget file xz-utils zip python3-pip dcfldd lsof
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
parted is already the newest version (3.6-4build1).
fdisk is already the newest version (2.39.3-9ubuntu6.4).
git is already the newest version (1:2.43.0-1ubuntu7.3).
wget is already the newest version (1.21.4-1ubuntu4.1).
file is already the newest version (1:5.45-3build1).
xz-utils is already the newest version (5.6.1+really5.4.5-1ubuntu0.2).
zip is already the newest version (3.0-13ubuntu0.2).
python3-pip is already the newest version (24.0+dfsg-1ubuntu1.3).
dcfldd is already the newest version (1.9.1-1ubuntu2).
lsof is already the newest version (4.95.0-1build3).
0 upgraded, 0 newly installed, 0 to remove and 64 not upgraded.
+ command -v pip3
/usr/bin/pip3
+ install_packages_pip3
+ echo 'Installing python3 dependencies...'
Installing python3 dependencies...
+ pip3 install --upgrade -r /home/eskod/ot-reference-release/requirements.txt
error: externally-managed-environment

 This environment is externally managed
> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

[yangsong-cnyn]

@EskoDijk Looks that bootstrap.bash is not covered with virtual env so we get the error. I am going to file a PR to also mandate virutal env on host.