LNT should install using current "best practice" for Python

[DavidSpickett]

Quoting @ldionne :

Eventually, we can move away from setup.py and replace it by pyproject.toml which is the current best practice, but that should be done separately.

A colleague of mine is an expert here, opening this issue so they can tell us what they recommend.

The current installation instructions are: https://llvm.org/docs/lnt/quickstart.html#installation

Checkout the LNT sources:
git clone https://github.com/llvm/llvm-lnt.git ~/lnt

Create a new virtual environment for the LNT application and activate it:
python3 -m venv .venv
source .venv/bin/activate

Install LNT into the virtual environment:
pip install -r requirements.txt

There are 3 requirements files:

• One for the client (this is what we will soon be using for the buildbots)
• One for the server (that includes the client file, creating a superset of it)
• One with the standard requirements.txt name. Purely so that the "obvious" install command works.

[DavidSpickett]

Useful to know that we're considering moving the minimum to Python 3.10 (#62). In case that provides any new things.

[DaftanoPro]

Python 3.9 end of life is this month, so 3.10 should be the bare minimum. To reduce maintenance I'd suggest to try to move up to 3.12.

For your setup I would suggest to use uv which is uses pyproject.toml, which would simplify the client installation to

uv sync

and the server installation to

uv sync --group server

[DaftanoPro]

Looking at the dependencies, it appears to me that many are just dependencies of dependencies, and you should get the same result with the following groups:

common

dependencies = [
    "Flask==0.12.2",
    "Flask-RESTful==0.3.4",
    "Flask-WTF==0.12",
    "python-gnupg==0.3.7",
    "pyyaml",
    "requests",
    "SQLAlchemy==1.3.24",
    "typing",
]

additional

server = [
    "psycopg2==2.9.10",
    "gunicorn==19.9.0",
    "progressbar2",
]
dev = [
    "flake8",
    "sphinx>=7",
    "sphinx_bootstrap_theme",
    "filecheck",
    "lit"
]

Below are the versions installed in my virtual environment for the packages that don't specify a version in setup.py.

pyyaml = 6.0.3
requests = 2.32.4
typing = 3.10.0.0
progressbar2 = 4.2.0
flake8 = 5.0.4
sphinx = 7.1.2
sphinx_bootstrap_theme = 0.8.1
filecheck = 0.0.24
lit = 18.1.8

[DavidSpickett]

For your setup I would suggest to use uv which is uses pyproject.toml

Is it possible to just switch to pyproject.toml, as a first step?

it appears to me that many are just dependencies of dependencies

So for example if we specify these two things:

• package A version X.Y
• package B version Y.Z

And package A itself already requires package B Y.Z, we are just re-stating the same thing?

Is there any advantage to doing that, clarity, security perhaps? I suppose package A X.Y could be replaced with a new build that requires B Y.Z+1, but equally, they could replace it with a build that does any number of malicious things without changing the requirements.

If you know any explanations or guides on how to approach this, that would be useful.

[DaftanoPro]

setuptools can use pyproject.toml (it was introduced in PEP 518 and then expanded in PEP 517, PEP 621 and PEP 660.) and nearly everything that is in the setup.py can be moved to the pyproject.toml.
There are few things that would require some work from the setup.py, but I would suggest to move to uv, which vastly simplify the setup and offers a lot of flexibility.

If package A requires package B==Y.Z and you specify both, you risk to create a conflict when package A will require a version of B greater than Y.Z, because you limited B to Y.Z. Another possible issue is another dependency using package B, and you may end up with A requiring B at W.Z, you specified B at Y.Z and package C requiring B at A.Z. The matter becomes more complicated if package C is a dependency of package A.

Moving to uv would provide an additional benefit, that it creates an uv.lock which lists the whole tree of dependencies, with version number and packages hashes. You want to publish the uv.lock file in the repository. That way you get a consistent installation every time. UV usually resolves conflicts automatically, if not prints an exhaustive message that helps you understand and fix the issue. The information I posted above come from the uv.lock I got from implementing pyproject.toml for lnt.

If you directly use package B and you specifically require Y.Z, uv will find out the best option, i.e. a version that works will all the dependencies (both listed and dependencies of dependencies).
Upgrading all the dependencies or just a single package is a piece of cake with uv.

[DavidSpickett]

setuptools can use pyproject.toml

Cool. One of the links in this system is the buildbots that take on the order of days to update (for reasons I won't go into). So having incremental steps is good.

If package A requires package B==Y.Z and you specify both, you risk to create a conflict when package A will require a version of B greater than Y.Z, because you limited B to Y.Z.

Ok, understood. If you let the packages define the version or version ranges, the install tool can find a version that satisfies all the constraints. If we step and say B must be a specific version, we are preventing it from doing that.

Moving to uv would provide an additional benefit, that it creates an uv.lock which lists the whole tree of dependencies, with version number and packages hashes.

This is a comprehensive answer to the concerns I was thinking about. Sounds much better than solving half the issues by pinning versions only.