Manage dependencies with uv (#4241)

* tox.ini is outdated and likely not used

* No need for extra pip-install

Dependencies are already installed by the setup script in the venv, and the pip install would install into the user env if not already in the venv.

* Removed traces of isort, black and flake8 since using ruff instead

* Moved requirements into dev dependency-group

* Added pre-commit hook to check that uv.lock is up to date

* Updated README with dependency mangement using uv

* Created dependency-group `ci`

* Fixed wording in README

* Updated dependency management in README

Author: carlolars

.pre-commit-config.yaml

@@ -15,11 +15,18 @@ repos:
     rev: v1.14.1
     hooks:
       - id: mypy
-  
+
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.4 # Keep this in sync with requirements_test.txt
+    rev: v0.9.4 # Keep this in sync with pyproject.toml
     hooks:
       - id: ruff
         args: ["--fix", "--exit-non-zero-on-fix", "--config", "pyproject.toml"]
       - id: ruff-format
         args: ["--config", "pyproject.toml"]
+
+  - repo: https://github.com/astral-sh/uv-pre-commit
+    # uv version.
+    rev: 0.8.6
+    hooks:
+      - id: uv-lock
+        args: ["--check"]

.vscode/settings.default.json

@@ -1,5 +1,4 @@
 {
-  "python.formatting.provider": "black",
   // Added --no-cov to work around TypeError: message must be set
   // https://github.com/microsoft/vscode-python/issues/14067
   "python.testing.pytestArgs": ["--no-cov"],

README.md

@@ -404,7 +404,7 @@ Now lets put this all together. If you examine the device definition above you w
 
 # Contribution Guidelines
 
-- All code is formatted with black. The check format script that runs in CI will ensure that code meets this requirement and that it is correctly formatted with black. Instructions for installing black in many editors can be found here: <https://github.com/psf/black#editor-integration>
+- All code is formatted with ruff. The check format script that runs in CI will ensure that code meets this requirement and that it is correctly formatted with ruff. Instructions for integrating ruff in many editors can be found here: <https://docs.astral.sh/ruff/editors/>
 
 - Capture the SimpleDescriptor log entries for each endpoint on the device. These can be found in the HA logs after joining a device and they look like this: `<SimpleDescriptor endpoint=1 profile=260 device_type=1026 device_version=0 input_clusters=[0, 1, 3, 32, 1026, 1280, 2821] output_clusters=[25]>`. This information can also be obtained from the zigbee.db if you want to take the time to query the tables and reconstitute the log entry. I find it easier to just remove and rejoin the device. ZHA entity ids are stable for the most part so it _shouldn't_ disrupt anything you have configured. These need to match what the device reports EXACTLY or zigpy will not match them when a device joins and the handler will not be used for the device. You can also obtain this information from the device screen in HA for the device. The `Zigbee Device Signature` button will launch a dialog that contains all of the information necessary to create quirks.
 
@@ -449,15 +449,9 @@ You can see a pattern that illustrates how to match a more complex event. In thi
 
 Open a terminal at the root of the project and run the setup script: `script/setup` This script will install all necessary dependencies and it will install the precommit hook.
 
-The tests use the [pytest](https://docs.pytest.org/en/latest/) framework.
-
-### Getting started
-
-To get set up, you need install the test dependencies:
-
-```bash
-pip install -r requirements_test.txt
-```
+All the dependencies are installed using the locked versions in the `uv.lock` file.
+To keep the environment up to date with the locked versions of the dependencies, run `uv sync` make sure the
+environment matches the `uv.lock` file.
 
 ### Running the tests
 
@@ -560,6 +554,51 @@ def test_ts0121_signature(assert_signature_matches_quirk):
     assert_signature_matches_quirk(zhaquirks.tuya.ts0121_plug.Plug, signature)
 ```
 
+## Managing dependencies
+
+This project uses [uv] to manage dependencies and ensure reproducible environments.
+
+[uv]: https://docs.astral.sh/uv/
+
+### Sync the virtual environment
+
+Running `uv sync` will install all packages using the locked version from the `uv.lock` file into the virtual environment.
+It will also *remove* any package that is not listed in the `uv.lock` file.
+
+If/when the `uv.lock` file has been changed, which can be the case when for example switching branch or after merging `dev`
+branch into the current branch, you will usually need to re-sync the virtual environment using `uv sync` to ensure that the
+environment is up to date with the locked versions of the dependencies.
+The `script/setup` script will do this for you (and some more), so use the script once and then use `uv sync` to keep the
+environment up to date.
+
+The `dev` dependency group is installed by default by `uv sync`.
+
+### Updating locked dependencies
+
+After modifying a constraint for a dependency in the `pyproject.toml` file, for example bumping the minimum version
+of `zigpy`, use `uv lock` to update the locked versions for the dependency and all its dependencies so that the new
+constraint is fulfilled.
+Note that the `lock` command will not update packages already in the lockfile that already fulfills the constraints,
+even if there are newer versions available.
+
+To update a *single* package to its latest available version that fulfills the constraints use `uv lock --upgrade-package <package>`.
+
+To update *all* packages to their latest version that fulfills the constraints use `uv lock --upgrade`.
+
+### Adding new dependencies
+
+Use `uv add <package>` to add a project dependency, `uv add --dev <package>` to add to the `dev` group,
+or `uv add --group ci <package>` to add to the `ci` group.
+Alternatively just add it to the appropriate field in the `pyproject.toml` file and run `uv lock`
+to update the lockfile. Only the new package and its dependencies will be added/updated in the lockfile.
+
+In short, project dependencies, packages that should be installed when
+installing the project, go into `dependencies`. Packages used only for development go into the `dev`
+group in `dependency-groups` and packages used only for CI go into the `ci` group.
+See [uv dependency fields] for details.
+
+[uv dependency fields]: https://docs.astral.sh/uv/concepts/projects/dependencies/#dependency-fields
+
 # Thanks
 
 - Special thanks to damarco for the majority of the device tracker code

pyproject.toml

@@ -6,23 +6,36 @@ build-backend = "setuptools.build_meta"
 name = "zha-quirks"
 dynamic = ["version"]
 description = "Library implementing Zigpy quirks for ZHA in Home Assistant"
-urls = {repository = "https://github.com/zigpy/zha-device-handlers"}
-authors = [
-    {name = "David F. Mulcahey", email = "[email protected]"}
-]
+urls = { repository = "https://github.com/zigpy/zha-device-handlers" }
+authors = [{ name = "David F. Mulcahey", email = "[email protected]" }]
 readme = "README.md"
-license = {text = "Apache-2.0"}
+license = { text = "Apache-2.0" }
 requires-python = ">=3.12"
-dependencies = [
-    "zigpy>=0.82.2",
-]
+dependencies = ["zigpy>=0.82.2"]
 
 [tool.setuptools.packages.find]
 exclude = ["tests", "tests.*"]
 
-[project.optional-dependencies]
-testing = [
-    "pytest",
+[dependency-groups]
+ci = [
+    "pytest-github-actions-annotate-failures>=0.3.0",
+    "pytest-xdist>=3.8.0",
+]
+dev = [
+    "codecov",
+    "codespell>=2.3.0",
+    "colorlog",
+    "coveralls",
+    "mypy==0.942",
+    "pre-commit",
+    "pylint",
+    "pytest>=7.1.3",
+    "pytest-asyncio",
+    "pytest-cov",
+    "pytest-sugar",
+    "pytest-timeout",
+    "ruff==0.9.4",        # Keep this in sync with .pre-commit-config.yaml
+    "time-machine>=2,<3",
 ]
 
 [tool.setuptools-git-versioning]
@@ -78,7 +91,7 @@ testpaths = "tests"
 norecursedirs = ".git testing_config"
 
 [tool.codespell]
-skip = "Contributors.md"
+skip = "Contributors.md,uv.lock"
 ignore-words-list = "hass, dout, potentiels, checkin"
 quiet-level = 2
 
@@ -87,70 +100,70 @@ target-version = "py312"
 
 [tool.ruff.lint]
 select = [
-    "B002", # Python does not support the unary prefix increment
-    "B007", # Loop control variable {name} not used within loop body
-    "B014", # Exception handler with duplicate exception
-    "B023", # Function definition does not bind loop variable {name}
-    "B026", # Star-arg unpacking after a keyword argument is strongly discouraged
-    "B904", # Use raise from to specify exception cause
-    "C", # complexity
-    "COM818", # Trailing comma on bare tuple prohibited
-    "D", # docstrings
-    "DTZ003", # Use datetime.now(tz=) instead of datetime.utcnow()
-    "DTZ004", # Use datetime.fromtimestamp(ts, tz=) instead of datetime.utcfromtimestamp(ts)
-    "E", # pycodestyle
-    "F", # pyflakes/autoflake
-    "G", # flake8-logging-format
-    "I", # isort
-    "ICN001", # import concentions; {name} should be imported as {asname}
-    "N804", # First argument of a class method should be named cls
-    "N805", # First argument of a method should be named self
-    "N815", # Variable {name} in class scope should not be mixedCase
+    "B002",    # Python does not support the unary prefix increment
+    "B007",    # Loop control variable {name} not used within loop body
+    "B014",    # Exception handler with duplicate exception
+    "B023",    # Function definition does not bind loop variable {name}
+    "B026",    # Star-arg unpacking after a keyword argument is strongly discouraged
+    "B904",    # Use raise from to specify exception cause
+    "C",       # complexity
+    "COM818",  # Trailing comma on bare tuple prohibited
+    "D",       # docstrings
+    "DTZ003",  # Use datetime.now(tz=) instead of datetime.utcnow()
+    "DTZ004",  # Use datetime.fromtimestamp(ts, tz=) instead of datetime.utcfromtimestamp(ts)
+    "E",       # pycodestyle
+    "F",       # pyflakes/autoflake
+    "G",       # flake8-logging-format
+    "I",       # isort
+    "ICN001",  # import concentions; {name} should be imported as {asname}
+    "N804",    # First argument of a class method should be named cls
+    "N805",    # First argument of a method should be named self
+    "N815",    # Variable {name} in class scope should not be mixedCase
     "PERF101", # Do not cast an iterable to list before iterating over it
     "PERF102", # When using only the {subset} of a dict use the {subset}() method
     "PERF203", # try-except within a loop incurs performance overhead
-    "PGH004", # Use specific rule codes when using noqa
+    "PGH004",  # Use specific rule codes when using noqa
     "PLC0414", # Useless import alias. Import alias does not rename original package.
-    "PLC", # pylint
-    "PLE", # pylint
-    "PLR", # pylint
-    "PLW", # pylint
-    "Q000", # Double quotes found but single quotes preferred
-    "RUF006", # Store a reference to the return value of asyncio.create_task
-    "S102", # Use of exec detected
-    "S103", # bad-file-permissions
-    "S108", # hardcoded-temp-file
-    "S306", # suspicious-mktemp-usage
-    "S307", # suspicious-eval-usage
-    "S313", # suspicious-xmlc-element-tree-usage
-    "S314", # suspicious-xml-element-tree-usage
-    "S315", # suspicious-xml-expat-reader-usage
-    "S316", # suspicious-xml-expat-builder-usage
-    "S317", # suspicious-xml-sax-usage
-    "S318", # suspicious-xml-mini-dom-usage
-    "S319", # suspicious-xml-pull-dom-usage
-    "S320", # suspicious-xmle-tree-usage
-    "S601", # paramiko-call
-    "S602", # subprocess-popen-with-shell-equals-true
-    "S604", # call-with-shell-equals-true
-    "S608", # hardcoded-sql-expression
-    "S609", # unix-command-wildcard-injection
-    "SIM105", # Use contextlib.suppress({exception}) instead of try-except-pass
-    "SIM114", # Combine if branches using logical or operator
-    "SIM117", # Merge with-statements that use the same scope
-    "SIM118", # Use {key} in {dict} instead of {key} in {dict}.keys()
-    "SIM201", # Use {left} != {right} instead of not {left} == {right}
-    "SIM208", # Use {expr} instead of not (not {expr})
-    "SIM212", # Use {a} if {a} else {b} instead of {b} if not {a} else {a}
-    "SIM300", # Yoda conditions. Use 'age == 42' instead of '42 == age'.
-    "SIM401", # Use get from dict with default instead of an if block
-    "T100", # Trace found: {name} used
-    "T20", # flake8-print
-    "TID251", # Banned imports
-    "TRY004", # Prefer TypeError exception for invalid type
-    "TRY302", # Remove exception handler; error is immediately re-raised
-    "UP", # pyupgrade
-    "W", # pycodestyle
+    "PLC",     # pylint
+    "PLE",     # pylint
+    "PLR",     # pylint
+    "PLW",     # pylint
+    "Q000",    # Double quotes found but single quotes preferred
+    "RUF006",  # Store a reference to the return value of asyncio.create_task
+    "S102",    # Use of exec detected
+    "S103",    # bad-file-permissions
+    "S108",    # hardcoded-temp-file
+    "S306",    # suspicious-mktemp-usage
+    "S307",    # suspicious-eval-usage
+    "S313",    # suspicious-xmlc-element-tree-usage
+    "S314",    # suspicious-xml-element-tree-usage
+    "S315",    # suspicious-xml-expat-reader-usage
+    "S316",    # suspicious-xml-expat-builder-usage
+    "S317",    # suspicious-xml-sax-usage
+    "S318",    # suspicious-xml-mini-dom-usage
+    "S319",    # suspicious-xml-pull-dom-usage
+    "S320",    # suspicious-xmle-tree-usage
+    "S601",    # paramiko-call
+    "S602",    # subprocess-popen-with-shell-equals-true
+    "S604",    # call-with-shell-equals-true
+    "S608",    # hardcoded-sql-expression
+    "S609",    # unix-command-wildcard-injection
+    "SIM105",  # Use contextlib.suppress({exception}) instead of try-except-pass
+    "SIM114",  # Combine if branches using logical or operator
+    "SIM117",  # Merge with-statements that use the same scope
+    "SIM118",  # Use {key} in {dict} instead of {key} in {dict}.keys()
+    "SIM201",  # Use {left} != {right} instead of not {left} == {right}
+    "SIM208",  # Use {expr} instead of not (not {expr})
+    "SIM212",  # Use {a} if {a} else {b} instead of {b} if not {a} else {a}
+    "SIM300",  # Yoda conditions. Use 'age == 42' instead of '42 == age'.
+    "SIM401",  # Use get from dict with default instead of an if block
+    "T100",    # Trace found: {name} used
+    "T20",     # flake8-print
+    "TID251",  # Banned imports
+    "TRY004",  # Prefer TypeError exception for invalid type
+    "TRY302",  # Remove exception handler; error is immediately re-raised
+    "UP",      # pyupgrade
+    "W",       # pycodestyle
 ]
 
 ignore = [
@@ -175,8 +188,8 @@ ignore = [
     "PLR0915", # Too many statements ({statements} > {max_statements})
     "PLR2004", # Magic value used in comparison, consider replacing {value} with a constant variable
     "PLW2901", # Outer {outer_kind} variable {name} overwritten by inner {inner_kind} target
-    "UP006", # keep type annotation style as is
-    "UP007", # keep type annotation style as is
+    "UP006",   # keep type annotation style as is
+    "UP007",   # keep type annotation style as is
     # Ignored due to performance: https://github.com/charliermarsh/ruff/issues/2923
     "UP038", # Use `X | Y` in `isinstance` call instead of `(X, Y)`
 
@@ -205,10 +218,7 @@ fixture-parentheses = false
 
 [tool.ruff.lint.isort]
 force-sort-within-sections = true
-known-first-party = [
-    "zhaquirks",
-    "tests",
-]
+known-first-party = ["zhaquirks", "tests"]
 combine-as-imports = true
 split-on-trailing-comma = false
 

requirements_test.txt

@@ -17,18 +17,17 @@ fi
 
 if [ ! -n "$VIRTUAL_ENV" ]; then
   if [ -x "$(command -v uv)" ]; then
-    uv venv venv
+    uv venv
   else
-    python3 -m venv venv
+    python3 -m venv .venv
   fi
-  source venv/bin/activate
+  source .venv/bin/activate
 fi
 
 if ! [ -x "$(command -v uv)" ]; then
   python3 -m pip install uv
 fi
 
-uv pip install -r requirements_test.txt
-uv pip install -e .
+uv sync
 
 pre-commit install