Use prek and ruff for linting and update rule set (#1952)

This pull request updates code style tooling and configuration,
streamlines code formatting in notebooks, and improves import
organization and configuration files. The most impactful changes are the
modernization of the pre-commit setup and the consolidation of code
formatting tools, alongside broad cleanups and formatting
simplifications across Jupyter notebooks and Python files.

**Tooling and Configuration Updates:**

* Updated `.pre-commit-config.yaml` to use the latest `ruff-pre-commit`
repo, replaced `black` and `isort` with Ruff's formatter and import
sorter, and adjusted hooks and dependencies for consistency and modern
standards.
[[1]](diffhunk://#diff-63a9c44a44acf85fea213a857769990937107cf072831e1a26808cfde9d096b9L17-R28)
[[2]](diffhunk://#diff-63a9c44a44acf85fea213a857769990937107cf072831e1a26808cfde9d096b9L63-R58)
* Overhauled `pyproject.toml` to remove legacy `black` and `isort`
settings, expanded Ruff configuration for linting, formatting, and
per-file ignores, and added mypy configuration for static type checking.
[[1]](diffhunk://#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711L128-L132)
[[2]](diffhunk://#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711L185-R182)
[[3]](diffhunk://#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711L234-R259)

**Code Formatting and Import Organization:**

* Refactored multi-line imports and function calls into single lines in
both Python and Jupyter notebook files for improved readability and
consistency.
[[1]](diffhunk://#diff-008dcb3426febd767787b1521f1fe33086313b927ea37eaab86df5fa88a51698L182-R182)
[[2]](diffhunk://#diff-aa879055ed1e1322b67939f639d3bc3c12d5bc295d7c4e834effbc4f1a53fd47L36-R36)
[[3]](diffhunk://#diff-77d278999778b6ae6f7aeb8a0ea8d5abbaf3e79a4ff53526648d343874eaf098L7-R7)
* Standardized import order and grouping in multiple notebooks, moving
library imports to the top and ensuring proper separation between
standard, third-party, and local imports.
[[1]](diffhunk://#diff-9ec1db58f8064a03a3bd0a6f18fe17675093c4396bf1e3dd24e929b7f06a79cdL162-R165)
[[2]](diffhunk://#diff-c84d3a222eb8e8c95893972a3e136fb420615d6eba35e05fb8ac1fd0ab0fe2b5L424-R423)
[[3]](diffhunk://#diff-c84d3a222eb8e8c95893972a3e136fb420615d6eba35e05fb8ac1fd0ab0fe2b5L644-R646)
[[4]](diffhunk://#diff-f78c8ee0adaf4e4c73403184ec90e74ecc8704b06602e3ac283a4eb0aa31e189L52-R53)
[[5]](diffhunk://#diff-90f4d52ca00dc82b18fa350790970ad209a3b5648d5740c56ae215619a2677e2R165)
[[6]](diffhunk://#diff-90f4d52ca00dc82b18fa350790970ad209a3b5648d5740c56ae215619a2677e2L237-R242)
[[7]](diffhunk://#diff-c84d3a222eb8e8c95893972a3e136fb420615d6eba35e05fb8ac1fd0ab0fe2b5L945-R941)

**Notebook Code Simplification:**

* Simplified function calls and list comprehensions to single lines in
notebooks to enhance clarity and maintainability.
[[1]](diffhunk://#diff-7038db84d18585b0f7da5f29a8b74c49d11868f4dac32cff9c7b36b373873949L165-R166)
[[2]](diffhunk://#diff-43702435914ce70eb5b8f94e455bdadee7eef74f794db43c3b926618c870c586L431-R431)
[[3]](diffhunk://#diff-65cccf7b378e42bad759a58749367aefb5ef7d672bc8f1791c721865b8970456L3298-R3298)
[[4]](diffhunk://#diff-44ada6db5a5f3838c865cc4be02b2550cdeaa528f0379d1f2042455f5626b352L181-R181)
[[5]](diffhunk://#diff-780024224c8402230299f07fd7170df50dbcffa25a77718fd27f5ab40a9d746cL142-R142)
[[6]](diffhunk://#diff-49df280ccdfe00a9f5bb767c6e9a7cba44b992af016e9d7a0bed54d652477db1L311-R311)
[[7]](diffhunk://#diff-61f535306c629a7f07ad9a9c0e7b99d4bb808840d2a267a82fd86ce3e74ca697L45478-R45478)
[[8]](diffhunk://#diff-c84d3a222eb8e8c95893972a3e136fb420615d6eba35e05fb8ac1fd0ab0fe2b5L187-R187)
[[9]](diffhunk://#diff-c84d3a222eb8e8c95893972a3e136fb420615d6eba35e05fb8ac1fd0ab0fe2b5L745-R744)
[[10]](diffhunk://#diff-c84d3a222eb8e8c95893972a3e136fb420615d6eba35e05fb8ac1fd0ab0fe2b5L757-R754)

**Minor Code Quality Fixes:**

* Fixed logic and formatting errors, such as the generator length check
in `docs/source/conf.py` and string concatenation in CLI help messages,
improving correctness and clarity.
[[1]](diffhunk://#diff-008dcb3426febd767787b1521f1fe33086313b927ea37eaab86df5fa88a51698L182-R182)
[[2]](diffhunk://#diff-6f1f61fbe373037fdf31139d07acfd57161d3557eaf7369807fea9f6ec65293fL58-R58)
[[3]](diffhunk://#diff-6f1f61fbe373037fdf31139d07acfd57161d3557eaf7369807fea9f6ec65293fL114-R115)
[[4]](diffhunk://#diff-4357d21435d95de54958f2aa1214e0af88de1df968b314c3f0a1736cd91d8d8aL70-R70)
[[5]](diffhunk://#diff-4357d21435d95de54958f2aa1214e0af88de1df968b314c3f0a1736cd91d8d8aL79-R79)

**Dependency and Optional Package Updates:**

* Adjusted optional dependencies in `pyproject.toml`, removing
`pre-commit` and adding `prek` for development.

These changes collectively modernize the project's code style
infrastructure, simplify code in notebooks, and improve overall
maintainability and developer experience.

#1935 

### Checklist
<!-- Put an 'x' in all the boxes that apply -->
- [ ] I have added tests to cover my changes or documented any manual
tests.
- [ ] I have updated the
[documentation](https://github.com/open-edge-platform/datumaro/tree/develop/docs)
accordingly

---------

Signed-off-by: Jort Bergfeld <[email protected]>
Signed-off-by: Albert van Houten <[email protected]>
Signed-off-by: Jort Bergfeld <[email protected]>
Co-authored-by: Albert van Houten <[email protected]>

Author: JortBergfeld

.github/workflows/linter.yml

@@ -35,12 +35,12 @@ jobs:
         shell: bash
         run: |
           python -m pip install --upgrade pip
-          pip install pre-commit
+          pip install prek
 
       # Execute pre-commit checks
       - name: Run pre-commit checks
         shell: bash
-        run: pre-commit run --all-files
+        run: prek run --all-files
 
   Zizmor-Scan-PR:
     runs-on: ubuntu-latest

.pre-commit-config.yaml

@@ -4,7 +4,7 @@ default_language_version:
 repos:
   # EOF and whitespace checker
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
+    rev: v6.0.0
     hooks:
       - id: end-of-file-fixer
         exclude: |
@@ -14,24 +14,14 @@ repos:
           )$
 
   # Ruff
-  - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: "v0.5.0"
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.14.4
     hooks:
-      - id: ruff
-        exclude: "tests"
-
-  # python code formatting
-  - repo: https://github.com/psf/black
-    rev: 23.1.0
-    hooks:
-      - id: black
-
-  # Sort Python import
-  - repo: https://github.com/pycqa/isort
-    rev: 5.11.5
-    hooks:
-      - id: isort
-        name: isort (python)
+    - id: ruff
+      name: Ruff linter on all Python files
+      args: ["--fix"]
+    - id: ruff-format
+      name: Ruff formatter on all Python files
 
   # TODO: Enable mypy as soon as possible
   # python static type checking
@@ -58,22 +48,22 @@ repos:
 
   # notebooks
   - repo: https://github.com/nbQA-dev/nbQA
-    rev: 1.8.5
+    rev: 1.9.1
     hooks:
-      - id: nbqa-black
       - id: nbqa-ruff
-        additional_dependencies: [ruff==0.4.10]
+        additional_dependencies: [ruff==0.14.4]
+        args: [--fix]
 
   # zizmor detects security issues in GitHub Actions workflows.
   - repo: https://github.com/woodruffw/zizmor-pre-commit
-    rev: v1.11.0
+    rev: v1.16.3
     hooks:
       - id: zizmor
         args: ["--min-severity", "low", "--min-confidence", "low"]
 
   # add bandit for security checks
   - repo: https://github.com/PyCQA/bandit
-    rev: 1.8.3
+    rev: 1.8.6
     hooks:
       - id: bandit
         args:

docs/source/conf.py

@@ -179,7 +179,7 @@ def replace(app, what, name, obj, options, lines):
             for b in exclude_plugins_name:
                 if a.lower() == b:
                     names.pop(n)
-    if all(1 == len(a) for a in names):
+    if all(len(a) == 1 for a in names):
         prog_name = "".join(names).lower()
     else:
         prog_name = "_".join(names).lower()
@@ -188,9 +188,8 @@ def replace(app, what, name, obj, options, lines):
             prog = str("%(prog)s")
             lines[i] = lines[i].replace(prog, prog_name)
             lines[i] = lines[i].replace("'frame_'", r"'frame\_'")  # fix unwanted link
-            if "'|n'" not in lines[i]:
-                if "'|s'" not in lines[i]:
-                    lines[i] = lines[i].replace("|n", "\n").replace("|s", " ")
+            if "'|n'" not in lines[i] and "'|s'" not in lines[i]:
+                lines[i] = lines[i].replace("|n", "\n").replace("|s", " ")
 
 
 def setup(app):

notebooks/01_merge_multiple_datasets_for_classification.ipynb

@@ -162,12 +162,8 @@
     }
    ],
    "source": [
-    "eurosat_label_names = [\n",
-    "    label_cat.name for label_cat in eurosat.categories()[dm.AnnotationType.label]\n",
-    "]\n",
-    "uc_merced_label_names = [\n",
-    "    label_cat.name for label_cat in uc_merced.categories()[dm.AnnotationType.label]\n",
-    "]\n",
+    "eurosat_label_names = [label_cat.name for label_cat in eurosat.categories()[dm.AnnotationType.label]]\n",
+    "uc_merced_label_names = [label_cat.name for label_cat in uc_merced.categories()[dm.AnnotationType.label]]\n",
     "\n",
     "print(\"EuroSAT label names:\")\n",
     "print(eurosat_label_names)\n",

notebooks/03_visualize.ipynb

@@ -124,7 +124,7 @@
     "\n",
     "dataset = dm.Dataset.import_from(\"coco_dataset\", format=\"coco_instances\")\n",
     "print(\"Subset candidates:\", dataset.subsets().keys())\n",
-    "subset = list(dataset.subsets().keys())[0]  # val2017\n",
+    "subset = next(dataset.subsets().keys())  # val2017\n",
     "print(\"Subset:\", subset)"
    ]
   },

notebooks/04_filter.ipynb

@@ -428,9 +428,7 @@
    ],
    "source": [
     "print(\"There is no item with occlusion\")\n",
-    "filtered = dataset.clone().filter(\n",
-    "    '/item/annotation[occluded=\"False\"]', filter_annotations=True, remove_empty=True\n",
-    ")\n",
+    "filtered = dataset.clone().filter('/item/annotation[occluded=\"False\"]', filter_annotations=True, remove_empty=True)\n",
     "for item in filtered:\n",
     "    print(f\"ID: {item.id}\")\n",
     "\n",

notebooks/05_transform.ipynb

@@ -3295,9 +3295,7 @@
    "source": [
     "# from datumaro.components.dataset import Dataset\n",
     "\n",
-    "val_dataset = dataset.filter(\n",
-    "    '/item[subset=\"val2017\"]'\n",
-    ")  # or Dataset(dataset.get_subset(subsets[0]))\n",
+    "val_dataset = dataset.filter('/item[subset=\"val2017\"]')  # or Dataset(dataset.get_subset(subsets[0]))\n",
     "val_dataset"
    ]
   },

notebooks/06_tiling.ipynb

@@ -178,9 +178,7 @@
    "source": [
     "from datumaro.plugins.tiling import Tile\n",
     "\n",
-    "tiled_dataset = dataset.transform(\n",
-    "    Tile, grid_size=(2, 2), overlap=(0.1, 0.1), threshold_drop_ann=0.5\n",
-    ")\n",
+    "tiled_dataset = dataset.transform(Tile, grid_size=(2, 2), overlap=(0.1, 0.1), threshold_drop_ann=0.5)\n",
     "target_ids = [tiled_id for tiled_id in get_ids(tiled_dataset, subset) if target_id in tiled_id]\n",
     "print(target_ids)"
    ]

notebooks/08_e2e_example_yolo_ultralytics_trainer.ipynb

@@ -139,9 +139,7 @@
     }
    ],
    "source": [
-    "splited_dataset = dataset.transform(\n",
-    "    \"random_split\", splits=[(\"train\", 0.5), (\"val\", 0.2), (\"test\", 0.3)]\n",
-    ")\n",
+    "splited_dataset = dataset.transform(\"random_split\", splits=[(\"train\", 0.5), (\"val\", 0.2), (\"test\", 0.3)])\n",
     "splited_dataset"
    ]
   },

notebooks/11_validate.ipynb

@@ -159,9 +159,10 @@
     }
    ],
    "source": [
-    "import numpy as np\n",
     "import cv2\n",
+    "import numpy as np\n",
     "from matplotlib import pyplot as plt\n",
+    "\n",
     "from datumaro.components.annotation import AnnotationType, LabelCategories\n",
     "\n",
     "label_categories = dataset.categories().get(AnnotationType.label, LabelCategories())\n",
@@ -189,7 +190,7 @@
     "for report in reports[\"validation_reports\"]:\n",
     "    if report[\"anomaly_type\"] == \"NegativeLength\":\n",
     "        item = dataset.get(report[\"item_id\"], report[\"subset\"])\n",
-    "        label_id = [int(s) for s in str.split(report[\"description\"], \"'\") if s.isdigit()][0]\n",
+    "        label_id = next(int(s) for s in str.split(report[\"description\"], \"'\") if s.isdigit())\n",
     "        visualize_label_id(item, label_id)"
    ]
   },
@@ -228,13 +229,13 @@
     "    if report[\"anomaly_type\"] == \"UndefinedAttribute\":\n",
     "        item = dataset.get(report[\"item_id\"], report[\"subset\"])\n",
     "        for ann in item.annotations:\n",
-    "            for k in ann.attributes.keys():\n",
+    "            for k in ann.attributes:\n",
     "                label_categories[ann.label].attributes.add(k)\n",
     "\n",
     "    if report[\"anomaly_type\"] == \"NegativeLength\":\n",
     "        item = dataset.get(report[\"item_id\"], report[\"subset\"])\n",
     "        print(report[\"description\"])\n",
-    "        label_id = [int(s) for s in str.split(report[\"description\"], \"'\") if s.isdigit()][0]\n",
+    "        label_id = next(int(s) for s in str.split(report[\"description\"], \"'\") if s.isdigit())\n",
     "        neg_len_anns = []\n",
     "        for ann in item.annotations:\n",
     "            if ann.id == label_id:\n",
@@ -431,7 +432,7 @@
     "    if report[\"anomaly_type\"] == \"FarFromLabelMean\":\n",
     "        print(report[\"description\"])\n",
     "        item = dataset.get(report[\"item_id\"], report[\"subset\"])\n",
-    "        label_id = [int(s) for s in str.split(report[\"description\"], \"'\") if s.isdigit()][0]\n",
+    "        label_id = next(int(s) for s in str.split(report[\"description\"], \"'\") if s.isdigit())\n",
     "        visualize_label_id(item, label_id)"
    ]
   },