Switch mypy prek hooks to be executed on pre-push, not on pre-commit (apache#56829)

potiuk · web-flow · commit 8836376a778a · 2025-10-18T21:48:14.000+02:00
The MyPy checks are slow and they require breeze image to be built,
so they are slowing down commit operations when you have prek
installed and your image is not built or your change contains
a number of files and mypy cache is not warmed up.

Moving mypy checks to pre-push makes it easier to accept by
developers to run `prek install` - and they might also opt-in to
use `prek install --hook-type pre-push` if they want to (by default)
run mypy checks when pushing their changes.

Still - even if you install prek with hook-stage `pre-push` you can
skip it by `git push --no-verify` - same as in case of commit.

This should make "prek" experience quite a bit better for casual usage
or when people are not keeping their breeze image updated.

This should also help in case of cherry-picking, when cherry-picking,
and you need to resolve conflicts, git does not have option to run
`cherry-pick --continue --no-verify` and you need to manually
uninstall prek to skip running prek hooks - which often is slow
due to image not being fresh enough and mypy being generally slow.

This might be quite a QOL improvement for many contributors
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1349,6 +1349,7 @@ repos:
         ## ADD MOST PREK HOOK ABOVE THAT LINE
         # The below prek hooks are those requiring CI image to be built
       - id: mypy-dev
+        stages: ['pre-push']
         name: Run mypy for dev
         language: python
         entry: ./scripts/ci/prek/mypy.py
@@ -1363,6 +1364,7 @@ repos:
         files: ^.*\.py$
         require_serial: true
       - id: mypy-airflow-core
+        stages: ['pre-push']
         name: Run mypy for airflow-core
         language: python
         entry: ./scripts/ci/prek/mypy.py
@@ -1377,6 +1379,7 @@ repos:
         files: ^airflow-core/.*\.py$
         require_serial: true
       - id: mypy-providers
+        stages: ['pre-push']
         name: Run mypy for providers
         language: python
         entry: ./scripts/ci/prek/mypy.py
@@ -1391,6 +1394,7 @@ repos:
         files: ^.*\.py$
         require_serial: true
       - id: mypy-task-sdk
+        stages: ['pre-push']
         name: Run mypy for task-sdk
         language: python
         entry: ./scripts/ci/prek/mypy.py
@@ -1405,6 +1409,7 @@ repos:
         files: ^.*\.py$
         require_serial: true
       - id: mypy-devel-common
+        stages: ['pre-push']
         name: Run mypy for devel-common
         language: python
         entry: ./scripts/ci/prek/mypy.py
@@ -1419,6 +1424,7 @@ repos:
         files: ^.*\.py$
         require_serial: true
       - id: mypy-airflow-ctl
+        stages: ['pre-push']
         name: Run mypy for airflow-ctl
         language: python
         entry: ./scripts/ci/prek/mypy.py
diff --git a/contributing-docs/03a_contributors_quick_start_beginners.rst b/contributing-docs/03a_contributors_quick_start_beginners.rst
@@ -50,7 +50,9 @@ For Breeze (local development):
 .. code-block:: bash
 
     uv tool install prek
-    prek install -f
+    prek install --force
+    prek install --force --hook-type pre-push
+
 * 4GB RAM, 40GB disk space, and at least 2 CPU cores
 
 .. note::
@@ -151,7 +153,8 @@ Option B – One-Click GitHub Codespaces
 
       curl -LsSf https://astral.sh/uv/install.sh | sh
       uv tool install prek
-      prek install -f
+      prek install --force
+      prek install --force --hook-type pre-push # for running mypy checks when pushing to repo
       uv tool install -e ./dev/breeze
       uv run setup_vscode.py
       breeze start-airflow
diff --git a/contributing-docs/08_static_code_checks.rst b/contributing-docs/08_static_code_checks.rst
@@ -22,7 +22,7 @@ The static code checks in Airflow are used to verify that the code meets certain
 All the static code checks can be run through prek hooks.
 
 The prek hooks perform all the necessary installation when you run them
-for the first time. See the table below to identify which prek checks require the Breeze Docker images.
+for the first time.
 
 You can also run the checks via `Breeze <../dev/breeze/doc/README.rst>`_ environment.
 
@@ -116,7 +116,7 @@ To install the checks also for ``pre-push`` operations, enter:
 
 .. code-block:: bash
 
-    prek install -t pre-push
+    prek install --hook-type pre-push
 
 For details on advanced usage of the install method, use:
 
@@ -157,7 +157,12 @@ Using prek
 ----------
 
 After installation, prek hooks are run automatically when you commit the
-code. But you can run prek hooks manually as needed.
+code or push it to the repository (depending on stages configured for the hooks). Some of the
+hooks are configured to run on "manual" stage only and are not run automatically.
+
+By default when you run ``prek``, the ``pre-commit`` stage hooks are run.
+
+But you can run prek hooks manually as needed.
 
 -   Run all checks on your staged files by using:
 
@@ -170,33 +175,36 @@ code. But you can run prek hooks manually as needed.
 
 .. code-block:: bash
 
-    prek mypy-airflow-core mypy-dev
+    prek mypy-airflow-core mypy-dev  --hook-stage pre-push
 
 -   Run only mypy airflow checks on all "airflow-core" files by using:
 
 .. code-block:: bash
 
-    prek mypy-airflow-core --all-files
+    prek mypy-airflow-core --all-files --hook-stage pre-push
 
--   Run all checks on all files by using:
+-   Run all pre-commit stage hooks on all files by using:
 
 .. code-block:: bash
 
     prek --all-files
 
--   Run all checks only on files modified in the last locally available commit in your checked out branch:
+-   Run all pre-commit stage hooks only on files modified in the last locally available
+    commit in your checked out branch:
 
 .. code-block:: bash
 
     prek --last-commit
 
--   Run all checks only on files modified in your last branch that is targeted to be merged into the main branch:
+-   Run all pre-commit stage hooks only on files modified in your last branch that is targeted
+    to be merged into the main branch:
 
 .. code-block:: bash
 
     prek --from-ref main
 
--   Show files modified automatically by prek when prek automatically fix errors
+-   Show files modified automatically by prek when prek automatically fixes errors (after running all
+    ``pre-commit`` stage hooks on locally modified files):
 
 .. code-block:: bash
 
@@ -207,7 +215,7 @@ code. But you can run prek hooks manually as needed.
 
 .. code-block:: bash
 
-    SKIP=mypy-airflow-core,ruff prek --all-files
+    SKIP=ruff,rst-backticks prek --all-files
 
 
 You can always skip running the tests by providing ``--no-verify`` flag to the
@@ -262,9 +270,8 @@ enter the terminal.
 Manual prek hooks
 -----------------
 
-Most of the checks we run are configured to run automatically when you commit the code. However,
-there are some checks that are not run automatically and you need to run them manually. Those
-checks are marked with ``manual`` in the ``Description`` column in the table below. You can run
+Most of the checks we run are configured to run automatically when you commit the code or push PR. However,
+there are some checks that are not run automatically and you need to run them manually. You can run
 them manually by running ``prek --hook-stage manual <hook-id>``.
 
 Special pin-versions prek
@@ -288,11 +295,13 @@ manually by running:
 Mypy checks
 -----------
 
-When we run mypy checks locally when committing a change, one of the ``mypy-*`` checks is run, ``mypy-airflow``,
+When we run mypy checks locally when pushing a change to PR, the ``mypy-*`` checks is run, ``mypy-airflow``,
 ``mypy-dev``, ``mypy-providers``, ``mypy-airflow-ctl``, depending on the files you are changing. The mypy checks
 are run by passing those changed files to mypy. This is way faster than running checks for all files (even
 if mypy cache is used - especially when you change a file in Airflow core that is imported and used by many
-files). However, in some cases, it produces different results than when running checks for the whole set
+files). You also need to have ``breeze ci-image build --python 3.10`` built locally to run the mypy checks.
+
+However, in some cases, it produces different results than when running checks for the whole set
 of files, because ``mypy`` does not even know that some types are defined in other files and it might not
 be able to follow imports properly if they are dynamic. Therefore in CI we run ``mypy`` check for whole
 directories (``airflow`` - excluding providers, ``providers``, ``dev`` and ``docs``) to make sure
diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -1215,23 +1215,6 @@ def docs_list_as_string(self) -> str | None:
     def skip_prek_hooks(self) -> str:
         prek_hooks_to_skip = set()
         prek_hooks_to_skip.add("identity")
-        # Skip all mypy "individual" file checks if we are running mypy checks in CI
-        # In the CI we always run mypy for the whole "package" rather than for `--all-files` because
-        # The prek will semi-randomly skip such list of files into several groups and we want
-        # to make sure that such checks are always run in CI for whole "group" of files - i.e.
-        # whole package rather than for individual files. That's why we skip those checks in CI
-        # and run them via `mypy-all` command instead and dedicated CI job in matrix
-        # This will also speed up static-checks job usually as the jobs will be running in parallel
-        prek_hooks_to_skip.update(
-            {
-                "mypy-providers",
-                "mypy-airflow-core",
-                "mypy-dev",
-                "mypy-task-sdk",
-                "mypy-airflow-ctl",
-                "mypy-devel-common",
-            }
-        )
         if self._default_branch != "main":
             # Skip those tests on all "release" branches
             prek_hooks_to_skip.update(
diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
@@ -105,56 +105,40 @@
 )
 
 ALL_SKIPPED_COMMITS_ON_NO_CI_IMAGE = (
-    "check-provider-yaml-valid,flynt,identity,lint-helm-chart,mypy-airflow-core,mypy-airflow-ctl,"
-    "mypy-dev,mypy-devel-common,mypy-providers,mypy-task-sdk,"
+    "check-provider-yaml-valid,flynt,identity,lint-helm-chart,"
     "ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
 )
 
-ALL_SKIPPED_COMMITS_BY_DEFAULT_ON_ALL_TESTS_NEEDED = (
-    "identity,mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-providers,mypy-task-sdk"
-)
+ALL_SKIPPED_COMMITS_BY_DEFAULT_ON_ALL_TESTS_NEEDED = "identity"
 
-ALL_SKIPPED_COMMITS_IF_NO_UI = (
-    "identity,mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,"
-    "mypy-providers,mypy-task-sdk,ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
-)
-ALL_SKIPPED_COMMITS_IF_NO_HELM_TESTS = (
-    "identity,lint-helm-chart,mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,"
-    "mypy-providers,mypy-task-sdk"
-)
+ALL_SKIPPED_COMMITS_IF_NO_UI = "identity,ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
+ALL_SKIPPED_COMMITS_IF_NO_HELM_TESTS = "identity,lint-helm-chart"
 
 ALL_SKIPPED_COMMITS_IF_NO_UI_AND_HELM_TESTS = (
-    "identity,lint-helm-chart,mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,"
-    "mypy-providers,mypy-task-sdk,ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
+    "identity,lint-helm-chart,ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
 )
 
 ALL_SKIPPED_COMMITS_IF_NO_PROVIDERS_AND_UI = (
-    "check-provider-yaml-valid,identity,mypy-airflow-core,mypy-airflow-ctl,"
-    "mypy-dev,mypy-devel-common,mypy-providers,mypy-task-sdk,"
-    "ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
+    "check-provider-yaml-valid,identity,ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
 )
 
 ALL_SKIPPED_COMMITS_IF_NO_PROVIDERS = (
-    "check-provider-yaml-valid,identity,lint-helm-chart,mypy-airflow-core,mypy-airflow-ctl,"
-    "mypy-dev,mypy-devel-common,mypy-providers,mypy-task-sdk,"
+    "check-provider-yaml-valid,identity,lint-helm-chart,"
     "ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
 )
 
 
 ALL_SKIPPED_COMMITS_IF_NO_PROVIDERS_UI_AND_HELM_TESTS = (
-    "check-provider-yaml-valid,identity,lint-helm-chart,mypy-airflow-core,mypy-airflow-ctl,"
-    "mypy-dev,mypy-devel-common,mypy-providers,mypy-task-sdk,"
+    "check-provider-yaml-valid,identity,lint-helm-chart,"
     "ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
 )
 
 ALL_SKIPPED_COMMITS_IF_NO_CODE_PROVIDERS_AND_HELM_TESTS = (
-    "check-provider-yaml-valid,flynt,identity,lint-helm-chart,mypy-airflow-core,mypy-airflow-ctl,"
-    "mypy-dev,mypy-devel-common,mypy-providers,mypy-task-sdk"
+    "check-provider-yaml-valid,flynt,identity,lint-helm-chart"
 )
 
 ALL_SKIPPED_COMMITS_IF_NOT_IMPORTANT_FILES_CHANGED = (
-    "check-provider-yaml-valid,flynt,identity,lint-helm-chart,mypy-airflow-core,mypy-airflow-ctl,"
-    "mypy-dev,mypy-devel-common,mypy-providers,mypy-task-sdk,"
+    "check-provider-yaml-valid,flynt,identity,lint-helm-chart,"
     "ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui"
 )
 
@@ -163,8 +147,7 @@
     "check-airflow-provider-compatibility,check-airflow-providers-bug-report-template,"
     "check-extra-packages-references,check-provider-yaml-valid,"
     "compile-fab-assets,generate-openapi-spec-fab,identity,"
-    "lint-helm-chart,mypy-airflow-core,mypy-airflow-ctl,mypy-dev,"
-    "mypy-devel-common,mypy-providers,mypy-task-sdk,validate-operators-init"
+    "lint-helm-chart,validate-operators-init"
 )
 
 
@@ -1112,7 +1095,7 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
                 "run-unit-tests": "true",
                 "run-amazon-tests": "false",
                 "docs-build": "true",
-                "skip-prek-hooks": "check-provider-yaml-valid,flynt,identity,mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-providers,mypy-task-sdk,ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui",
+                "skip-prek-hooks": "check-provider-yaml-valid,flynt,identity,ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui",
                 "upgrade-to-newer-dependencies": "false",
                 "core-test-types-list-as-strings-in-json": None,
                 "providers-test-types-list-as-strings-in-json": None,
