Restore prek cache in a more robust way (apache#56796)

Apparently the prek cache mechanism has been somewhat broken for a
while - after we split prek to monorepo. The hash files used to
determine prek-cache was different for save and restore step
(the `**/` has been missing in the save cache step. Which means
that we always failed to restore cache and created it from the
scratch.

Also, it seems that the prek cache-when prepared refers to the uv
version that is pre-installed for it in case uv is not installed
in the system. And it refers to the uv version when creating the
virtual environments used by prek, and we first attempted to
install prek and create cache, and only after we installed uv, which
had a side-effect that in some cases the installed venvs referred
to a missing python binary.

Finally - there is a bug in prek j178/prek#918
that pygrep cache contains reference to a non-existing python binary
that should be run when pygrep runs.

Also it's possible that some of the cache installed in workspace by the
github worker remained, and we did not preemptively clean the cache when
we attempted to restore it and failed.

This PR attempts to restore the cache usage in a more robust way:

* fixed cache key on save to save cache with proper name
* added uv version to cache key for prek
* always install uv in desired version before installing prek
* if we faile to cache-hit and restore the cache, we clean-up
  the .cache/prek folder
* we do not look at skipped hooks when installing prek and restoring
  or saving cache. There is very little saving on some hooks and
  since we are preparing the cache in "build-info" now - it's better
  to always use the same cache, no matter if some checks are skipped
* upgraded to prek 0.2.10 that fixed the issue with pygrep cache

Author: potiuk

.github/actions/breeze/action.yml

@@ -22,6 +22,9 @@ inputs:
   python-version:
     description: 'Python version to use'
     default: "3.10"
+  uv-version:
+    description: 'uv version to use'
+    default: "0.9.3"  # Keep this comment to allow automatic replacement of uv version
 outputs:
   host-python-version:
     description: Python version used in host
@@ -33,6 +36,11 @@ runs:
       uses: actions/setup-python@v5
       with:
         python-version: ${{ inputs.python-version }}
+    - name: "Install uv"
+      shell: bash
+      run: curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | sh
+      env:
+        UV_VERSION: ${{ inputs.uv-version }}
     # NOTE! Installing Breeze without using cache is FASTER than when using cache - uv is so fast and has
     # so low overhead, that just running upload cache/restore cache is slower than installing it from scratch
     - name: "Install Breeze"

.github/actions/install-pre-commit/action.yml

@@ -27,10 +27,7 @@ inputs:
     default: "0.9.3"  # Keep this comment to allow automatic replacement of uv version
   prek-version:
     description: 'prek version to use'
-    default: "0.2.9"  # Keep this comment to allow automatic replacement of prek version
-  skip-prek-hooks:
-    description: "Skip some prek hooks from installation"
-    default: ""
+    default: "0.2.10"  # Keep this comment to allow automatic replacement of prek version
   save-cache:
     description: "Whether to save prek cache"
     required: true
@@ -40,14 +37,17 @@ inputs:
 runs:
   using: "composite"
   steps:
-    - name: Install uv and prek
+    - name: "Install uv"
+      shell: bash
+      run: curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | sh
+      env:
+        UV_VERSION: ${{ inputs.uv-version }}
+    - name: Install prek
       shell: bash
       env:
-        UV_VERSION: ${{inputs.uv-version}}
         PREK_VERSION: ${{inputs.prek-version}}
-        SKIP: ${{ inputs.skip-prek-hooks }}
+        UV_VERSION: ${{ inputs.uv-version }}
       run: |
-        curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | sh
         uv tool install prek==${PREK_VERSION} --with uv==${UV_VERSION}
       working-directory: ${{ github.workspace }}
     # We need to use tar file with archive to restore all the permissions and symlinks
@@ -64,7 +64,7 @@ runs:
       uses: apache/infrastructure-actions/stash/restore@1c35b5ccf8fba5d4c3fdf25a045ca91aa0cbc468
       with:
         # yamllint disable rule:line-length
-        key: cache-prek-v6-${{ inputs.platform }}-${{ inputs.python-version }}-${{inputs.skip-prek-hooks}}-${{ hashFiles('**/.pre-commit-config.yaml') }}
+        key: cache-prek-v8-${{ inputs.platform }}-python${{ inputs.python-version }}-uv${{ inputs.uv-version }}-${{ hashFiles('**/.pre-commit-config.yaml') }}
         path: /tmp/
       id: restore-prek-cache
     - name: "Check if prek cache tarball exists"
@@ -88,12 +88,28 @@ runs:
         echo
       shell: bash
       if: steps.restore-prek-cache.outputs.stash-hit == 'true'
+    - name: "Prepare local venv for pygrep"
+      # Prek cache restore seems to have a bug where removed temporary python is
+      # used in cached pygrep installation. It seems prek can fallback to uv sync installed python
+      # So let's install it. See https://github.com/j178/prek/issues/918:
+      run: |
+        uv sync --no-dev --no-install-workspace
+      shell: bash
+      if: steps.restore-prek-cache.outputs.stash-hit == 'true'
+    - name: "Make sure cache is cleared on cache miss"
+      run: |
+        echo "Cleaning up prek cache in case of cache miss (in case of pre-installed-cache from the system)"
+        ls -la ~/.cache/prek || true
+        rm -rf ~/.cache/prek
+      shell: bash
+      if: steps.restore-prek-cache.outputs.stash-hit != 'true'
     - name: Install prek hooks
       shell: bash
-      run: prek install-hooks || (cat ~/.cache/prek/prek.log && exit 1)
+      run: prek install-hooks
       working-directory: ${{ github.workspace }}
-      env:
-        SKIP: ${{ inputs.skip-prek-hooks }}
+    - name: "Show prek log"
+      shell: bash
+      run: cat ~/.cache/prek/prek.log || true
     - name: "Prepare .tar file from prek cache"
       run: |
         tar -C ~ -czf /tmp/cache-prek.tar.gz .cache/prek
@@ -103,7 +119,7 @@ runs:
       uses: apache/infrastructure-actions/stash/save@1c35b5ccf8fba5d4c3fdf25a045ca91aa0cbc468
       with:
         # yamllint disable rule:line-length
-        key: cache-prek-v6-${{ inputs.platform }}-${{ inputs.python-version }}-${{ inputs.skip-prek-hooks }}-${{ hashFiles('.pre-commit-config.yaml') }}
+        key: cache-prek-v8-${{ inputs.platform }}-python${{ inputs.python-version }}-uv${{ inputs.uv-version }}-${{ hashFiles('**/.pre-commit-config.yaml') }}
         path: /tmp/cache-prek.tar.gz
         if-no-files-found: 'error'
         retention-days: '2'

.github/actions/install-prek/action.yml

@@ -100,8 +100,6 @@ jobs:
       matrix:
         shared-distribution: ${{ fromJSON(inputs.shared-distributions-as-json) }}
     runs-on: ${{ fromJSON(inputs.runners) }}
-    env:
-      UV_VERSION: ${{inputs.uv-version}}
     steps:
       - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
         uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2
@@ -110,6 +108,8 @@ jobs:
           persist-credentials: false
       - name: "Install uv"
         run: curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | sh
+        env:
+          UV_VERSION: ${{ inputs.uv-version }}
       - name: "Run shared ${{ matrix.shared-distribution }} tests"
         run: uv run --group dev pytest --color=yes -n auto
         working-directory: shared/${{ matrix.shared-distribution }}
@@ -193,7 +193,6 @@ jobs:
         id: prek
         with:
           python-version: ${{steps.breeze.outputs.host-python-version}}
-          skip-prek-hooks: ${{ inputs.skip-prek-hooks }}
           platform: ${{ inputs.platform }}
           save-cache: false
       - name: "Check translation completeness"
@@ -229,7 +228,6 @@ jobs:
         id: prek
         with:
           python-version: ${{ steps.breeze.outputs.host-python-version }}
-          skip-prek-hooks: ${{ inputs.skip-prek-hooks }}
           platform: ${{ inputs.platform }}
           save-cache: false
       - name: Fetch incoming commit ${{ github.sha }} with its parent
@@ -282,7 +280,6 @@ jobs:
         id: prek
         with:
           python-version: ${{ steps.breeze.outputs.host-python-version }}
-          skip-prek-hooks: ${{ inputs.skip-prek-hooks }}
           platform: ${{ inputs.platform }}
           save-cache: false
       - name: "Autoupdate all prek hooks"
@@ -404,7 +401,6 @@ jobs:
     name: "Test Airflow standalone commands"
     runs-on: ${{ fromJSON(inputs.runners) }}
     env:
-      UV_VERSION: ${{inputs.uv-version}}
       AIRFLOW_HOME: ~/airflow
       FORCE_COLOR: 1
     steps:
@@ -414,6 +410,8 @@ jobs:
           persist-credentials: false
       - name: "Install uv"
         run: curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | sh
+        env:
+          UV_VERSION: ${{ inputs.uv-version }}
       - name: "Set up Airflow home directory"
         run: |
           echo "Setting AIRFLOW_HOME to $AIRFLOW_HOME"

.github/workflows/basic-tests.yml

@@ -170,7 +170,6 @@ jobs:
         id: prek
         with:
           python-version: ${{ steps.breeze.outputs.host-python-version }}
-          skip-prek-hooks: ${{ needs.build-info.outputs.skip-prek-hooks }}
           platform: "linux/amd64"
           save-cache: true
   run-pin-versions-hook:
@@ -190,7 +189,6 @@ jobs:
           python-version: "3.11"
           platform: "linux/amd64"
           save-cache: true
-          skip-prek-hooks: ""
       - name: "Run pin-versions"
         run: >
           prek -c dev/.pre-commit-config.yaml --all-files --verbose --hook-stage manual

.github/workflows/ci-amd.yml

@@ -162,7 +162,6 @@ jobs:
         id: prek
         with:
           python-version: ${{ steps.breeze.outputs.host-python-version }}
-          skip-prek-hooks: ${{ needs.build-info.outputs.skip-prek-hooks }}
           platform: "linux/arm64"
           save-cache: true
   basic-tests:

.github/workflows/ci-arm.yml

@@ -155,17 +155,17 @@ jobs:
           platform: ${{ inputs.platform }}
           save-cache: false
       - name: "Static checks"
-        # We have added cache cleaning here as otherwise it was failing on rst-backticks hook
-        # This increases the time of this step from ~9 minutes 25 seconds to ~12 minutes 37 seconds
-        # If we want to remove the first part, we need to find root cause of the problem
-        run: prek cache clean && prek --all-files --show-diff-on-failure --color always
+        run: prek --all-files --show-diff-on-failure --color always
         env:
           VERBOSE: "false"
           SKIP: ${{ inputs.skip-prek-hooks }}
           COLUMNS: "202"
           SKIP_GROUP_OUTPUT: "true"
           DEFAULT_BRANCH: ${{ inputs.branch }}
           RUFF_FORMAT: "github"
+      - name: "Show prek log on failure"
+        run: cat ~/.cache/prek/prek.log || true
+        if: failure()
 
   mypy:
     timeout-minutes: 45

.github/workflows/ci-image-checks.yml

@@ -87,19 +87,19 @@ jobs:
         uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2
         with:
           persist-credentials: false
-      - name: "Install uv"
-        run: curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | sh
-      - name: "Check airflow version"
-        id: check-airflow-version
-        shell: bash
-        run: uv run scripts/ci/airflow_version_check.py "${AIRFLOW_VERSION}" >> "${GITHUB_OUTPUT}"
       - name: "Install Breeze"
         uses: ./.github/actions/breeze
+        with:
+          uv-version: ${{ env.UV_VERSION }}
       - name: Selective checks
         id: selective-checks
         env:
           VERBOSE: "false"
         run: breeze ci selective-check 2>> ${GITHUB_OUTPUT}
+      - name: "Check airflow version"
+        id: check-airflow-version
+        shell: bash
+        run: uv run scripts/ci/airflow_version_check.py "${AIRFLOW_VERSION}" >> "${GITHUB_OUTPUT}"
       - name: "Determine build matrix"
         shell: bash
         id: determine-matrix

.github/workflows/release_dockerhub_image.yml

@@ -1678,7 +1678,7 @@ COPY --from=scripts common.sh install_packaging_tools.sh install_additional_depe
 ARG AIRFLOW_PIP_VERSION=25.2
 # ARG AIRFLOW_PIP_VERSION="git+https://github.com/pypa/pip.git@main"
 ARG AIRFLOW_UV_VERSION=0.9.3
-ARG AIRFLOW_PREK_VERSION="0.2.9"
+ARG AIRFLOW_PREK_VERSION="0.2.10"
 
 # UV_LINK_MODE=copy is needed since we are using cache mounted from the host
 ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \

Dockerfile.ci

@@ -135,6 +135,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PREK.
 
 ---------------------------------------------------------------------------------------------------------
 
-Package config hash: eeecbf0df75ded82558188c801a214f66c89da758f4bf58fd6381234bcd222fe9b53154e10331c6975f5521377d93f8f3a8f39e3d76e6874510df80511096a00
+Package config hash: 809191a413ca650c171f5c64561742bf391cf170207308be2605c063afd115f5ae5ece39da6f1d193a990009bcb1b54abb8598f17dae9733cedfd9baa8aa6fbb
 
 ---------------------------------------------------------------------------------------------------------