Merge branch 'ci/fix-pipeline-commit-sha-in-build-docker' into 'master'

Ci/fix pipeline commit sha in build docker

Closes IDFCI-3918

See merge request espressif/esp-idf!42381

Author: hfudev

.gitlab/ci/build.yml

@@ -284,17 +284,11 @@ build_docker:
   stage: host_test
   needs: []
   image: espressif/docker-builder:1
-  tags:
-    - build_docker_amd64_brno
+  tags: [shiny, dind]
   variables:
     DOCKER_TMP_IMAGE_NAME: "idf_tmp_image"
   script:
-    - export LOCAL_CI_REPOSITORY_URL=$CI_REPOSITORY_URL
-    - if [ -n "$LOCAL_GITLAB_HTTPS_HOST" ]; then export LOCAL_CI_REPOSITORY_URL="https://gitlab-ci-token:${CI_JOB_TOKEN}@${LOCAL_GITLAB_HTTPS_HOST}/${CI_PROJECT_PATH}"; fi
-    - if [ -n "$LOCAL_GIT_MIRROR" ]; then export LOCAL_CI_REPOSITORY_URL="${LOCAL_GIT_MIRROR}/${CI_PROJECT_PATH}"; fi
-    - echo "Using repository at $LOCAL_CI_REPOSITORY_URL"
-    - export DOCKER_BUILD_ARGS="--build-arg IDF_CLONE_URL=${LOCAL_CI_REPOSITORY_URL} --build-arg IDF_CLONE_BRANCH_OR_TAG=${CI_COMMIT_REF_NAME} --build-arg IDF_CHECKOUT_REF=${CI_COMMIT_TAG:-$PIPELINE_COMMIT_SHA}"
-    # Build
+    - export DOCKER_BUILD_ARGS="--build-arg IDF_CLONE_URL=${CI_REPOSITORY_URL} --build-arg IDF_CLONE_BRANCH_OR_TAG=${CI_COMMIT_REF_NAME} --build-arg IDF_CHECKOUT_REF=${CI_COMMIT_TAG:-$CI_COMMIT_SHA} --build-arg IDF_CLONE_SHALLOW=1 --build-arg IDF_GITHUB_ASSETS=${INTERNAL_GITHUB_ASSETS}"
     - docker build --tag ${DOCKER_TMP_IMAGE_NAME} ${DOCKER_BUILD_ARGS} tools/docker/
     # We can't mount $PWD/examples/get-started/blink into the container, see https://gitlab.com/gitlab-org/gitlab-ce/issues/41227.
     # The workaround mentioned there works, but leaves around directories which need to be cleaned up manually.

.gitlab/ci/pre_check.yml

@@ -142,6 +142,10 @@ pipeline_variables:
         MR_MODIFIED_COMPONENTS=$(run_cmd python tools/ci/ci_get_mr_info.py components --modified-files $MODIFIED_FILES | tr '\n' ';')
         echo "MR_MODIFIED_COMPONENTS=\"$MR_MODIFIED_COMPONENTS\"" >> pipeline.env
       fi
+    - |
+      if [ -n "$PIPELINE_COMMIT_SHA" ]; then
+        echo "PIPELINE_COMMIT_SHA=$PIPELINE_COMMIT_SHA" >> pipeline.env
+      fi
     - echo "OOCD_DISTRO_URL_ARMHF=$OOCD_DISTRO_URL_ARMHF" >> pipeline.env
     - echo "OOCD_DISTRO_URL_ARM64=$OOCD_DISTRO_URL_ARM64" >> pipeline.env
     - run_cmd idf-ci gitlab pipeline-variables >> pipeline.env

docs/en/api-guides/tools/idf-docker-image.rst

@@ -152,6 +152,7 @@ The Docker file in ESP-IDF repository provides several build arguments which can
 - ``IDF_CLONE_SHALLOW``: If this argument is set to a non-empty value, ``--depth=1 --shallow-submodules`` arguments are used when performing ``git clone``. Depth can be customized using ``IDF_CLONE_SHALLOW_DEPTH``. Doing a shallow clone significantly reduces the amount of data downloaded and the size of the resulting Docker image. However, if switching to a different branch in such a "shallow" repository is necessary, an additional ``git fetch origin <branch>`` command must be executed first.
 - ``IDF_CLONE_SHALLOW_DEPTH``: This argument specifies the depth value to use when doing a shallow clone. If not set, ``--depth=1`` will be used. This argument has effect only if ``IDF_CLONE_SHALLOW`` is used. Use this argument if you are building a Docker image for a branch, and the image has to contain the latest tag on that branch. To determine the required depth, run ``git describe`` for the given branch and note the offset number. Increment it by 1, then use it as the value of this argument. The resulting image will contain the latest tag on the branch, and consequently ``git describe`` command inside the Docker image will work as expected.
 - ``IDF_INSTALL_TARGETS``: Comma-separated list of ESP-IDF targets to install toolchains for, or ``all`` to install toolchains for all targets. Selecting specific targets reduces the amount of data downloaded and the size of the resulting Docker image. The default is ``all``.
+- ``IDF_GITHUB_ASSETS``: Hostname used to download ESP-IDF tools from a mirror server instead of directly from ``github.com``. This is useful when ``github.com`` is not accessible from your network. The default value is empty.
 
 To use these arguments, pass them via the ``--build-arg`` command line option. For example, the following command builds a Docker image with a shallow clone of ESP-IDF v4.4.1 and tools for ESP32-C3 only:
 

docs/zh_CN/api-guides/tools/idf-docker-image.rst

@@ -152,6 +152,7 @@ ESP-IDF 库中的 Docker 文件提供了以下构建参数，可用于构建自
 - ``IDF_CLONE_SHALLOW``：如果将此参数设置为非空值，则会在执行 ``git clone`` 时使用 ``--depth=1 --shallow-submodules`` 参数。浅克隆的深度可以使用 ``IDF_CLONE_SHALLOW_DEPTH`` 设置。浅克隆可以极大减少下载的数据量及生成的 Docker 镜像大小。然而，如果需要切换到此类“浅层”存储库中的其他分支，必须先执行额外的 ``git fetch origin <branch>`` 命令。
 - ``IDF_CLONE_SHALLOW_DEPTH``：此参数指定进行浅克隆时要使用的深度值。如未设置，将使用 ``--depth=1``。此参数仅在使用 ``IDF_CLONE_SHALLOW`` 时有效。如果要为分支构建 Docker 镜像，并且该镜像必须包含该分支上的最新标签，则需使用此参数。要确定所需的深度，请在特定的分支运行 ``git describe`` 命令，并注意偏移值。将偏移值加 1 后即可将其用作 ``IDF_CLONE_SHALLOW_DEPTH`` 参数的值。此过程将确保生成的镜像包含分支上的最新标签，且 Docker 镜像内部的 ``git describe`` 命令也会按预期工作。
 - ``IDF_INSTALL_TARGETS``：以逗号分隔的 ESP-IDF 目标列表，用于安装工具链，或者使用 ``all`` 安装所有目标的工具链。选择特定目标可以减少下载的数据量和生成的 Docker 镜像的大小。该参数默认值为 ``all``。
+- ``IDF_GITHUB_ASSETS``：如果将此参数设置为非空值，则会从设置的镜像站点下载 ESP-IDF 工具，而不是从 ``github.com`` 下载。这样可以在网络无法访问 ``github.com`` 时使用镜像站点。该参数的默认值为空。
 
 要使用以上参数，请通过 ``--build-arg`` 命令行选项传递。例如，以下命令使用 ESP-IDF v4.4.1 的浅克隆以及仅适用于 ESP32-C3 的工具链构建了 Docker 镜像：
 

tools/docker/Dockerfile

@@ -54,9 +54,11 @@ ARG IDF_CHECKOUT_REF=
 ARG IDF_CLONE_SHALLOW=
 ARG IDF_CLONE_SHALLOW_DEPTH=1
 ARG IDF_INSTALL_TARGETS=all
+ARG IDF_GITHUB_ASSETS=
 
 ENV IDF_PATH=/opt/esp/idf
 ENV IDF_TOOLS_PATH=/opt/esp
+ENV IDF_GITHUB_ASSETS=${IDF_GITHUB_ASSETS}
 
 # install build essential needed for linux target apps, which is a preview target so it is installed with "all" only
 RUN if [ "$IDF_INSTALL_TARGETS" = "all" ]; then \