Merge branch 'feat/core_components' into 'master'

feat: add `COMPONENT_SOURCE` property to component targets

Closes DOC-7429

See merge request espressif/esp-idf!26586

Author: hfudev

docs/en/api-guides/build-system.rst

@@ -307,7 +307,14 @@ When CMake runs to configure the project, it logs the components included in the
 Multiple Components with the Same Name
 --------------------------------------
 
-When ESP-IDF is collecting all the components to compile, it will do this in the order specified by ``COMPONENT_DIRS``; by default, this means ESP-IDF's internal components first (``IDF_PATH/components``), then any components in directories specified in ``EXTRA_COMPONENT_DIRS``, and finally the project's components (``PROJECT_DIR/components``). If two or more of these directories contain component sub-directories with the same name, the component in the last place searched is used. This allows, for example, overriding ESP-IDF components with a modified version by copying that component from the ESP-IDF components directory to the project components directory and then modifying it there. If used in this way, the ESP-IDF directory itself can remain untouched.
+When ESP-IDF is collecting all the components to compile, the search precedence is as follows (from highest to lowest):
+
+* Project components
+* Components from ``EXTRA_COMPONENT_DIRS``
+* Project managed components, downloaded by the IDF Component Manager into ``PROJECT_DIR/managed_components``, unless the IDF Component Manager is disabled.
+* ESP-IDF components (``IDF_PATH/components``)
+
+If two or more of these directories contain component sub-directories with the same name, the component with higher precedence is used. This allows, for example, overriding ESP-IDF components with a modified version by copying that component from the ESP-IDF components directory to the project components directory and then modifying it there. If used in this way, the ESP-IDF directory itself can remain untouched.
 
 .. note::
 
@@ -1257,12 +1264,26 @@ Set a :ref:`build property <cmake-build-properties>` *property* with value *val*
 
 .. code-block:: none
 
-  idf_build_component(component_dir)
+  idf_build_component(component_dir [component_source])
 
 Present a directory *component_dir* that contains a component to the build system. Relative paths are converted to absolute paths with respect to current directory.
-All calls to this command must be performed before `idf_build_process`.
 
-This command does not guarantee that the component will be processed during build (see the `COMPONENTS` argument description for `idf_build_process`)
+An optional *component_source* argument can be specified to indicate the source of the component. (default: "project_components")
+
+This argument determines the overriding priority for components with the same name. For detailed information, see :ref:`cmake-components-same-name`.
+
+This argument supports the following values (from highest to lowest priority):
+
+- "project_components" - project components
+- "project_extra_components" - components from ``EXTRA_COMPONENT_DIRS``
+- "project_managed_components" - custom project dependencies managed by the IDF Component Manager
+- "idf_components" - ESP-IDF built-in components, typically under :idf:`/components`
+
+For instance, if a component named "json" is present as both "idf_components", and "project_components", the component as "project_components" takes precedence over the one as "idf_components".
+
+.. warning::
+
+    All calls to this command must be performed before `idf_build_process`. This command does not guarantee that the component will be processed during build (see the `COMPONENTS` argument description for `idf_build_process`).
 
 .. code-block:: none
 
@@ -1422,6 +1443,7 @@ These are properties that describe a component. Values of component properties c
 - COMPONENT_LIB - name for created component static/interface library; set by ``idf_build_component`` and library itself is created by ``idf_component_register``
 - COMPONENT_NAME - name of the component; set by ``idf_build_component`` based on the component directory name
 - COMPONENT_TYPE - type of the component, whether LIBRARY or CONFIG_ONLY. A component is of type LIBRARY if it specifies source files or embeds a file
+- COMPONENT_SOURCE - source of the component, one of "idf_components", "project_managed_components", "project_components", "project_extra_components". This is used to determine the override precedence of components with the same name.
 - EMBED_FILES - list of files to embed in component; set from ``idf_component_register`` EMBED_FILES argument
 - EMBED_TXTFILES - list of text files to embed in component; set from ``idf_component_register`` EMBED_TXTFILES argument
 - INCLUDE_DIRS - list of component include directories; set from ``idf_component_register`` INCLUDE_DIRS argument

docs/zh_CN/api-guides/build-system.rst

@@ -307,9 +307,16 @@ ESP-IDF 适用于 Python 3.8 以上版本。
 同名组件
 --------
 
-ESP-IDF 在搜索所有待构建的组件时，会按照 ``COMPONENT_DIRS`` 指定的顺序依次进行，这意味着在默认情况下，首先搜索 ESP-IDF 内部组件（``IDF_PATH/components``），然后是 ``EXTRA_COMPONENT_DIRS`` 中的组件，最后是项目组件（``PROJECT_DIR/components``）。如果这些目录中的两个或者多个包含具有相同名字的组件，则使用搜索到的最后一个位置的组件。这就允许将组件复制到项目目录中再修改以覆盖 ESP-IDF 组件，如果使用这种方式，ESP-IDF 目录本身可以保持不变。
+ESP-IDF 在搜索所有待构建的组件时，会按照以下优先级搜索组件目录（从高到低）：
 
-.. 注解::
+* 项目目录下的组件
+* ``EXTRA_COMPONENT_DIRS`` 中的组件
+* 项目目录下 ``managed_components`` 目录中的组件。这些组件由 IDF Component Manager 下载并管理。（除非 IDF Component Manager 被禁用）
+* ``IDF_PATH/components`` 目录下的组件
+
+如果有两个及以上同名组件，构建系统会使用优先级更高的组件。这使得我们可以在项目中覆盖 ESP-IDF 提供的组件。只需要复制 ESP-IDF 组件到项目目录下，然后修改它。这样可以在修改组件的同时，不修改 ESP-IDF 的源代码。
+
+.. note::
 
   如果在现有项目中通过将组件移动到一个新位置来覆盖它，项目不会自动看到新组件的路径。请运行 ``idf.py reconfigure`` 命令后（或删除项目构建文件夹）再重新构建。
 
@@ -1257,23 +1264,37 @@ ESP-IDF 构建命令
 
 .. code-block:: none
 
-  idf_build_component(component_dir)
+  idf_build_component(component_dir [component_source])
 
 向构建系统提交一个包含组件的 *component_dir* 目录。相对路径会被转换为相对于当前目录的绝对路径。
-所有对该命令的调用必须在`idf_build_process`之前执行。
 
-该命令并不保证组件在构建过程中会被处理（参见 `idf_build_process` 中 `COMPONENTS` 参数说明）
+一个可选的 *component_source* 参数可以用于指定组件源。（默认为 "project_components"）
+
+这个参数决定了同名组件的优先级。详细信息请参考 :ref:`cmake-components-same-name`。
+
+该参数可以指定如下组件源（优先级从高到低排序）：
+
+- "project_components" - 项目目录中的组件
+- "project_extra_components" - 通过 ``EXTRA_COMPONENT_DIRS`` 指定的额外组件
+- "project_managed_components" - 通过 IDF Component Manager 管理的组件
+- "idf_components" - ESP-IDF 中的组件，通常在 :idf:`/components` 目录中
+
+举个例子，如果有两个组件，组件名都为 "json"。一个组件源被定义为 "project_components"，另一个组件源被定义为 "idf_components"，那么 "project_components" 中的 "json" 组件会被优先选择。
+
+.. warning::
+
+    所有对该命令的调用必须在 `idf_build_process` 之前执行。该命令并不保证组件在构建过程中会被处理（参见 `idf_build_process` 中 `COMPONENTS` 参数说明）。
 
 .. code-block:: none
 
-  idf_build_process(target
-                    [PROJECT_DIR project_dir]
-                    [PROJECT_VER project_ver]
-                    [PROJECT_NAME project_name]
-                    [SDKCONFIG sdkconfig]
-                    [SDKCONFIG_DEFAULTS sdkconfig_defaults]
-                    [BUILD_DIR build_dir]
-                    [COMPONENTS component1 component2 ...])
+    idf_build_process(target
+                      [PROJECT_DIR project_dir]
+                      [PROJECT_VER project_ver]
+                      [PROJECT_NAME project_name]
+                      [SDKCONFIG sdkconfig]
+                      [SDKCONFIG_DEFAULTS sdkconfig_defaults]
+                      [BUILD_DIR build_dir]
+                      [COMPONENTS component1 component2 ...])
 
 为导入 ESP-IDF 组件执行大量的幕后工作，包括组件配置、库创建、依赖性扩展和解析。在这些功能中，对于用户最重要的可能是通过调用每个组件的 ``idf_component_register`` 来创建库。该命令为每个组件创建库，这些库可以使用别名来访问，其形式为 idf::*component_name*。
 这些别名可以用来将组件链接到用户自己的目标、库或可执行文件上。
@@ -1422,6 +1443,7 @@ ESP-IDF 组件属性
 - COMPONENT_LIB - 所创建的组件静态/接口库的名称；由 ``idf_build_component`` 设置，库本身由 ``idf_component_register`` 创建。
 - COMPONENT_NAME - 组件的名称；由 ``idf_build_component`` 根据组件的目录名设置。
 - COMPONENT_TYPE - 组件的类型（LIBRARY 或 CONFIG_ONLY）。如果一个组件指定了源文件或嵌入了一个文件，那么它的类型就是 LIBRARY。
+- COMPONENT_SOURCE - 组件源。可选值为 "idf_components"，"project_managed_components"，"project_components"，"project_extra_components". 用于决定同名组件的优先级。
 - EMBED_FILES - 要嵌入组件的文件列表；由 ``idf_component_register`` EMBED_FILES 参数设置。
 - EMBED_TXTFILES - 要嵌入组件的文本文件列表；由 ``idf_component_register`` EMBED_TXTFILES 参数设置。
 - INCLUDE_DIRS - 组件 include 目录列表；由 ``idf_component_register`` INCLUDE_DIRS 参数设置。

tools/cmake/build.cmake

@@ -253,7 +253,7 @@ function(__build_init idf_path)
         if(IS_DIRECTORY ${component_dir})
             __component_dir_quick_check(is_component ${component_dir})
             if(is_component)
-                __component_add(${component_dir} ${prefix})
+                __component_add(${component_dir} ${prefix} "idf_components")
             endif()
         endif()
     endforeach()
@@ -283,9 +283,29 @@ endfunction()
 #        during build (see the COMPONENTS argument description for command idf_build_process)
 #
 # @param[in] component_dir directory of the component
+# @param[in, optional] component_source source of the component, defaults to "project_components"
 function(idf_build_component component_dir)
     idf_build_get_property(prefix __PREFIX)
-    __component_add(${component_dir} ${prefix} 0)
+
+    # if argvc is 1, then component_source is not specified
+    # this should only happen when users call this function directly
+    if(${ARGC} EQUAL 1)
+        set(component_source "project_components")
+    else()
+        set(component_source ${ARGV1})
+    endif()
+
+    # component_source must be one of the following (sorted by the override order):
+    set(valid_component_sources "idf_components"
+                              "project_managed_components"
+                              "project_extra_components"
+                              "project_components")
+
+    if(NOT component_source IN_LIST valid_component_sources)
+        message(FATAL_ERROR "Invalid component source '${component_source}'.")
+    endif()
+
+    __component_add(${component_dir} ${prefix} ${component_source})
 endfunction()
 
 #

tools/cmake/component.cmake

@@ -138,7 +138,7 @@ endfunction()
 # Add a component to process in the build. The components are keeped tracked of in property
 # __COMPONENT_TARGETS in component target form.
 #
-function(__component_add component_dir prefix)
+function(__component_add component_dir prefix component_source)
     # For each component, two entities are created: a component target and a component library. The
     # component library is created during component registration (the actual static/interface library).
     # On the other hand, component targets are created early in the build
@@ -186,6 +186,9 @@ function(__component_add component_dir prefix)
     __component_set_property(${component_target} COMPONENT_NAME ${component_name})
     __component_set_property(${component_target} COMPONENT_DIR ${component_dir})
     __component_set_property(${component_target} COMPONENT_ALIAS ${component_alias})
+    if(component_source)
+        __component_set_property(${component_target} COMPONENT_SOURCE ${component_source})
+    endif()
 
     __component_set_property(${component_target} __PREFIX ${prefix})
 

tools/cmake/project.cmake

@@ -61,7 +61,7 @@ if(NOT "$ENV{IDF_COMPONENT_MANAGER}" EQUAL "0")
     idf_build_set_property(IDF_COMPONENT_MANAGER 1)
 endif()
 # Set component manager interface version
-idf_build_set_property(__COMPONENT_MANAGER_INTERFACE_VERSION 2)
+idf_build_set_property(__COMPONENT_MANAGER_INTERFACE_VERSION 3)
 
 #
 # Parse and store the VERSION argument provided to the project() command.
@@ -420,11 +420,11 @@ function(__project_init components_var test_components_var)
     idf_build_set_property(CXX_COMPILE_OPTIONS "${extra_cxxflags}" APPEND)
     idf_build_set_property(COMPILE_OPTIONS "${extra_cppflags}" APPEND)
 
-    function(__project_component_dir component_dir)
+    function(__project_component_dir component_dir component_source)
         get_filename_component(component_dir "${component_dir}" ABSOLUTE)
         # The directory itself is a valid idf component
         if(EXISTS ${component_dir}/CMakeLists.txt)
-            idf_build_component(${component_dir})
+            idf_build_component(${component_dir} ${component_source})
         else()
             idf_build_get_property(exclude_dirs EXTRA_COMPONENT_EXCLUDE_DIRS)
             # otherwise, check whether the subfolders are potential idf components
@@ -433,7 +433,7 @@ function(__project_init components_var test_components_var)
                 if(IS_DIRECTORY ${component_dir} AND NOT ${component_dir} IN_LIST exclude_dirs)
                     __component_dir_quick_check(is_component ${component_dir})
                     if(is_component)
-                        idf_build_component(${component_dir})
+                        idf_build_component(${component_dir} ${component_source})
                     endif()
                 endif()
             endforeach()
@@ -451,11 +451,11 @@ function(__project_init components_var test_components_var)
             if(NOT EXISTS ${component_abs_path})
                 message(FATAL_ERROR "Directory specified in COMPONENT_DIRS doesn't exist: ${component_abs_path}")
             endif()
-            __project_component_dir(${component_dir})
+            __project_component_dir(${component_dir} "project_components")
         endforeach()
     else()
         if(EXISTS "${CMAKE_CURRENT_LIST_DIR}/main")
-            __project_component_dir("${CMAKE_CURRENT_LIST_DIR}/main")
+            __project_component_dir("${CMAKE_CURRENT_LIST_DIR}/main" "project_components")
         endif()
 
         paths_with_spaces_to_list(EXTRA_COMPONENT_DIRS)
@@ -464,12 +464,12 @@ function(__project_init components_var test_components_var)
             if(NOT EXISTS ${component_abs_path})
                 message(FATAL_ERROR "Directory specified in EXTRA_COMPONENT_DIRS doesn't exist: ${component_abs_path}")
             endif()
-            __project_component_dir("${component_dir}")
+            __project_component_dir("${component_dir}" "project_extra_components")
         endforeach()
 
         # Look for components in the usual places: CMAKE_CURRENT_LIST_DIR/main,
         # extra component dirs, and CMAKE_CURRENT_LIST_DIR/components
-        __project_component_dir("${CMAKE_CURRENT_LIST_DIR}/components")
+        __project_component_dir("${CMAKE_CURRENT_LIST_DIR}/components" "project_components")
     endif()
 
     # For bootloader components, we only need to set-up the Kconfig files.
@@ -521,7 +521,7 @@ function(__project_init components_var test_components_var)
                     set(include 0)
                 endif()
                 if(include AND EXISTS ${component_dir}/test)
-                    __component_add(${component_dir}/test ${component_name})
+                    __component_add(${component_dir}/test ${component_name} "project_components")
                     list(APPEND test_components ${component_name}::test)
                 endif()
             endif()

tools/cmake/scripts/component_get_requirements.cmake

@@ -27,7 +27,7 @@ endfunction()
 function(__component_get_target var name_or_alias)
     idf_build_get_property(component_targets __COMPONENT_TARGETS)
 
-    # Assume first that the paramters is an alias.
+    # Assume first that the parameters is an alias.
     string(REPLACE "::" "_" name_or_alias "${name_or_alias}")
     set(component_target ___${name_or_alias})
 
@@ -119,7 +119,40 @@ function(__component_get_requirements)
 endfunction()
 
 set(CMAKE_BUILD_EARLY_EXPANSION 1)
+
+# smaller number means lower priority
+set(__TARGETS_IDF_COMPONENTS "")                # 0
+set(__TARGETS_PROJECT_MANAGED_COMPONENTS "")    # 1
+set(__TARGETS_PROJECT_EXTRA_COMPONENTS "")      # 2
+set(__TARGETS_PROJECT_COMPONENTS "")            # 3
+
 foreach(__component_target ${__component_targets})
+    __component_get_property(__component_source ${__component_target} COMPONENT_SOURCE)
+
+    if("${__component_source}" STREQUAL "idf_components")
+        list(APPEND __TARGETS_IDF_COMPONENTS ${__component_target})
+    elseif("${__component_source}" STREQUAL "project_managed_components")
+        list(APPEND __TARGETS_PROJECT_MANAGED_COMPONENTS ${__component_target})
+    elseif("${__component_source}" STREQUAL "project_extra_components")
+        list(APPEND __TARGETS_PROJECT_EXTRA_COMPONENTS ${__component_target})
+    elseif("${__component_source}" STREQUAL "project_components")
+        list(APPEND __TARGETS_PROJECT_COMPONENTS ${__component_target})
+    else()
+        message(FATAL_ERROR "Unknown component source for component ${__component_target}: ${__component_source}")
+    endif()
+endforeach()
+
+# priority higher ones goes first
+set(__sorted_component_targets "")
+foreach(__target IN LISTS __TARGETS_PROJECT_COMPONENTS
+                          __TARGETS_PROJECT_EXTRA_COMPONENTS
+                          __TARGETS_PROJECT_MANAGED_COMPONENTS
+                          __TARGETS_IDF_COMPONENTS)
+    __component_get_property(__component_name ${__target} COMPONENT_NAME)
+    list(APPEND __sorted_component_targets ${__target})
+endforeach()
+
+foreach(__component_target ${__sorted_component_targets})
     set(__component_requires "")
     set(__component_priv_requires "")
     set(__component_registered 0)
@@ -141,11 +174,14 @@ foreach(__component_target ${__component_targets})
         list(REMOVE_ITEM __component_priv_requires ${__component_alias} ${__component_name})
     endif()
 
+    __component_get_property(__component_source ${__component_target} COMPONENT_SOURCE)
+
     set(__contents
 "__component_set_property(${__component_target} REQUIRES \"${__component_requires}\")
 __component_set_property(${__component_target} PRIV_REQUIRES \"${__component_priv_requires}\")
 __component_set_property(${__component_target} __COMPONENT_REGISTERED ${__component_registered})
-__component_set_property(${__component_target} INCLUDE_DIRS \"${__component_include_dirs}\")"
+__component_set_property(${__component_target} INCLUDE_DIRS \"${__component_include_dirs}\")
+__component_set_property(${__component_target} __COMPONENT_SOURCE \"${__component_source}\")"
     )
 
     if(__component_kconfig)

tools/test_build_system/conftest.py

@@ -51,7 +51,7 @@ def _session_work_dir(request: FixtureRequest) -> typing.Generator[typing.Tuple[
     work_dir = request.config.getoption('--work-dir')
 
     if work_dir:
-        work_dir = os.path.join(work_dir, datetime.datetime.utcnow().strftime('%Y-%m-%d_%H-%M-%S'))
+        work_dir = os.path.join(work_dir, datetime.datetime.now(datetime.timezone.utc).strftime('%Y-%m-%d_%H-%M-%S'))
         logging.debug(f'using work directory: {work_dir}')
         os.makedirs(work_dir, exist_ok=True)
         clean_dir = None