Merge branch 'feat/load_idf_ext_from_components' into 'master'

feat(tools): Load idf.py extensions from project's component directories and python packages

Closes IDF-5736 and IDF-4121

See merge request espressif/esp-idf!39875

Author: dobairoland

docs/en/api-guides/tools/idf-py.rst

@@ -301,7 +301,112 @@ Arguments from a file can be combined with additional command line arguments, an
 
 A further example of how this argument file can be used, e.g., creating configuration profile files via @filename, is in the :example_file:`Multiple Build Configurations Example <build_system/cmake/multi_config/README.md>`.
 
+Extending ``idf.py``
+====================
+
+``idf.py`` can be extended with additional subcommands, global options, and callbacks provided by extension files in your project and components which participate in the build, as well as by external Python packages exposing entry points.
+
+- **From components participating in the build**: Place a file named ``idf_ext.py`` in the project root or in a component's root directory that is registered in the project's ``CMakeLists.txt``. Component extensions are discovered after the project is configured - run ``idf.py build`` or ``idf.py reconfigure`` to make newly added commands available.
+- **From Python entry points**: Any installed Python package may contribute extensions by defining an entry point in the ``idf_extension`` group. Package installation is sufficient, no project build is required.
+
+.. important::
+
+   Extensions must not define subcommands or options that have the same names as the core ``idf.py`` commands. Custom actions and options are checked for name collisions, overriding defaults is not possible and a warning is printed. For Python entry points, use unique identifiers as duplicate entry point names will be ignored with a warning.
+
+Extension File Example
+----------------------
+
+An extension file defines an ``action_extensions`` function which returns additional actions/options. The same structure is used for component-based extensions (``idf_ext.py``) and for package-based extensions (e.g., ``<package_name>_ext.py``):
+
+.. code-block:: python
+
+  from typing import Any
+  import click
+
+  def action_extensions(base_actions: dict, project_path: str) -> dict:
+      def hello_test(subcommand_name: str, ctx: click.Context, global_args: dict, **action_args: Any) -> None:
+          message = action_args.get('message')
+          print(f"Running action: {subcommand_name}. Message: {message}")
+
+      def global_callback_detail(ctx: click.Context, global_args: dict, tasks: list) -> None:
+          if getattr(global_args, 'detail', False):
+              print(f"About to execute {len(tasks)} task(s): {[t.name for t in tasks]}")
+
+      return {
+          "version": "1",
+          "global_options": [
+              {
+                  "names": ["--detail", "-d"],
+                  "is_flag": True,
+                  "help": "Enable detailed output",
+              }
+          ],
+          "global_action_callbacks": [global_callback_detail],
+          "actions": {
+              "hello": {
+                  "callback": hello_test,
+                  "short_help": "Hello from component",
+                  "help": "Test command from component extension",
+                  "options": [
+                      {
+                          "names": ["--message", "-m"],
+                          "help": "Custom message to display",
+                          "default": "Hi there!",
+                          "type": str,
+                      }
+                  ]
+              },
+          },
+      }
+
+
+Extension API Reference
+-----------------------
+
+The ``action_extensions`` function takes arguments ``base_actions`` (all currently registered commands) and ``project_path`` (absolute project directory) and returns a dictionary with up to four keys:
+
+- ``version``: A string representing the interface version of the extension. Currently, the API version is ``1``. **This key is mandatory** and must be provided.
+- ``global_options``: A list of options available globally for all commands. Each option is a dictionary with fields such as ``names``, ``help``, ``type``, ``is_flag``, ``scope``, etc.
+- ``global_action_callbacks``: A list of functions called once before any task execution. Each global action callback function accepts three arguments:
+
+   - ``ctx`` — The `click context`_
+   - ``global_args`` — All available global arguments
+   - ``tasks`` — The list of tasks to be executed. Task refer to the action / sub-command used with `idf.py`
+
+- ``actions``: A dictionary of new subcommands. Each action has a ``callback`` function and may also include ``options``, ``arguments``, ``dependencies``, etc. Each action callback function accepts three to four arguments:
+
+   - ``subcommand_name`` — the name of the command (useful if multiple commands share the same callback)
+   - ``ctx`` — the `click context`_
+   - ``global_args`` — all available global arguments,
+   - ``**action_args`` — (optional) arguments passed to the action
+
+Basic Usage Examples
+--------------------
+
+1) Provide an extension from a component in your project
+
+  Create ``idf_ext.py`` in the project root or in a registered component (for example ``components/my_component/idf_ext.py``). Use the extension file example above as your ``idf_ext.py`` implementation.
+
+  Run ``idf.py build`` or ``idf.py reconfigure`` to load the new command, then ``idf.py --help`` will show the new extension.
+
+2) Provide an extension via a Python package entry point
+
+  Implement your extension in a module named ``<package_name>_ext.py`` using the extension file example above, and expose the ``action_extensions`` function via the ``idf_extension`` entry-point group. For example, with ``pyproject.toml``:
+
+  .. code-block:: TOML
+
+    [project]
+    name = "my_comp"
+    version = "0.1.0"
+
+    [project.entry-points.idf_extension]
+    my_pkg_ext = "my_component.my_ext:action_extensions"
+
+
+  Install the package into the same Python environment as ``idf.py`` (for example with ``pip install -e .`` in the package directory). It is recommended to use a unique module name (e.g., ``<package_name>_ext.py``) to avoid name conflicts. After successful installation, ``idf.py --help`` will show the new extension.
+
 .. _cmake: https://cmake.org
 .. _ninja: https://ninja-build.org
 .. _esptool.py: https://github.com/espressif/esptool/#readme
 .. _CCache: https://ccache.dev/
+.. _click context: https://click.palletsprojects.com/en/stable/api/#context

docs/zh_CN/api-guides/tools/idf-py.rst

@@ -301,7 +301,112 @@ uf2 二进制文件也可以通过 :ref:`idf.py uf2 <generate-uf2-binary>` 生
 
 关于参数文件的更多示例，如通过 @filename 创建配置文件概要，请参阅 :example_file:`多个构建配置示例 <build_system/cmake/multi_config/README.md>`。
 
+扩展 ``idf.py``
+====================
+
+``idf.py`` 支持扩展功能。通过项目中的扩展文件以及参与构建的组件中的扩展文件，可以增加额外的子命令、全局选项和回调函数；通过暴露入口点的外部 Python 包，可以提供新的扩展功能。
+
+- **参与构建的组件**：在项目根目录，或注册在项目 ``CMakeLists.txt`` 中的组件根目录，放置名为 ``idf_ext.py`` 的文件，该文件会在项目配置完成后得到识别。运行 ``idf.py build`` 或 ``idf.py reconfigure``，新添加的命令即可生效。
+- **Python 入口点**：对于任何已安装的 Python 包，在 ``idf_extension`` 组中定义入口点后，就可以提供扩展功能。只要安装了 Python 包就可以使用扩展功能，无需重新构建项目。
+
+.. important::
+
+   扩展不能定义与 ``idf.py`` 命令同名的子命令或选项。系统会检查自定义的动作和选项名称是否存在冲突，不允许覆盖默认命令，如有冲突会打印警告。对于 Python 入口点，必须使用唯一标识符，否则会忽略重复的入口点名称并发出警告。
+
+扩展文件示例
+----------------------
+
+扩展文件需要定义一个 ``action_extensions`` 函数，用于返回扩展的动作或选项。组件扩展 ``idf_ext.py`` 和基于包的扩展（例如 ``<package_name>_ext.py``）使用相同的结构，如下所示：
+
+.. code-block:: python
+
+  from typing import Any
+  import click
+
+  def action_extensions(base_actions: dict, project_path: str) -> dict:
+      def hello_test(subcommand_name: str, ctx: click.Context, global_args: dict, **action_args: Any) -> None:
+          message = action_args.get('message')
+          print(f"Running action: {subcommand_name}. Message: {message}")
+
+      def global_callback_detail(ctx: click.Context, global_args: dict, tasks: list) -> None:
+          if getattr(global_args, 'detail', False):
+              print(f"About to execute {len(tasks)} task(s): {[t.name for t in tasks]}")
+
+      return {
+          "version": "1",
+          "global_options": [
+              {
+                  "names": ["--detail", "-d"],
+                  "is_flag": True,
+                  "help": "Enable detailed output",
+              }
+          ],
+          "global_action_callbacks": [global_callback_detail],
+          "actions": {
+              "hello": {
+                  "callback": hello_test,
+                  "short_help": "Hello from component",
+                  "help": "Test command from component extension",
+                  "options": [
+                      {
+                          "names": ["--message", "-m"],
+                          "help":  "Custom message to display",
+                          "default": "Hi there!",
+                          "type": str,
+                      }
+                  ]
+              },
+          },
+      }
+
+
+扩展 API 参考
+-----------------------
+
+``action_extensions`` 函数接收两个参数： ``base_actions`` 表示当前已注册的所有命令， ``project_path`` 表示项目的绝对路径。该函数返回一个包含最多四个键的字典：
+
+- ``version``：表示扩展接口版本。当前 API 版本为 ``1``。此键为必填项。
+- ``global_options``：一组全局选项，适用于所有命令。每个选项都是一个字典，包含 ``names``、 ``help``、 ``type``、 ``is_flag``、 ``scope`` 等字段。
+- ``global_action_callbacks``：表示一组全局回调函数，在执行任何任务之前都会调用一次。每个全局回调函数接受三个参数：
+
+   - ``ctx``：即 `click context`_
+   - ``global_args``：所有可用的全局参数
+   - ``tasks``：将要执行的任务列表。任务指的是运行 ``idf.py`` 时所调用的具体动作或子命令
+
+- ``actions``：子命令字典，用于定义新的子命令。每个子命令都有一个 ``callback`` 函数，并且可以包含 ``options``、 ``arguments``、 ``dependencies`` 等。每个回调函数接受三到四个参数：
+
+   - ``subcommand_name``：命令的名称（在多个命令共享同一回调时很有用）
+   - ``ctx``：即 `click context`_
+   - ``global_args``：所有可用的全局参数
+   - ``**action_args``：传递给该子命令的具体参数，可选
+
+基本用法示例
+--------------------
+
+1) **通过项目组件提供扩展**
+
+  在项目根目录或某个已注册的组件目录下创建 ``idf_ext.py`` （例如 ``components/my_component/idf_ext.py`` ）。实现内容可参考上面的扩展文件示例。
+
+  运行 ``idf.py build`` 或 ``idf.py reconfigure`` 加载新命令，然后执行 ``idf.py --help`` 即可看到新扩展。
+
+2) **通过 Python 包入口点提供扩展**
+
+  使用上述扩展文件示例，在名为 ``<package_name>_ext.py`` 的模块中实现扩展，并通过 ``idf_extension`` 入口点组暴露 ``action_extensions`` 函数。例如，在 ``pyproject.toml`` 中配置：
+
+  .. code-block:: TOML
+
+    [project]
+    name = "my_comp"
+    version = "0.1.0"
+
+    [project.entry-points.idf_extension]
+    my_pkg_ext = "my_component.my_ext:action_extensions"
+
+
+  将该包安装到与 ``idf.py`` 相同的 Python 环境中（例如在包目录下执行 ``pip install -e .``）。建议使用唯一的模块名（例如 ``<package_name>_ext.py``）避免命名冲突。安装成功后，运行 ``idf.py --help`` 就可以看到新扩展命令。
+
 .. _cmake: https://cmake.org
 .. _ninja: https://ninja-build.org
 .. _esptool.py: https://github.com/espressif/esptool/#readme
 .. _CCache: https://ccache.dev/
+.. _click context: https://click.palletsprojects.com/en/stable/api/#context

tools/idf.py

@@ -14,6 +14,8 @@
 # their specific function instead.
 import codecs
 import glob
+import importlib.metadata
+import importlib.util
 import json
 import locale
 import os.path
@@ -246,6 +248,8 @@ def __call__(
             self.callback(self.name, context, global_args, **action_args)
 
     class Action(click.Command):
+        callback: Callable
+
         def __init__(
             self,
             name: str | None = None,
@@ -721,6 +725,90 @@ def execute_tasks(self, tasks: list, **kwargs: str) -> OrderedDict:
 
             return tasks_to_run
 
+    def load_cli_extension_from_dir(ext_dir: str) -> Any | None:
+        """Load extension 'idf_ext.py' from directory and return the action_extensions function"""
+        ext_file = os.path.join(ext_dir, 'idf_ext.py')
+        if not os.path.exists(ext_file):
+            return None
+
+        try:
+            module_name = f'idf_ext_{os.path.basename(ext_dir)}'
+            spec = importlib.util.spec_from_file_location(module_name, ext_file)
+            if spec is None or spec.loader is None:
+                raise ImportError('Failed to load python module')
+            ext_module = importlib.util.module_from_spec(spec)
+            sys.modules[module_name] = ext_module
+            spec.loader.exec_module(ext_module)
+
+            if hasattr(ext_module, 'action_extensions'):
+                return ext_module.action_extensions
+            else:
+                print_warning(f"Warning: Extension {ext_file} has no attribute 'action_extensions'")
+
+        except (ImportError, SyntaxError) as e:
+            print_warning(f'Warning: Failed to import extension {ext_file}: {e}')
+
+        return None
+
+    def load_cli_extensions_from_entry_points() -> list[tuple[str, Any]]:
+        """Load extensions from Python entry points"""
+        extensions: list[tuple[str, Any]] = []
+        eps = importlib.metadata.entry_points(group='idf_extension')
+
+        # declarative value is the path-like identifier of entry point defined in the components config file
+        # having same declarative value for multiple entry points results in loading only one of them (undeterministic)
+        eps_declarative_values: list[str] = []
+        for ep in eps:
+            if ep.value in eps_declarative_values:
+                conflicting_names = [e.name for e in eps if e.value == ep.value]
+                print_warning(
+                    f"Warning: Entry point's declarative value [extension_file_name:method_name] "
+                    f'name collision detected for - {ep.value}. The same {ep.value} is used by '
+                    f'{conflicting_names} entry points. To ensure successful loading, please use'
+                    ' a different extension file name or method name for the entry point.'
+                )
+                # Remove any already loaded extensions with conflicting names
+                extensions[:] = [ext for ext in extensions if ext[0] not in conflicting_names]
+                continue
+
+            if ep.value == 'idf_ext:action_extensions':
+                print_warning(
+                    f'Entry point "{ep.name}" has declarative value "{ep.value}". For external components, '
+                    'it is recommended to use name like <<COMPONENT_NAME>>_ext:action_extensions, '
+                    "so it does not interfere with the project's idf_ext.py file."
+                )
+
+            eps_declarative_values.append(ep.value)
+            try:
+                extension_func = ep.load()
+                extensions.append((ep.name, extension_func))
+            except Exception as e:
+                print_warning(f'Warning: Failed to load entry point extension "{ep.name}": {e}')
+
+        return extensions
+
+    def resolve_build_dir() -> str:
+        """Resolve build directory from command line arguments
+        return build path if explicitly set, otherwise default build path"""
+        import argparse
+
+        parser = argparse.ArgumentParser(add_help=False)
+        parser.add_argument('-B', '--build-dir', default=os.path.join(project_dir, 'build'))
+        args, _ = parser.parse_known_args()
+        build_dir: str = args.build_dir
+        return os.path.abspath(build_dir)
+
+    def _extract_relevant_path(path: str) -> str:
+        """
+        Returns part of the path starting from 'components' or 'managed_components'.
+        If neither is found, returns the full path.
+        """
+        for keyword in ('components', 'managed_components'):
+            # arg path is loaded from project_description.json, where paths are always defined with '/'
+            if keyword in path.split('/'):
+                return keyword + path.split(keyword, 1)[1]
+        return path
+
     # That's a tiny parser that parse project-dir even before constructing
     # fully featured click parser to be sure that extensions are loaded from the right place
     @click.command(
@@ -774,21 +862,40 @@ def parse_project_dir(project_dir: str) -> Any:
         except AttributeError:
             print_warning(f'WARNING: Cannot load idf.py extension "{name}"')
 
-    # Load extensions from project dir
-    if os.path.exists(os.path.join(project_dir, 'idf_ext.py')):
-        sys.path.append(project_dir)
+    component_idf_ext_dirs = []
+    # Get component directories with idf extensions that participate in the build
+    build_dir_path = resolve_build_dir()
+    project_description_json_file = os.path.join(build_dir_path, 'project_description.json')
+    if os.path.exists(project_description_json_file):
         try:
-            from idf_ext import action_extensions
-        except ImportError:
-            print_warning('Error importing extension file idf_ext.py. Skipping.')
-            print_warning(
-                "Please make sure that it contains implementation (even if it's empty) of add_action_extensions"
-            )
+            with open(project_description_json_file, encoding='utf-8') as f:
+                project_desc = json.load(f)
+                all_component_info = project_desc.get('build_component_info', {})
+                for _, comp_info in all_component_info.items():
+                    comp_dir = comp_info.get('dir')
+                    if comp_dir and os.path.isdir(comp_dir) and os.path.exists(os.path.join(comp_dir, 'idf_ext.py')):
+                        component_idf_ext_dirs.append(comp_dir)
+        except (OSError, json.JSONDecodeError) as e:
+            print_warning(f'Warning: Failed to read component info from project_description.json: {e}')
+    # Load extensions from directories that participate in the build (components and project)
+    for ext_dir in component_idf_ext_dirs + [project_dir]:
+        extension_func = load_cli_extension_from_dir(ext_dir)
+        if extension_func:
+            try:
+                all_actions = merge_action_lists(all_actions, custom_actions=extension_func(all_actions, project_dir))
+            except Exception as e:
+                print_warning(f'WARNING: Cannot load directory extension from "{ext_dir}": {e}')
+            else:
+                if ext_dir != project_dir:
+                    print(f'INFO: Loaded component extension from "{_extract_relevant_path(ext_dir)}"')
 
+    # Load extensions from Python entry points
+    entry_point_extensions = load_cli_extensions_from_entry_points()
+    for name, extension_func in entry_point_extensions:
         try:
-            all_actions = merge_action_lists(all_actions, action_extensions(all_actions, project_dir))
-        except NameError:
-            pass
+            all_actions = merge_action_lists(all_actions, custom_actions=extension_func(all_actions, project_dir))
+        except Exception as e:
+            print_warning(f'WARNING: Cannot load entry point extension "{name}": {e}')
 
     cli_help = (
         'ESP-IDF CLI build management tool. '