updates

* docs: fix minor issues and improve vague statements

* feat(esp32h21): Add Microsoft UF2 family ID

* fix(logging): Avoid crashes when flushing if sys.stdout is not available

Closes espressif#1064

Closes espressif#1063

* change(github): Add Github workflow for bot responses to issues

* fix(elf2image): Print correct MMU page size in error message

* fix(elf2image): Try to correct MMU page size if not specified

This commit fixes issue with using elf2image command without --flash-mmu-page-size.
In that case, app info segment might be incorrectly placed in the image.
This is fixed by checking if app info segment is present and if so,
use page size from it or from its alignment.
Closes espressif#1062

* fix(elf2image): support --flash-mmu-page-config for all chips

This commit fixes the issue where the `--flash-mmu-page-config` option
did not work for chips that do not have configurable MMU page sizes.

* fix(esptool): Fix efuse base address for esp32h21

* remove: Deprecate Python versions 3.7, 3.8 and 3.9

BREAKING CHANGE

* docs(logger): Fix custom logger example code

Closes espressif#1066

* feat(esp32h4): add ESP32H4 esptool support

* feat(espefuse): Updates esp32h4 efuse table and fixes tests

- Added esp32h4 efuse table
- Fixed h4 test
- Enabled some tests for p4 chip.

* feat(esp32h4): update the ESP32H4StubLoader

* change: fix issues reported by ruff linter

* ci: update pre-commit hooks to the latest versions

* feat(flash_attach): Encapsulate logic for flash attaching and configuration

* refactor(cli_mode): Improve CLI mode workflow code

* feat(cmds): Rework the public API to work as a Python module

BREAKING CHANGE

Closes espressif#208

* feat(cmds): Allow commands to output bytes, as well as write to a file

* fix(lint): Correct type annotations issues reported by mypy

* Tasmota changes

* update manifest

* Update build_esptool.yml

* Update build_esptool.yml

* docs(logger): Fix custom logger example code

Closes espressif#1066

---------

Co-authored-by: Peter Dragun <[email protected]>
Co-authored-by: Radim Karniš <[email protected]>
Co-authored-by: Roland Dobai <[email protected]>
Co-authored-by: Jaroslav Burian <[email protected]>
Co-authored-by: Konstantin Kondrashov <[email protected]>
Co-authored-by: Chen Jichang <[email protected]>

Author: 7 people

.git-blame-ignore-revs

@@ -13,3 +13,6 @@ b57f69bd13222b1753446a0f7c17386eda1dc2c9
 
 # Refactoring documentation
 6282f98dbfca58add4992c259c0aac3c3ec64d3f
+
+# Fixing linter issues
+2e4e77cde269379f3aba2e1344c912f47c5974de

.github/ISSUE_TEMPLATE/bug-report-no-hw.yml

@@ -23,7 +23,7 @@ body:
     attributes:
       label: Python Version
       description: Which Python version are you using? Run `python -V` to check this.
-      placeholder: ex. Python 3.9.6
+      placeholder: ex. Python 3.10.6
     validations:
       required: true
   - type: input

.github/ISSUE_TEMPLATE/issue-with-hw.yml

@@ -30,7 +30,7 @@ body:
     attributes:
       label: Python Version
       description: Which Python version are you using? Run `python -V` to check this.
-      placeholder: ex. Python 3.9.6
+      placeholder: ex. Python 3.10.6
     validations:
       required: true
   - type: input

.github/workflows/build_esptool.yml

@@ -27,7 +27,7 @@ jobs:
         name: Build and test in ${{ matrix.platform }}
         with:
           arch: ${{ matrix.platform }}
-          distro: ubuntu20.04
+          distro: ubuntu22.04
           shell: /bin/bash
           # Not required, but speeds up builds by storing container images in
           # a GitHub package registry.
@@ -45,8 +45,7 @@ jobs:
             export PATH=\$PATH:/home/builder/.local/bin
             # Install dependencies
             python -m pip install --upgrade pip
-            # PyInstaller >=6.0 results in significantly more antivirus false positives
-            pip install pyinstaller==5.13.2
+            pip install pyinstaller==6.11.1
             pip install --user -e .
 
             # Build with PyInstaller
@@ -90,7 +89,7 @@ jobs:
         - platform: linux-amd64
           TARGET: linux-amd64
           SEPARATOR: ':'
-          RUN_ON: ubuntu-20.04
+          RUN_ON: ubuntu-24.04
     env:
       DISTPATH: esptool-${{ matrix.TARGET }}
       STUBS_DIR: ./esptool/targets/stub_flasher/

.github/workflows/issue_bot.yml

@@ -0,0 +1,30 @@
+name: Bot response to issues
+
+on:
+  issues:
+    types: [opened, edited]
+
+jobs:
+  docs_bot:
+    name: Generate automated response by docs bot
+    runs-on: ubuntu-latest
+    steps:
+      - name: Docs bot action
+        if: ${{ github.repository_owner == 'espressif' }}
+        uses: espressif/docs-bot-action@master
+        env:
+            BOT_API_KEY: ${{ secrets.BOT_API_KEY }}
+            BOT_INTEGRATION_ID: ${{ secrets.BOT_INTEGRATION_ID }}
+            BOT_API_ENDPOINT: ${{ secrets.BOT_API_ENDPOINT }}
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          github_repository: ${{ github.repository }}
+          github_issue_number: ${{ github.event.issue.number }}
+          title: ${{ github.event.issue.title }}
+          in_msg: ${{ github.event.issue.body }}
+          prefix_out_msg: >
+            Hi @${{ github.event.issue.user.login }}! Please be aware that 
+            (1) the following suggestions are generated by a bot and haven't been fact-checked by Espressif
+            Systems,
+            (2) burning eFuses and enabling security features are irreversible operations and can damage your ESP32 device.
+            We hope that this message will help you until an Espressif Engineer looks at your issue.

.github/workflows/test_esptool.yml

@@ -4,11 +4,11 @@ on: [push, pull_request]
 
 jobs:
   test_esptool:
-    runs-on: ubuntu-22.04  # 22.04 is the last version with Python 3.7
+    runs-on: ubuntu-22.04
 
     strategy:
       matrix:
-        python-version: ['3.9', '3.10', '3.11', '3.12']
+        python-version: ['3.10', '3.11', '3.12', '3.13']
 
     steps:
     - name: Checkout ref commit
@@ -62,14 +62,14 @@ jobs:
             git diff --exit-code
 
   lint_esptool:
-      runs-on: ubuntu-22.04  # 22.04 is the last version with Python 3.7
+      runs-on: ubuntu-22.04
       steps:
         - name: Checkout
           uses: actions/checkout@master
-        - name: Set up Python 3.9
+        - name: Set up Python 3.10
           uses: actions/setup-python@master
           with:
-            python-version: '3.9'
+            python-version: '3.10'
         - name: Lint and format with ruff
           run: |
             pip install --extra-index-url https://dl.espressif.com/pypi -e .[dev]

.gitlab-ci.yml

@@ -38,7 +38,7 @@ cache:
 
 .test_template: &test_template
   stage: test
-  image: python:3.7-bullseye
+  image: python:3.10-bullseye
   tags:
     - host_test
   dependencies: []
@@ -117,6 +117,7 @@ host_tests_espefuse:
         - esp32s2
         - esp32s3
         - esp32h21
+        - esp32h4
   script:
     - coverage run --parallel-mode -m pytest ${CI_PROJECT_DIR}/test/test_espefuse.py --chip ${TARGET}
     # some .coverage files in sub-directories are not collected on some runners, move them first
@@ -137,7 +138,7 @@ host_tests_latest_python:
     -  pytest ${CI_PROJECT_DIR}/test/test_espefuse.py --chip esp32
 
 # A new job "host_test_hsm" is created for the test "test_espsecure_hsm.py" which runs an ubuntu image,
-# because python-pkcs11 (v0.7.0) package is compiled using GLIBC_2.34 but docker image python:3.7-bullseye
+# because python-pkcs11 (v0.7.0) package is compiled using GLIBC_2.34 but docker image python:3.10-bullseye
 # support versions only upto GLIBC_2.31.
 host_tests_hsm:
   <<: *host_tests_template
@@ -157,7 +158,7 @@ host_tests_hsm:
 
 run_pre_commit_hooks:
   stage: test
-  image: python:3.7-bullseye
+  image: python:3.10-bullseye
   rules:
     - if: $CI_PIPELINE_SOURCE != "schedule"
   tags:
@@ -189,7 +190,6 @@ check_install_coverage:
       - ".coverage*"
     expire_in: 1 week
   script:
-    - coverage run --parallel-mode setup.py install
     - coverage run --parallel-mode esptool.py --help
     - coverage run --parallel-mode espefuse.py --help
     - coverage run --parallel-mode espsecure.py --help
@@ -250,7 +250,7 @@ check_stub_build:
 
 .target_esptool_test:
   stage: test
-  image: python:3.7-bullseye
+  image: python:3.10-bullseye
   variables:
     PYTHONPATH: "$PYTHONPATH:${CI_PROJECT_DIR}/test"
     COVERAGE_PROCESS_START: "${CI_PROJECT_DIR}/test/.covconf"
@@ -479,7 +479,7 @@ target_tests_windows:
 
 combine_reports:
   stage: report
-  image: python:3.7-bullseye
+  image: python:3.10-bullseye
   tags:
     - host_test
   artifacts:
@@ -509,7 +509,7 @@ combine_reports:
 
 build_docs:
   stage: build_docs
-  image: python:3.7-bullseye
+  image: python:3.12-bookworm  # 3.12 is the last version with imghdr
   tags:
     - build_docs
   rules:
@@ -518,6 +518,7 @@ build_docs:
     - changes:
       - "docs/**/*"
       - "CONTRIBUTING.rst"
+      - "esptool/cmds.py"
   needs: []
   artifacts:
     when: always
@@ -526,13 +527,14 @@ build_docs:
       - docs/_build/*/*/html/*
     expire_in: 4 days
   script:
+    - pip install .  # esptool is needed for the automatic API documentation generation
     - cd docs
     - pip install -r requirements.txt --prefer-binary
     - build-docs -l en -t {esp8266,esp32,esp32s2,esp32c3,esp32s3,esp32c2,esp32c6,esp32h2,esp32p4,esp32c5,esp32c61}
 
 .deploy_docs_template:
   stage: deploy_docs
-  image: python:3.7-bullseye
+  image: python:3.12-bookworm  # 3.12 is the last version with imghdr
   tags:
     - deploy
   needs:
@@ -553,6 +555,7 @@ deploy_docs_preview:
     - changes:
       - "docs/**/*"
       - "CONTRIBUTING.rst"
+      - "esptool/cmds.py"
   variables:
     TYPE: "preview"
     DOCS_BUILD_DIR: "${CI_PROJECT_DIR}/docs/_build/"
@@ -570,6 +573,7 @@ deploy_docs_production:
       changes:
       - "docs/**/*"
       - "CONTRIBUTING.rst"
+      - "esptool/cmds.py"
   variables:
     TYPE: "production"
     DOCS_BUILD_DIR: "${CI_PROJECT_DIR}/docs/_build/"

.pre-commit-config.yaml

@@ -1,25 +1,25 @@
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.1.14
+    rev: v0.9.6
     hooks:
       - id: ruff # Runs ruff linter (replaces flake8)
         args: [--fix, --exit-non-zero-on-fix] # --fix for fixing errors
       - id: ruff-format
   - repo: https://github.com/sphinx-contrib/sphinx-lint
-    rev: v0.6.8
+    rev: v1.0.0
     hooks:
       - id: sphinx-lint
         name: Lint RST files in docs folder using Sphinx Lint
         files: ^((docs/en)/.*\.(rst|inc))|CONTRIBUTING.rst$
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.4.1  # the last version running on py3.7
+    rev: v1.15.0
     hooks:
       - id: mypy
         additional_dependencies: ['types-PyYAML<=6.0.12.12']
         # ignore wrapper scripts because of name colision with efuse/__init__.py etc.
         exclude: test/|docs/|espefuse.py|espsecure.py|esptool.py|esp_rfc2217_server.py
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.2.5
+    rev: v2.4.1
     hooks:
     - id: codespell
       additional_dependencies:
@@ -31,5 +31,5 @@ repos:
         stages: [commit-msg]
         args:
           - --allow-breaking
-default_stages: [commit]
+default_stages: [pre-commit]
 default_install_hook_types: [pre-commit, commit-msg]

CONTRIBUTING.rst

@@ -44,6 +44,8 @@ Please report bugs in ``esptool.py`` if you find them. However, before reporting
 
 If you don’t find anything, please `open a new issue <https://github.com/espressif/esptool/issues/new/choose>`_.
 
+.. _feature-requests:
+
 Sending Feature Requests
 ------------------------
 
@@ -73,7 +75,7 @@ Code Style & Static Analysis
 
 Please follow these coding standards when writing code for ``esptool.py``:
 
-Pre-commit checks
+Pre-Commit Checks
 """""""""""""""""
 
 `pre-commit <https://pre-commit.com/>`_ is a framework for managing pre-commit hooks. These hooks help to identify simple issues before committing code for review.
@@ -104,13 +106,13 @@ Ruff
 
 When you submit a Pull Request, the GitHub Actions automated build system will run automated checks using these tools.
 
-Shinx-lint
-""""""""""
+Sphinx-Lint
+"""""""""""
 
 The documentation is checked for stylistic and formal issues by ``sphinx-lint``.
 
 
-Codespell check
+Codespell Check
 """""""""""""""
 
 This repository utilizes an automatic `spell checker <https://github.com/codespell-project/codespell>`_ integrated into the pre-commit process. If any spelling issues are detected, the recommended corrections will be applied automatically to the file, ready for commit.
@@ -143,7 +145,7 @@ The following tests are not run automatically by GitHub Actions, because they ne
 
    ``pytest test_esptool.py --port /dev/ttyUSB0 --chip esp32 --baud 230400``
 
-   Or to run the TestFlashing suite only (using the pytest ``-k`` option to select tests based on their name) on an ESP8266 board connected to /dev/ttyUSB2, at 460800bps:
+   Or to run the ``TestFlashing`` suite only (using the pytest ``-k`` option to select tests based on their name) on an ESP8266 board connected to /dev/ttyUSB2, at 460800bps:
 
    ``pytest test_esptool.py --port /dev/ttyUSB2 --chip esp8266 --baud 460800 -k TestFlashing``
 
@@ -171,9 +173,7 @@ The whole test suite (without the tests needing an actual hardware or installati
 Pull Request Process
 --------------------
 
-.. note::
-
-   If you are developing the stub flasher and plan to send a pull request, please use the latest toolchains available.
+If you would like to contribute to the flasher stub, please see the `Flasher stub repository <https://github.com/espressif/esptool-legacy-flasher-stub>`_.
 
 After you open the Pull Request, there will probably be some discussion in the comments field of the request itself.
 

ci/download_flasher_stubs.py

@@ -26,7 +26,8 @@
             "esp32s3",
             "esp8266",
         ),
-        "LICENSE": "released as Free Software under GNU General Public License Version 2 or later",
+        "LICENSE": "released as Free Software under GNU General Public License "
+        "Version 2 or later",
     },
     {
         "STUB_SET_VERSION": "2",
@@ -42,7 +43,8 @@
             "esp32s2",
             "esp32s3",
         ),
-        "LICENSE": "dual licensed under the Apache License Version 2.0 or the MIT license",
+        "LICENSE": "dual licensed under the Apache License Version 2.0 or the MIT "
+        "license",
     },
 )
 
@@ -51,16 +53,17 @@
 README_TEMPLATE = """# Licensing
 
 The binaries in JSON format distributed in this directory are {LICENSE}. They were released at {URL} from where the sources can be obtained.
-"""
+"""  # noqa: E501
 
 
 def main():
     for stub_set in STUBS:
         dest_sub_dir = os.path.join(DESTINATION_DIR, stub_set["STUB_SET_VERSION"])
 
-        """ The directory is cleaned up so we would detect if a stub was just committed into the repository but the
-        name was not added into the FILE_LIST of STUBS. This would be an unwanted state because the checker would not
-        detect any changes in that stub."""
+        """ The directory is cleaned up so we would detect if a stub was just committed
+        into the repository but the name was not added into the FILE_LIST of STUBS.
+        This would be an unwanted state because the checker would not detect any
+        changes in that stub."""
         for old_file in glob.glob(os.path.join(dest_sub_dir, "*.json")):
             print(f"Removing old file {old_file}")
             os.remove(old_file)