Merge branch 'feat/usb_wireless_disk_4bit_sdcard' into 'master'

feat: improve usb wireless disk performance

Closes AEG-2710

See merge request ae_group/esp-iot-solution!1358

Author: leeebo

.gitignore

@@ -123,3 +123,5 @@ output.json
 
 #components cache
 **/dist/
+
+*.log

.pre-commit-config.yaml

@@ -6,17 +6,17 @@ default_install_hook_types: [pre-commit,commit-msg]
 
 repos:
   - repo: https://github.com/espressif/check-copyright/
-    rev: v1.0.3
+    rev: v1.1.1
     hooks:
       - id: check-copyright
         args: ['--config', 'tools/ci/check_copyright_config.yaml', '--ignore', 'tools/ci/check_copyright_ignore.txt']
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.6.0
+    rev: v6.0.0
     hooks:
       - id: file-contents-sorter
         files: 'tools\/ci\/(executable-list\.txt|check_copyright_ignore\.txt)'
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.6.0
+    rev: v6.0.0
     hooks:
       - id: trailing-whitespace
         # note: whitespace exclusions use multiline regex, see https://pre-commit.com/#regular-expressions
@@ -53,7 +53,7 @@ repos:
         name: Do not use uppercase letters in the branch name
         args: ['--pattern', '^[^A-Z]*[A-Z]']
   - repo: https://github.com/espressif/astyle_py.git
-    rev: v1.0.5
+    rev: v1.1.0
     hooks:
       - id: astyle_py
         # If you are modifying astyle version, update tools/format.sh as well
@@ -83,15 +83,15 @@ repos:
           - pyyaml
         always_run: true
   - repo: https://github.com/espressif/conventional-precommit-linter
-    rev: v1.8.0
+    rev: v1.10.0
     hooks:
       - id: conventional-precommit-linter
         stages: [commit-msg]
         args:
           - --subject-min-length=15
           - --body-max-line-length=200
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.3.0
+    rev: v2.4.1
     hooks:
     - id: codespell
       args: [-w, "--ignore-words=codespell-ignore-list", "--skip=*.js,*.vue,*/pnpm-lock.yaml,*/package-lock.json,*/yarn.lock,*.base64"]

components/sensors/humiture/aht20/README.md

@@ -5,7 +5,7 @@ I2C driver and definition of AHT20 humidity and temperature sensor.
 
 Components compatible with AHT30 and AHT21 (AHT21 is deprecated).
 
-See [AHT20 datasheet](http://www.aosong.com/en/products-32.html), [AHT30 datasheet](http://www.aosong.com/en/products-131.html).
+See [AHT20 datasheet](https://asairsensors.com/wp-content/uploads/2021/09/Data-Sheet-AHT20-Humidity-and-Temperature-Sensor-ASAIR-V1.0.03.pdf), [AHT30 datasheet](https://asairsensors.com/product/aht30-integrated-temperature-and-humidity-sensor/).
 
 ## Usage
 

examples/camera/README.md

@@ -78,7 +78,7 @@ The APIs included in the [esp32-camera](https://github.com/espressif/esp32-camer
 Convert image buffer to RGB565 buffer.
 
 ### Header File
-- [esp32-camera/conversions/include/esp_jpg_decode.h](https://github.com/espressif/esp32-camera/blob/master/conversions/include/esp_jpg_decode.h)
+- [esp32-camera/conversions/include/esp_jpg_decode.h](https://github.com/espressif/esp32-camera/blob/v2.0.0/conversions/include/esp_jpg_decode.h)
 #### Functions
 - `esp_err_t esp_jpg_decode(size_t len, jpg_scale_t scale, jpg_reader_cb reader, jpg_writer_cb writer, void * arg)`  
 JPEG scaling function.

examples/usb/device/usb_extend_screen/components/espressif__esp_lvgl_port/examples/touchscreen/README.md

@@ -4,11 +4,11 @@ Very simple example for demonstration of initialization and usage of the `esp_lv
 
 ## 1. LCD HW initialization - `app_lcd_init()`
 
-Standard HW initialization of the LCD using [`esp_lcd`](https://github.com/espressif/esp-idf/tree/master/components/esp_lcd) component. Settings of this example are fully compatible with [ESP-BOX](https://github.com/espressif/esp-bsp/tree/master/esp-box) board.
+Standard HW initialization of the LCD using [`esp_lcd`](https://github.com/espressif/esp-idf/tree/master/components/esp_lcd) component. Settings of this example are fully compatible with [ESP-BOX](https://github.com/espressif/esp-bsp/tree/master/bsp/esp-box-3) board.
 
 ## 2. Touch HW initialization - `app_touch_init()`
 
-Standard HW initialization of the LCD touch using [`esp_lcd_touch`](https://github.com/espressif/esp-bsp/tree/master/components/lcd_touch/esp_lcd_touch) component. Settings of this example are fully compatible with [ESP-BOX](https://github.com/espressif/esp-bsp/tree/master/esp-box) board.
+Standard HW initialization of the LCD touch using [`esp_lcd_touch`](https://github.com/espressif/esp-bsp/tree/master/components/lcd_touch/esp_lcd_touch) component. Settings of this example are fully compatible with [ESP-BOX](https://github.com/espressif/esp-bsp/tree/master/bsp/esp-box-3) board.
 
 ## 3. LVGL port initialization - `app_lvgl_init()`
 

examples/usb/device/usb_msc_wireless_disk/README.md

@@ -49,3 +49,46 @@ This demo has enabled FATFS OEM multi-code page support. To manually adjust mult
   - This option controls the encoding method used by the filenames read by the FATFS API. For more details, please refer to the configuration help.
   - For web development, it is recommended to use UTF-8 encoding.
 
+### Wireless Transfer Rate Optimization
+
+**For optimal wireless access speeds, it is strongly recommended to use chips or modules with PSRAM, particularly ESP32-S3 equipped with 8MB or larger octal (OCT) PSRAM.**
+
+#### Performance Test Results
+
+The following chart shows the HTTP file upload speed analysis results with optimized configurations:
+
+![ESP HTTP File Upload Speed Analysis](./tools/upload_test_20250826_150059_analysis_20250826_151202.png)
+
+*Test results show an average upload speed of 4.24 MB/s with a maximum speed of 4.61 MB/s across 100 upload tests.*
+
+You can further improve wireless file upload and download speeds using the following methods:
+
+1. **Wi-Fi Configuration Optimization** (RAM impact: High)
+   - The following parameters have been configured in the `sdkconfig.defaults.r8` file:
+     - `CONFIG_ESP_WIFI_STATIC_RX_BUFFER_NUM=16`: Increases the number of static receive buffers (approx. 1.6KB each)
+     - `CONFIG_ESP_WIFI_DYNAMIC_RX_BUFFER_NUM=85`: Increases the number of dynamic receive buffers (approx. 1.6KB each)
+     - `CONFIG_ESP_WIFI_DYNAMIC_TX_BUFFER_NUM=64`: Increases the number of dynamic transmit buffers (approx. 1.6KB each)
+     - `CONFIG_ESP_WIFI_TX_BA_WIN=32` and `CONFIG_ESP_WIFI_RX_BA_WIN=32`: Enlarges the Block ACK window size
+     - `CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y`: Enables additional IRAM optimization (reduces RAM usage but increases IRAM usage)
+
+2. **TCP/IP Stack Optimization** (RAM impact: Medium-High)
+   - `CONFIG_LWIP_TCPIP_TASK_PRIO=23`: Increases TCP/IP task priority (no RAM impact)
+   - `CONFIG_LWIP_IRAM_OPTIMIZATION=y` and `CONFIG_LWIP_EXTRA_IRAM_OPTIMIZATION=y`: Enables LWIP IRAM optimization (increases IRAM usage)
+   - `CONFIG_LWIP_TCP_SND_BUF_DEFAULT=65535` and `CONFIG_LWIP_TCP_WND_DEFAULT=65535`: Enlarges TCP send and receive windows (significantly increases RAM usage per connection)
+   - `CONFIG_LWIP_TCPIP_RECVMBOX_SIZE=64`: Increases TCP/IP receive mailbox size (moderate RAM impact)
+   - `CONFIG_LWIP_TCP_RECVMBOX_SIZE=64` and `CONFIG_LWIP_UDP_RECVMBOX_SIZE=64`: Increases TCP and UDP receive mailbox sizes (moderate RAM impact)
+
+3. **File Operation Buffer Optimization** (RAM impact: High)
+   - `CONFIG_FILE_WRITE_BUFFER_COUNT=10`: Increases the number of write buffers
+   - `CONFIG_FILE_WRITE_BUFFER_SIZE=128`: Increases the size of write buffers
+   - `CONFIG_FILE_DMA_BUFFER_SIZE=32`: Sets DMA buffer size
+   - Total RAM impact approx: 10 × 128 + 32 = 1312 KB
+
+## Changelog
+
+### 2025-08-25
+
+- Support for 4-bit SD card interface
+- The file system employs asynchronous writing to enhance performance.
+- A section on optimizing wireless transmission rates has been added, along with detailed configuration guidelines.
+

examples/usb/device/usb_msc_wireless_disk/README_cn.md

@@ -48,3 +48,47 @@
   - 本例程默认为 <kbd>API uses UTF-8 encoding</kbd> (`FATFS_API_ENCODING_UTF_8`)。
   - 该选项控制 FATFS API 读出的文件名所使用的编码方式，详见该配置项的说明。
   - 若进行 Web 开发，建议使用 UTF 8 编码。
+
+### 无线传输速率优化
+
+**如果您希望获得最佳的无线访问速度，强烈推荐选择带有 PSRAM 的芯片或模组，特别是配备 8MB 及以上容量、8 线（OCT）PSRAM 的 ESP32-S3。**
+
+#### 性能测试结果
+
+以下图表显示了使用优化配置后的 HTTP 文件上传速度分析结果：
+
+![ESP HTTP File Upload Speed Analysis](./tools/upload_test_20250826_150059_analysis_20250826_151202.png)
+
+*测试结果显示，在 100 次上传测试中，平均上传速度为 4.24 MB/s，最大速度达到 4.61 MB/s。*
+
+通过以下方式可以进一步提高无线文件上传和下载的速度：
+
+1. **Wi-Fi 配置优化** (RAM 开销: 高)
+   - 在 `sdkconfig.defaults.r8` 文件中已经配置了以下参数：
+     - `CONFIG_ESP_WIFI_STATIC_RX_BUFFER_NUM=16`：增加静态接收缓冲区数量（每个缓冲区约 1.6KB）
+     - `CONFIG_ESP_WIFI_DYNAMIC_RX_BUFFER_NUM=85`：增加动态接收缓冲区数量（每个缓冲区约 1.6KB）
+     - `CONFIG_ESP_WIFI_DYNAMIC_TX_BUFFER_NUM=64`：增加动态发送缓冲区数量（每个缓冲区约 1.6KB）
+     - `CONFIG_ESP_WIFI_TX_BA_WIN=32` 和 `CONFIG_ESP_WIFI_RX_BA_WIN=32`：增大 Block ACK 窗口大小
+     - `CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y`：启用额外的 IRAM 优化（降低 RAM 使用但增加 IRAM 使用）
+
+2. **TCP/IP 协议栈优化** (RAM 开销: 中高)
+   - `CONFIG_LWIP_TCPIP_TASK_PRIO=23`：提高 TCP/IP 任务优先级（无 RAM 影响）
+   - `CONFIG_LWIP_IRAM_OPTIMIZATION=y` 和 `CONFIG_LWIP_EXTRA_IRAM_OPTIMIZATION=y`：启用 LWIP 的 IRAM 优化（增加 IRAM 使用）
+   - `CONFIG_LWIP_TCP_SND_BUF_DEFAULT=65535` 和 `CONFIG_LWIP_TCP_WND_DEFAULT=65535`：增大 TCP 发送和接收窗口（显著增加每个连接的 RAM 使用）
+   - `CONFIG_LWIP_TCPIP_RECVMBOX_SIZE=64`：增大 TCP/IP 接收邮箱大小（适度增加 RAM 使用）
+   - `CONFIG_LWIP_TCP_RECVMBOX_SIZE=64` 和 `CONFIG_LWIP_UDP_RECVMBOX_SIZE=64`：增大 TCP 和 UDP 接收邮箱大小（适度增加 RAM 使用）
+
+3. **文件操作缓冲区优化** (RAM 开销: 高)
+   - `CONFIG_FILE_WRITE_BUFFER_COUNT=10`：增加写入缓冲区数量
+   - `CONFIG_FILE_WRITE_BUFFER_SIZE=128`：增加写入缓冲区大小
+   - `CONFIG_FILE_DMA_BUFFER_SIZE=32`：设置 DMA 缓冲区大小
+   - 总计 RAM 开销约为: 10 × 128 + 32 = 1312 KB
+
+## 变更日志
+
+### 2025-08-25
+
+- 支持 4-bit SD 卡接口
+- 文件系统使用异步写入以提升性能
+- 增加了无线传输速率优化章节，提供详细的配置指南
+

examples/usb/device/usb_msc_wireless_disk/main/Kconfig.projbuild

@@ -23,6 +23,7 @@ menu "USB MSC Device Demo"
 
     choice STORAGE_MEDIA
         prompt "Storage Media"
+        default USE_EXTERNAL_SDCARD if ESP32_S3_USB_OTG
         default USE_INTERNAL_FLASH
 
         config USE_INTERNAL_FLASH
@@ -34,6 +35,7 @@ menu "USB MSC Device Demo"
         choice SDCARD_INTERFACE
             prompt "SD Card Interface"
             depends on USE_EXTERNAL_SDCARD
+            default SDCARD_INTFC_SDIO if ESP32_S3_USB_OTG
             default SDCARD_INTFC_SPI
 
             config SDCARD_INTFC_SPI
@@ -46,51 +48,184 @@ menu "USB MSC Device Demo"
         config DISK_BLOCK_SIZE
             int "disk block size used for format"
             depends on USE_EXTERNAL_SDCARD
-            default 512
+            range 512 32768
+            default 16384
+            help
+                Block size used for formatting the disk. A larger block size can improve performance
+                but may waste space on small files. Typical values are 4096 or 16384.
+
+        config FORMAT_IF_MOUNT_FAILED
+            bool "Format file system if mount failed"
+            default y
+            help
+                If this option is enabled, the file system will be formatted if mounting fails.
+
+        config FATFS_MAX_FILES
+            int "Maximum number of open files"
+            range 1 10
+            default 3
+            help
+                Maximum number of files that can be open at the same time.
 
         menu "SD Card PIN CONFIG"
 
         config SDCARD_SPI_CS_PIN
             int "SD Card SPI CS pin"
             depends on SDCARD_INTFC_SPI
-            default 34
+            default 34 if ESP32_S3_USB_OTG
+            default 10
 
         config SDCARD_SPI_CLK_PIN
             int "SD Card SPI CLK pin"
             depends on SDCARD_INTFC_SPI
-            default 36
+            default 36 if ESP32_S3_USB_OTG
+            default 12
 
         config SDCARD_SPI_DI_PIN
             int "SD Card SPI DI pin"
             depends on SDCARD_INTFC_SPI
-            default 35
+            default 35 if ESP32_S3_USB_OTG
+            default 11
 
         config SDCARD_SPI_DO_PIN
             int "SD Card SPI DO pin"
             depends on SDCARD_INTFC_SPI
-            default 37
+            default 37 if ESP32_S3_USB_OTG
+            default 13
 
-        config SDCARD_SDIO_DO_PIN
-            int "SD Card SDIO DO pin"
+        config SDCARD_SDIO_D0_PIN
+            int "SD Card SDIO D0 pin"
             depends on SDCARD_INTFC_SDIO
-            default 37
+            default 37 if ESP32_S3_USB_OTG
+            default 13
 
         config SDCARD_SDIO_CMD_PIN
             int "SD Card SDIO CMD pin"
             depends on SDCARD_INTFC_SDIO
-            default 35
+            default 35 if ESP32_S3_USB_OTG
+            default 11
 
         config SDCARD_SDIO_CLK_PIN
             int "SD Card SDIO CLK pin"
             depends on SDCARD_INTFC_SDIO
-            default 36
+            default 36 if ESP32_S3_USB_OTG
+            default 12
+
+        choice SDCARD_SDIO_DATA_WIDTH
+            prompt "SDIO Data Width"
+            depends on SDCARD_INTFC_SDIO
+            default SDCARD_SDIO_DATA_WIDTH_4 if ESP32_S3_USB_OTG
+            default SDCARD_SDIO_DATA_WIDTH_1
+            help
+                Select SDIO bus width: 1-line or 4-line.
+
+            config SDCARD_SDIO_DATA_WIDTH_1
+                bool "1-line mode"
+
+            config SDCARD_SDIO_DATA_WIDTH_4
+                bool "4-line mode"
+        endchoice
+
+        config SDCARD_SDIO_D1_PIN
+            int "SD Card SDIO D1 pin (for 4-line)"
+            depends on SDCARD_INTFC_SDIO && SDCARD_SDIO_DATA_WIDTH_4
+            default 38 if ESP32_S3_USB_OTG
+            default 14
+
+        config SDCARD_SDIO_D2_PIN
+            int "SD Card SDIO D2 pin (for 4-line)"
+            depends on SDCARD_INTFC_SDIO && SDCARD_SDIO_DATA_WIDTH_4
+            default 33 if ESP32_S3_USB_OTG
+            default 9
+
+        config SDCARD_SDIO_D3_PIN
+            int "SD Card SDIO D3 pin (for 4-line)"
+            depends on SDCARD_INTFC_SDIO && SDCARD_SDIO_DATA_WIDTH_4
+            default 34 if ESP32_S3_USB_OTG
+            default 10
         endmenu
     endchoice
 
     config WIFI_HTTP_ACCESS
         bool "Enable WiFi HTTP files server"
         default y
 
+    config HTTP_UPLOAD_MAX_FILE_SIZE_MB
+        int "Max upload file size (MB)"
+        default -1
+        depends on WIFI_HTTP_ACCESS
+        help
+            Set the maximum allowed file upload size in megabytes. Default is 20MB.
+            Set to -1 for no limit.
+
+    config HTTP_REDIRECT_TO_ROOT
+        bool "Redirect to root after file operations"
+        default n
+        depends on WIFI_HTTP_ACCESS
+        help
+            If enabled, after file upload or delete operations, the server will redirect to root path ("/").
+            This will trigger a full disk scan to update the file list. Disable this on large file systems
+            where scanning is slow.
+
+    config FILE_READ_BUFFER_SIZE
+        int "Size of HTTP file read buffer (KB)"
+        default 16
+        range 4 128
+        depends on WIFI_HTTP_ACCESS
+        help
+            Size of buffer used when reading files from storage during HTTP downloads in KB.
+            This affects the chunk size sent to clients. Larger values may improve download speed
+            but require more memory.
+
+    config FILE_WRITE_TASK_STACK_SIZE
+        int "File write task stack size"
+        default 4096
+        depends on WIFI_HTTP_ACCESS
+        help
+            Stack size in bytes for the file write task.
+            The task handles receiving data from the RingBuffer and
+            writing it to the filesystem.
+
+    config FILE_WRITE_BUFFER_COUNT
+        int "Number of RingBuffer items for file writes"
+        default 3
+        range 3 16
+        depends on WIFI_HTTP_ACCESS
+        help
+            Number of items that can be stored in the RingBuffer.
+            Higher values allow for more pending file chunks in the buffer,
+            which can improve throughput with slower SD cards.
+
+    config FILE_WRITE_BUFFER_SIZE
+        int "Size of each RingBuffer item (KB)"
+        default 16
+        range 16 512
+        depends on WIFI_HTTP_ACCESS
+        help
+            Size of each RingBuffer item in KB. This determines the maximum
+            size of each data chunk during file transfers. Larger values
+            may improve throughput but require more memory.
+
+    config FILE_DMA_BUFFER_SIZE
+        int "Size of DMA buffer for fwrite operations (KB)"
+        default 32
+        range 8 64
+        depends on WIFI_HTTP_ACCESS
+        help
+            Size of DMA buffer for file write operations in KB (kilobytes).
+            This buffer is used by the file system to improve write performance.
+            Increasing this value can improve SD card write throughput.
+
+    config FILE_WRITE_TIMEOUT_MS
+        int "File write timeout (milliseconds)"
+        default -1
+        range -1 300000
+        depends on WIFI_HTTP_ACCESS
+        help
+            Maximum time to wait for file write completion in milliseconds.
+            After this timeout, the HTTP upload will fail with a timeout error.
+            Default is -1, which means wait indefinitely.
+
     menu "WiFi Settings"
         config ESP_WIFI_ROUTE_SSID
             string "WiFi Route SSID"