Merge pull request #368 from Ljzd-PRO/devel

Ljzd-PRO · web-flow · commit 0fe34d212536 · 2025-12-28T23:14:11.000+08:00
Bump to v0.23.0
diff --git a/.github/workflows/codecov.yml b/.github/workflows/codecov.yml
@@ -32,11 +32,25 @@ jobs:
       cancel-in-progress: true
     strategy:
       fail-fast: false
+      # macos-14 aka. macos-latest has switched to being an ARM runner, only supporting newer versions of Python.
+      # https://github.com/actions/setup-python/issues/855#issuecomment-2096792205
       matrix:
-        # macos-14 aka. macos-latest has switched to being an ARM runner, only supporting newer versions of Python.
-        # https://github.com/actions/setup-python/issues/855#issuecomment-2096792205
-        os: [ ubuntu-latest, windows-latest, macos-13 ]
-        python-version: [ "3.8", "3.9", "3.10", "3.11", "3.12", "3.13" ]
+        os: [ ubuntu-latest, windows-latest, macos-15 ]
+        python-version: [ "3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14" ]
+        exclude:
+          - os: macos-15
+            python-version: "3.8"
+          - os: macos-15
+            python-version: "3.9"
+          - os: macos-15
+            python-version: "3.10"
+        include:
+          - os: macos-15-intel
+            python-version: "3.8"
+          - os: macos-15-intel
+            python-version: "3.9"
+          - os: macos-15-intel
+            python-version: "3.10"
     env:
       PYTEST_REPORT_FILENAME: report-${{ matrix.os }}-${{ matrix.python-version }}.html
 
diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml
@@ -42,7 +42,7 @@ jobs:
           - os: macos-latest
             arch: arm64
             python-arch: "x64"
-          - os: macos-13
+          - os: macos-15
             arch: x64
             python-arch: "x64"
           
@@ -70,7 +70,7 @@ jobs:
         if: matrix.qemu != true
         uses: ./.github/actions/setup-python
         with:
-          python-version: "3.13"
+          python-version: "3.14"
           architecture: ${{ matrix.python-arch }}
 
       - name: Create Executable (Native)
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,86 +1,30 @@
 ## Changes
 
-![Downloads](https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/v0.23.0/total)
+![Downloads](https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/v0.24.0/total)
 
 ### ✨ Features
 
-- Added configuration options to support **filtering** downloads by **file size** - #343
-  - You can set the minimum and maximum file size (in bytes) via `job.min_file_size` and `job.max_file_size`
-  - Both options can be set together to define a file size range
-  - Configure these options using the graphical config editor, or set them in the dotenv file `.env` or via system environment variables:
-    ```dotenv
-    # Skip files smaller than 1 MB (to avoid downloading thumbnails)
-    KTOOLBOX_JOB__MIN_FILE_SIZE=1048576
-
-    # Skip files larger than 50 MB (to save disk space)
-    KTOOLBOX_JOB__MAX_FILE_SIZE=52428800
-    ```
-  - 📖 More info: [Configuration-Reference-JobConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.JobConfiguration)
-- Improved progress bar output - #345
-  - Fixed the issue of the download file progress bar **constantly reordering**
-  - Added **visual overall progress bar**
-  - Added display of **total download speed**
-  - Enhanced the **color rendering** of the progress bar
-    ```
-    🔄   [==>---------------------------] 9% | Jobs: 173/1870 | 3 running | 1694 waiting | 5.7MB/s
-    
-    ⠹ 0bh1EKTGt5Zg9nNaDAi25P...    |███████████████████████████░░░| 3.7MB/4.0MB  92.5% ⚡ 1.9MB/s  
-    ⠹ YV30J8ftUbE9dUkkJVCqvN...    |███░░░░░░░░░░░░░░░░░░░░░░░░░░░| 527.0KB/4.1MB  12.5% ⚡ 1.9MB/s  
-    ⠹ KvKMSpwB4rRknTPKhEiXle...    |░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| 95.0KB/3.8MB   2.5% ⚡ 1.9MB/s  
-    ```
+- Track and **log files that already exist** during download progress, and add tests to cover file-exists handling
+- Throttle the progress display to **0.1s** and only refresh when content changes to reduce flicker and CPU usage
+- Add **Python 3.14** support and update CI/workflows to test against 3.14
 
 ### 🪲 Fixes
 
-- **Increased** the default **tps limit** (maximum number of connections established per second)
-  - This setting is optional. To **improve download efficiency** in general cases, the default value has been increased from `1.0` to `5.0`
-  - If you frequently encounter **403** errors during downloads, try setting this value lower, such as `1.0`
-  - Run `ktoolbox config-editor` to edit this setting (`Downloader -> tps_limit`)
-  - Or manually edit the `KTOOLBOX_DOWNLOADER__TPS_LIMIT` in the `.env` file or set it via environment variables
-    ```dotenv
-    KTOOLBOX_DOWNLOADER__TPS_LIMIT=1.0
-    ```
-  - 📖 More info: [Configuration-Reference-DownloaderConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.DownloaderConfiguration)
+- Ensure that the program **can terminate immediately** when all downloads are complete, rather than having to wait for a period of time before ending.
+- Prevent **overcounting completed jobs** when multiple files already exist
 
 - - -
 
 ### ✨ 新特性
 
-- 增加配置项以支持**按文件大小过滤**下载 - #343
-  - 通过配置 `job.min_file_size` 和 `job.max_file_size` 来设置最小和最大文件大小（单位：字节）
-  - 你可以同时设置这两个选项来定义一个文件大小范围
-  - 通过图形化配置编辑器或在 dotenv 文件 `.env` 或系统环境变量中设置这些配置：
-    ```dotenv
-    # 跳过小于 1 MB 的文件 （避免下载到缩略图）
-    KTOOLBOX_JOB__MIN_FILE_SIZE=1048576
-    
-    # 跳过大于 50 MB 的文件 （节省磁盘空间）
-    KTOOLBOX_JOB__MAX_FILE_SIZE=52428800
-    ```
-  - 📖 更多信息：[Configuration-Reference-JobConfiguration](https://ktoolbox.readthedocs.io/latest/zh/configuration/reference/#ktoolbox._configuration_zh.JobConfiguration)
-- 改进进度条输出 - #345
-  - 修复了下载文件进度条**不断重新排序**的问题
-  - 增加了**可视化的总进度条**
-  - 增加了下载**总速度**显示
-  - 增加了进度条的**颜色渲染**
-  ```
-  🔄   [==>---------------------------] 9% | Jobs: 173/1870 | 3 running | 1694 waiting | 5.7MB/s
-
-  ⠹ 0bh1EKTGt5Zg9nNaDAi25P...    |███████████████████████████░░░| 3.7MB/4.0MB  92.5% ⚡ 1.9MB/s
-  ⠹ YV30J8ftUbE9dUkkJVCqvN...    |███░░░░░░░░░░░░░░░░░░░░░░░░░░░| 527.0KB/4.1MB  12.5% ⚡ 1.9MB/s
-  ⠹ KvKMSpwB4rRknTPKhEiXle...    |░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| 95.0KB/3.8MB   2.5% ⚡ 1.9MB/s
-  ```
+- 在下载进度中**记录已存在的文件**并在进度中反映，添加了针对文件已存在处理的测试
+- 将进度刷新节流为**0.1秒**且仅在内容变更时刷新，以减少抖动和 CPU 使用
+- 添加 **Python 3.14** 支持，并在 CI/workflow 中加入对 3.14 的测试
 
 ### 🪲 修复
 
-- **提高**了默认的 **tps limit** （每秒最多建立的连接数）
-  - 这项配置是可选的，为了**提高一般情况下的下载效率**，现在的默认值从 `1.0` 提升到了 `5.0`
-  - 当下载频繁出现 **403** 错误时，可尝试将此设置改为较低值，如 `1.0`
-  - 执行 `ktoolbox config-editor` 来编辑这项配置 (`Downloader -> tps_limit`)
-  - 或手动编辑 `.env` 文件中的 `KTOOLBOX_DOWNLOADER__TPS_LIMIT` 或环境变量来设置这项配置
-    ```dotenv
-    KTOOLBOX_DOWNLOADER__TPS_LIMIT=1.0
-    ```
-  - 📖更多信息：[Configuration-Reference-DownloaderConfiguration](https://ktoolbox.readthedocs.io/latest/zh/configuration/reference/#ktoolbox._configuration_zh.DownloaderConfiguration)
+- 确保下载全部完成时程序**能即刻结束**，而不是要等待一段时间才能结束
+- 修复当多个文件已存在时**完成计数被重复计算**的问题
 
 ## Upgrade
 
@@ -89,4 +33,4 @@ Use this command to upgrade if you are using **pipx**:
 pipx upgrade ktoolbox
 ```
 
-**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.22.0...v0.23.0
+**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.23.0...v0.24.0
diff --git a/SECURITY.md b/SECURITY.md
@@ -4,7 +4,8 @@
 
 | Version | Supported          |
 |---------|--------------------|
-| 3.14.x  | :x:                |
+| 3.15.x  | :x:                |
+| 3.14.x  | :white_check_mark: |
 | 3.13.x  | :white_check_mark: |
 | 3.12.x  | :white_check_mark: |
 | 3.11.x  | :white_check_mark: |
diff --git a/ktoolbox/__init__.py b/ktoolbox/__init__.py
@@ -1,4 +1,4 @@
 __title__ = "KToolBox"
 # noinspection SpellCheckingInspection
 __description__ = "A useful CLI tool for downloading posts in Kemono.cr / .su / .party"
-__version__ = "v0.23.0"
+__version__ = "v0.24.0"
diff --git a/ktoolbox/job/runner.py b/ktoolbox/job/runner.py
@@ -139,11 +139,19 @@ async def processor(self) -> int:
                 if not exception:  # raise Exception when cancelled or other exceptions
                     ret = task_done.result()
                     if ret.code == RetCodeEnum.FileExisted:
-                        logger.info(ret.message)
-                        # Treat file existed as successful download
+                        logger.debug(ret.message)
+                        # Treat file existed as successful download but mark as existed
                         if self._progress_manager:
+                            # Increment existed count atomically and update completed based on current done_size
+                            try:
+                                self._progress_manager.increment_existed(1)
+                            except AttributeError:
+                                # Fallback if older ProgressManager doesn't have increment_existed
+                                self._progress_manager.update_job_progress(
+                                    existed=self._progress_manager._existed_jobs + 1
+                                )
                             self._progress_manager.update_job_progress(
-                                completed=self.done_size + 1
+                                completed=self.done_size
                             )
                     elif ret.code != RetCodeEnum.Success:
                         logger.error(ret.message)
@@ -157,7 +165,7 @@ async def processor(self) -> int:
                         # Update progress manager with completed job
                         if self._progress_manager:
                             self._progress_manager.update_job_progress(
-                                completed=self.done_size + 1
+                                completed=self.done_size
                             )
                 elif isinstance(exception, CancelledError):
                     logger.warning(
@@ -188,12 +196,21 @@ async def _watch_status(self):
         """
         Watch running, completed, failed jobs
         """
-        while not self._job_queue.empty():
-            await asyncio.sleep(30)
-            logger.info(f"Waiting: {self.waiting_size} / "
-                        f"Running: {self.processing_size} / "
-                        f"Completed: {self.done_size} "
-                        f"({(self.done_size / (self.waiting_size + self.processing_size + self.done_size)) * 100:.2f}%)")
+        try:
+            while not self._job_queue.empty():
+                await asyncio.sleep(30)
+                existed = self._progress_manager._existed_jobs if self._progress_manager else 0
+                total = (self.waiting_size + self.processing_size + self.done_size)
+                percent = (self.done_size / total) * 100 if total > 0 else 0
+                logger.info(
+                    f"Waiting: {self.waiting_size} / "
+                    f"Running: {self.processing_size} / "
+                    f"Completed: {self.done_size} "
+                    f"({percent:.2f}%) | Existed: {existed}"
+                )
+        except asyncio.CancelledError:
+            # Exit promptly when cancelled to allow fast shutdown
+            return
 
     async def start(self) -> int:
         """
@@ -224,10 +241,21 @@ async def start(self) -> int:
             if self._progress_manager:
                 display_task = asyncio.create_task(self._update_display_loop())
 
-            _, (task_done_set, _) = await asyncio.gather(
-                self._watch_status(),
-                asyncio.wait(self._concurrent_tasks)
-            )
+            # Start watcher as a background task so we can cancel it promptly when downloads finish
+            watch_task = None
+            if self._progress_manager:
+                watch_task = asyncio.create_task(self._watch_status())
+
+            # Wait for all concurrent processor tasks to finish
+            task_done_set, _ = await asyncio.wait(self._concurrent_tasks)
+
+            # Cancel watcher promptly to avoid waiting for its sleep interval
+            if watch_task:
+                watch_task.cancel()
+                try:
+                    await watch_task
+                except asyncio.CancelledError:
+                    pass
 
             if display_task:
                 display_task.cancel()
diff --git a/ktoolbox/progress.py b/ktoolbox/progress.py
@@ -179,6 +179,10 @@ class ProgressState:
     unit_scale: bool = False
     rate: Optional[float] = None
     last_update: float = field(default_factory=time.time)
+    # Tracks the last state timestamp that was actually rendered.
+    # Used to avoid advancing animation frames or forcing redraws
+    # when nothing meaningful has changed.
+    _last_render_seen: float = 0.0
     finished: bool = False
     failed: bool = False
     paused: bool = False
@@ -191,7 +195,8 @@ class ProgressManager:
     """
 
     def __init__(self, max_workers: int = 5, file: Optional[TextIO] = None,
-                 use_colors: bool = True, use_emojis: bool = True):
+                 use_colors: bool = True, use_emojis: bool = True,
+                 update_interval: float = 0.1):
         """
         Initialize the progress manager.
         
@@ -212,29 +217,41 @@ def __init__(self, max_workers: int = 5, file: Optional[TextIO] = None,
         self._total_jobs = 0
         self._completed_jobs = 0
         self._failed_jobs = 0
+        self._existed_jobs = 0
 
         # Terminal control
         self._lines_written = 0
         self._last_display_time = 0
-        self._update_interval = 0.1  # Update every 100ms
+        # Update interval (seconds). When downloads change, refresh at most
+        # once per `update_interval`. Default is 1.0s to avoid excessive redraws.
+        self._update_interval = float(update_interval)
 
         # Display deduplication
         self._last_display_content = ""
 
-    def set_job_totals(self, total: int, completed: int = 0, failed: int = 0):
+    def set_job_totals(self, total: int, completed: int = 0, failed: int = 0, existed: int = 0):
         """Set the total number of jobs for overall progress tracking"""
         with self._lock:
             self._total_jobs = total
             self._completed_jobs = completed
             self._failed_jobs = failed
+            self._existed_jobs = existed
 
-    def update_job_progress(self, completed: int = None, failed: int = None):
+    def update_job_progress(self, completed: int = None, failed: int = None, existed: int = None):
         """Update overall job progress"""
         with self._lock:
             if completed is not None:
                 self._completed_jobs = completed
             if failed is not None:
                 self._failed_jobs = failed
+            if existed is not None:
+                self._existed_jobs = existed
+
+    def increment_existed(self, n: int = 1) -> int:
+        """Atomically increment the existed count by n and return the new value"""
+        with self._lock:
+            self._existed_jobs += n
+            return self._existed_jobs
 
     def create_progress_bar(self, desc: str, total: Optional[int] = None,
                           unit: str = "B", unit_scale: bool = True) -> 'ManagedTqdm':
@@ -434,11 +451,13 @@ def _render_overall_progress(self) -> List[str]:
                 total_colored = ColorTheme.colorize(str(self._total_jobs), ColorTheme.BRIGHT_WHITE)
                 running_colored = ColorTheme.colorize(str(running), ColorTheme.BRIGHT_CYAN)
                 waiting_colored = ColorTheme.colorize(str(waiting), ColorTheme.BRIGHT_YELLOW)
+                existed_colored = ColorTheme.colorize(str(self._existed_jobs), ColorTheme.BRIGHT_WHITE)
             else:
                 completed_colored = str(self._completed_jobs)
                 total_colored = str(self._total_jobs)
                 running_colored = str(running)
                 waiting_colored = str(waiting)
+                existed_colored = str(self._existed_jobs)
 
             # Calculate overall download speed from active progress bars
             total_rate = 0
@@ -468,7 +487,8 @@ def _render_overall_progress(self) -> List[str]:
                 pct_colored,
                 f"| Jobs: {completed_colored}/{total_colored}",
                 f"| {running_colored} running",
-                f"| {waiting_colored} waiting"
+                f"| {waiting_colored} waiting",
+                f"| {existed_colored} existed"
             ])
             
             if speed_str:
@@ -521,10 +541,15 @@ def _render_single_progress_bar(self, state: ProgressState) -> str:
                 status_emoji = ColorTheme.WAITING
             else:
                 # Animated spinner for active downloads
+                # Advance spinner only when this progress state's content changed
+                # since the last render to avoid continuous terminal refreshes
+                # caused solely by animation frames.
                 current_time = time.time()
-                if current_time - _animation_state['last_update'] > 0.1:
+                last_seen = getattr(state, '_last_render_seen', 0.0)
+                if state.last_update != last_seen:
                     _animation_state['frame'] = (_animation_state['frame'] + 1) % len(ColorTheme.SPINNER_FRAMES)
                     _animation_state['last_update'] = current_time
+                    state._last_render_seen = state.last_update
                 status_emoji = ColorTheme.SPINNER_FRAMES[_animation_state['frame']]
         else:
             status_emoji = ""
diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
diff --git a/tests/ktoolbox/test_progress.py b/tests/ktoolbox/test_progress.py
