Merge pull request #347 from Ljzd-PRO/devel

Bump to v0.23.0

Author: Ljzd-PRO

CHANGELOG.md

@@ -1,52 +1,86 @@
 ## Changes
 
-![Downloads](https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/v0.22.0/total)
+![Downloads](https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/v0.23.0/total)
 
 ### ✨ Features
 
-- Added `job.download_file` and `job.download_attachments` configuration options to control **whether to download files and attachments in posts**
-  - Related configuration options:
-    - `job.download_file` (default: `True`): Whether to download the file corresponding to the `file` field of the post (usually the cover image)
-    - `job.download_attachments` (default: `True`): Whether to download files corresponding to the `attachments` field of the post (usually original images, compressed packages, etc.)
-  - These options are enabled by default to maintain the same behavior as previous versions
-  - You can edit these options via `ktoolbox config-editor` (`Job -> ...`)
-  - Or manually edit them in the `.env` file or environment variables:
+- Added configuration options to support **filtering** downloads by **file size** - #343
+  - You can set the minimum and maximum file size (in bytes) via `job.min_file_size` and `job.max_file_size`
+  - Both options can be set together to define a file size range
+  - Configure these options using the graphical config editor, or set them in the dotenv file `.env` or via system environment variables:
     ```dotenv
-    # Whether to download the file corresponding to the `file` field of the post (usually the cover image)
-    # Download enabled by default
-    KTOOLBOX_JOB__DOWNLOAD_FILE=False
+    # Skip files smaller than 1 MB (to avoid downloading thumbnails)
+    KTOOLBOX_JOB__MIN_FILE_SIZE=1048576
 
-    # Whether to download files corresponding to the `attachments` field of the post (usually original images, compressed packages, etc.)
-    # Download enabled by default
-    KTOOLBOX_JOB__DOWNLOAD_ATTACHMENTS=True
+    # Skip files larger than 50 MB (to save disk space)
+    KTOOLBOX_JOB__MAX_FILE_SIZE=52428800
+    ```
+  - 📖 More info: [Configuration-Reference-JobConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.JobConfiguration)
+- Improved progress bar output - #345
+  - Fixed the issue of the download file progress bar **constantly reordering**
+  - Added **visual overall progress bar**
+  - Added display of **total download speed**
+  - Enhanced the **color rendering** of the progress bar
+    ```
+    🔄   [==>---------------------------] 9% | Jobs: 173/1870 | 3 running | 1694 waiting | 5.7MB/s
+    
+    ⠹ 0bh1EKTGt5Zg9nNaDAi25P...    |███████████████████████████░░░| 3.7MB/4.0MB  92.5% ⚡ 1.9MB/s  
+    ⠹ YV30J8ftUbE9dUkkJVCqvN...    |███░░░░░░░░░░░░░░░░░░░░░░░░░░░| 527.0KB/4.1MB  12.5% ⚡ 1.9MB/s  
+    ⠹ KvKMSpwB4rRknTPKhEiXle...    |░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| 95.0KB/3.8MB   2.5% ⚡ 1.9MB/s  
     ```
-  - 📖 More information: [Configuration Reference - JobConfiguration](https://ktoolbox.readthedocs.io/latest/zh/configuration/reference/#ktoolbox._configuration_zh.JobConfiguration)
 
-[//]: # (### 🪲 Fixes)
+### 🪲 Fixes
+
+- **Increased** the default **tps limit** (maximum number of connections established per second)
+  - This setting is optional. To **improve download efficiency** in general cases, the default value has been increased from `1.0` to `5.0`
+  - If you frequently encounter **403** errors during downloads, try setting this value lower, such as `1.0`
+  - Run `ktoolbox config-editor` to edit this setting (`Downloader -> tps_limit`)
+  - Or manually edit the `KTOOLBOX_DOWNLOADER__TPS_LIMIT` in the `.env` file or set it via environment variables
+    ```dotenv
+    KTOOLBOX_DOWNLOADER__TPS_LIMIT=1.0
+    ```
+  - 📖 More info: [Configuration-Reference-DownloaderConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.DownloaderConfiguration)
 
 - - -
 
 ### ✨ 新特性
 
-- 新增 `job.download_file` 和 `job.download_attachments` 配置项，控制**是否下载帖子中的文件和附件**
-  - 相关配置项：
-    - `job.download_file`（默认值：`True`）：是否下载帖子的 `file` 字段对应的文件（通常是帖子封面图）
-    - `job.download_attachments`（默认值：`True`）：是否下载帖子的 `attachments` 字段对应的文件（通常是帖子中的正式原图、压缩包等附件）
-  - 该配置项默认均为开启状态，保持与之前版本一致的行为
-  - 可通过运行 `ktoolbox config-editor` 编辑这些配置（`Job -> ...`）
-  - 或手动在 `.env` 文件或环境变量中编辑：
+- 增加配置项以支持**按文件大小过滤**下载 - #343
+  - 通过配置 `job.min_file_size` 和 `job.max_file_size` 来设置最小和最大文件大小（单位：字节）
+  - 你可以同时设置这两个选项来定义一个文件大小范围
+  - 通过图形化配置编辑器或在 dotenv 文件 `.env` 或系统环境变量中设置这些配置：
     ```dotenv
-    # 是否下载帖子的 `file` 字段对应的文件（通常是帖子封面图）
-    # 默认开启下载
-    KTOOLBOX_JOB__DOWNLOAD_FILE=False
+    # 跳过小于 1 MB 的文件 （避免下载到缩略图）
+    KTOOLBOX_JOB__MIN_FILE_SIZE=1048576
     
-    # 是否下载帖子的 `attachments` 字段对应的文件（通常是帖子中的正式原图、压缩包等附件）
-    # 默认开启下载
-    KTOOLBOX_JOB__DOWNLOAD_ATTACHMENTS=True
+    # 跳过大于 50 MB 的文件 （节省磁盘空间）
+    KTOOLBOX_JOB__MAX_FILE_SIZE=52428800
     ```
-  - 📖更多信息：[配置参考-JobConfiguration](https://ktoolbox.readthedocs.io/latest/zh/configuration/reference/#ktoolbox._configuration_zh.JobConfiguration)
+  - 📖 更多信息：[Configuration-Reference-JobConfiguration](https://ktoolbox.readthedocs.io/latest/zh/configuration/reference/#ktoolbox._configuration_zh.JobConfiguration)
+- 改进进度条输出 - #345
+  - 修复了下载文件进度条**不断重新排序**的问题
+  - 增加了**可视化的总进度条**
+  - 增加了下载**总速度**显示
+  - 增加了进度条的**颜色渲染**
+  ```
+  🔄   [==>---------------------------] 9% | Jobs: 173/1870 | 3 running | 1694 waiting | 5.7MB/s
+
+  ⠹ 0bh1EKTGt5Zg9nNaDAi25P...    |███████████████████████████░░░| 3.7MB/4.0MB  92.5% ⚡ 1.9MB/s
+  ⠹ YV30J8ftUbE9dUkkJVCqvN...    |███░░░░░░░░░░░░░░░░░░░░░░░░░░░| 527.0KB/4.1MB  12.5% ⚡ 1.9MB/s
+  ⠹ KvKMSpwB4rRknTPKhEiXle...    |░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| 95.0KB/3.8MB   2.5% ⚡ 1.9MB/s
+  ```
+
+### 🪲 修复
 
-[//]: # (### 🪲 修复)
+- **提高**了默认的 **tps limit** （每秒最多建立的连接数）
+  - 这项配置是可选的，为了**提高一般情况下的下载效率**，现在的默认值从 `1.0` 提升到了 `5.0`
+  - 当下载频繁出现 **403** 错误时，可尝试将此设置改为较低值，如 `1.0`
+  - 执行 `ktoolbox config-editor` 来编辑这项配置 (`Downloader -> tps_limit`)
+  - 或手动编辑 `.env` 文件中的 `KTOOLBOX_DOWNLOADER__TPS_LIMIT` 或环境变量来设置这项配置
+    ```dotenv
+    KTOOLBOX_DOWNLOADER__TPS_LIMIT=1.0
+    ```
+  - 📖更多信息：[Configuration-Reference-DownloaderConfiguration](https://ktoolbox.readthedocs.io/latest/zh/configuration/reference/#ktoolbox._configuration_zh.DownloaderConfiguration)
 
 ## Upgrade
 
@@ -55,4 +89,4 @@ Use this command to upgrade if you are using **pipx**:
 pipx upgrade ktoolbox
 ```
 
-**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.21.1...v0.22.0
+**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.22.0...v0.23.0

README.md

@@ -67,6 +67,9 @@
     - Such as `[2025-01-02]_TheTitle_1.jpg`, `[2025-01-02]_TheTitle_2.jpg`, `[2025-01-02]_TheTitle_3.jpg`, etc.
 - Support excluding **specified file formats** or downloading only specified formats
   - For example, if you don't want to download large and duplicate PSD or archive files, you can exclude `.psd` and `.zip` files in the config
+- Support filtering downloads by **file size**
+  - For example, if you want to avoid downloading large video files when running out of disk space, you can set a maximum file size limit in the config
+  - You can also set a minimum file size to skip downloading thumbnail or preview images
 - Support filtering downloads by post **title keywords**
   - For example, if you only want to download posts whose titles contain "表情" or "効果音差分", you can use the `sync-creator` command with the `--keywords` option
   - You can also exclude posts with specific keywords in the title using the `--keywords-exclude` option

README_zh-CN.md

@@ -68,6 +68,9 @@
     - 如 `[2025-01-02]_TheTitle_1.jpg`、`[2025-01-02]_TheTitle_2.jpg`、`[2025-01-02]_TheTitle_3.jpg` 等
 - 支持排除 **指定格式** 的文件或仅下载指定格式的文件
   - 例如当你不想下载庞大重复的 PSD 和压缩包文件时，可以在配置中排除 `.psd` 和 `.zip` 文件
+- 支持按**文件大小**过滤下载
+  - 例如，如果你想在磁盘空间不足时避免下载大型视频文件，可以在配置中设置最大文件大小限制
+  - 你也可以设置最小文件大小，以跳过下载缩略图或预览图片
 - 支持按帖子**标题关键词**过滤下载
   - 例如你只想下载标题中包含“表情、効果音差分”的帖子，可以使用 `sync-creator` 命令的 `--keywords` 选项
   - 如果你想排除标题中包含指定关键词的帖子，可以使用 `--keywords-exclude` 选项

demo_colorama_progress.py

@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+"""
+Demonstration script for the new colorama-based progress bars
+Shows the new Unicode progress bar style: ━━━╺━━━━━━━
+"""
+
+import time
+import sys
+from ktoolbox.progress import ProgressManager
+
+def demo_colorama_progress():
+    """Demo the new colorama-based progress bar display"""
+    print("🎨 Colorama Progress Bar Demo")
+    print("=" * 50)
+    
+    # Create progress manager with colors enabled
+    manager = ProgressManager(max_workers=3, use_colors=True, use_emojis=True)
+    manager.set_job_totals(10, completed=2, failed=0)
+    
+    print(f"Colorama available: {manager.use_colors}")
+    print(f"Terminal supports colors: {manager.use_colors}")
+    print()
+    
+    # Start display
+    manager.start_display()
+    
+    try:
+        # Create progress bars to demonstrate the new Unicode style
+        pbar1 = manager.create_progress_bar("image1.jpg", total=1000)
+        pbar2 = manager.create_progress_bar("video.mp4", total=5000)
+        pbar3 = manager.create_progress_bar("document.pdf", total=800)
+        
+        # Simulate progress with different speeds
+        for i in range(50):
+            # Update progress bars at different rates
+            pbar1.update(20)
+            if i % 2 == 0:
+                pbar2.update(100)
+            if i % 3 == 0:
+                pbar3.update(16)
+            
+            # Update the display
+            manager.update_display()
+            time.sleep(0.1)
+            
+            # Update job progress occasionally
+            if i % 10 == 0:
+                manager.update_job_progress(completed=2 + (i // 10))
+        
+        # Mark some as finished
+        pbar1.close()
+        manager.update_job_progress(completed=4)
+        
+        # Continue with remaining
+        for i in range(30):
+            pbar2.update(50)
+            pbar3.update(10)
+            manager.update_display()
+            time.sleep(0.05)
+        
+        pbar2.close()
+        pbar3.close()
+        manager.update_job_progress(completed=10)
+        
+        # Show final state briefly
+        time.sleep(1)
+        
+    finally:
+        manager.stop_display()
+    
+    print("\n✅ Demo completed!")
+    print("You should have seen:")
+    print("- Pink/magenta progress bars with ━━━╺━━━━━━━ style (when colorama available)")
+    print("- Gray remaining sections")
+    print("- Overall job progress with visual bar")
+    print("- Fallback to █░ characters when colorama not available")
+
+if __name__ == "__main__":
+    demo_colorama_progress()

docs/en/index.md

@@ -55,6 +55,9 @@
     - Such as `[2025-01-02]_TheTitle_1.jpg`, `[2025-01-02]_TheTitle_2.jpg`, `[2025-01-02]_TheTitle_3.jpg`, etc.
 - Support excluding **specified file formats** or downloading only specified formats
   - For example, if you don't want to download large and duplicate PSD or archive files, you can exclude `.psd` and `.zip` files in the config
+- Support filtering downloads by **file size**
+  - For example, if you want to avoid downloading large video files when running out of disk space, you can set a maximum file size limit in the config
+  - You can also set a minimum file size to skip downloading thumbnail or preview images
 - Support filtering downloads by post **title keywords**
   - For example, if you only want to download posts whose titles contain "表情" or "効果音差分", you can use the `sync-creator` command with the `--keywords` option
   - You can also exclude posts with specific keywords in the title using the `--keywords-exclude` option

docs/zh/index.md

@@ -56,6 +56,9 @@
     - 如 `[2025-01-02]_TheTitle_1.jpg`、`[2025-01-02]_TheTitle_2.jpg`、`[2025-01-02]_TheTitle_3.jpg` 等
 - 支持排除 **指定格式** 的文件或仅下载指定格式的文件
   - 例如当你不想下载庞大重复的 PSD 和压缩包文件时，可以在配置中排除 `.psd` 和 `.zip` 文件
+- 支持按**文件大小**过滤下载
+  - 例如，如果你想在磁盘空间不足时避免下载大型视频文件，可以在配置中设置最大文件大小限制
+  - 你也可以设置最小文件大小，以跳过下载缩略图或预览图片
 - 支持按帖子**标题关键词**过滤下载
   - 例如你只想下载标题中包含“表情、効果音差分”的帖子，可以使用 `sync-creator` 命令的 `--keywords` 选项
   - 如果你想排除标题中包含指定关键词的帖子，可以使用 `--keywords-exclude` 选项

ktoolbox/__init__.py

@@ -1,4 +1,4 @@
 __title__ = "KToolBox"
 # noinspection SpellCheckingInspection
 __description__ = "A useful CLI tool for downloading posts in Kemono.cr / .su / .party"
-__version__ = "v0.22.0"
+__version__ = "v0.23.0"

ktoolbox/_configuration_zh.py

@@ -146,6 +146,8 @@ class JobConfiguration(ktoolbox.configuration.JobConfiguration):
     :ivar keywords_exclude: 按帖子标题关键词排除（不区分大小写）
     :ivar download_file: 是否下载帖子文件（通常为封面图片）。设置为 False 可跳过文件下载。
     :ivar download_attachments: 是否下载帖子附件。设置为 False 可跳过附件下载。
+    :ivar min_file_size: 最小文件大小（字节）。小于此大小的文件将被跳过。设置为 None 禁用最小文件大小过滤。
+    :ivar max_file_size: 最大文件大小（字节）。大于此大小的文件将被跳过。设置为 None 禁用最大文件大小过滤。
     """
     post_structure: PostStructureConfiguration = PostStructureConfiguration()
 

ktoolbox/configuration.py

@@ -78,7 +78,7 @@ class DownloaderConfiguration(BaseModel):
     retry_times: int = 10
     retry_stop_never: bool = False
     retry_interval: float = 3.0
-    tps_limit: float = 1.0
+    tps_limit: float = 5.0
     use_bucket: bool = False
     bucket_path: Path = Path("./.ktoolbox/bucket_storage")
     reverse_proxy: str = "{}"
@@ -224,6 +224,10 @@ class JobConfiguration(BaseModel):
     :ivar keywords_exclude: keywords to exclude posts by title (case-insensitive)
     :ivar download_file: Download post file (usually cover image). Set to False to skip file downloads.
     :ivar download_attachments: Download post attachments. Set to False to skip attachment downloads.
+    :ivar min_file_size: Minimum file size in bytes to download. Files smaller than this will be skipped. \
+    Set to None to disable minimum size filtering.
+    :ivar max_file_size: Maximum file size in bytes to download. Files larger than this will be skipped. \
+    Set to None to disable maximum size filtering.
     """
     count: int = 4
     include_revisions: bool = False
@@ -297,6 +301,8 @@ class JobConfiguration(BaseModel):
     keywords_exclude: Set[str] = Field(default_factory=set)
     download_file: bool = True
     download_attachments: bool = True
+    min_file_size: Optional[int] = None
+    max_file_size: Optional[int] = None
 
 
 class LoggerConfiguration(BaseModel):