Merge pull request #286 from Ljzd-PRO/devel

Bump to v0.17.0

Author: Ljzd-PRO

CHANGELOG.md

@@ -1,38 +1,69 @@
 ## Changes
 
-![Downloads](https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/v0.16.0/total)
+![Downloads](https://img.shields.io/github/downloads/Ljzd-PRO/KToolBox/v0.17.0/total)
 
 ### 💡 Feature
 
-- Add auto-managed cookies to bypass **DDoS Guard** - #269 (@CanglanXYA)
-- Add comprehensive **revision post** support with enhanced API and configuration - #240, #241
-  - For posts like: `https://kemono.cr/{service}/user/{user_id}/post/{post_id}/revision/{revision_id}`
-  - This feature is disabled by default
-  - Run `ktoolbox config-editor` to edit this configurations (`Job -> include_revisions`)
-  - Or manually edit it in `.env` file or environment variables
+- Support download **images embedded in post HTML content** - #218
+- Add external links extraction feature for **cloud storage URLs** - #232 (@xxkzn)
+  - New configurations:
+    - `job.extract_external_links`: Whether to extract external file sharing links from post content and save to separate file
+    - `job.external_link_patterns`: Regex patterns for extracting external links
+  - These configuration are **optional**, with the feature enabled by default. The regular expression includes the following services:
+    - Google Drive
+    - MEGA
+    - Dropbox
+    - OneDrive
+    - MediaFire
+    - And other common file hosting services
+  - Run `ktoolbox config-editor` to edit these configurations (`Job -> extract_external_links`, `Job -> external_link_patterns`)
+  - Or manually edit them `.env` file or environment variables
     ```dotenv
-    # Set this to `True` to enable revisions download
-    KTOOLBOX_JOB__INCLUDE_REVISIONS=True
+    # This feature is enabled by default
+    KTOOLBOX_JOB__EXTRACT_EXTERNAL_LINKS=True
+    # Setting up lists and regular expressions in dotenv is relatively complex and cumbersome. It is recommended to use the aforementioned graphical configuration editor for these settings.
+    KTOOLBOX_JOB__EXTERNAL_LINK_PATTERNS='["https?://drive\\.google\\.com/[^\\s]+", "https?://docs\\.google\\.com/[^\\s]+", "https?://mega\\.nz/[^\\s]+", "https?://mega\\.co\\.nz/[^\\s]+", "https?://(?:www\\.)?dropbox\\.com/[^\\s]+", "https?://db\\.tt/[^\\s]+", "https?://onedrive\\.live\\.com/[^\\s]+", "https?://1drv\\.ms/[^\\s]+", "https?://(?:www\\.)?mediafire\\.com/[^\\s]+", "https?://(?:www\\.)?wetransfer\\.com/[^\\s]+", "https?://we\\.tl/[^\\s]+", "https?://(?:www\\.)?sendspace\\.com/[^\\s]+", "https?://(?:www\\.)?4shared\\.com/[^\\s]+", "https?://(?:www\\.)?zippyshare\\.com/[^\\s]+", "https?://(?:www\\.)?uploadfiles\\.io/[^\\s]+", "https?://(?:www\\.)?box\\.com/[^\\s]+", "https?://(?:www\\.)?pcloud\\.com/[^\\s]+", "https?://disk\\.yandex\\.[a-z]+/[^\\s]+", "https?://[^\\s]*(?:file|upload|share|download|drive|storage)[^\\s]*\\.[a-z]{2,4}/[^\\s]+"]'
     ```
+  - 📖More information: [Configuration-Reference-JobConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.JobConfiguration)
 
-[//]: # (### 🪲 Fix)
+### 🪲 Fix
+
+- Removed the deprecated configuration `job.post_structure.content_filepath`, use `job.post_structure.content` instead
+- Fixed an issue where the `sync-creator` command lacked handling for 404 responses when fetching post revisions
+  (i\.e\. no revision version exists), which caused **slow task creation** - #294
+- Fixed the issue of **duplicate Cookies** in DDoS Guard management (manual management is no longer performed)
 
 - - -
 
 ### 💡 新特性
 
-- 新增自动管理 Cookie 功能以绕过 **DDoS Guard** - #269 (@CanglanXYA)
-- 新增全面的**修订作品**支持，增强 API 和配置功能 - #240, #241
-  - 适用于如下作品：`https://kemono.cr/{service}/user/{user_id}/post/{post_id}/revision/{revision_id}`
-  - 此功能默认关闭
-  - 运行 `ktoolbox config-editor` 可编辑此配置项（`Job -> include_revisions`）
-  - 或在 `.env` 文件或环境变量中手动编辑
+- 支持下载**帖子 HTML 内容中嵌入的图片** - #218
+- 新增**云存储 URL 外链提取**功能 - #232 (@xxkzn)
+  - 新增配置项：
+    - `job.extract_external_links`：是否从帖子内容中提取外部文件分享链接并保存到单独文件
+    - `job.external_link_patterns`：用于提取外链的正则表达式模式
+  - 这些配置项为**可选**，该功能默认启用。正则表达式已包含以下服务：
+    - Google Drive
+    - MEGA
+    - Dropbox
+    - OneDrive
+    - MediaFire
+    - 及其他常见文件托管服务
+  - 可运行 `ktoolbox config-editor` 编辑这些配置（`Job -> extract_external_links`，`Job -> external_link_patterns`）
+  - 或手动编辑 `.env` 文件或环境变量
     ```dotenv
-    # 设置为 `True` 以启用修订下载
-    KTOOLBOX_JOB__INCLUDE_REVISIONS=True
+    # 此功能默认启用
+    KTOOLBOX_JOB__EXTRACT_EXTERNAL_LINKS=True
+    # 在 dotenv 中设置列表和正则表达式较为复杂，推荐使用上述图形化配置编辑器进行设置。
+    KTOOLBOX_JOB__EXTERNAL_LINK_PATTERNS='["https?://drive\\.google\\.com/[^\\s]+", "https?://docs\\.google\\.com/[^\\s]+", "https?://mega\\.nz/[^\\s]+", "https?://mega\\.co\\.nz/[^\\s]+", "https?://(?:www\\.)?dropbox\\.com/[^\\s]+", "https?://db\\.tt/[^\\s]+", "https?://onedrive\\.live\\.com/[^\\s]+", "https?://1drv\\.ms/[^\\s]+", "https?://(?:www\\.)?mediafire\\.com/[^\\s]+", "https?://(?:www\\.)?wetransfer\\.com/[^\\s]+", "https?://we\\.tl/[^\\s]+", "https?://(?:www\\.)?sendspace\\.com/[^\\s]+", "https?://(?:www\\.)?4shared\\.com/[^\\s]+", "https?://(?:www\\.)?zippyshare\\.com/[^\\s]+", "https?://(?:www\\.)?uploadfiles\\.io/[^\\s]+", "https?://(?:www\\.)?box\\.com/[^\\s]+", "https?://(?:www\\.)?pcloud\\.com/[^\\s]+", "https?://disk\\.yandex\\.[a-z]+/[^\\s]+", "https?://[^\\s]*(?:file|upload|share|download|drive|storage)[^\\s]*\\.[a-z]{2,4}/[^\\s]+"]'
     ```
+  - 📖更多信息：[配置参考-JobConfiguration](https://ktoolbox.readthedocs.io/latest/configuration/reference/#ktoolbox.configuration.JobConfiguration)
     
-[//]: # (### 🪲 修复)
+### 🪲 修复
+
+- 移除了过时的配置 `job.post_structure.content_filepath`，请用 `job.post_structure.content` 代替
+- 修复 `sync-creator` 命令在 **获取帖子修订（revision）** 时缺少 404 响应的处理（即帖子无修订版本）导致的**任务创建缓慢**的问题 - #294
+- 修复 DDoS Guard Cookies 管理出现**重复 Cookie** 的问题（不再进行手动管理）
 
 ## Upgrade
 
@@ -41,4 +72,4 @@ Use this command to upgrade if you are using **pipx**:
 pipx upgrade ktoolbox
 ```
 
-**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.15.1...v0.16.0
+**Full Changelog**: https://github.com/Ljzd-PRO/KToolBox/compare/v0.16.0...v0.17.0

docs/en/configuration/guide.md

@@ -28,6 +28,15 @@
 ## `.env` / `prod.env` file example
 
 ```dotenv
+##############################################################################
+#  It is recommended to use the graphical configuration editor for editing.  #
+#  Run `ktoolbox config-editor` to launch it.                                #
+##############################################################################
+
+# (Optional) Session key that can be found in cookies after a successful login
+# Use when 403 Error
+#KTOOLBOX_API__SESSION_KEY=xxxxx
+
 # Download 10 files at the same time.
 KTOOLBOX_JOB__COUNT=10
 
@@ -43,7 +52,7 @@ KTOOLBOX_JOB__SEQUENTIAL_FILENAME=True
 # For example: `{title}_{}` > `HelloWorld_b4b41de2-8736-480d-b5c3-ebf0d917561b`, etc.
 # You can also use it with `sequential_filename`. For instance,
 # `[{published}]_{}` > `[2024-1-1]_1.png`, `[2024-1-1]_2.png`, etc.
-KTOOLBOX_JOB__FILENAME_FORMAT=[{published}]_{}
+KTOOLBOX_JOB__FILENAME_FORMAT=[{published}]_{title}_{id}_{}
 
 # Prefix the post directory name with its release/publish date, e.g. `[2024-1-1]HelloWorld`
 KTOOLBOX_JOB__POST_DIRNAME_FORMAT=[{published}]{title}

docs/zh/configuration/guide.md

@@ -27,23 +27,32 @@
 ## `.env` / `prod.env` 文件示例
 
 ```dotenv
-# 可同时下载10个文件
+##############################################################################
+#  推荐使用图形化配置编辑器进行编辑。                                         #
+#  运行 `ktoolbox config-editor` 启动编辑器。                                #
+##############################################################################
+
+# （可选）会话密钥，可在成功登录后的 cookies 中找到
+# 403 错误时使用
+#KTOOLBOX_API__SESSION_KEY=xxxxx
+
+# 同时下载 10 个文件。
 KTOOLBOX_JOB__COUNT=10
 
-# 设置作品附件目录为 `./`, 这意味着所有附件将直接保存在作品目录下
-# 而不会创建一个子目录来储存
+# 设置帖子附件目录路径为 `./`，表示将所有附件文件保存在帖子目录下
+# 不会为附件单独创建子目录
 KTOOLBOX_JOB__POST_STRUCTURE__ATTACHMENTS=./
 
-# 按照数字顺序重命名附件, 例如 `1.png`, `2.png`, ...
+# 附件按数字顺序重命名，例如 `1.png`、`2.png`、……
 KTOOLBOX_JOB__SEQUENTIAL_FILENAME=True
 
-# 通过插入一个代表了基本文件名的空白的 `{}` 以自定义文件名格式 
-# 与 `post_dirname_format` 类似，你可以使用一些 `Post` 类里的属性
-# 例如 `{title}_{}` > `HelloWorld_b4b41de2-8736-480d-b5c3-ebf0d917561b`
-# 你也可以和 `sequential_filename` 搭配使用
-# 例如 `[{published}]_{}` > `[2024-1-1]_1.png`, `[2024-1-1]_2.png`
-KTOOLBOX_JOB__FILENAME_FORMAT=[{published}]_{}
+# 通过插入空的 `{}` 自定义文件名格式，表示基础文件名。
+# 类似于 `post_dirname_format`，可以使用 `Post` 中的一些属性。
+# 例如：`{title}_{}` > `HelloWorld_b4b41de2-8736-480d-b5c3-ebf0d917561b` 等。
+# 也可以与 `sequential_filename` 一起使用。例如，
+# `[{published}]_{}` > `[2024-1-1]_1.png`、`[2024-1-1]_2.png` 等。
+KTOOLBOX_JOB__FILENAME_FORMAT=[{published}]_{title}_{id}_{}
 
-# 将发布日期作为作品目录名的开头，例如 `[2024-1-1]HelloWorld`
+# 帖子目录名以发布时间作为前缀，例如 `[2024-1-1]HelloWorld`
 KTOOLBOX_JOB__POST_DIRNAME_FORMAT=[{published}]{title}
 ```

ktoolbox/__init__.py

@@ -1,4 +1,4 @@
 __title__ = "KToolBox"
 # noinspection SpellCheckingInspection
 __description__ = "A useful CLI tool for downloading posts in Kemono.cr / .su / .party"
-__version__ = "v0.16.0"
+__version__ = "v0.17.0"

ktoolbox/action/job.py

@@ -16,6 +16,7 @@
 from ktoolbox.api.posts import get_post_revisions as get_post_revisions_api
 from ktoolbox.configuration import config, PostStructureConfiguration
 from ktoolbox.job import Job, CreatorIndices
+from ktoolbox.utils import extract_external_links
 
 __all__ = ["create_job_from_post", "create_job_from_creator"]
 
@@ -46,9 +47,12 @@ async def create_job_from_post(
         attachments_path.mkdir(exist_ok=True)
         content_path = post_path / post_structure.content  # content
         content_path.parent.mkdir(exist_ok=True)
+        external_links_path = post_path / post_structure.external_links  # external_links
+        external_links_path.parent.mkdir(exist_ok=True)
     else:
         attachments_path = post_path
         content_path = None
+        external_links_path = None
 
     # Filter and create jobs for ``Post.attachment``
     jobs: List[Job] = []
@@ -110,6 +114,16 @@ async def create_job_from_post(
     if content_path and post.content:
         async with aiofiles.open(content_path, "w", encoding=config.downloader.encoding) as f:
             await f.write(post.content)
+
+    # Extract and write external links file
+    if config.job.extract_external_links and external_links_path and post.content:
+        external_links = extract_external_links(post.content, config.job.external_link_patterns)
+        if external_links:
+            async with aiofiles.open(external_links_path, "w", encoding=config.downloader.encoding) as f:
+                # Write each link on a separate line
+                for link in sorted(external_links):
+                    await f.write(f"{link}\n")
+
     if dump_post_data:
         async with aiofiles.open(str(post_path / DataStorageNameEnum.PostData.value), "w", encoding="utf-8") as f:
             await f.write(
@@ -194,6 +208,9 @@ async def create_job_from_creator(
             ) as f:
                 await f.write(indices.model_dump_json(indent=config.json_dump_indent))
 
+    logger.info("`job.include_revisions` is enabled and will fetch post revisions, "
+                "which may take time. Disable if not needed.")
+
     job_list: List[Job] = []
     for post in post_list:
         # Get post path

ktoolbox/api/base.py

@@ -12,7 +12,6 @@
 from ktoolbox._enum import RetCodeEnum
 from ktoolbox.configuration import config
 from ktoolbox.utils import BaseRet, generate_msg
-from ktoolbox.ddos_guard import DDoSGuardCookieManager, merge_cookies
 
 __all__ = ["APITenacityStop", "APIRet", "BaseAPI"]
 
@@ -69,30 +68,9 @@ class BaseAPI(ABC, Generic[_T]):
     path: str = "/"
     method: Literal["get", "post"]
     extra_validator: Optional[Callable[[str], BaseModel]] = None
-    
-    # Initialize DDoS Guard cookie manager
-    _ddos_cookie_manager: Optional[DDoSGuardCookieManager] = None
-    
-    @classmethod
-    def _get_ddos_cookie_manager(cls) -> DDoSGuardCookieManager:
-        """Get or create DDoS Guard cookie manager"""
-        if cls._ddos_cookie_manager is None:
-            cls._ddos_cookie_manager = DDoSGuardCookieManager()
-            logger.debug("Initialized DDoS Guard cookie manager")
-        return cls._ddos_cookie_manager
-    
-    @classmethod
-    def _build_cookies(cls) -> Optional[dict]:
-        """Build cookies including session and DDoS Guard cookies"""
-        session_cookies = {"session": config.api.session_key} if config.api.session_key else None
-        ddos_manager = cls._get_ddos_cookie_manager()
-        ddos_cookies = ddos_manager.cookies
-        
-        return merge_cookies(session_cookies, ddos_cookies)
-    
     client = httpx.AsyncClient(
         verify=config.ssl_verify,
-        cookies=None  # We'll set cookies dynamically
+        cookies={"session": config.api.session_key} if config.api.session_key else None
     )
 
     Response = BaseModel
@@ -113,7 +91,7 @@ def handle_res(cls, res: httpx.Response) -> APIRet[_T]:
                     message=str(e),
                     exception=e
                 )
-            elif isinstance(e, ValidationError):
+            else:
                 return APIRet(
                     code=RetCodeEnum.ValidationError,
                     message=str(e),
@@ -135,12 +113,6 @@ async def request(cls, path: str = None, **kwargs) -> APIRet[_T]:
             path = cls.path
         url_parts = [config.api.scheme, config.api.netloc, f"{config.api.path}{path}", '', '', '']
         url = str(urlunparse(url_parts))
-        
-        # Build cookies dynamically to include any updates
-        cookies = cls._build_cookies()
-        if cookies:
-            kwargs.setdefault('cookies', {}).update(cookies)
-        
         try:
             res = await cls.client.request(
                 method=cls.method,
@@ -149,11 +121,6 @@ async def request(cls, path: str = None, **kwargs) -> APIRet[_T]:
                 follow_redirects=True,
                 **kwargs
             )
-            
-            # Update DDoS Guard cookies from response
-            ddos_manager = cls._get_ddos_cookie_manager()
-            ddos_manager.update_from_response(res)
-                
         except Exception as e:
             return APIRet(
                 code=RetCodeEnum.NetWorkError,

ktoolbox/api/posts/get_post_revisions.py

@@ -1,11 +1,15 @@
-from pydantic import BaseModel, RootModel
-from typing import List
+from typing import List, TypeVar
+
+import httpx
+from pydantic import RootModel
 
 from ktoolbox.api import BaseAPI, APIRet
 from ktoolbox.api.model import Revision
 
 __all__ = ["GetPostRevisions", "get_post_revisions"]
 
+_T = TypeVar('_T')
+
 
 class GetPostRevisions(BaseAPI):
     path = "/{service}/user/{creator_id}/post/{post_id}/revisions"
@@ -28,8 +32,12 @@ async def __call__(cls, service: str, creator_id: str, post_id: str) -> APIRet[R
             creator_id=creator_id,
             post_id=post_id
         )
-        
+
         return await cls.request(path=path)
 
+    @classmethod
+    def handle_res(cls, res: httpx.Response) -> APIRet[_T]:
+        return APIRet(data=[]) if res.status_code == 404 else super().handle_res(res)
+
 
-get_post_revisions = GetPostRevisions.__call__
+get_post_revisions = GetPostRevisions.__call__

ktoolbox/cli.py

@@ -12,7 +12,7 @@
 from ktoolbox.action import create_job_from_post, create_job_from_creator, generate_post_path_name
 from ktoolbox.action import search_creator as search_creator_action, search_creator_post as search_creator_post_action
 from ktoolbox.api.misc import get_app_version
-from ktoolbox.api.posts import get_post as get_post_api, get_post_revisions as get_post_revisions_api
+from ktoolbox.api.posts import get_post as get_post_api
 from ktoolbox.configuration import config
 from ktoolbox.job import JobRunner
 from ktoolbox.utils import dump_search, parse_webpage_url, generate_msg
@@ -228,7 +228,7 @@ async def download_post(
 
                 for revision_order, revision_data in ret.data.props.revisions:
                     if revision_data.revision_id:  # Only process actual revisions, not the main post
-                        revision_path = post_path / "revision" / str(revision_data.revision_id)
+                        revision_path = post_path / config.job.post_structure.revisions / generate_post_path_name(revision_data)
                         revision_jobs = await create_job_from_post(
                             post=revision_data,
                             post_path=revision_path,