Dev to 3.0.0

[BlueSkyXN]

This pull request introduces a major enhancement to AI-DataFlux: the addition of a Web GUI control panel and a new dual build system supporting both "Full" (with GUI) and "CLI-only" (without GUI) versions. The CLI is now adaptive, exposing the gui subcommand only when the GUI backend is available. The workflows for both PyInstaller and Nuitka have been updated to build and package both variants across platforms, and the documentation has been expanded to explain the differences and usage of each version.

The most important changes are:

1. Web GUI Control Panel Integration

• Added a new gui subcommand to cli.py that starts a local Web GUI control panel for managing services, editing configs, and viewing logs. The command is registered only if the src.control.server module is present, ensuring CLI-only builds remain clean. (cli.py, README.md, [1] F2d50726L508R508, [2] [3]

2. Dual Build System: Full vs CLI-only

• Updated build workflows (build-pyinstaller.yml and build-nuitka.yml) to produce both Full (with GUI and frontend assets) and CLI-only (without GUI/frontend) binaries for all supported platforms. The CLI-only variant excludes the GUI modules and static assets, resulting in a smaller footprint. (.github/workflows/build-pyinstaller.yml, .github/workflows/build-nuitka.yml, [1] [2] [3] [4] [5]

3. Documentation and User Guidance

• Added docs/BUILD_VARIANTS.md, and expanded the README.md with clear explanations, usage instructions, and feature comparisons between Full and CLI-only versions, including how the CLI adapts to the available features. (docs/BUILD_VARIANTS.md, README.md, README.mdR105-R137)

4. CLI Improvements and Validation

• Enhanced CLI argument validation for ports and config paths, and made the process subcommand optionally accept a progress file for GUI integration. (cli.py, [1] [2] [3] [4] [5]

5. Frontend Build and Packaging

• Integrated frontend build steps (Node.js, npm) into the build workflows and ensured static assets are bundled with Full builds for both PyInstaller and Nuitka. (.github/workflows/build-pyinstaller.yml, .github/workflows/build-nuitka.yml, [1] [2]

These changes provide a more flexible distribution model, improve usability for both end-users and developers, and lay the groundwork for further modularization and optional feature support.

• Web GUI Control Panel Integration [1] F2d50726L508R508, [2] [3]
• Dual Build System: Full vs CLI-only [1] [2] [3] [4] [5]
• Documentation and User Guidance [1] [2]
• CLI Improvements and Validation [1] [2] [3] [4] [5]
• Frontend Build and Packaging [1] [2]

In general, to fix uncontrolled path usage you should (1) define a trusted root directory, (2) build all user-supplied paths relative to that root, (3) normalize the resulting path with os.path.realpath or os.path.normpath, and (4) verify that the normalized path is still inside the trusted root using a robust check such as os.path.commonpath. Avoid relying on simple string prefix checks or un-normalized roots, and ensure you handle edge cases like different drive letters on Windows.

For this codebase, the best targeted fix is to make _validate_path fully normalize both PROJECT_ROOT and the computed path, and then compare using their normalized forms. Concretely:

• Normalize PROJECT_ROOT once when computing it, e.g. PROJECT_ROOT = os.path.realpath(str(runtime.get_project_root())).
• In _validate_path, compute real as os.path.realpath(os.path.join(PROJECT_ROOT, path)) as before, but then normalize both PROJECT_ROOT and real via os.path.realpath (or use the already-normalized global) when calling os.path.commonpath.
• Compare common with the normalized root (ROOT_REAL) rather than the possibly non-normalized PROJECT_ROOT.

This still keeps behavior the same (user is restricted to files under the project root), but removes subtle mismatches that could confuse static analysis and closes edge cases where a non-normalized root might behave unexpectedly. No changes are needed in read_config or write_config, since they already funnel access through _validate_path; we just strengthen the helper and the PROJECT_ROOT definition. The only file to edit is src/control/config_api.py.

---

In general, to fix uncontrolled-path issues, ensure that any user-supplied path is interpreted strictly relative to a trusted root, normalize both the root and the combined path to canonical absolute form, and reject any path that escapes the root or is absolute on its own.

For this codebase, the best targeted fix is: (1) normalize PROJECT_ROOT once to an absolute canonical path (via os.path.realpath), and (2) adjust _validate_path so that it robustly computes the canonical path of the requested file and verifies that it lies under the canonical project root. This means:

• Define PROJECT_ROOT as os.path.realpath(runtime.get_project_root()) instead of just str(...). This ensures it is absolute and normalized.
• In _validate_path, build target = os.path.join(PROJECT_ROOT, path), then compute real = os.path.realpath(target).
• Use os.path.commonpath([PROJECT_ROOT, real]) and verify that it equals PROJECT_ROOT. On Windows, continue to catch ValueError for differing drives.
• Optionally, strip leading path separators from path to avoid os.path.join(PROJECT_ROOT, "/etc/passwd") discarding the root (though the current commonpath+realpath check would already prevent this if PROJECT_ROOT is canonical; still, it’s harmless and clarifies intent). Given the instruction to avoid changing functionality, we will not add additional rejections beyond the strengthened canonical-root handling.

No changes are needed in write_config or read_config beyond relying on the improved _validate_path. The behavior remains the same for valid in-tree paths, but malicious or malformed paths are more reliably rejected.

---

In general, to fix “uncontrolled data used in path expression” issues you must fully normalize user‑controlled paths, anchor them under a trusted root, and reject any path that escapes that root before using it with filesystem operations. The sanitizer should be applied exactly once at the boundary where untrusted data enters your file‑API layer, and all internal code should then only use the validated, absolute path.

For this codebase, the best fix is to strengthen _validate_path so that:

• PROJECT_ROOT is normalized to an absolute, real path in a canonical form.
• The user‑controlled path is joined to that canonical root using os.path.join, and the result is normalized with os.path.realpath.
• We then use os.path.commonpath on the normalized root and the normalized candidate path and verify they are equal.
• Optionally, we can slightly restructure _validate_path to make it more obvious to CodeQL that it is a dedicated validation/sanitization step, which may help it treat real as “untainted” afterwards.

We will keep the API and external behavior the same: read_config and write_config will still accept a path relative to the project root, and write_config will still create .bak and .tmp files next to the target. The changes are limited to src/control/config_api.py:

1. Normalize PROJECT_ROOT at definition time by wrapping runtime.get_project_root() in os.path.realpath(os.path.abspath(...)). This ensures the root used in the commonpath check is canonical.
2. In _validate_path, introduce a project_root_real local variable that reuses the normalized root (for clarity), build candidate with os.path.join(project_root_real, path), normalize it with os.path.realpath, and then compute common = os.path.commonpath([project_root_real, candidate]). If common != project_root_real, raise 403. This keeps the security behavior but makes the invariants explicit.

No changes are needed in write_config itself: once _validate_path guarantees that real_path is inside PROJECT_ROOT, real_path + ".tmp" and real_path + ".bak" are also safely inside the project root.

---

In general, the fix is to ensure that any path derived from untrusted input is strictly validated before being used in file operations. For directory-based validation, this means resolving the candidate path (using os.path.realpath or os.path.normpath) and then confirming it is still within an intended root directory. If it is not, or if any resolution error occurs, the server should return an error instead of accessing the file.

For this specific case, the best fix with minimal functional change is to (a) explicitly reject absolute paths provided by the client, and (b) keep the realpath/commonpath based containment check as the authoritative guard. Rejecting absolute paths makes the intent clearer and narrows the accepted input; the containment check then ensures that even crafted relative paths cannot escape WEB_DIST_DIR. We can do this entirely within serve_static in src/control/server.py. Concretely:

• Inside serve_static, before computing file_path, check if os.path.isabs(path): raise HTTPException(404, "Not found") so that any attempt to use an absolute path is denied.
• Keep the existing base_dir = os.path.realpath(WEB_DIST_DIR), file_path = os.path.realpath(os.path.join(base_dir, path)), and os.path.commonpath check unchanged, since they already enforce that the resolved file lives inside WEB_DIST_DIR.
  No new imports are necessary, since os is already imported and provides path.isabs. This preserves existing behavior for legitimate relative paths and tightens/clarifies the validation on the tainted path variable that flows into file_path.

In general, to fix uncontrolled path usage you should (1) treat the incoming path as relative, (2) reject dangerous patterns like absolute paths or .. segments, (3) normalize/resolve the path against a trusted base directory, and (4) only serve files if the final canonical path is inside that base directory.

For this specific endpoint, the best way to harden it without changing functionality is:

1. Normalize and validate the raw path segment as a relative path:
   • Reject empty or whitespace-only strings.
   • Use os.path.isabs(path) and reject absolute paths.
   • Use os.path.normpath(path) and ensure it does not start with .. or contain any .. segments that would escape upward.
2. After that, keep the existing realpath + commonpath check, which protects against symlink tricks and further ensures file_path is under WEB_DIST_DIR.
3. Only if all checks pass, serve the file or fall back to the SPA index.html.

All required imports (os) are already present in src/control/server.py, so we only need to update the serve_static function body (lines 385–403) within the file.

In general, to fix information exposure via exceptions you should avoid sending raw exception messages or stack traces to clients. Instead, log the detailed error on the server (for debugging) and return a generic, non-sensitive error response to the user, optionally including minimal structured metadata (like “line” and “column”) that is safe to expose.

For this concrete case in src/control/server.py, the best fix without changing overall behavior is:

• Catch yaml.YAMLError as we already do.
• Extract only safe, high-level information from the exception (such as line and column of the error) if available, but do NOT return str(e) itself.
• Optionally log the full exception (including stack trace) using the existing logging setup so developers still get diagnostics.
• Return a JSON structure like {"valid": False, "error": "YAML parsing failed", "details": {...}} where details only contains non-sensitive, structured fields.

Concretely:

• Inside api_validate_config, replace return {"valid": False, "error": str(e)} with logic that:
  • Logs the exception using logging.exception("Invalid YAML content").
  • Builds a safe error payload, possibly inspecting attributes like e.problem_mark if present.
• Do not change imports beyond what’s already there; logging is already imported at the top of the file, so we can reuse it.

All changes are confined to the api_validate_config function in src/control/server.py.

[BlueSkyXN]

PR 概述：AI-DataFlux Web GUI 与双重构建系统
此 PR 为 AI-DataFlux 引入了一项重大增强：新增了 Web GUI 控制面板以及支持 "完整版" (Full) 和 "纯命令行版" (CLI-only) 的全新 双重构建系统。

CLI 现已具备自适应能力，仅在 GUI 后端可用时才会暴露 gui 子命令。PyInstaller 和 Nuitka 的工作流均已更新，支持在各平台上构建和打包这两种变体。此外，文档也进行了扩展，详细说明了两个版本的区别及用法。

核心变更详情

1. 集成 Web GUI 控制面板
   功能描述：在 cli.py 中新增了 gui 子命令，用于启动本地 Web GUI 控制面板。该面板支持服务管理、配置编辑和日志查看。

自适应设计：该命令仅在 src.control.server 模块存在时才会注册，从而确保 "纯命令行版" (CLI-only) 的构建保持纯净。

相关文件：cli.py, README.md

2. 双重构建系统：完整版 (Full) vs 纯命令行版 (CLI-only)
   工作流更新：更新了 build-pyinstaller.yml 和 build-nuitka.yml，现在可以为所有支持的平台同时生成两种二进制文件。

体积优化："纯命令行版" (CLI-only) 变体排除了 GUI 模块和静态资源，因此体积更小，占用资源更少。

相关文件：.github/workflows/build-pyinstaller.yml, .github/workflows/build-nuitka.yml

3. 文档与用户指南
   新增文档：添加了 docs/BUILD_VARIANTS.md。

README 更新：在 README.md 中增加了清晰的说明、使用指南以及 "完整版" 与 "纯命令行版" 的功能对比，并解释了 CLI 如何根据可用功能进行自适应。

相关文件：docs/BUILD_VARIANTS.md, README.md

4. CLI 改进与验证
   验证增强：增强了 CLI 对端口和配置路径参数的验证逻辑。

进度集成：process 子命令现在支持可选的进度文件参数，以便更好地与 GUI 集成。

相关文件：cli.py

5. 前端构建与打包
   流程集成：将前端构建步骤（Node.js, npm）集成到了构建工作流中。

资源捆绑：确保静态资源在 PyInstaller 和 Nuitka 的 "完整版" 构建中被正确捆绑。

相关文件：.github/workflows/build-pyinstaller.yml, .github/workflows/build-nuitka.yml

变更影响
这些更改提供了一种更灵活的分发模式，同时提升了最终用户和开发者的使用体验，并为未来的模块化和可选功能支持奠定了基础。

[Copilot]

Pull request overview

This pull request introduces a comprehensive Web GUI control panel for AI-DataFlux version 3.0.0, adding visual management capabilities for Gateway and Process services.

Changes:

• Adds complete React-based web frontend with Dashboard, Config Editor, and Logs viewer
• Implements FastAPI-based control server for process management and configuration API
• Introduces dual build variants (Full with GUI, CLI-only without GUI)
• Adds bilingual support (English/Chinese) throughout the interface

Reviewed changes

Copilot reviewed 56 out of 59 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
web/*	Complete React/TypeScript frontend application with Vite build system
src/control/*	FastAPI control server, process manager, and configuration API
src/core/processor.py	Added progress file tracking for GUI status display
cli.py	Added gui subcommand with conditional registration based on module availability
tests/*	New control server tests and updated CLI tests
docs/GUI.md	Comprehensive documentation for the Web GUI feature
docs/BUILD_VARIANTS.md	Documentation for Full vs CLI-only build variants
.github/workflows/*	Updated CI/CD to build both Full and CLI-only versions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The Dashboard component has an extra closing </div> tag at line 358 that doesn't match any opening tag. Line 194 opens <div className="h-full flex flex-col"> and line 357 closes it with </div>, but line 358 has another </div> which is incorrect. This will cause a React rendering error.

[Copilot]

Security concern: The browser file/folder chooser implementation using webkitRelativePath only gets the relative path from the user's file selection, not an absolute path usable by the backend. When a user selects a folder, the code extracts dirPath = filePath.split('/')[0] which will be just the folder name, not a full path. This means the working directory will be set to a relative path that may not exist or may refer to the wrong location when passed to the backend.

The same issue affects the file chooser which uses file.name - this is just the filename without any path information. These values should either be validated or the feature should use a proper file system API that can provide actual paths.

[Copilot]

Race condition: The _start_log_reader method creates an asyncio task to read logs, but wraps the task creation in a try-except that catches RuntimeError when there's no running event loop. In this case, it simply passes and doesn't create the task. This means if the process manager starts processes before the FastAPI event loop is running, log reading will silently fail and logs won't be captured.

A better approach would be to defer task creation until the event loop is available, or ensure the event loop is running before starting processes.

Suggested change

	# 创建任务 (需要在事件循环中)
	try:
	loop = asyncio.get_running_loop()
	self._read_tasks[name] = loop.create_task(read_logs())
	except RuntimeError:
	# 没有运行中的事件循环，延迟创建
	pass
	# 创建任务：如果事件循环尚未运行，任务会在事件循环启动后执行
	loop = asyncio.get_event_loop()
	self._read_tasks[name] = loop.create_task(read_logs())

[Copilot]

The config_path validation function checks for .yaml or .yml extensions, but the function is named _validate_config_path and is applied to the gateway --config argument. However, it's only checking the extension and not validating if the file exists. The docstring states "仅检查基本格式，不检查存在性" (only checks basic format, doesn't check existence), which is intentional for startup scenarios.

However, the validation is inconsistent: the function is applied to gateway's --config argument but NOT to process's --config argument (line 474 vs line 463). This means gateway gets validation but process doesn't, leading to inconsistent behavior.

[Copilot]

The progress file write operation in _write_progress() lacks proper error handling for the os.replace() call. While there's a broad try-except that catches all exceptions and logs them as debug messages, if os.replace() fails, the temporary file is not cleaned up. This could lead to accumulation of .tmp files over time.

Recommended fix: Add cleanup of the temporary file in the except block before logging the error.

[Copilot]

'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

	except Exception:
	pass
	except Exception as e:
	logging.warning("Failed to close stdout for process %s: %s", name, e)

[Copilot]

'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

	except OSError:
	pass
	except OSError as e:
	# 进度文件清理失败对主流程无影响，忽略错误，但记录调试日志便于排查
	logging.debug(f"Failed to remove progress file {self.progress_file!r}: {e}")

[Copilot]

'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

	except Exception:
	except Exception:
	# Best-effort: if resolving sys.executable fails, just skip this candidate.

[Copilot]

'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

	except Exception:
	except Exception:
	# Best-effort cleanup: failures while waiting for the writer to close
	# are non-fatal and can be safely ignored here.

[Copilot]

'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

	except RuntimeError:
	except RuntimeError:
	# 当没有运行中的事件循环（例如应用关闭阶段）时，get_running_loop 会抛出 RuntimeError，
	# 这种情况下我们静默忽略并不广播日志是可接受的行为。