feat: Enhance protocol specification with streaming support and module interface improvements

tercel · tercel · commit 36860da8e7f8 · 2026-02-23T09:57:27.000+08:00
diff --git a/PROTOCOL_SPEC.md b/PROTOCOL_SPEC.md
diff --git a/README.md b/README.md
@@ -264,27 +264,17 @@ pip install apcore
 ```
 
 ```python
-from apcore import Module, Registry, Executor, Context
-from pydantic import BaseModel, Field
-
-# 1. Define module (Schema-driven)
-class GreetInput(BaseModel):
-    name: str = Field(..., description="User name")
-
-class GreetOutput(BaseModel):
-    message: str
+from apcore import module, Registry, Executor
 
-class GreetModule(Module):
+# 1. Define module with @module decorator (Schema auto-inferred from type annotations)
+@module(id="executor.greet", tags=["greeting"])
+def greet(name: str) -> dict:
     """Generate greeting message"""
-    input_schema = GreetInput
-    output_schema = GreetOutput
-
-    def execute(self, inputs: dict, context: Context) -> dict:
-        return {"message": f"Hello, {inputs['name']}!"}
+    return {"message": f"Hello, {name}!"}
 
 # 2. Register & call
 registry = Registry()
-registry.register("executor.greet", GreetModule())
+registry.register("executor.greet", greet)
 
 executor = Executor(registry=registry)
 result = executor.call("executor.greet", {"name": "World"})
@@ -838,7 +828,7 @@ Any language SDK implementation can choose different conformance levels:
 |------|------|------|
 | **Level 0 (Core)** | Minimally viable | ID mapping, Schema loading, Registry, Executor |
 | **Level 1 (Standard)** | Production ready | + ACL, middleware, error handling, observability |
-| **Level 2 (Full)** | Complete implementation | + Hot reload, distributed execution, advanced monitoring |
+| **Level 2 (Full)** | Complete implementation | + Extension point framework, async task management, W3C Trace Context, Prometheus metrics, version negotiation, schema migration, module isolation, multi-version coexistence |
 
 **Reference Implementation**: [apcore-python](https://github.com/aipartnerup/apcore-python)
 
@@ -896,7 +886,7 @@ The apcore ecosystem uses a **core + independent adapters** architecture. The co
 
 | Type | Examples | Description |
 |------|------|------|
-| **Web Frameworks** | `apcore-fastapi`, `apcore-flask`, `apcore-express` | Expose modules as HTTP APIs |
+| **Web Frameworks** | `nestjs-apcore`, `flask-apcore`, `express-apcore` | Expose modules as HTTP APIs |
 | **AI Protocols** | `apcore-mcp`, `apcore-openai-tools` | Expose modules as AI tools |
 | **RPC** | `apcore-grpc`, `apcore-thrift` | Expose modules as RPC services |
 
diff --git a/SCOPE.md b/SCOPE.md
@@ -94,9 +94,9 @@ Things explicitly **not within the project scope**:
 | **Distributed Execution** | Advanced runtime feature | Independent extension projects |
 | **UI/CLI Applications** | Focus on core protocol | Upstream projects |
 | **Specific Cloud Service Integrations** | Stay neutral | Extension packages |
-| **Framework Adapters** (Flask/FastAPI/Django) | Keep core pure, belongs to ecosystem projects | Independent repositories (e.g., apcore-fastapi) |
+| **Framework Adapters** (Flask/FastAPI/Django) | Keep core pure, belongs to ecosystem projects | Independent repositories (e.g., django-apcore) |
 
-> apcore only defines the adapter interface specification (see [Adapter Development Guide](./docs/guides/adapter-development.md)); it does not include any framework-specific implementations. The community or official team can develop adapters in independent repositories (e.g., `apcore-fastapi`, `apcore-flask`, `apcore-django`), built on top of the core `module()` and External Binding mechanisms.
+> apcore only defines the adapter interface specification (see [Adapter Development Guide](./docs/guides/adapter-development.md)); it does not include any framework-specific implementations. The community or official team can develop adapters in independent repositories (e.g., `flask-apcore`, `django-apcore`), built on top of the core `module()` and External Binding mechanisms.
 
 ---
 
diff --git a/docs/api/registry-api.md b/docs/api/registry-api.md
@@ -677,7 +677,7 @@ Steps:
 
 ### 9.1 Custom Discoverer
 
-> **Reserved — Not Implemented.** The `set_discoverer()` API below is a reserved design; current SDKs do not support it. Documentation is retained for future reference.
+> Implemented in apcore-python v0.5.1+ and apcore-typescript v0.3.0+.
 
 ```python
 from apcore import Registry, ModuleDiscoverer
@@ -706,7 +706,7 @@ registry.discover()
 
 ### 9.2 Module Validator
 
-> **Reserved — Not Implemented.** The `set_validator()` API below is a reserved design; current SDKs do not support it. Documentation is retained for future reference.
+> Implemented in apcore-python v0.5.1+ and apcore-typescript v0.3.0+.
 
 ```python
 from apcore import Registry, ModuleValidator
@@ -735,7 +735,7 @@ registry.discover()
 
 ### 9.3 Hot Reload (Development Mode)
 
-> **Reserved — Not Implemented.** The `watch()` / `unwatch()` API below is a reserved design; current SDKs do not support it. Documentation is retained for future reference.
+> Implemented in apcore-python v0.5.1+ and apcore-typescript v0.3.0+. Python requires the optional `watchdog` dependency.
 
 ```python
 from apcore import Registry
diff --git a/docs/architecture.md b/docs/architecture.md
@@ -87,26 +87,39 @@ extensions/
 
 ### 2.1 Module
 
-Modules are the smallest execution unit in apcore.
+Modules are the smallest execution unit in apcore. Modules can be defined via the `@module` decorator (primary approach) or by providing a class with an `execute()` method and required schema attributes. No abstract base class inheritance is required.
+
+**Decorator approach (recommended):**
+
+```python
+from apcore import module
+
+@module(id="executor.greet", tags=["greeting"])
+def greet(name: str) -> dict:
+    """Generate greeting message"""
+    return {"message": f"Hello, {name}!"}
+```
+
+**Class-based approach (no ABC required):**
 
 ```python
-class Module(ABC):
-    """Module base class"""
+class SendEmailModule:
+    """Send email module"""
 
     # ====== Core Layer (Required) ======
-    input_schema: ClassVar[Type[BaseModel]]
-    output_schema: ClassVar[Type[BaseModel]]
-    description: ClassVar[str]
+    input_schema = SendEmailInput       # Type[BaseModel] or JSON Schema dict
+    output_schema = SendEmailOutput     # Type[BaseModel] or JSON Schema dict
+    description = "Send email to specified recipient"
 
     # ====== Annotation Layer (Optional) ======
-    name: ClassVar[str | None]
-    tags: ClassVar[list[str]]
-    version: ClassVar[str]
-    annotations: ClassVar[ModuleAnnotations | None]  # Behavior annotations
-    examples: ClassVar[list[ModuleExample]]           # Usage examples
+    name = None                         # Human-readable name
+    tags = ["email"]                    # Tag list
+    version = "1.0.0"                   # Semantic version
+    annotations = None                  # ModuleAnnotations instance
+    examples = []                       # Usage examples
 
     # ====== Extension Layer (Optional) ======
-    metadata: ClassVar[dict[str, Any]]                # Free metadata
+    metadata = {}                       # Free metadata
 
     # Core methods (def or async def both supported, framework auto-detects)
     def execute(self, inputs: dict, context: Context) -> dict: ...
@@ -488,10 +501,49 @@ my-project/
 
 ## 5. Extension Points
 
-### 5.1 Custom Discoverer
+apcore provides a formal `ExtensionManager` API for managing pluggable extension points. The five built-in extension points are: `discoverer`, `middleware`, `acl`, `span_exporter`, and `module_validator`.
+
+### 5.1 ExtensionManager API
+
+The `ExtensionManager` provides a unified interface for registering, retrieving, and applying extensions:
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `register` | `register(point_name, extension)` | Register an extension for a named extension point |
+| `get` | `get(point_name)` | Retrieve the single registered extension, or `None` |
+| `get_all` | `get_all(point_name)` | Retrieve all registered extensions for a point as a list |
+| `unregister` | `unregister(point_name, extension)` | Remove a registered extension; returns `bool` |
+| `apply` | `apply(registry, executor)` | Apply all registered extensions to the given registry and executor |
+| `list_points` | `list_points()` | List all registered extension points as a list of `ExtensionPoint` |
+
+```python
+from apcore import ExtensionManager
+
+manager = ExtensionManager()
+
+# Register a custom discoverer
+manager.register("discoverer", remote_discoverer)
+
+# Register a custom module validator
+manager.register("module_validator", strict_validator)
+
+# Retrieve all discoverers
+discoverers = manager.get_all("discoverer")
+
+# Retrieve single extension (or None)
+acl_ext = manager.get("acl")
+
+# Remove an extension
+removed = manager.unregister("discoverer", remote_discoverer)  # True
 
-> **Note:** This example shows the conceptual interface pattern. The actual
-> Python SDK uses a function-based API (`apcore.registry.scanner.scan_extensions`).
+# Apply all extensions to registry and executor
+manager.apply(registry, executor)
+
+# List registered points
+points = manager.list_points()  # [ExtensionPoint(...), ...]
+```
+
+### 5.2 Custom Discoverer
 
 ```python
 from apcore.registry.scanner import scan_extensions
@@ -510,10 +562,7 @@ class RemoteDiscoverer:
         return modules
 ```
 
-### 5.2 Custom Validator
-
-> **Note:** This example shows the conceptual interface pattern. The actual
-> Python SDK uses a function-based API (`apcore.registry.validation.validate_module`).
+### 5.3 Custom Validator
 
 ```python
 from apcore.registry.validation import validate_module
@@ -532,7 +581,7 @@ class StrictValidator:
         return errors
 ```
 
-### 5.3 Custom ACL
+### 5.4 Custom ACL
 
 ```python
 from apcore import ACL
@@ -550,6 +599,8 @@ class RBACAuthorizer(ACL):
         return bool(set(context.identity.roles) & set(required_roles))
 ```
 
+> **Backward compatibility note:** The informal patterns shown above (subclassing, function-based APIs) continue to work. The `ExtensionManager` provides a formal unified API on top of these patterns for implementations that need programmatic extension point management.
+
 ---
 
 ## 6. Design Principles
@@ -666,13 +717,13 @@ def sync_caller():
 - **ACL check timeout**: independent timing (default 1 second)
 - Timing starts from first `before()` middleware
 
-**Cancellation Strategy:** **Future (Not Implemented)** — The following is a planned design; current SDKs have not yet implemented this.
+**Cancellation Strategy:** Cooperative cancellation is implemented in both SDKs via `CancelToken`. Forced termination fallback is not yet implemented.
 
-1. **Cooperative cancellation** (recommended):
+1. **Cooperative cancellation** (implemented):
    - Module checks `context.cancel_token.is_cancelled()` and actively exits
    - Suitable for long-running tasks (loops, I/O, etc.)
 
-2. **Forced termination** (fallback):
+2. **Forced termination** (fallback, not yet implemented):
    - After cooperative cancellation fails, wait grace period (default 5 seconds)
    - If still not exited, force terminate thread/coroutine (may cause resource leaks)
 
diff --git a/docs/features/observability.md b/docs/features/observability.md
@@ -25,7 +25,9 @@ Comprehensive observability with distributed tracing, metrics collection, and st
 - Implement `ContextLogger` as a standalone structured logger with JSON and text output formats.
 - Support log levels: trace, debug, info, warn, error, fatal.
 - Inject trace context (trace_id, module_id, caller_id) into every log entry.
-- Automatically redact fields with `_secret_` prefix when `redact_sensitive=True`.
+- Automatically redact sensitive data using two mechanisms:
+  1. `x-sensitive` schema annotation: used by the Executor to redact annotated fields from input/output before logging.
+  2. `_secret_` key prefix: used by `ContextLogger` to redact matching keys in log extras when `redact_sensitive=True`.
 - Provide `ContextLogger.from_context()` factory for automatic context extraction.
 - Implement `ObsLoggingMiddleware` using `ContextLogger` with stack-based timing and configurable input/output logging.
 
@@ -97,6 +99,35 @@ As documented in the package `__init__.py`:
 2. `MetricsMiddleware` -- Captures execution timing.
 3. `ObsLoggingMiddleware` -- Logs with timing already set up (innermost).
 
+### W3C Trace Context
+
+apcore supports [W3C Trace Context](https://www.w3.org/TR/trace-context/) for distributed tracing interoperability. The `TraceContext` class provides methods to inject and extract `traceparent` headers, enabling trace propagation across service boundaries.
+
+| Method | Description |
+|--------|-------------|
+| `TraceContext.inject(context)` | Convert an apcore `Context` into a headers dict (`dict[str, str]` / `Record<string, string>`) containing a `traceparent` key |
+| `TraceContext.extract(headers)` | Parse a `traceparent` header from an incoming request headers dict and return a `TraceParent` |
+| `TraceContext.from_traceparent(str)` | Strict parsing of a `traceparent` string into a `TraceParent` object |
+
+Integration with `Context.create()`:
+
+```python
+from apcore import Context
+from apcore.observability import TraceContext
+
+# Extract trace parent from incoming request
+trace_parent = TraceContext.extract(request.headers)
+
+# Create context with propagated trace
+context = Context.create(trace_parent=trace_parent)
+
+# Inject trace parent into outgoing request headers
+outgoing_headers = TraceContext.inject(context)
+# outgoing_headers = {"traceparent": "00-<trace_id>-<span_id>-01"}
+```
+
+The `traceparent` header follows the W3C format: `{version}-{trace_id}-{parent_id}-{trace_flags}`.
+
 ## Key Files
 
 | File | Lines | Purpose |
diff --git a/docs/guides/adapter-development.md b/docs/guides/adapter-development.md
@@ -14,8 +14,8 @@ The apcore core remains pure and **does not include** any web framework-specific
                            ↑ Built on core mechanisms
       ┌──────────┬──────────┬──────────┬──────────┐
       │          │          │          │          │
-   apcore-      apcore-    apcore-    apcore-    ...
-   fastapi      flask      django     express
+   tiptap-      flask-    django-    express-    ...
+   apcore       apcore    apcore     apcore
    (separate)   (separate)  (separate)  (separate)
 ```
 
@@ -39,16 +39,16 @@ Adapters **should not**:
 ### Repository Naming
 
 ```
-apcore-{framework}
+{framework}-apcore
 ```
 
-Examples: `apcore-fastapi`, `apcore-flask`, `apcore-django`, `apcore-express`
+Examples: `flask-apcore`, `django-apcore`, `express-apcore`
 
 ### Package Naming
 
 ```
-pip install apcore-{framework}
-npm install apcore-{framework}
+pip install {framework}-apcore
+npm install {framework}-apcore
 ```
 
 ## 4. Adapter Interface Reference
diff --git a/docs/spec/conformance.md b/docs/spec/conformance.md
@@ -110,7 +110,7 @@ Level 1 adds permission control, middleware, basic observability, and structured
 | **Context complete implementation** | `trace_id`, `caller_id`, `call_chain`, `executor`, `identity`, `data` | PROTOCOL_SPEC §5.7 |
 | **Circular call detection** | Detect circular calls based on `call_chain` | PROTOCOL_SPEC §5.7 |
 | **Call depth limit** | Limit maximum call depth based on `call_chain` | PROTOCOL_SPEC §5.7 |
-| **Error hierarchy system** | Complete error type hierarchy (ApCoreError → ModuleError/SchemaError/ACLError/...) | PROTOCOL_SPEC §7.7 |
+| **Error hierarchy system** | Flat error hierarchy under `ModuleError` base class | PROTOCOL_SPEC §7.7 |
 | **Custom error codes** | Module custom error code registration and collision detection | PROTOCOL_SPEC §7.4 |
 | **ID Map all-language conversion** | Support ID conversion for all five languages | PROTOCOL_SPEC §2.2 |
 | **Dependency resolution** | `resolve_dependencies()` topological sort and circular dependency detection | PROTOCOL_SPEC §5.3 |
@@ -164,11 +164,11 @@ Level 2 adds all extension points, async modules, hot loading, and advanced obse
 
 | Component | Responsibility | Reference Section |
 |------|------|---------|
-| **Extension point framework** | All five extension points (SchemaLoader, ModuleLoader, IDConverter, ACLChecker, Executor) | PROTOCOL_SPEC §10.3, §10.6 |
+| **Extension point framework** | All five extension points (discoverer, middleware, acl, span_exporter, module_validator) via `ExtensionManager` with `register()`, `get()`, `get_all()`, `unregister()`, `apply()`, `list_points()`. Note: these names map to actual runtime extension needs rather than the original theoretical design names (SchemaLoader, ModuleLoader, IDConverter, ACLChecker, Executor). | PROTOCOL_SPEC §10.3, §10.6 |
 | **Extension point chain** | `first_success`, `all`, `fallback` strategies | PROTOCOL_SPEC §10.3 |
 | **Extension loading order** | `load_extensions()` algorithm | PROTOCOL_SPEC §10.7 |
-| **Async modules** | `execute_async()`, `get_status()`, `cancel()` | PROTOCOL_SPEC §5.8 |
-| **Async state machine** | State transition rules (idle → pending → running → completed/failed/cancelled) | PROTOCOL_SPEC §5.8 |
+| **Async modules** | `submit()`, `get_status()`, `cancel()`, `list_tasks()` via `AsyncTaskManager` | PROTOCOL_SPEC §5.8 |
+| **Async state machine** | State transition rules (PENDING → RUNNING → COMPLETED/FAILED/CANCELLED) via `TaskStatus` enum | PROTOCOL_SPEC §5.8 |
 | **Middleware state machine** | Complete state transitions (init → before → execute → after → done, with error branches) | PROTOCOL_SPEC §10.5 |
 | **Version negotiation** | `negotiate_version()` algorithm | PROTOCOL_SPEC §12.3 |
 | **Schema migration** | `migrate_schema()` algorithm | PROTOCOL_SPEC §12.4 |
@@ -539,10 +539,8 @@ conformance:
       total: 10
 
   # Known deviations (if any)
-  known_deviations:
-    - id: "T03-012"
-      reason: "$ref circular reference detection is Level 0 non-mandatory, not yet implemented (target: v0.2.0)"
-      severity: "minor"
+  # Format: - { id: "T03-xxx", reason: "...", severity: "minor|major" }
+  known_deviations: []
 
   # Optional feature support
   optional_features:
@@ -576,7 +574,26 @@ apcore Conformant — Level 2 (Full)
 
 ---
 
-## 7. References
+## 7. Known Deviations
+
+The following features are specified in PROTOCOL_SPEC but not yet fully implemented in current SDK releases (apcore-python, apcore-typescript). Implementers **should** document these deviations in their conformance declarations.
+
+| Feature | Spec Reference | Current Status |
+|---------|---------------|----------------|
+| `Config` class (YAML loading, env override, schema validation) | PROTOCOL_SPEC §8.1, §8.2, §8.3 | Stub implementation only. YAML loading, environment variable override, and `validate_config()` schema validation are not implemented. |
+| Registry schema query/export methods (`get_schema`, `export_schema`, etc.) | PROTOCOL_SPEC §11.2 | Not on `Registry`. A standalone `SchemaExporter` class is available for schema export. |
+| Error codes `GENERAL_NOT_IMPLEMENTED` and `DEPENDENCY_NOT_FOUND` | PROTOCOL_SPEC §7.2, §7.7 | Error code constants defined but corresponding error classes not yet implemented. |
+| Version negotiation | PROTOCOL_SPEC §12.3 | `negotiate_version()` algorithm not yet implemented. |
+| Schema migration | PROTOCOL_SPEC §12.4 | `migrate_schema()` algorithm not yet implemented. |
+| Module isolation | PROTOCOL_SPEC §5.5 | Process-level or container-level isolation not yet implemented. |
+| Multi-version coexistence | PROTOCOL_SPEC §5.4 | Multiple versions of the same module running concurrently not yet implemented. |
+| `AsyncTaskManager.submit()` / `cancel()` sync vs async | PROTOCOL_SPEC §5.8 | Python `AsyncTaskManager.submit()` and `cancel()` are async methods; TypeScript equivalents are synchronous. |
+
+Implementations declaring conformance **must** list any of these deviations that apply in their `known_deviations` section.
+
+---
+
+## 8. References
 
 - [PROTOCOL_SPEC §11 — SDK Implementation Guide](../../PROTOCOL_SPEC.md#11-sdk-implementation-guide)
 - [PROTOCOL_SPEC §11.4 — Conformance Testing Requirements](../../PROTOCOL_SPEC.md#114-conformance-testing-requirements)
