Merge pull request #174 from vshn/adr-framework-2

Add ADR AppCat Framework 2.0

Author: zugao

docs/modules/ROOT/assets/images/framework2-architecture.png

@@ -1,4 +1,4 @@
-= ADR 0049 - Migrate MinIO to Rclone
+= ADR 0041 - Migrate MinIO to Rclone
 :adr_author:    Mike Ditton
 :adr_owner:     Schedar
 :adr_reviewers: Schedar

docs/modules/ROOT/assets/images/simple-architecture-2-0.png

@@ -1,4 +1,4 @@
-= ADR 0050 - Wazuh Agent for AppCat Services
+= ADR 0043 - Wazuh Agent for AppCat Services
 :adr_author:    Simon Beck
 :adr_owner:     Schedar
 :adr_reviewers:

…es/adr/0049-migrate-minio-to-rclone.adoc …es/adr/0041-migrate-minio-to-rclone.adocdocs/modules/ROOT/pages/adr/0049-migrate-minio-to-rclone.adoc renamed to docs/modules/ROOT/pages/adr/0041-migrate-minio-to-rclone.adoc docs/modules/ROOT/pages/adr/0049-migrate-minio-to-rclone.adoc renamed to docs/modules/ROOT/pages/adr/0041-migrate-minio-to-rclone.adoc

@@ -0,0 +1,102 @@
+= ADR 0044 - Configuration Language: KCL vs CUE
+:adr_author: Gabriel Saratura
+:adr_owner: Schedar
+:adr_reviewers: Schedar
+:adr_date:
+:adr_upd_date:
+:adr_status: draft
+:adr_tags: framework,framework2,kcl,cue,configuration
+
+include::partial$adr-meta.adoc[]
+
+[NOTE]
+.Summary
+====
+We choose https://www.kcl-lang.io/[KCL] over https://cuelang.org/[CUE] for type-safe, constraint-based configuration tool.
+KCL satisfies our needs for service definitions, helm release mapping, and platform schemas in Framework 2.0.
+====
+
+== Problem
+
+Need a type-safe, constraint-based configuration language for defining service plans, Helm value mappings, and cluster configurations.
+Must support validation, composition, and reusability while being maintainable by Service Maintainers and Framework Engineers, and align with how each persona works:
+
+* *Service Maintainer*: author services and define parameter-to-Helm mappings without Go, with clear validation errors and minimal boilerplate.
+* *Framework Engineer*: enforce platform base schemas, facilitates generation of service bundles, handles platform resources via appcat-catalogs in GitOps style.
+* *Service Operator*: rely on consistent API surfaces for troubleshooting and upgrades; need deterministic mappings and readable errors when validation fails.
+* *Service User*: see only safe, validated parameters in the portal; prevent invalid combinations from ever reaching the runtime.
+
+=== Solutions
+
+==== Option A: KCL (Kubernetes Configuration Language)
+
+**Description:**
+
+https://www.kcl-lang.io/[KCL] is a constraint-based configuration language designed for Kubernetes.
+Features include strong typing with compile-time validation, lambda functions for transformations, and native Kubernetes resource understanding.
+
+**Advantages:**
+
+* *Strong Typing*: Compile-time validation of schemas and data structures catches configuration errors before deployment
+* *Constraint-Based*: Define validation rules and constraints (for example, CPU must match regex pattern, memory within allowed ranges)
+* *Built for Kubernetes*: Native understanding of Kubernetes resource structures reduces boilerplate
+* *Lambda Functions*: Natural expression of parameters/plan-to-Helm-value mapping functions, critical for service configuration
+* *Good Error Messages*: Clear validation errors with line numbers guide Service Maintainers to issues
+* *Managed Resources Support*: Import system enables base configurations with distribution-specific overrides
+* *Override Pattern*: Kapitan style of resolving hierarchical configuration structure in the platform repository
+* *Active Development*: Backed by Ant Group with growing community and ecosystem
+
+**Disadvantages:**
+
+* *Smaller Ecosystem*: Fewer tools and IDE support compared to more established languages like CUE
+* *Learning Curve*: Service Maintainers/Framework Engineers need to learn new syntax and constraint-based thinking
+* *Less Mature*: Newer language with potential for breaking changes as it evolves
+* *Limited Resources*: Fewer tutorials and examples
+
+==== Option B: CUE (Configure Unify Execute)
+
+**Description:**
+
+https://cuelang.org/[CUE] is a data constraint language where types and values are unified.
+Configurations compose through constraint unification.
+Excels at validation but lacks native lambda functions - transformation logic must be expressed through constraint systems, making some operations such as Helm value mappings verbose.
+
+**Advantages:**
+
+* *Unification Model*: Types and values unified - constraints automatically propagate and validate across compositions
+* *Exceptional Data Validation*: Catch configuration errors through mathematical constraint solving before any runtime execution
+* *Mature Tooling*: Language Server Protocol, IDE support, comprehensive CLI tools
+* *Constraint Propagation*: Automatically validates that all configuration layers satisfy constraints without explicit checks
+* *Export Flexibility*: Can export to multiple formats (JSON, YAML, Protocol Buffers) natively
+
+**Disadvantages:**
+
+* *Steep Learning Curve*: Unification model requires different thinking - declarative constraints vs procedural logic
+* *Less Imperative*: Difficult to express procedural transformations like Helm value mapping with conditional logic
+* *No Native Lambda Functions*: Must encode logic as constraints
+* *Verbose Transformations*: Complex mapping functions require extensive boilerplate, reducing Service Maintainer velocity
+* *Limited Procedural Logic*: "If-then-else" patterns become constraint systems that are harder to read and maintain
+
+=== Decision
+
+*We use KCL.*
+
+==== Rationale
+
+1. *Lambda Functions*: Service Maintainers need to define their services with constraints defined by Framework Engineers. KCL's lambda support makes this natural and concise.
+
+2. *Kubernetes-Native*: KCL is designed specifically for Kubernetes configuration, with built-in understanding of K8s resource structures.
+
+3. *Procedural Logic*: Custom Service configuration often requires conditional logic. KCL handles this more naturally than CUE.
+
+4. *Simpler Mental Model*: For Service Maintainers and Framework Engineers, KCL's imperative style is closer to traditional programming, reducing learning curve.
+
+5. *Crossplane Integration*: KCL Crossplane https://artifacthub.io/packages/kcl/kcl-module/crossplane[modules] available for Service Maintainers.
+
+6. *Override and Packaging Fit*: KCL works with profile overrides (Kapitan substitution).
+
+7. *Error Quality for Maintainers*: KCL's validation errors help Service Maintainers debug services quickly, reducing turnaround time versus CUE’s more abstract unification errors.
+
+8. *Future Service Evolution*: As Framework 2.0 targets a single generic composition function consuming unstructured inputs, keeping service logic in KCL avoids per-service Go changes while still providing expressive mapping hooks.
+
+**Trade-off Accepted:** Smaller ecosystem and tooling compared to CUE, but better fit for our use case.

…050-wazuh-agent-for-appcat-services.adoc …043-wazuh-agent-for-appcat-services.adocdocs/modules/ROOT/pages/adr/0050-wazuh-agent-for-appcat-services.adoc renamed to docs/modules/ROOT/pages/adr/0043-wazuh-agent-for-appcat-services.adoc docs/modules/ROOT/pages/adr/0050-wazuh-agent-for-appcat-services.adoc renamed to docs/modules/ROOT/pages/adr/0043-wazuh-agent-for-appcat-services.adoc

@@ -0,0 +1,135 @@
+= ADR 0045 - Service Orchestration: Crossplane 2.x
+:adr_author: Gabriel Saratura
+:adr_owner: Schedar
+:adr_reviewers: Schedar
+:adr_date:
+:adr_upd_date:
+:adr_status: draft
+:adr_tags: framework,framework2,crossplane,composition,orchestration
+
+include::partial$adr-meta.adoc[]
+
+[NOTE]
+.Summary
+====
+This ADR defines the service orchestration mechanism for AppCat Framework 2.0, evaluating Crossplane 2.x composition functions against a custom Kubernetes operator.
+
+The decision selects https://www.crossplane.io/[Crossplane] 2.0 with composition functions as the orchestration layer, providing generic service management with built-in revision control and clear separation between framework code (https://go.dev/[Go]) and service configuration (https://www.kcl-lang.io/[KCL]).
+====
+
+== Context
+
+=== Problem Statement
+
+The AppCat Framework 2.0 requires a service orchestration mechanism that:
+
+* Renders Helm charts based on service configuration and user-selected parameters
+* Manages complete service lifecycle (create, update, delete)
+* Handles connection secrets and billing/maintenance/backup/metrics integration
+* Allows Framework Engineers to maintain core logic in Go
+* Enables Service Maintainers to add new services without touching Go code
+* Provides versioning and revision management for controlled rollouts
+
+=== Architectural Context
+
+The framework architecture separates concerns across three layers:
+
+1. **Framework Engineers** maintain the orchestration runtime (Go composition functions)
+2. **Service Maintainers** define service configurations (KCL compiled managed resource objects)
+3. **Service Users** consume services via declarative API (XRD/Composite resources)
+
+The orchestration layer sits at the boundary between framework runtime and service configuration, responsible for:
+
+* Loading service configurations at runtime
+* Merging static defaults with dynamic user parameters
+* Generating Kubernetes resources (HelmRelease, K8up backup schedules, ServiceMonitor, secrets, etc.)
+* Tracking revisions for upgrade workflows
+* Propagating status and errors to users
+
+== Options
+
+=== Option A: Crossplane 2.x with Composition Functions
+
+==== Description
+
+https://www.crossplane.io/[Crossplane] 2.0 composition functions execute in pipeline mode during reconciliation, receiving composite state and returning desired resources.
+A single generic function handles all services by loading service-specific configuration Kubernetes like object (ex:, `RedisServiceConfig`) at runtime, merging user parameters, and generating resources.
+Other functions can be added to increase framework quality such as increased service resilience and error handling.
+
+==== Advantages
+
+* **Generic Function Pattern**: Single function handles all services without Go code changes
+* **Built-in Composition Revisions**: Native versioning and gradual rollout capabilities
+* **Clear Separation**: Framework code (Go) completely separated from service configuration (KCL)
+* **Provider Ecosystem**: Leverage existing Crossplane providers (Helm, SQL) instead of custom implementations
+* **Status Aggregation**: Automatic status propagation from managed resources
+* **Existing Team Knowledge**: Team already proficient with Crossplane patterns
+
+==== Disadvantages
+
+* **Learning Curve**: Crossplane concepts (XRD, Composition, Revisions) require initial learning investment, though team already has experience
+* **Abstraction Overhead**: Multiple layers (Composite → Composition Function → HelmRelease → Helm Chart) vs direct operator reconcile loop
+* **Debugging Complexity**: Tracing issues through pipeline requires understanding Crossplane request/response flow and function logs
+* **Function Packaging**: Each function version requires OCI image build and registry push
+* **Performance**: Function invoked via gRPC adds latency compared to in-process operator reconciliation
+
+=== Option B: Custom Kubernetes Operator
+
+==== Description
+
+Build a custom Kubernetes operator using https://github.com/operator-framework/operator-sdk[operator-sdk] that directly reconciles service CRDs and renders Helm charts.
+The operator would either use per-service controllers or a generic controller loading service configuration at runtime, with custom implementations for revision management and upgrade workflows.
+
+==== Advantages
+
+* **Simpler Mental Model**: Direct reconciliation loop easier to understand than pipeline abstraction
+* **Faster Debugging**: Single operator process, direct logging, simpler to trace issues with standard Go debugging
+* **More Control**: Full control over reconciliation logic, status updates, error handling, retry behavior
+* **No Crossplane Dependency**: One less external dependency to manage and upgrade
+* **Familiar Pattern**: Standard operator pattern widely used across Kubernetes ecosystem
+* **Performance**: In-process reconciliation potentially faster than gRPC function invocation
+
+==== Disadvantages
+
+* **Framework Coupling**: Risk of service-specific logic creeping into operator code over time
+* **No Built-in Revisions**: Must implement version management, gradual rollouts, and revision tracking ourselves (complex)
+* **Reinventing Wheels**: Helm integration, status aggregation, provider patterns already exist in Crossplane
+* **More Code to Maintain**: Custom implementations needed for operator, CRD management, status propagation, connection secrets all custom implementations
+* **Migration Cost**: Existing Crossplane investment (team knowledge, component-appcat patterns) cannot be leveraged
+* **Revision Management**: Composition revisions essential for maintenance workflows would require significant custom development
+
+=== Options Not Investigated
+
+The following orchestration approaches were considered but excluded early as they do not meet core requirements:
+
+* **https://fluxcd.io/[Helm Operator (Flux)]**: Flux's Helm controller reconciles HelmRelease resources but lacks service configuration logic, connection secret generation, billing integration - essentially requires rebuilding Crossplane capabilities.
+
+* **https://developer.hashicorp.com/terraform[Terraform Operator]**: Terraform excels at cloud infrastructure provisioning but adds unnecessary complexity for Kubernetes-native workloads.
+Requires maintaining Terraform modules alongside Helm charts, mixing declarative models, and does not provide native Kubernetes resource management.
+
+* **https://kubevela.io/[KubeVela]**: Application-centric platform with promising capabilities but immature ecosystem compared to Crossplane.
+Application model doesn't align with "Helm Chart in" vision, and has less adoption in managed service space.
+Provides similar capabilities to Crossplane but with smaller community and fewer production deployments.
+
+These options lack critical features (revision management, status aggregation, provider ecosystem) or introduce unnecessary abstraction layers without clear benefits over Crossplane 2.x composition functions.
+
+== Decision
+
+**We use Crossplane 2.x with Composition Functions** as the service orchestration mechanism for AppCat Framework 2.x.
+
+=== Rationale
+
+The decision prioritizes long-term maintainability and scalability over short-term simplicity:
+
+1. **Built-in Revision Management**: Crossplane composition revisions provide native versioning and gradual rollout capabilities essential for maintenance workflows.
+Building equivalent functionality in a custom operator would be complex and require significant ongoing maintenance.
+
+2. **Generic Function Pattern**: A single composition function handles all services by loading runtime configuration, eliminating the need for service-specific Go code.
+This enables Service Maintainers to add services independently while keeping the framework stable.
+
+3. **Leverage Existing Ecosystem**: Crossplane providers (Helm, SQL) provide mature implementations for chart rendering, resource management, and status aggregation.
+Reimplementing these capabilities in a custom operator offers no clear benefit.
+
+4. **Existing Team Knowledge**: The team is already proficient with Crossplane from component-appcat, reducing adoption risk and enabling faster implementation.
+
+**Trade-off Accepted:** We accept higher abstraction complexity in exchange for built-in revision management, provider ecosystem leverage, and generic function scalability.

docs/modules/ROOT/pages/adr/0044-configuration-language-kcl-vs-cue.adoc

@@ -0,0 +1,99 @@
+= ADR 0046 - Secret Management (Framework 2.0)
+:adr_author: Gabriel Saratura
+:adr_owner: Schedar
+:adr_reviewers: Schedar
+:adr_date:
+:adr_upd_date:
+:adr_status: draft
+:adr_tags: framework,framework2,secrets,crossplane,vault,openbao
+
+include::partial$adr-meta.adoc[]
+
+[NOTE]
+.Summary
+====
+The decision selects https://docs.crossplane.io/latest/guides/connection-details-composition/[Crossplane 2.x Connection Secret] as connection credentials
+====
+
+== Problem
+
+Services need to securely store and distribute connection credentials (passwords, endpoints, TLS certificates) to Service Users in their claim namespaces.
+
+=== Solutions
+
+==== Option A: Crossplane Connection Secrets (Current)
+
+**Description:**
+
+Crossplane's native https://docs.crossplane.io/latest/guides/connection-details-composition/[connection secret] mechanism.
+Composition functions use `AddConnectionDetails()` to export credentials, which Crossplane writes to the claim namespace.
+
+**Advantages:**
+
+* *Native Crossplane*: Built-in mechanism, no additional dependencies
+* *Proven Pattern*: Existing appcat repository uses this, though architectural simplification is required
+* *Automatic Propagation*: Crossplane handles secret creation in composite namespace
+* *Lifecycle Management*: Secrets deleted when composite deleted
+* *Simple Integration*: Servala portal fetches secrets directly via K8s API
+
+**Disadvantages:**
+
+* *No Rotation*: Secrets are static, no automatic rotation
+* *No Centralization*: Each service instance has separate secret
+* *Limited Auditing*: No built-in audit trail for secret access
+* *Complexity*: High complexity in handling connection secrets through managed resources
+
+==== Option B: External Secrets Operator + Vault/OpenBao
+
+**Description:**
+
+Use https://github.com/openbao/openbao-secrets-operator[External Secrets Operator] to sync secrets from a centralized secret store (https://developer.hashicorp.com/vault[Vault] / https://openbao.org/[OpenBao]) to Kubernetes secrets.
+
+**Architecture:**
+
+[source]
+----
+Composition Function creates secret A in claim namespace with static content such as username
+        ↓
+Creates ExternalSecret CR such as VaultDynamicSecret with reference to secret A
+        ↓
+External Secrets Operator generates password and certificates
+        ↓
+Syncs from Vault to secret A in claim namespace
+----
+
+**Advantages:**
+
+* *Centralized Management*: All secrets in one place (Vault/OpenBao)
+* *Rotation Support*: Can implement automatic password rotation
+* *Auditing*: Vault provides detailed audit logs
+* *RBAC*: Fine-grained access control policies
+* *Encryption at Rest*: Vault encrypts secrets
+
+**Disadvantages:**
+
+* *Additional Dependency*: Requires Vault/OpenBao + External Secrets Operator
+* *Additional Dependency*: Static content has to be handled outside the operator
+* *Complexity*: More moving parts, harder to debug
+* *Operational Overhead*: (Vault/OpenBao) requires management, backup, HA setup
+
+== Decision
+
+We use Crossplane Connection Secrets (continue current approach).
+
+=== Rationale
+
+1. *Proven Pattern*: Existing appcat repository uses Crossplane secret management though major simplification in secret propagation is necessary.
+
+2. *Simplicity*: No additional dependencies. Crossplane handles lifecycle automatically.
+
+3. *Service Credentials*: Secrets are service connection details (host, port, password). These don't require frequent rotation like API keys or tokens.
+
+4. *Servala Integration*: Servala portal already fetches secrets via K8s API. No changes needed.
+
+**Future Enhancement:**
+
+* Phase 2: Investigate External Secrets Operator for centralized secret management
+* https://miro.com/app/board/uXjVJiczJH0=/?moveToWidget=3458764653515989765&cot=14[Implement rotation] policies for long-lived instances
+* Add audit trail for secret access
+