feat(messaging): add manifest compiler

[sandl99]

Summary

Adds the phase-1 messaging manifest compiler that converts channel manifests into a serializable sandbox messaging plan. The compiler resolves channel inputs through env keys and interactive enrollment hooks, then delegates credential, policy, render, build-step, state-update, and health-check planning to small pure engines.

Related Issue

Fixes #3994

Changes

• Add ManifestCompiler with interactive enrollment-hook input resolution and env-key input initialization.
• Add compiler plan engines for credential bindings, network policy, agent render fragments, build steps, state updates, and health checks.
• Expand SandboxMessagingPlan and related manifest plan types to the top-level plan shape required by [Messaging] Add pure manifest compiler and plan engines #3994.
• Add coverage for built-in Telegram/Discord/Slack/WeChat/WhatsApp plans, Hermes WeChat policy aliasing, non-interactive env input behavior, secret-free JSON plans, disabled channels, and a synthetic non-built-in channel.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Verification

• npx prek run --all-files passes
• npm test passes
• Tests added or updated for new or changed behavior
• No secrets, API keys, or credentials committed
• Docs updated for user-facing behavior changes
• make docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

Additional verification performed:

• npm test -- --project cli src/lib/messaging passes.
• npm run typecheck:cli passes.
• npm run lint -- src/lib/messaging passes with the existing unrelated warning in src/lib/onboard/child-exit-tracker.test.ts.
• git diff --check passes.
• npm run source-shape:check passes.
• npx prek run --all-files and the normal pre-push hook were attempted and currently fail in unrelated full CLI doctor/debug/snapshot tests outside the messaging compiler changes.

---

Signed-off-by: San Dang sdang@nvidia.com

Summary by CodeRabbit

• New Features

  • Added manifest compilation system for messaging channels with support for multiple agents and workflows
  • Implemented credential binding and authentication management
  • Added network policy configuration and agent rendering capabilities
  • Introduced health check and build step planning
  • Added state persistence and hydration management
  • Implemented placeholder resolution for sandbox names and credentials

• Tests

  • Added comprehensive test suite validating compilation behavior, credential handling, and plan serialization

[github-actions]

E2E Advisor Recommendation

Required E2E: messaging-providers-e2e, channels-stop-start-e2e, network-policy-e2e, hermes-slack-e2e, hermes-discord-e2e
Optional E2E: token-rotation-e2e, channels-add-remove-e2e, credential-sanitization-e2e, messaging-compatible-endpoint-e2e

Dispatch hint: messaging-providers-e2e,channels-stop-start-e2e,network-policy-e2e,hermes-slack-e2e,hermes-discord-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/u/sdang/messaging-hooks-channels-3993-3994
Head: HEAD
Confidence: high

Required E2E

• messaging-providers-e2e (high): Primary coverage for messaging credential provider creation, placeholder rendering, L7 proxy rewrite, no-secret leakage, channel config, and policy/no-provider behavior across Telegram, Discord, Slack, WeChat, and WhatsApp.
• channels-stop-start-e2e (high): Exercises OpenClaw and Hermes channel lifecycle across telegram, discord, wechat, slack, and whatsapp, including stop/start/remove, cached credential reattachment, provider deletion, policy unapply, and rebuild behavior impacted by active/configured/disabled channel planning.
• network-policy-e2e (medium): The new policy resolver maps messaging presets and agent aliases to concrete policy keys; this job validates policy boundary behavior, deny-by-default, whitelist/hot-reload, and SSRF protections.
• hermes-slack-e2e (medium): Validates Hermes-specific Slack onboarding, policy, dual Slack credential placeholders, and OpenShell credential rewrite path, which are directly adjacent to new agent render and credential binding planning.
• hermes-discord-e2e (medium): Validates Hermes Discord config shape and placeholder isolation in ~/.hermes outputs, covering the new agent-specific env-lines render path for a second Hermes messaging provider.

Optional E2E

• token-rotation-e2e (medium): Useful additional confidence for credential availability and provider rebinding semantics after re-onboard, but the PR primarily introduces compile-time plan structures rather than token rotation logic.
• channels-add-remove-e2e (high): Adjacent coverage for channel add/remove applying and unapplying policy presets and preserving rebuild behavior; channels-stop-start-e2e is broader and should be merge-blocking first.
• credential-sanitization-e2e (medium): Additional end-to-end leak scanning for credential material after sandbox onboarding; useful because compiled plans intentionally carry placeholders and availability booleans instead of raw secret values.
• messaging-compatible-endpoint-e2e (medium): Optional regression for Telegram config plus inference.local routing with an OpenAI-compatible endpoint; relevant to agent config rendering but less direct than provider and Hermes-specific messaging jobs.

New E2E recommendations

• messaging manifest compiler integration (high): Existing E2E jobs validate current runtime messaging behavior, but there is no dedicated E2E that asserts a compiled SandboxMessagingPlan is consumed by the installer/onboarding/channel applier path end-to-end.
  • Suggested test: Add an E2E that enables the manifest compiler path during non-interactive onboard and verifies the compiled plan fields drive provider attachment, policy presets, rendered agent config, build steps, state persistence, and health gates.
• custom channel manifest coverage (medium): Unit tests cover a non-built-in Matrix-like manifest, but existing E2E jobs focus on built-in channels and do not prove custom manifests can be compiled and applied without runtime side effects or secret leakage.
  • Suggested test: Add a hermetic custom messaging channel E2E using a fake HTTP provider to verify generic manifest compilation, policy preset resolution, credential binding, and agent render application.
• WeChat build-step and hook-output planning (medium): The compiler now emits build-file/build-arg plans from hook outputs, especially for WeChat-style seeded account files, but current E2E coverage may not assert the compiler-produced build steps are honored directly.
  • Suggested test: Add an E2E assertion in the WeChat messaging flow that build-step hook outputs from the compiled plan are materialized into the sandbox build context and survive rebuild hydration.

Dispatch hint

• Workflow: nightly-e2e.yaml
• jobs input: messaging-providers-e2e,channels-stop-start-e2e,network-policy-e2e,hermes-slack-e2e,hermes-discord-e2e

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e85aa396-32db-4734-bba0-5dd7e6830c18

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• ✅ Review completed - (🔄 Check again to review again)

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch u/sdang/messaging-manifest-compiler-3994

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

PR Review Advisor

Recommendation: blocked
Confidence: medium
Analyzed HEAD: f38810447f366ce49e69c2357d51310729e0de1b
Findings: 3 blocker(s), 3 warning(s), 1 suggestion(s)

This is an automated advisory review. A human maintainer must make the final merge decision.

Limitations: Review was based on trusted gathered context, the provided diff, and read-only inspection of key changed/current files; no scripts, package-manager commands, tests, or E2E jobs were executed by this advisor.; The linked issue had no comments in gathered context; acceptance coverage maps the issue body clauses and the PR/E2E advisor context only.; The diff in the prompt was truncated outside the displayed files, though the main compiler, engine, type, hook, and registry files relevant to the findings were inspected from the worktree.; CodeRabbit auto review was skipped because the base branch is not the default branch, so there is no automated line-review signal to incorporate.

Workflow run

Full advisor summary

PR Review Advisor

Base: origin/u/sdang/messaging-hooks-channels-3993-3994
Head: HEAD
Analyzed SHA: f38810447f366ce49e69c2357d51310729e0de1b
Recommendation: blocked
Confidence: medium

The manifest compiler shape is well tested for serialization and secret-free plans, but merge should wait on required messaging/network E2E jobs and resolution of purity/input-validation/policy-boundary concerns.

Gate status

• CI: pass — 5 required status context(s) completed with no failures for f388104: checks, commit-lint, dco-check, check-hash, changes. Non-required contexts were still pending in gathered context: test-e2e-ollama-proxy queued, build-sandbox-images in progress, build-sandbox-images-arm64 in progress.
• Mergeability: warning — GitHub reported mergeable=true but mergeStateStatus=UNSTABLE / mergeable_state=unstable for head f388104.
• Review threads: pass — GraphQL reviewThreads.nodes was empty and review_comments=0. CodeRabbit posted that auto review was skipped for this non-default base branch, so there are no CodeRabbit threads to resolve, but also no CodeRabbit line review signal.
• Risky code tested: warning — Risky areas detected: credentials/inference/network. Unit tests were added, but E2E Advisor required messaging-providers-e2e, channels-stop-start-e2e, network-policy-e2e, hermes-slack-e2e, and hermes-discord-e2e; no matching passed job contexts were present for this head SHA.

🔴 Blockers

• Required messaging/network E2E jobs are missing for this head SHA: The E2E Advisor explicitly required messaging-providers-e2e, channels-stop-start-e2e, network-policy-e2e, hermes-slack-e2e, and hermes-discord-e2e because the PR changes credential, provider, render, lifecycle, and network-policy planning. The status rollup for f388104 does not show those required jobs as passed.
  • Recommendation: Run the E2E Advisor-required jobs for the current head SHA and review failures before merge.
  • Evidence: E2E Advisor comment lists required jobs: messaging-providers-e2e, channels-stop-start-e2e, network-policy-e2e, hermes-slack-e2e, hermes-discord-e2e. Status rollup contains E2E recommendation success but not passed contexts for those job names.
• Compiler can execute enrollment hooks, which weakens the pure-planning boundary (src/lib/messaging/compiler/manifest-compiler.ts:154): The linked issue asks for a pure manifest compiler / architecture-only planning engine. compile() can call runMessagingHook for active create+interactive channels, so passing a production hook registry could run prompt, QR, credential, filesystem, or network side effects during compilation instead of only producing a plan.
  • Recommendation: Keep the compiler pure: make enrollment outputs explicit compiler inputs, add a separate side-effectful enrollment phase outside ManifestCompiler, or harden compile() so it only accepts dry-run/pure hook adapters and documents/enforces that boundary.
  • Evidence: resolveChannelInputs filters enroll hooks and calls runMessagingHook(hook, hooks, ...) at src/lib/messaging/compiler/manifest-compiler.ts:154 when runEnrollment is true.
• New 480-line compiler test monolith exceeds repository growth budget signal (src/lib/messaging/compiler/manifest-compiler.test.ts:1): Trusted monolith analysis flagged src/lib/messaging/compiler/manifest-compiler.test.ts as a new 480-line monolith with blocker severity because the current monolith grew by 20 or more lines.
  • Recommendation: Split the test file into focused suites, fixtures, and engine-specific tests, or offset the monolith growth according to the repository source-shape guidance before merge.
  • Evidence: monolithDeltas reported baseLines=0, headLines=480, delta=480, severity=blocker for src/lib/messaging/compiler/manifest-compiler.test.ts.

🟡 Warnings

• Config input values from environment are copied into plans without validation (src/lib/messaging/compiler/manifest-compiler.ts:202): Manifest inputs can declare validValues, but readInputEnvValue returns raw process.env strings and resolveChannelInput stores config values in SandboxMessagingInputReference.value without checking validValues, expected format, or length. These values may later drive rendered config, rebuild hydration, or hook inputs, so invalid or injection-shaped config can cross the compiler boundary.
  • Recommendation: Validate non-secret env/config inputs against manifest allowlists and per-field format/length rules before adding them to plans or hook inputs; add negative tests for invalid validValues and malformed IDs/URLs.
  • Evidence: ChannelInputBaseSpec includes validValues in src/lib/messaging/manifest/types.ts:68, but readInputEnvValue at src/lib/messaging/compiler/manifest-compiler.ts:202 only reads process.env and resolveChannelInput stores config env values directly.
• Unknown policy presets are passed through as concrete policy keys (src/lib/messaging/compiler/engines/policy-resolver.ts:59): For presets not in the built-in map or agent alias map, the resolver emits policyKeys: [presetName] with source: manifest. If custom manifests are ever user- or plugin-controlled, this can turn a manifest string into a requested network policy key and broaden egress unless a later applier independently validates the key against an allowlist.
  • Recommendation: Resolve policy presets through a trusted policy registry/allowlist and fail closed for unknown presets, or explicitly mark unknown custom policy keys as requiring privileged approval before application.
  • Evidence: planManifestPolicyEntries returns policyKeys: builtinKeys ?? [presetName] and source: builtinKeys ? 'builtin' : 'manifest' at src/lib/messaging/compiler/engines/policy-resolver.ts:59-60.
• Active messaging PR overlap on public messaging exports and manifest types (src/lib/messaging/index.ts:4): PR feat(messaging): add channel enrollment manifests #4050 overlaps this PR on src/lib/messaging/index.ts and src/lib/messaging/manifest/types.test.ts. This PR also targets the feature branch from that work, so reviewers should check for ordering assumptions and rebase drift before landing.
  • Recommendation: Coordinate merge order with PR feat(messaging): add channel enrollment manifests #4050 and re-run type/tests/E2E after the base branch changes or this branch is rebased.
  • Evidence: openPrOverlaps reported PR feat(messaging): add channel enrollment manifests #4050 'feat(messaging): add channel enrollment manifests' with sameFiles src/lib/messaging/index.ts and src/lib/messaging/manifest/types.test.ts.

🔵 Suggestions

• Add negative compiler tests for unsupported agents, missing manifests, invalid config, and policy denial (src/lib/messaging/compiler/manifest-compiler.test.ts:1): The added tests cover happy paths, disabled channels, non-interactive env behavior, serialization, no functions, and secret-free JSON. Missing negative tests would be useful for unsupported selected channels, invalid validValues, unknown policy presets, missing required hook outputs in the compiler path, and side-effect isolation.
  • Recommendation: Add focused negative tests for compiler failure/deny paths and keep them separated from large fixture assertions.
  • Evidence: Existing tests exercise deterministic built-ins, Hermes aliasing, enrollment, non-interactive env behavior, no-secret serialization, disabled channels, and a custom Matrix manifest, but no invalid-input or policy-denial cases are present.

Acceptance coverage

• unknown — Part of Refactor messaging integrations into a manifest-first planning architecture #3896 phase 1.: PR links Fixes [Messaging] Add pure manifest compiler and plan engines #3994, whose body says it is part of Refactor messaging integrations into a manifest-first planning architecture #3896 phase 1. No parent issue details beyond this reference were available in the gathered context.
• partial — Add the pure planning engine that converts validated channel manifests into a serializable SandboxMessagingPlan.: ManifestCompiler returns SandboxMessagingPlan and tests verify JSON round-trip/no functions. Purity is questionable because resolveChannelInputs can execute enrollment hooks, and manifest/config input validation is not evident for validValues.
• partial — This is architecture-only work: no production workflow should use the compiler yet.: No installer/onboarding/channel-applier workflow changes are in the changed files. However src/lib/messaging/index.ts now exports ./compiler, making the compiler part of the public messaging surface before integration/E2E coverage.
• met — Finalize SandboxMessagingPlan so it contains top-level channels, credentialBindings, networkPolicy, agentRender, buildSteps, stateUpdates, and healthChecks.: src/lib/messaging/manifest/types.ts defines these top-level fields on SandboxMessagingPlan, and manifest-compiler.ts returns all of them.
• partial — Add compiler context types for sandbox name, agent, workflow, selected channels, configured channels, disabled channels, persisted state, config inputs, and credential availability.: ManifestCompilerContext includes sandboxName, agent, workflow, selectedChannels, configuredChannels, disabledChannels, supportedChannelIds, and credentialAvailability. It does not include an explicit persisted state or config input map; config values are read from process.env instead.
• partial — Add pure engines: input-resolver.ts, credential-binding-engine.ts, policy-resolver.ts, agent-render-engine.ts, state-update-engine.ts, build-step-engine.ts, and health-check-engine.ts.: credential-binding-engine.ts, policy-resolver.ts, agent-render-engine.ts, state-update-engine.ts, build-step-engine.ts, and health-check-engine.ts exist. There is no input-resolver.ts; input resolution is embedded in manifest-compiler.ts and can execute hooks.
• met — Add ManifestCompiler.: src/lib/messaging/compiler/manifest-compiler.ts adds exported class ManifestCompiler.
• met — Add tests for plan JSON round-trip, no secrets, no function-valued fields, and deterministic output.: src/lib/messaging/compiler/manifest-compiler.test.ts verifies deterministic equality, jsonRoundTrip, findFunctionPaths, and absence of raw secret strings. src/lib/messaging/manifest/types.test.ts also covers serialization and function-free contracts.
• met — Telegram, Discord, Slack, WeChat, and WhatsApp manifests compile into stable SandboxMessagingPlan fixtures.: The built-in compiler test compiles selectedChannels for slack, telegram, wechat, discord, and whatsapp and checks deterministic ordering and plan content.
• met — Provider names and placeholders match current behavior.: manifest-compiler.test.ts asserts provider names such as demo-telegram-bridge/demo-discord-bridge/demo-wechat-bridge and placeholders such as openshell:resolve:env:TELEGRAM_BOT_TOKEN plus Slack xoxb/xapp placeholders.
• met — Policy plans match current channel policy intent, including Hermes WeChat alias behavior.: OpenClaw policy entries are asserted for telegram/discord/wechat/slack/whatsapp, and the Hermes test asserts WeChat source agent-alias with policyKeys ['wechat_bridge'].
• met — OpenClaw and Hermes render fragments are represented in the plan.: OpenClaw test asserts json-fragment render entries; Hermes test asserts ~/.hermes/.env and ~/.hermes/config.yaml targets for multiple channels.
• partial — Build args, build files, and rebuild hydration are represented without applying them.: Build-file steps and rebuild-hydration state updates are represented in plan tests. No build-arg fixture assertion is visible in the provided diff, and the compiler can execute enrollment hooks for interactive create plans.
• met — Health checks are planned before any lifecycle success can be reported.: planHealthChecks emits requiredBefore: lifecycle-success, and the built-in test verifies all health checks have requiredBefore === 'lifecycle-success'.
• met — ManifestCompiler contains no channel-specific branches like if channel === "wechat".: ManifestCompiler logic uses manifest data and generic active/selected/configured/disabled handling; channel-specific WeChat behavior is in manifests/engines/tests, not conditional branches in ManifestCompiler.
• partial — Compiler and engines do not import production side-effect modules.: types.test.ts scans production messaging modules for forbidden imports and the compiler engines only import manifest/compiler types or pure helpers. However ManifestCompiler imports runMessagingHook and can execute registered hook handlers, so the runtime purity boundary depends on the registry supplied by callers.
• met — Do not write registry or session state.: Changed compiler files do not import state/registry and state-update-engine only produces state update plan objects.
• met — Do not create providers.: credential-binding-engine only emits provider binding plan entries; no provider creation APIs or credential adapters are imported in the changed compiler files.
• met — Do not patch Dockerfiles.: build-step-engine only emits build-step plan metadata from hook output declarations; no filesystem or Dockerfile patching modules are imported.
• met — Do not modify OpenClaw or Hermes config files.: agent-render-engine only emits json-fragment/env-lines render plans; no node:fs imports or agent config write paths appear in compiler files.

Security review

• pass — Category 1: Secrets and Credentials: Secret env values and secret hook outputs are not copied into the plan; tests assert raw token strings are absent and placeholders/credentialAvailable booleans are used instead. No hardcoded real secrets were observed in changed files.
• warning — Category 2: Input Validation and Data Sanitization: Config input values are read from process.env and stored in plans without enforcing validValues or format/length constraints. This is relevant because config values can later affect hook inputs, persisted state, or rendered agent configuration.
• pass — Category 3: Authentication and Authorization: No new endpoints, auth checks, token validation, or authorization boundaries are introduced. The compiler only models credential availability and plan metadata.
• pass — Category 4: Dependencies and Third-Party Libraries: No new dependencies or package registry changes are present in the changed files.
• pass — Category 5: Error Handling and Logging: No logging of secrets or sensitive values was added. Errors for missing manifests contain channel IDs only; however human review should ensure future callers do not surface untrusted IDs in unsafe contexts.
• pass — Category 6: Cryptography and Data Protection: Not applicable — no cryptographic operations, algorithms, or data-at-rest/in-transit handling are added in this change.
• warning — Category 7: Configuration and Security Headers: Network policy planning passes unknown manifest presets through as concrete policy keys. If custom manifests become user/plugin controlled, later appliers must fail closed against a trusted policy registry to avoid egress policy expansion.
• warning — Category 8: Security Testing: Unit tests cover no-secret serialization and several plan cases, but required messaging/network E2E jobs have not passed for the head SHA, and negative tests for invalid config values or unknown policy presets are missing.
• warning — Category 9: Holistic Security Posture: The compiler largely keeps side effects out of engine files and avoids raw secret material, but executing enrollment hooks inside compile() weakens the intended pure-planning boundary and could become a side-effect or credential-flow risk if production hook registries are used.

Test / E2E status

• Test depth: e2e_required — Runtime/sandbox/infrastructure paths need real execution coverage: src/lib/messaging/compiler/engines/agent-render-engine.ts, src/lib/messaging/compiler/engines/build-step-engine.ts, src/lib/messaging/compiler/engines/credential-binding-engine.ts, src/lib/messaging/compiler/engines/health-check-engine.ts, src/lib/messaging/compiler/engines/policy-resolver.ts, src/lib/messaging/compiler/engines/state-update-engine.ts, src/lib/messaging/compiler/engines/template.ts, src/lib/messaging/compiler/index.ts. Unit tests give useful deterministic/secret-free coverage, but they cannot prove provider attachment, network policy behavior, sandbox rebuild materialization, or Hermes/OpenClaw runtime rendering.
• E2E Advisor: missing
• Required E2E jobs: messaging-providers-e2e, channels-stop-start-e2e, network-policy-e2e, hermes-slack-e2e, hermes-discord-e2e
• Missing for analyzed SHA: messaging-providers-e2e, channels-stop-start-e2e, network-policy-e2e, hermes-slack-e2e, hermes-discord-e2e

✅ What looks good

• The plan shape is explicit and serializable, with top-level sections for channels, credential bindings, network policy, render, build steps, state updates, and health checks.
• Tests assert deterministic output, JSON round-trip behavior, absence of function-valued fields, and no raw secret leakage for representative scenarios.
• Disabled configured channels are tested to avoid credential/policy/render/build/state/health side-effect planning.
• The compiler engines are small and mostly data-driven, and the custom Matrix-like manifest test helps guard against channel-specific branches.
• Side-effect module import scanning in manifest/types.test.ts is a useful guardrail for keeping messaging manifest production modules isolated.

Review completeness

• Review was based on trusted gathered context, the provided diff, and read-only inspection of key changed/current files; no scripts, package-manager commands, tests, or E2E jobs were executed by this advisor.
• The linked issue had no comments in gathered context; acceptance coverage maps the issue body clauses and the PR/E2E advisor context only.
• The diff in the prompt was truncated outside the displayed files, though the main compiler, engine, type, hook, and registry files relevant to the findings were inspected from the worktree.
• CodeRabbit auto review was skipped because the base branch is not the default branch, so there is no automated line-review signal to incorporate.
• Human maintainer review required: yes

[wscurran]

✨

Related open issues:

• #3994 [Messaging] Add pure manifest compiler and plan engines