refactor: restructure inference context to separate metadata from runtime state

[AshishKumar4]

Summary

Refactors the inference context system to separate persistent metadata (agentId, userId) from runtime-only state (API keys, abort signals, model configs), preventing accidental serialization of sensitive data in Durable Object state.

Changes

• Split InferenceContext into InferenceMetadata (persistent) and InferenceContext (runtime-assembled)
• Add InferenceRuntimeOverrides type for BYOK API keys and AI Gateway overrides
• Move userModelConfigs and runtimeOverrides from state to behavior instance properties
• Add credentialsToRuntimeOverrides() utility for SDK credential payload conversion
• Update getInferenceContext() method in BaseCodingBehavior to assemble runtime context on-demand
• Rename state field inferenceContext -> metadata with migration support for legacy state
• Propagate runtimeOverrides through inference pipeline to support BYOK keys

Motivation

The previous design persisted runtime-only data (like user API keys and abort signals) in Durable Object state, which was:

1. A security concern - API keys should never be persisted to storage
2. Architecturally incorrect - mixing persistent and ephemeral data
3. Inefficient - serializing/deserializing unnecessary data

This refactor cleanly separates concerns by:

• Persisting only immutable metadata (agentId, userId)
• Assembling runtime context on-demand when needed for inference

Testing

• Existing agent tests should pass with migration handling legacy state
• Test BYOK functionality with runtime API key overrides
• Verify state migration from inferenceContext to metadata for existing apps

Breaking Changes

• state.inferenceContext renamed to state.metadata (migration handles legacy data)
• Code accessing inference context must now use behavior.getInferenceContext()

Related Issues

• None identified

[github-actions]

Claude finished @AshishKumar4's task in 1m 43s —— View job

---

Code Quality & Security Review (Critical Path)

Recommendation: APPROVE

PR Summary

Refactors the inference context system to separate persistent metadata (agentId, userId) from runtime-only state (API keys, abort signals, model configs), preventing accidental serialization of sensitive data in Durable Object state.

PR Description

• Updated: No (already comprehensive)
• Notes: The existing PR description is well-written and complete with clear sections for Summary, Changes, Motivation, Testing, Breaking Changes, and Related Issues.

Addressed Issues

• None identified (security/architecture improvement)

---

Code Quality

Severity	Count	Details
Critical	0	-
High	0	-
Medium	1	TODO comments in base.ts:316-317 should be addressed or converted to tracked issues
Low	1	Duplicate constraint validation logic in infer.ts:98-145 (lines 98-114 and 128-145 are nearly identical)

Positive Observations:

• Clean type definitions with proper JSDoc documentation
• Good separation of concerns between persistent and ephemeral data
• Proper migration handling for legacy state with migrateCommon() and migratePhasic()
• The getInferenceContext() method correctly assembles runtime context on-demand
• readonly modifier on metadata field provides compile-time protection

---

Security

Severity	Count	Details
Critical	0	-
High	0	-
Medium	0	-
Low	0	-

Security Improvements in This PR:

1. API Keys No Longer Persisted - userApiKeys moved from state to runtime-only InferenceRuntimeOverrides. This is the primary security improvement.

2. Clean Type Separation:

   • InferenceMetadata (persistent): Only agentId, userId
   • InferenceRuntimeOverrides (runtime-only): userApiKeys, aiGatewayOverride
   • InferenceContext (assembled on-demand): Combines metadata + runtime state

3. Safe Migration - extractInferenceMetadata() in stateMigration.ts:44-52 properly extracts only safe fields and validates them before migration.

4. No Sensitive Data Logging - Verified that console.log("Getting API key for provider: ", provider) at core.ts:253 only logs the provider name, not the actual API key.

5. Proper BYOK Flow - Runtime overrides are correctly propagated through the inference pipeline:

   • controller.ts creates context with metadata nested properly
   • infer.ts passes runtimeOverrides to core infer()
   • core.ts checks runtimeOverrides?.userApiKeys first in getApiKey()

---

Testing

• Existing agent tests should pass with migration handling
• Recommend manual testing:
  • BYOK functionality with runtime API key overrides
  • State migration from legacy inferenceContext to metadata for existing apps
  • Verify API keys are not appearing in DO state dumps

---

Architecture Notes

The refactor correctly implements the pattern:

┌─────────────────────────────────────────────────────────────┐
│                    Durable Object State                      │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ metadata: { agentId, userId }  (PERSISTED)          │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                  Behavior Instance (Memory)                  │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ userModelConfigs   (EPHEMERAL - loaded on start)    │    │
│  │ runtimeOverrides   (EPHEMERAL - per-request BYOK)   │    │
│  │ abortSignal        (EPHEMERAL - per-operation)      │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│              getInferenceContext() (Assembled on-demand)     │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ InferenceContext = metadata + all runtime fields     │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

---

Final Recommendation

APPROVE - This PR implements a well-designed security improvement by ensuring API keys and other sensitive runtime data are never persisted to Durable Object storage. The migration path is clean, and the type system now enforces the separation between persistent and ephemeral data at compile time.

Minor suggestions (non-blocking):

1. Consider converting the TODO comments at base.ts:316-317 to tracked issues
2. The duplicate validation logic in infer.ts could be refactored, but this is pre-existing and outside the scope of this PR