Implement new fetchProgram built-in using HTTPProgramResolver (commontoolsinc#2049)

* Implement fetchProgram builtin for URL-based program execution

Add new fetchProgram builtin that enables patterns to fetch and resolve
programs from URLs, composing cleanly with the existing compileAndRun builtin.

Implementation:
- Created fetch-program.ts builtin following fetchData pattern
- Uses HttpProgramResolver to fetch main file and resolve dependencies
- Uses resolveProgram to handle multi-file imports
- Returns { pending, result: { files, main }, error } structure
- Registered in builtins/index.ts
- Added type definitions to api/index.ts
- Added builder API in builder/built-in.ts
- Created test pattern in patterns/fetch-program-test.tsx

Usage example:
const { result: program } = fetchProgram({ url });
const { result } = compileAndRun({ ...program, input: data });

This enables loading and executing patterns from GitHub URLs or any
HTTP-accessible location, supporting the goal of distributing patterns
as public, reusable components.

* Refactor: Extract shared fetch utilities to eliminate duplication

Extracted common fetch-related utilities (tryClaimMutex, tryWriteResult,
computeInputHash, internalSchema) into a shared fetch-utils.ts module.

Changes:
- Created packages/runner/src/builtins/fetch-utils.ts with shared utilities
- Updated fetch-data.ts to import from fetch-utils.ts
- Updated fetch-program.ts to import from fetch-utils.ts
- Eliminated ~100 lines of duplicated code

Benefits:
- DRY principle - single source of truth for fetch patterns
- Easier maintenance - changes only need to be made once
- Consistent behavior across all fetch-style builtins
- Makes fetch-program.ts configurable with longer timeout (10s vs 5s)

The shared utilities handle:
- Input hashing for change detection
- Mutex claiming for multi-tab coordination
- Result writing with input validation

* Simplify fetchProgram - remove unnecessary complexity

Stripped down fetchProgram from ~300 lines to ~115 lines by removing:
- Mutex/multi-tab coordination (over-engineered for initial version)
- Input hashing and change detection
- AbortController logic
- All the shared fetch-utils machinery

Core logic is now clear:
1. Initialize cells for pending/result/error
2. Fetch URL and resolve program dependencies
3. Update cells with result or error

The builtin now does exactly what it needs to:
- Fetch a program from a URL using HttpProgramResolver
- Resolve all dependencies via resolveProgram
- Return { files, main } structure for compileAndRun

Can always add optimizations later if needed.

* Use shared fetch utilities in fetchProgram for consistency

Updated fetchProgram to use the same fetch-utils.ts patterns as fetchData:
- Multi-tab coordination via tryClaimMutex (prevents duplicate fetches)
- Input hashing via computeInputHash (detects URL changes)
- Safe result writing via tryWriteResult (any tab can write if inputs match)
- AbortController support for cancellation
- Proper cleanup on recipe stop

This ensures both fetch-style builtins follow the same pattern for:
- Avoiding duplicate network requests across tabs
- Handling race conditions properly
- Consistent timeout and retry behavior

fetchProgram uses 10s timeout (vs fetchData's 5s) since program
resolution with dependencies can take longer than simple HTTP fetches.

* It's alive!

* Format pass

* Fix lint + format

* Exclude result from hash computation

* Tighten retry and de-duplication logic

* Clear result if input to fetchProgram cleared

* Format + lint

* Fix tests

* Address code review

* Action code review feedback

* One more pass over the state machine

---------

Co-authored-by: Claude <[email protected]>

Author: bfollington

packages/api/index.ts

@@ -1183,6 +1183,17 @@ export type FetchDataFunction = <T>(
   }>,
 ) => OpaqueRef<{ pending: boolean; result: T; error: any }>;
 
+export type FetchProgramFunction = (
+  params: Opaque<{ url: string }>,
+) => OpaqueRef<{
+  pending: boolean;
+  result: {
+    files: Array<{ name: string; contents: string }>;
+    main: string;
+  } | undefined;
+  error: any;
+}>;
+
 export type StreamDataFunction = <T>(
   params: Opaque<{
     url: string;
@@ -1255,6 +1266,7 @@ export declare const llmDialog: LLMDialogFunction;
 export declare const generateObject: GenerateObjectFunction;
 export declare const generateText: GenerateTextFunction;
 export declare const fetchData: FetchDataFunction;
+export declare const fetchProgram: FetchProgramFunction;
 export declare const streamData: StreamDataFunction;
 export declare const compileAndRun: CompileAndRunFunction;
 export declare const navigateTo: NavigateToFunction;

packages/js-runtime/typescript/mod.ts

@@ -7,4 +7,9 @@ export {
   type CompilationErrorType,
   CompilerError,
 } from "./diagnostics/mod.ts";
-export { getCompilerOptions } from "./options.ts";
+export { getCompilerOptions, TARGET } from "./options.ts";
+export {
+  type ResolveModuleConfig,
+  resolveProgram,
+  type UnresolvedModuleHandling,
+} from "./resolver.ts";

packages/patterns/fetch-program-test.tsx

@@ -0,0 +1,63 @@
+/// <cts-enable />
+import {
+  cell,
+  compileAndRun,
+  derive,
+  fetchProgram,
+  NAME,
+  recipe,
+  UI,
+} from "commontools";
+
+/**
+ * Test pattern for fetchProgram builtin.
+ * Fetches a program from a URL and compiles it.
+ */
+export default recipe("Fetch Program Test", () => {
+  // URL to a simple pattern file
+  const url = cell(
+    "https://raw.githubusercontent.com/commontoolsinc/labs/main/packages/patterns/counter.tsx",
+  );
+
+  // Step 1: Fetch the program from URL
+  const { pending: fetchPending, result: program, error: fetchError } =
+    fetchProgram({ url });
+
+  // Step 2: Compile and run the fetched program
+  // Explicitly map program fields to compileAndRun params
+  const compileParams = derive(program, (p) => ({
+    files: p?.files ?? [],
+    main: p?.main ?? "",
+    input: { value: 10 },
+  }));
+  const { pending: compilePending, result, error: compileError } =
+    compileAndRun(compileParams);
+
+  return {
+    [NAME]: "Fetch Program Test",
+    [UI]: (
+      <div>
+        <h1>Fetch Program Test</h1>
+        <div>
+          <label>URL:</label>
+          <ct-input type="text" $value={url} />
+        </div>
+        {fetchPending && <div>Fetching program...</div>}
+        {compilePending && <div>Compiling...</div>}
+        {fetchError && <div style="color: red">Fetch error: {fetchError}</div>}
+        {compileError && (
+          <div style="color: red">Compile error: {compileError}</div>
+        )}
+        {result && (
+          <div style="color: green">
+            Successfully compiled recipe! Charm ID: {result}
+            <pre>{derive(result, r => JSON.stringify(r, null, 2))}</pre>
+          </div>
+        )}
+      </div>
+    ),
+    url,
+    program,
+    result,
+  };
+});

packages/runner/src/builder/built-in.ts

@@ -71,6 +71,20 @@ export const fetchData = createNodeFactory({
   }>,
 ) => OpaqueRef<{ pending: boolean; result: T; error: unknown }>;
 
+export const fetchProgram = createNodeFactory({
+  type: "ref",
+  implementation: "fetchProgram",
+}) as (
+  params: Opaque<{ url: string }>,
+) => OpaqueRef<{
+  pending: boolean;
+  result: {
+    files: Array<{ name: string; contents: string }>;
+    main: string;
+  } | undefined;
+  error: unknown;
+}>;
+
 export const streamData = createNodeFactory({
   type: "ref",
   implementation: "streamData",

packages/runner/src/builder/factory.ts

@@ -27,6 +27,7 @@ import { byRef, computed, derive, handler, lift } from "./module.ts";
 import {
   compileAndRun,
   fetchData,
+  fetchProgram,
   generateObject,
   generateText,
   ifElse,
@@ -92,6 +93,7 @@ export const createBuilder = (): {
       generateObject,
       generateText,
       fetchData,
+      fetchProgram,
       streamData,
       compileAndRun,
       navigateTo,

packages/runner/src/builder/types.ts

@@ -15,6 +15,7 @@ import type {
   ComputedFunction,
   DeriveFunction,
   FetchDataFunction,
+  FetchProgramFunction,
   GenerateObjectFunction,
   GenerateTextFunction,
   GetRecipeEnvironmentFunction,
@@ -254,6 +255,7 @@ export interface BuilderFunctionsAndConstants {
   generateObject: GenerateObjectFunction;
   generateText: GenerateTextFunction;
   fetchData: FetchDataFunction;
+  fetchProgram: FetchProgramFunction;
   streamData: StreamDataFunction;
   compileAndRun: CompileAndRunFunction;
   navigateTo: NavigateToFunction;

packages/runner/src/builtins/compile-and-run.ts

@@ -140,6 +140,27 @@ export function compileAndRun(
     // as previousCallHash) or when rehydrated from storage (same as the
     // contents of the requestHash doc).
     if (hash === previousCallHash) return;
+
+    // Check if inputs are undefined/empty (e.g., during rehydration before cells load)
+    const hasValidInputs = program.main && program.files &&
+      program.files.length > 0;
+
+    // Special case: if inputs are invalid AND this is the hash for empty inputs,
+    // the user intentionally cleared them - proceed to clear outputs
+    const emptyInputsHash = refer({ files: [], main: "" }).toString();
+    const isIntentionallyEmpty = !hasValidInputs && hash === emptyInputsHash;
+
+    // If we have a previous valid result and inputs are currently invalid (likely rehydrating),
+    // don't clear the outputs - just wait for real inputs to load
+    // BUT if inputs are intentionally empty, we should clear
+    if (
+      !hasValidInputs && previousCallHash && previousCallHash !== hash &&
+      !isIntentionallyEmpty
+    ) {
+      // Don't update previousCallHash - we'll wait for valid inputs
+      return;
+    }
+
     previousCallHash = hash;
 
     // Abort any in-flight compilation before starting a new one
@@ -153,7 +174,7 @@ export function compileAndRun(
     errorsWithLog.set(undefined);
 
     // Undefined inputs => Undefined output, not pending
-    if (!program.main || !program.files) {
+    if (!hasValidInputs) {
       pendingWithLog.set(false);
       return;
     }