Add multimodal tool outputs

Author: domenic

README.md

@@ -137,48 +137,6 @@ const result = await multiUserSession.prompt([
 
 Because of their special behavior of being preserved on context window overflow, system prompts cannot be provided this way.
 
-### Tool use
-
-The Prompt API supports **tool use** via the `tools` option, allowing you to define external capabilities that a language model can invoke in a model-agnostic way. Each tool is represented by an object that includes an `execute` member that specifies the JavaScript function to be called. When the language model initiates a tool use request, the user agent calls the corresponding `execute` function and sends the result back to the model.
-
-Here’s an example of how to use the `tools` option:
-
-```js
-const session = await LanguageModel.create({
-  initialPrompts: [
-    {
-      role: "system",
-      content: `You are a helpful assistant. You can use tools to help the user.`
-    }
-  ],
-  tools: [
-    {
-      name: "getWeather",
-      description: "Get the weather in a location.",
-      inputSchema: {
-        type: "object",
-        properties: {
-          location: {
-            type: "string",
-            description: "The city to check for the weather condition.",
-          },
-        },
-        required: ["location"],
-      },
-      async execute({ location }) {
-        const res = await fetch("https://weatherapi.example/?location=" + location);
-        // Returns the result as a JSON string.
-        return JSON.stringify(await res.json());
-      },
-    }
-  ]
-});
-
-const result = await session.prompt("What is the weather in Seattle?");
-```
-
-In this example, the `tools` array defines a `getWeather` tool, specifying its name, description, input schema, and `execute` implementation. When the language model determines that a tool call is needed, the user agent invokes the `getWeather` tool's `execute()` function with the provided arguments and returns the result to the model, which can then incorporate it into its response.
-
 ### Multimodal inputs
 
 All of the above examples have been of text prompts. Some language models also support other inputs. Our design initially includes the potential to support images and audio clips as inputs. This is done by using objects in the form `{ type: "image", content }` and `{ type: "audio", content }` instead of strings. The `content` values can be the following:
@@ -269,6 +227,101 @@ Details:
 
 Future extensions may include more ambitious multimodal inputs, such as video clips, or realtime audio or video. (Realtime might require a different API design, more based around events or streams instead of messages.)
 
+### Tool use
+
+The Prompt API supports **tool use** via the `tools` option, allowing you to define external capabilities that a language model can invoke in a model-agnostic way. Each tool is represented by an object that includes an `execute` member that specifies the JavaScript function to be called. When the language model initiates a tool use request, the user agent calls the corresponding `execute` function and sends the result back to the model.
+
+Here’s an example of how to use the `tools` option:
+
+```js
+const session = await LanguageModel.create({
+  initialPrompts: [
+    {
+      role: "system",
+      content: `You are a helpful assistant. You can use tools to help the user.`
+    }
+  ],
+  tools: [
+    {
+      name: "getWeather",
+      description: "Get the weather in a location.",
+      inputSchema: {
+        type: "object",
+        properties: {
+          location: {
+            type: "string",
+            description: "The city to check for the weather condition.",
+          },
+        },
+        required: ["location"],
+      },
+      async execute({ location }) {
+        const res = await fetch("https://weatherapi.example/?location=" + location);
+        // Returns the result as a JSON string.
+        return JSON.stringify(await res.json());
+      },
+    }
+  ]
+});
+
+const result = await session.prompt("What is the weather in Seattle?");
+```
+
+In this example, the `tools` array defines a `getWeather` tool, specifying its name, description, input schema, and `execute` implementation. When the language model determines that a tool call is needed, the user agent invokes the `getWeather` tool's `execute()` function with the provided arguments and returns the result to the model, which can then incorporate it into its response.
+
+#### Tool return values
+
+The above example shows tools returning a string. (In fact, stringified JSON.) Models which support [multimodal inputs](#multimodal-inputs) might also support interpreting image or audio results from tool calls.
+
+Just like the `content` option to a `prompt()` call can accept either a string or an array of `{ type, value }` objects, web developer-provided tools can return either a string or such an array. Here's an example:
+
+```js
+let mutex, resolveMutex;
+
+const session = await LanguageModel.create({
+  tools: [
+    {
+      name: "grabKeyframe",
+      description: "Grab a keyframe from the video we're analyzing at the given time",
+      inputSchema: {
+        type: "number",
+        minimum: 0,
+        exclusiveMaximum: videoEl.duration
+      },
+      expectedOutputs: {
+        types: ["image"]
+      },
+      async execute(timestamp) {
+        if (mutex) {
+          // Since we're seeking a single video element, guard against concurrent calls.
+          await mutex;
+        }
+        try {
+          mutex = new Promise(r => resolveMutex = r);
+
+          if (Math.abs(videoEl.currentTime - timestamp) > 0.001) {
+            videoEl.currentTime = timestamp;
+            await new Promise(r => videoEl.addEventListener("seeked", r, { once: true }));
+          }
+          await new Promise(r => videoEl.requestVideoFrameCallback(r));
+
+          return [{ type: "image", value: videoEl }];
+        } finally {
+          resolveMutex();
+          mutex = null;
+        }
+      }
+    }
+  ]
+});
+```
+
+Note how the output types need to be specified in the tool definition, so that session creation can fail early if the model doesn't support processing multimodal tool outputs. If the return value contains non-text components without them being present in the tool specification, then the tool call will fail at prompting time, even if the model could support it.
+
+Similarly, expected output languages can be provided (via `expectedOutputs: { languages: ["ja" ] }`) or similar, to get an early failure if the model doesn't support processing tool outputs in those languages. However, unlike modalities, there is no prompt-time checking of the tool call result's languages.
+
+The above example shows a single-item array, but just like with prompt inputs, it's allowed to include multiple tool outputs. The same rules are followed as for inputs, e.g., concatenation of adjacent text chunks is done with a single space character.
+
 ### Structured output with JSON schema or RegExp constraints
 
 To help with programmatic processing of language model responses, the prompt API supports constraining the response with either a JSON schema object or a `RegExp` passed as the `responseConstraint` option:

index.bs

@@ -82,6 +82,7 @@ callback LanguageModelToolFunction = Promise<DOMString> (any... arguments);
 dictionary LanguageModelTool {
   required DOMString name;
   required DOMString description;
+  LanguageModelExpected expectedOutputs;
   // JSON schema for the input parameters.
   required object inputSchema;
   // The function to be invoked by user agent on behalf of language model.
@@ -135,14 +136,17 @@ typedef (
 
 dictionary LanguageModelMessage {
   required LanguageModelMessageRole role;
-
-  // The DOMString branch is shorthand for `[{ type: "text", value: providedValue }]`
-  required (DOMString or sequence<LanguageModelMessageContent>) content;
-
+  required LanguageModelMessageContent content;
   boolean prefix = false;
 };
 
-dictionary LanguageModelMessageContent {
+typedef (
+  sequence<LanguageModelMessageContentChunk>
+  // Shorthand for `[{ type: "text", value: providedValue }]`
+  or DOMString
+) LanguageModelMessageContent;
+
+dictionary LanguageModelMessageContentChunk {
   required LanguageModelMessageType type;
   required LanguageModelMessageValue value;
 };