diff --git a/README.md b/README.md index aace534..b4e5586 100644 --- a/README.md +++ b/README.md @@ -137,60 +137,6 @@ const result = await multiUserSession.prompt([ Because of their special behavior of being preserved on context window overflow, system prompts cannot be provided this way. -### Tool use - -The Prompt API supports **tool use** via the `tools` option, allowing you to define external capabilities that a language model can invoke in a model-agnostic way. Each tool is represented by an object that includes an `execute` member that specifies the JavaScript function to be called. When the language model initiates a tool use request, the user agent calls the corresponding `execute` function and sends the result back to the model. - -Here’s an example of how to use the `tools` option: - -```js -const session = await LanguageModel.create({ - initialPrompts: [ - { - role: "system", - content: `You are a helpful assistant. You can use tools to help the user.` - } - ], - tools: [ - { - name: "getWeather", - description: "Get the weather in a location.", - inputSchema: { - type: "object", - properties: { - location: { - type: "string", - description: "The city to check for the weather condition.", - }, - }, - required: ["location"], - }, - async execute({ location }) { - const res = await fetch("https://weatherapi.example/?location=" + location); - // Returns the result as a JSON string. - return JSON.stringify(await res.json()); - }, - } - ] -}); - -const result = await session.prompt("What is the weather in Seattle?"); -``` - -In this example, the `tools` array defines a `getWeather` tool, specifying its name, description, input schema, and `execute` implementation. When the language model determines that a tool call is needed, the user agent invokes the `getWeather` tool's `execute()` function with the provided arguments and returns the result to the model, which can then incorporate it into its response. - -#### Concurrent tool use - -Developers should be aware that the model might call their tool multiple times, concurrently. For example, code such as - -```js -const result = await session.prompt("Which of these locations currently has the highest temperature? Seattle, Tokyo, Berlin"); -``` - -might call the above `"getWeather"` tool's `execute()` function three times. The model would wait for all tool call results to return, using the equivalent of `Promise.all()` internally, before it composes its final response. - -Similarly, the model might call multiple different tools, if it believes they all are relevant when responding to the given prompt. - ### Multimodal inputs All of the above examples have been of text prompts. Some language models also support other inputs. Our design initially includes the potential to support images and audio clips as inputs. This is done by using objects in the form `{ type: "image", content }` and `{ type: "audio", content }` instead of strings. The `content` values can be the following: @@ -281,6 +227,115 @@ Details: Future extensions may include more ambitious multimodal inputs, such as video clips, or realtime audio or video. (Realtime might require a different API design, more based around events or streams instead of messages.) +### Tool use + +The Prompt API supports **tool use** via the `tools` option, allowing you to define external capabilities that a language model can invoke in a model-agnostic way. Each tool is represented by an object that includes an `execute` member that specifies the JavaScript function to be called. When the language model initiates a tool use request, the user agent calls the corresponding `execute` function and sends the result back to the model. + +Here’s an example of how to use the `tools` option: + +```js +const session = await LanguageModel.create({ + initialPrompts: [ + { + role: "system", + content: `You are a helpful assistant. You can use tools to help the user.` + } + ], + tools: [ + { + name: "getWeather", + description: "Get the weather in a location.", + inputSchema: { + type: "object", + properties: { + location: { + type: "string", + description: "The city to check for the weather condition.", + }, + }, + required: ["location"], + }, + async execute({ location }) { + const res = await fetch("https://weatherapi.example/?location=" + location); + // Returns the result as a JSON string. + return JSON.stringify(await res.json()); + }, + } + ] +}); + +const result = await session.prompt("What is the weather in Seattle?"); +``` + +In this example, the `tools` array defines a `getWeather` tool, specifying its name, description, input schema, and `execute` implementation. When the language model determines that a tool call is needed, the user agent invokes the `getWeather` tool's `execute()` function with the provided arguments and returns the result to the model, which can then incorporate it into its response. + +#### Concurrent tool use + +Developers should be aware that the model might call their tool multiple times, concurrently. For example, code such as + +```js +const result = await session.prompt("Which of these locations currently has the highest temperature? Seattle, Tokyo, Berlin"); +``` + +might call the above `"getWeather"` tool's `execute()` function three times. The model would wait for all tool call results to return, using the equivalent of `Promise.all()` internally, before it composes its final response. + +Similarly, the model might call multiple different tools, if it believes they all are relevant when responding to the given prompt. + +If a developer's `execute()` function is not safe against being called multiple times concurrently, e.g., because it accesses some shared resource, then the developer is responsible for writing appropriate code to suspend execution until the resource is available. The following section contains an example of such code. + +#### Tool return values + +The above example shows tools returning a string. (In fact, stringified JSON.) Models which support [multimodal inputs](#multimodal-inputs) might also support interpreting image or audio results from tool calls. + +Just like the `content` option to a `prompt()` call can accept either a string or an array of `{ type, value }` objects, web developer-provided tools can return either a string or such an array. Here's an example: + +```js +let mutex, resolveMutex; + +const session = await LanguageModel.create({ + tools: [ + { + name: "grabKeyframe", + description: "Grab a keyframe from the video we're analyzing at the given time", + inputSchema: { + type: "number", + minimum: 0, + exclusiveMaximum: videoEl.duration + }, + expectedOutputs: { + type: ["image"] + }, + async execute(timestamp) { + if (mutex) { + // Since we're seeking a single video element, guard against concurrent calls. + await mutex; + } + try { + mutex = new Promise(r => resolveMutex = r); + + if (Math.abs(videoEl.currentTime - timestamp) > 0.001) { + videoEl.currentTime = timestamp; + await new Promise(r => videoEl.addEventListener("seeked", r, { once: true })); + } + await new Promise(r => videoEl.requestVideoFrameCallback(r)); + + return [{ type: "image", value: videoEl }]; + } finally { + resolveMutex(); + mutex = null; + } + } + } + ] +}); +``` + +Note how the output types need to be specified in the tool definition, so that session creation can fail early if the model doesn't support processing multimodal tool outputs. If the return value contains non-text components without them being present in the tool specification, then the tool call will fail at prompting time, even if the model could support it. + +Similarly, expected output languages can be provided (via `expectedOutputs: { languages: ["ja"] }`) or similar, to get an early failure if the model doesn't support processing tool outputs in those languages. However, unlike modalities, there is no prompt-time checking of the tool call result's languages. + +The above example shows a single-item array, but just like with prompt inputs, it's allowed to include multiple tool outputs. The same rules are followed as for inputs, e.g., concatenation of adjacent text chunks is simple string concatenation with no space or other characters inserted. + ### Structured output with JSON schema or RegExp constraints To help with programmatic processing of language model responses, the prompt API supports constraining the response with either a JSON schema object or a `RegExp` passed as the `responseConstraint` option: diff --git a/index.bs b/index.bs index 07f31be..4c6746e 100644 --- a/index.bs +++ b/index.bs @@ -82,6 +82,7 @@ callback LanguageModelToolFunction = Promise (any... arguments); dictionary LanguageModelTool { required DOMString name; required DOMString description; + LanguageModelExpected expectedOutputs; // JSON schema for the input parameters. required object inputSchema; // The function to be invoked by user agent on behalf of language model. @@ -135,14 +136,17 @@ typedef ( dictionary LanguageModelMessage { required LanguageModelMessageRole role; - - // The DOMString branch is shorthand for `[{ type: "text", value: providedValue }]` - required (DOMString or sequence) content; - + required LanguageModelMessageContent content; boolean prefix = false; }; -dictionary LanguageModelMessageContent { +typedef ( + sequence + // Shorthand for `[{ type: "text", value: providedValue }]` + or DOMString +) LanguageModelMessageContent; + +dictionary LanguageModelMessageContentChunk { required LanguageModelMessageType type; required LanguageModelMessageValue value; }; @@ -164,7 +168,8 @@ typedef (

This will be incorporated into a proper part of the specification later. For now, we're just writing out this algorithm as a full spec, since it's complicated.

- To validate and canonicalize a prompt given a {{LanguageModelPrompt}} |input|, a [=list=] of {{LanguageModelMessageType}}s |expectedTypes|, and a boolean |isInitial|, perform the following steps. The return value will be a non-empty [=list=] of {{LanguageModelMessage}}s in their "longhand" form. + + To validate and canonicalize a prompt given a {{LanguageModelPrompt}} |input|, a [=list=] of {{LanguageModelMessageType}}s |expectedTypes|, and a boolean |isInitial|, perform the following steps. The return value will be a non-empty [=list=] of {{LanguageModelMessage}}s in their "longhand" form. 1. [=Assert=]: |expectedTypes| [=list/contains=] "{{LanguageModelMessageType/text}}". @@ -173,8 +178,8 @@ typedef ( "{{LanguageModelMessage/role}}" → "{{LanguageModelMessageRole/user}}", "{{LanguageModelMessage/content}}" → « «[ - "{{LanguageModelMessageContent/type}}" → "{{LanguageModelMessageType/text}}", - "{{LanguageModelMessageContent/value}}" → |input| + "{{LanguageModelMessageContentChunk/type}}" → "{{LanguageModelMessageType/text}}", + "{{LanguageModelMessageContentChunk/value}}" → |input| ]» », "{{LanguageModelMessage/prefix}}" → false @@ -193,8 +198,8 @@ typedef ( "{{LanguageModelMessage/role}}" → |message|["{{LanguageModelMessage/role}}"], "{{LanguageModelMessage/content}}" → « «[ - "{{LanguageModelMessageContent/type}}" → "{{LanguageModelMessageType/text}}", - "{{LanguageModelMessageContent/value}}" → |message| + "{{LanguageModelMessageContentChunk/type}}" → "{{LanguageModelMessageType/text}}", + "{{LanguageModelMessageContentChunk/value}}" → |message| ]» », "{{LanguageModelMessage/prefix}}" → |message|["{{LanguageModelMessage/prefix}}"] @@ -218,21 +223,21 @@ typedef ( 1. If |message|["{{LanguageModelMessage/role}}"] is not "{{LanguageModelMessageRole/system}}", then set |seenNonSystemRole| to true. - 1. If |message|["{{LanguageModelMessage/role}}"] is "{{LanguageModelMessageRole/assistant}}" and |content|["{{LanguageModelMessageContent/type}}"] is not "{{LanguageModelMessageType/text}}", then throw a "{{NotSupportedError}}" {{DOMException}}. + 1. If |message|["{{LanguageModelMessage/role}}"] is "{{LanguageModelMessageRole/assistant}}" and |content|["{{LanguageModelMessageContentChunk/type}}"] is not "{{LanguageModelMessageType/text}}", then throw a "{{NotSupportedError}}" {{DOMException}}. - 1. If |content|["{{LanguageModelMessageContent/type}}"] is "{{LanguageModelMessageType/text}}" and |content|["{{LanguageModelMessageContent/value}}"] is not a [=string=], then throw a {{TypeError}}. + 1. If |content|["{{LanguageModelMessageContentChunk/type}}"] is "{{LanguageModelMessageType/text}}" and |content|["{{LanguageModelMessageContentChunk/value}}"] is not a [=string=], then throw a {{TypeError}}. - 1. If |content|["{{LanguageModelMessageContent/type}}"] is "{{LanguageModelMessageType/image}}", then: + 1. If |content|["{{LanguageModelMessageContentChunk/type}}"] is "{{LanguageModelMessageType/image}}", then: 1. If |expectedTypes| does not [=list/contain=] "{{LanguageModelMessageType/image}}", then throw a "{{NotSupportedError}}" {{DOMException}}. - 1. If |content|["{{LanguageModelMessageContent/value}}"] is not an {{ImageBitmapSource}} or {{BufferSource}}, then throw a {{TypeError}}. + 1. If |content|["{{LanguageModelMessageContentChunk/value}}"] is not an {{ImageBitmapSource}} or {{BufferSource}}, then throw a {{TypeError}}. - 1. If |content|["{{LanguageModelMessageContent/type}}"] is "{{LanguageModelMessageType/audio}}", then: + 1. If |content|["{{LanguageModelMessageContentChunk/type}}"] is "{{LanguageModelMessageType/audio}}", then: 1. If |expectedTypes| does not [=list/contain=] "{{LanguageModelMessageType/audio}}", then throw a "{{NotSupportedError}}" {{DOMException}}. - 1. If |content|["{{LanguageModelMessageContent/value}}"] is not an {{AudioBuffer}}, {{BufferSource}}, or {{Blob}}, then throw a {{TypeError}}. + 1. If |content|["{{LanguageModelMessageContentChunk/value}}"] is not an {{AudioBuffer}}, {{BufferSource}}, or {{Blob}}, then throw a {{TypeError}}. 1. Let |contentWithContiguousTextCollapsed| be an empty [=list=] of {{LanguageModelMessageContent}}s. @@ -240,7 +245,7 @@ typedef ( 1. [=list/For each=] |content| of |message|["{{LanguageModelMessage/content}}"]: - 1. If |content|["{{LanguageModelMessageContent/type}}"] is "{{LanguageModelMessageType/text}}": + 1. If |content|["{{LanguageModelMessageContentChunk/type}}"] is "{{LanguageModelMessageType/text}}": 1. If |lastTextContent| is null: @@ -248,9 +253,9 @@ typedef ( 1. Set |lastTextContent| to |content|. - 1. Otherwise, set |lastTextContent|["{{LanguageModelMessageContent/value}}"] to the concatenation of |lastTextContent|["{{LanguageModelMessageContent/value}}"] and |content|["{{LanguageModelMessageContent/value}}"]. + 1. Otherwise, set |lastTextContent|["{{LanguageModelMessageContentChunk/value}}"] to the concatenation of |lastTextContent|["{{LanguageModelMessageContentChunk/value}}"] and |content|["{{LanguageModelMessageContentChunk/value}}"]. -

No space or other character is added. Thus, « «[ "{{LanguageModelMessageContent/type}}" → "{{LanguageModelMessageType/text}}", "`foo`" ]», «[ "{{LanguageModelMessageContent/type}}" → "{{LanguageModelMessageType/text}}", "`bar`" ]» » is canonicalized to « «[ "{{LanguageModelMessageContent/type}}" → "{{LanguageModelMessageType/text}}", "`foobar`" ]».

+

No space or other character is added. Thus, « «[ "{{LanguageModelMessageContentChunk/type}}" → "{{LanguageModelMessageType/text}}", "`foo`" ]», «[ "{{LanguageModelMessageContentChunk/type}}" → "{{LanguageModelMessageType/text}}", "`bar`" ]» » is canonicalized to « «[ "{{LanguageModelMessageContentChunk/type}}" → "{{LanguageModelMessageType/text}}", "`foobar`" ]».

1. Otherwise: