Skip to content

Extend support for useFileOutput to stream #309

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 25, 2024
Merged

Extend support for useFileOutput to stream #309

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
254b7df
Break stream on done event without enqueuing it
mattt Sep 23, 2024
4430e6b
Fix doc comment for fetch parameter
mattt Sep 23, 2024
139fb89
Transform URLs in streaming responses when useFileOutput is enabled
mattt Sep 23, 2024
6a2156d
Revert 'Break stream on done event without enqueuing it'
mattt Sep 23, 2024
51d300b
Default to useFileOutput = true for readable streams
mattt Sep 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions lib/stream.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,18 @@ class ServerSentEvent {
* @param {string} config.url The URL to connect to.
* @param {typeof fetch} [config.fetch] The URL to connect to.
* @param {object} [config.options] The EventSource options.
* @param {boolean} [config.options.useFileOutput] Whether to use the file output stream.
* @returns {ReadableStream<ServerSentEvent> & AsyncIterable<ServerSentEvent>}
*/
function createReadableStream({ url, fetch, options = {} }) {
const { useFileOutput = false, headers = {}, ...initOptions } = options;

return new ReadableStream({
async start(controller) {
const init = {
...options,
...initOptions,
headers: {
...options.headers,
...headers,
Accept: "text/event-stream",
},
};
Expand Down Expand Up @@ -84,9 +87,15 @@ function createReadableStream({ url, fetch, options = {} }) {
break;
}

controller.enqueue(
new ServerSentEvent(event.event, event.data, event.id)
);
let data = event.data;
if (
useFileOutput &&
typeof data === "string" &&
(data.startsWith("https:") || data.startsWith("data:"))
Copy link
Contributor

@aron aron Sep 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want to explicitly match that data consists of only a valid data uri. There is a reasonable chance that a language model might emit a data: line that starts with "data:" but less so that it will emit a line that consists only of a well formed data-uri.

🤔 That said, I did think today after reading this post on other AI apis that we should move to structured outputs.

Perhaps the file stream should emit JSON.

data: {"type": "url", value: "data://..."}

And in future we refactor the text streaming interface to do the same:

data: {"type": "string", "data: and some more text"}

Copy link
Contributor Author

@mattt mattt Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want to explicitly match that data consists of only a valid data uri. There is a reasonable chance that a language model might emit a data: line that starts with "data:" but less so that it will emit a line that consists only of a well formed data-uri.

That's a good callout. The trick is finding a good way to validate without parsing the whole thing and throwing away the result. I think we can still read lazily if we use a regex to apply some heuristics about its first chunk of content.

🤔 That said, I did think today after reading this post on other AI apis that we should move to structured outputs.

Perhaps the file stream should emit JSON.

data: {"type": "url", value: "data://..."}

And in future we refactor the text streaming interface to do the same:

data: {"type": "string", "data: and some more text"}

I see the advantages of typed outputs, but also quite like the experience we have now of emitting raw tokens. In any case, structured outputs would be a backwards-incompatible change, so we'd have to be clever about a migration.

One way we could support both would be keeping Accept: text/event-stream as-is, but adding support for Accept: text/event-stream+json. The client libraries could start sending that as needed to opt into structured outputs.

) {
data = createFileOutput({ data, fetch });
}
controller.enqueue(new ServerSentEvent(event.event, data, event.id));

if (event.event === "done") {
break;
Expand All @@ -104,7 +113,7 @@ function createReadableStream({ url, fetch, options = {} }) {
*
* @param {object} config
* @param {string} config.url The URL to connect to.
* @param {typeof fetch} [config.fetch] The URL to connect to.
* @param {typeof fetch} [config.fetch] The fetch function.
* @returns {ReadableStream<Uint8Array>}
*/
function createFileOutput({ url, fetch }) {
Expand Down
Loading