Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions packages/types/src/provider-settings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,10 @@ const openAiNativeSchema = apiModelIdProviderModelSchema.extend({
// OpenAI Responses API service tier for openai-native provider only.
// UI should only expose this when the selected model supports flex/priority.
openAiNativeServiceTier: serviceTierSchema.optional(),
// Enable stateless mode for OpenAI Responses API (sets store: false)
// When enabled, responses won't be stored for 30 days and can't be referenced
// in future requests using previous_response_id
openAiNativeStatelessMode: z.boolean().optional(),
})

const mistralSchema = apiModelIdProviderModelSchema.extend({
Expand Down
103 changes: 103 additions & 0 deletions src/api/providers/__tests__/openai-native.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -859,6 +859,109 @@ describe("OpenAiNativeHandler", () => {
expect(secondCallBody.previous_response_id).toBe("resp_789")
})

it("should respect openAiNativeStatelessMode configuration", async () => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be helpful to add a test case for when openAiNativeStatelessMode is true AND metadata.store is explicitly set to true? This would verify that the global setting takes precedence over per-request settings as documented.

// Test with stateless mode enabled
const statelessHandler = new OpenAiNativeHandler({
...mockOptions,
openAiNativeStatelessMode: true,
})

// Mock fetch for Responses API
const mockFetch = vitest.fn().mockResolvedValue({
ok: true,
body: new ReadableStream({
start(controller) {
controller.enqueue(
new TextEncoder().encode('data: {"type":"response.text.delta","delta":"Test"}\n\n'),
)
controller.enqueue(new TextEncoder().encode("data: [DONE]\n\n"))
controller.close()
},
}),
})
global.fetch = mockFetch as any

// Mock SDK to fail
mockResponsesCreate.mockRejectedValue(new Error("SDK not available"))

const stream = statelessHandler.createMessage(systemPrompt, messages, { taskId: "test-task" })
const chunks = []
for await (const chunk of stream) {
chunks.push(chunk)
}

// Verify that store is set to false when stateless mode is enabled
const requestBody = JSON.parse(mockFetch.mock.calls[0][1].body)
expect(requestBody.store).toBe(false)
})

it("should use metadata.store when stateless mode is disabled", async () => {
// Test with stateless mode disabled (default)
const handler = new OpenAiNativeHandler(mockOptions)

// Mock fetch for Responses API
const mockFetch = vitest.fn().mockResolvedValue({
ok: true,
body: new ReadableStream({
start(controller) {
controller.enqueue(
new TextEncoder().encode('data: {"type":"response.text.delta","delta":"Test"}\n\n'),
)
controller.enqueue(new TextEncoder().encode("data: [DONE]\n\n"))
controller.close()
},
}),
})
global.fetch = mockFetch as any

// Mock SDK to fail
mockResponsesCreate.mockRejectedValue(new Error("SDK not available"))

// Test with metadata.store = false
const stream = handler.createMessage(systemPrompt, messages, { taskId: "test-task", store: false })
const chunks = []
for await (const chunk of stream) {
chunks.push(chunk)
}

// Verify that store is set to false when metadata.store is false
const requestBody = JSON.parse(mockFetch.mock.calls[0][1].body)
expect(requestBody.store).toBe(false)
})

it("should default to store:true when stateless mode is disabled and metadata.store is not set", async () => {
// Test with stateless mode disabled and no metadata.store
const handler = new OpenAiNativeHandler(mockOptions)

// Mock fetch for Responses API
const mockFetch = vitest.fn().mockResolvedValue({
ok: true,
body: new ReadableStream({
start(controller) {
controller.enqueue(
new TextEncoder().encode('data: {"type":"response.text.delta","delta":"Test"}\n\n'),
)
controller.enqueue(new TextEncoder().encode("data: [DONE]\n\n"))
controller.close()
},
}),
})
global.fetch = mockFetch as any

// Mock SDK to fail
mockResponsesCreate.mockRejectedValue(new Error("SDK not available"))

const stream = handler.createMessage(systemPrompt, messages, { taskId: "test-task" })
const chunks = []
for await (const chunk of stream) {
chunks.push(chunk)
}

// Verify that store defaults to true
const requestBody = JSON.parse(mockFetch.mock.calls[0][1].body)
expect(requestBody.store).toBe(true)
})

it("should retry with full conversation when previous_response_id fails", async () => {
// This test verifies the fix for context loss bug when previous_response_id becomes invalid
const mockFetch = vitest
Expand Down
6 changes: 4 additions & 2 deletions src/api/providers/openai-native.ts
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,8 @@ export class OpenAiNativeHandler extends BaseProvider implements SingleCompletio
model: model.id,
input: formattedInput,
stream: true,
store: metadata?.store !== false, // Default to true unless explicitly set to false
// Check if stateless mode is enabled in configuration, otherwise use metadata.store
store: this.options.openAiNativeStatelessMode ? false : metadata?.store !== false,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider extracting the store property logic into a helper method like getStoreValue(metadata) since this same logic appears in both line 259 and line 1291. This would improve maintainability and ensure consistency.

// Always include instructions (system prompt) for Responses API.
// Unlike Chat Completions, system/developer roles in input have no special semantics here.
// The official way to set system behavior is the top-level `instructions` field.
Expand Down Expand Up @@ -1286,7 +1287,8 @@ export class OpenAiNativeHandler extends BaseProvider implements SingleCompletio
},
],
stream: false, // Non-streaming for completePrompt
store: false, // Don't store prompt completions
// Use stateless mode if configured, otherwise don't store prompt completions
store: this.options.openAiNativeStatelessMode ? false : false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In completePrompt, the code sets the store field as:

store: this.options.openAiNativeStatelessMode ? false : false

This always yields false. If prompt completions are meant to be non-stored regardless of configuration, consider simplifying to a literal false and adding a comment to explain the rationale.

Suggested change
store: this.options.openAiNativeStatelessMode ? false : false,
\t\t\t\tstore: false, // prompt completions are never stored (see rationale above)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this intentional? The expression this.options.openAiNativeStatelessMode ? false : false always evaluates to false. Could we simplify this to just store: false since prompt completions shouldn't be stored regardless of the stateless mode setting?

}

// Include service tier if selected and supported
Expand Down
Loading