Skip to content

feat: add stateless mode configuration for OpenAI Native provider #7791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

feat: add stateless mode configuration for OpenAI Native provider #7791

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions packages/types/src/provider-settings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,9 @@ const openAiNativeSchema = apiModelIdProviderModelSchema.extend({
// OpenAI Responses API service tier for openai-native provider only.
// UI should only expose this when the selected model supports flex/priority.
openAiNativeServiceTier: serviceTierSchema.optional(),
// When true, forces the OpenAI Responses API to run in stateless mode (store: false)
// This prevents responses from being stored for 30 days in OpenAI's Responses API
openAiNativeStatelessMode: z.boolean().optional(),
Copy link
Contributor Author

@roomote roomote bot Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good addition to the schema! However, I notice there's no corresponding UI component to expose this setting to users. Would it be helpful to add a toggle in the provider settings UI so users don't have to manually edit configuration files?

Also, consider adding more detailed documentation about when users should enable this mode (e.g., for privacy-sensitive applications, compliance requirements, etc.).

})

const mistralSchema = apiModelIdProviderModelSchema.extend({
Expand Down
182 changes: 182 additions & 0 deletions src/api/providers/__tests__/openai-native.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1732,5 +1732,187 @@ describe("GPT-5 streaming event coverage (additional)", () => {
expect(bodyStr).not.toContain('"verbosity"')
})
})

describe("Stateless mode configuration", () => {
Copy link
Contributor Author

@roomote roomote bot Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great test coverage! The 6 test cases cover the main scenarios well. Consider adding a few edge case tests:

  • What happens if openAiNativeStatelessMode is set to a non-boolean value?
  • How does this interact with other providers that might accidentally have this property set?
  • Test the behavior when switching between stateless and stateful modes during runtime.

it("should use stateless mode when openAiNativeStatelessMode is true", async () => {
const mockFetch = vitest.fn().mockResolvedValue({
ok: true,
body: new ReadableStream({
start(controller) {
controller.enqueue(
new TextEncoder().encode('data: {"type":"response.done","response":{}}\n\n'),
)
controller.enqueue(new TextEncoder().encode("data: [DONE]\n\n"))
controller.close()
},
}),
})
;(global as any).fetch = mockFetch as any

// Force SDK path to fail so we use fetch fallback
mockResponsesCreate.mockRejectedValue(new Error("SDK not available"))

const handler = new OpenAiNativeHandler({
apiModelId: "gpt-5-2025-08-07",
openAiNativeApiKey: "test-api-key",
openAiNativeStatelessMode: true, // Enable stateless mode
})

const systemPrompt = "You are a helpful assistant."
const messages: Anthropic.Messages.MessageParam[] = [{ role: "user", content: "Hello!" }]
const stream = handler.createMessage(systemPrompt, messages)

for await (const _ of stream) {
// drain
}

const bodyStr = (mockFetch.mock.calls[0][1] as any).body as string
const parsedBody = JSON.parse(bodyStr)
expect(parsedBody.store).toBe(false) // Should be false when stateless mode is enabled
})

it("should default to store: true when openAiNativeStatelessMode is false", async () => {
const mockFetch = vitest.fn().mockResolvedValue({
ok: true,
body: new ReadableStream({
start(controller) {
controller.enqueue(
new TextEncoder().encode('data: {"type":"response.done","response":{}}\n\n'),
)
controller.enqueue(new TextEncoder().encode("data: [DONE]\n\n"))
controller.close()
},
}),
})
;(global as any).fetch = mockFetch as any

// Force SDK path to fail so we use fetch fallback
mockResponsesCreate.mockRejectedValue(new Error("SDK not available"))

const handler = new OpenAiNativeHandler({
apiModelId: "gpt-5-2025-08-07",
openAiNativeApiKey: "test-api-key",
openAiNativeStatelessMode: false, // Explicitly disable stateless mode
})

const systemPrompt = "You are a helpful assistant."
const messages: Anthropic.Messages.MessageParam[] = [{ role: "user", content: "Hello!" }]
const stream = handler.createMessage(systemPrompt, messages)

for await (const _ of stream) {
// drain
}

const bodyStr = (mockFetch.mock.calls[0][1] as any).body as string
const parsedBody = JSON.parse(bodyStr)
expect(parsedBody.store).toBe(true) // Should be true when stateless mode is disabled
})

it("should default to store: true when openAiNativeStatelessMode is not set", async () => {
const mockFetch = vitest.fn().mockResolvedValue({
ok: true,
body: new ReadableStream({
start(controller) {
controller.enqueue(
new TextEncoder().encode('data: {"type":"response.done","response":{}}\n\n'),
)
controller.enqueue(new TextEncoder().encode("data: [DONE]\n\n"))
controller.close()
},
}),
})
;(global as any).fetch = mockFetch as any

// Force SDK path to fail so we use fetch fallback
mockResponsesCreate.mockRejectedValue(new Error("SDK not available"))

const handler = new OpenAiNativeHandler({
apiModelId: "gpt-5-2025-08-07",
openAiNativeApiKey: "test-api-key",
// openAiNativeStatelessMode not set
})

const systemPrompt = "You are a helpful assistant."
const messages: Anthropic.Messages.MessageParam[] = [{ role: "user", content: "Hello!" }]
const stream = handler.createMessage(systemPrompt, messages)

for await (const _ of stream) {
// drain
}

const bodyStr = (mockFetch.mock.calls[0][1] as any).body as string
const parsedBody = JSON.parse(bodyStr)
expect(parsedBody.store).toBe(true) // Should default to true
})

it("should override metadata.store when openAiNativeStatelessMode is true", async () => {
const mockFetch = vitest.fn().mockResolvedValue({
ok: true,
body: new ReadableStream({
start(controller) {
controller.enqueue(
new TextEncoder().encode('data: {"type":"response.done","response":{}}\n\n'),
)
controller.enqueue(new TextEncoder().encode("data: [DONE]\n\n"))
controller.close()
},
}),
})
;(global as any).fetch = mockFetch as any

// Force SDK path to fail so we use fetch fallback
mockResponsesCreate.mockRejectedValue(new Error("SDK not available"))

const handler = new OpenAiNativeHandler({
apiModelId: "gpt-5-2025-08-07",
openAiNativeApiKey: "test-api-key",
openAiNativeStatelessMode: true, // Enable stateless mode
})

const systemPrompt = "You are a helpful assistant."
const messages: Anthropic.Messages.MessageParam[] = [{ role: "user", content: "Hello!" }]
// Even if metadata.store is true, stateless mode should override it
const stream = handler.createMessage(systemPrompt, messages, { taskId: "test", store: true })

for await (const _ of stream) {
// drain
}

const bodyStr = (mockFetch.mock.calls[0][1] as any).body as string
const parsedBody = JSON.parse(bodyStr)
expect(parsedBody.store).toBe(false) // Should be false even when metadata.store is true
})

it("should use stateless mode in completePrompt when openAiNativeStatelessMode is true", async () => {
// Mock the responses.create method
mockResponsesCreate.mockResolvedValue({
output: [
{
type: "message",
content: [
{
type: "output_text",
text: "Test response",
},
],
},
],
})

const handler = new OpenAiNativeHandler({
apiModelId: "gpt-5-2025-08-07",
openAiNativeApiKey: "test-api-key",
openAiNativeStatelessMode: true, // Enable stateless mode
})

await handler.completePrompt("Test prompt")

expect(mockResponsesCreate).toHaveBeenCalledWith(
expect.objectContaining({
store: false, // Should always be false in completePrompt with stateless mode
}),
)
})
})
})
})
6 changes: 4 additions & 2 deletions src/api/providers/openai-native.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,8 @@ export class OpenAiNativeHandler extends BaseProvider implements SingleCompletio
model: model.id,
input: formattedInput,
stream: true,
store: metadata?.store !== false, // Default to true unless explicitly set to false
// Use stateless mode if configured, otherwise respect metadata.store (default true)
store: this.options.openAiNativeStatelessMode ? false : metadata?.store !== false,
Copy link
Contributor Author

@roomote roomote bot Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation correctly overrides the metadata.store value when stateless mode is enabled. However, have you considered making this controllable per-request through metadata? Something like:

This would give users more granular control when needed.

// Always include instructions (system prompt) for Responses API.
// Unlike Chat Completions, system/developer roles in input have no special semantics here.
// The official way to set system behavior is the top-level `instructions` field.
Expand Down Expand Up @@ -1286,7 +1287,8 @@ export class OpenAiNativeHandler extends BaseProvider implements SingleCompletio
},
],
stream: false, // Non-streaming for completePrompt
store: false, // Don't store prompt completions
// Use stateless mode if configured, otherwise don't store prompt completions
store: this.options.openAiNativeStatelessMode ? false : false,
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ternary operator for the 'store' field is redundant here—both branches yield false. If prompt completions should never be stored (even when stateless mode is disabled), consider simply using 'store: false' with an explanatory comment. Otherwise, adjust the logic to allow storing when stateless mode is off.

Copy link
Contributor Author

@roomote roomote bot Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears to be redundant logic. The expression this.options.openAiNativeStatelessMode ? false : false always evaluates to false regardless of the configuration value.

Since completePrompt is meant for non-conversational completions, it makes sense to always use stateless mode (store: false) here.

}

// Include service tier if selected and supported
Expand Down
Loading