-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Add 'Use custom base URL' option to Vertex AI provider #4815
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add 'Use custom base URL' option to Vertex AI provider #4815
Conversation
daniel-lxs
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @jferrell-sayari, thanks for the PR!
I left a couple of questions and suggestions worth taking a look at.
Curious to hear your thoughts!
| const geminiSchema = apiModelIdProviderModelSchema.extend({ | ||
| geminiApiKey: z.string().optional(), | ||
| googleGeminiBaseUrl: z.string().optional(), | ||
| isVertex: z.boolean().optional(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm curious about adding isVertex to the geminiSchema here. From what I can see, this appears to be an internal implementation detail that's used to differentiate behavior when VertexHandler extends GeminiHandler.
Would this show up in the Gemini provider settings UI? It seems like users choose either Gemini OR Vertex as their provider, not both, so having an isVertex toggle in Gemini settings might be confusing.
Could we keep this as an internal flag that's only passed by the VertexHandler constructor instead of exposing it in the schema?
| "lmStudioUrl": "Default: http://localhost:1234", | ||
| "geminiUrl": "Default: https://generativelanguage.googleapis.com" | ||
| "geminiUrl": "Default: https://generativelanguage.googleapis.com", | ||
| "vertexUrl": "Default: http://localhost:1234" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I noticed the default URL for Vertex is set to http://localhost:1234. Is this intentional?
It seems like this might be a copy from the LM Studio default. For Vertex AI, wouldn't users typically use Google Cloud endpoints like https://{region}-aiplatform.googleapis.com or a custom proxy URL?
Maybe we could use a more representative placeholder that helps users understand what kind of URL to enter?
| const config: GenerateContentConfig = { | ||
| systemInstruction, | ||
| httpOptions: this.options.googleGeminiBaseUrl ? { baseUrl: this.options.googleGeminiBaseUrl } : undefined, | ||
| httpOptions: this.options.isVertex |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a bit unsure about the logic here. We're checking isVertex in GeminiHandler to choose between vertexBaseUrl and googleGeminiBaseUrl.
This works because VertexHandler extends GeminiHandler and sets isVertex: true, but it feels like a layering violation - GeminiHandler is now aware of Vertex-specific config.
It might be cleaner to let the Vertex provider handle its own configuration and keep the Gemini provider focused only on its own settings.
Related GitHub Issue
Closes: #3114
Description
This PR adds a "Use custom base URL" option to the Vertex AI provider just like the Google Gemini provider has. This will allow us to run requests to Vertex AI through a proxy. @shwuhk mentioned needing this feature in the linked issue as well to help solve an issue they're experiencing.
Test Procedure
Unit tests were added and manual testing was performed for:
Type of Change
srcor test files.Pre-Submission Checklist
npm run lint).console.log) has been removed.npm test).mainbranch.npm run changesetif this PR includes user-facing changes or dependency updates.Screenshots / Videos
Documentation Updates
The documentation should be updated to mention this capability. However, I noticed the documentation does not mention that the Google Gemini provider has this capability.
Additional Notes
Get in Touch
jferrell_sayari
Important
Adds custom base URL option for Vertex AI provider, updates handlers and UI, and includes tests and i18n updates.
vertexBaseUrloption to Vertex AI provider inprovider-settings.ts.VertexHandlerinvertex.tsto usevertexBaseUrlfor HTTP requests.Vertex.tsxto include checkbox and input for custom base URL.vertex.spec.tsto verifyvertexBaseUrlusage increateMessageandcompletePrompt.vertexUrl.This description was created by
for 5d774da. You can customize this summary. It will automatically update as commits are pushed.