-
Couldn't load subscription status.
- Fork 1.2k
feat: add oci genai service as chat inference provider #3876
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
feat: add oci genai service as chat inference provider #3876
Conversation
e17bf00 to
acd1008
Compare
|
@github-actions run precommit |
|
⏳ Running pre-commit hooks on PR #3876... |
🤖 Applied by @github-actions bot via pre-commit workflow
|
✅ Pre-commit hooks completed successfully! 🔧 Changes have been committed and pushed to the PR branch. |
|
Removing docs additions at request of @raghotham |
|
cc @mattf for a review since this touches the inference system |
|
Any updates here? |
|
|
||
|
|
||
| @json_schema_type | ||
| class OCIConfig(BaseModel): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| class OCIConfig(BaseModel): | |
| class OCIConfig(RemoteInferenceProviderConfig): |
| # log_probs=params.get("log_probs", 0), | ||
| # tool_choice=params.get("tool_choice", {}), # Unsupported | ||
| # tools=params.get("tools", {}), # Unsupported | ||
| # web_search_options=params.get("web_search_options", {}), # Unsupported | ||
| # stop=params.get("stop", []), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why comment out? Are all of them unsupported?
| ) | ||
| return chat_details | ||
|
|
||
| async def chat_completion( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have OpenAIMixin Class that exposes a lot of knobs, cna you look at it and see if you can use it? Instead of using "custom" code to do the completion requests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- does oci provide an openai compatible endpoint?
- please include output of the inference tests against the remote::oci provider
|
I think it would be much preferable if we can work against an OpenAI compatible endpoint. Otherwise, at the very least we need a set of recorded tests against the provider. But before recordings, let's make sure at least the tests pass "live". Here's a command to run (roughly): |
|
@leseb @ashwinb @mattf thanks for reviewing. If it would be strongly preferable for me to use an OpenAI compatible endpoint, I can make those changes. I'll refactor and re-request when this is done. Sorry also, I started the PR a few weeks ago before a conference, and when I got back inference providers had changed significantly, although it seems like for the better. I'll align on the changes and re-request! |
What does this PR do?
Adds OCI GenAI PaaS models for openai chat completion endpoints.
Test Plan
In an OCI tenancy with access to GenAI PaaS, perform the following steps:
modelsendpoint to list models after server is running:/chat/completionsrequest:/modelsendpoint.