feat: OpenAI-compatible API Endpoints for Embedding Models #104
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
yinggeh
force-pushed
the
yinggeh/tri-49-request-for-openai-compatible-api-endpoints-for-triton
branch
from
0acb76f to
7d06043
Compare
yinggeh
force-pushed
the
yinggeh/tri-49-request-for-openai-compatible-api-endpoints-for-triton
branch
from
7d06043 to
2c3e148
Compare
…end into yinggeh/tri-49-request-for-openai-compatible-api-endpoints-for-triton
src/utils/request.py
Outdated
| pooling_params = PoolingParams(dimensions=dims, task="embed") | ||
| return pooling_params | ||
|
|
||
| def create_response(self, request_output): |
There was a problem hiding this comment.
would be nice to have a type hint on request_output.
src/utils/request.py
Outdated
| async for response in response_iterator: | ||
| yield response | ||
|
|
||
| def create_response(self, request_output_state, request_output, prepend_input): |
There was a problem hiding this comment.
would be nice to have type hints on request_output_state, request_output, and prepend_input.
src/utils/request.py
Outdated
|
|
||
|
|
||
| class RequestBase: | ||
| def __init__(self, request, executor_callback, output_dtype): |
There was a problem hiding this comment.
would be nice to have type hints on request, executor_callback, and output_dtype.
| from abc import abstractmethod | ||
| from io import BytesIO | ||
|
|
||
| import numpy as np |
There was a problem hiding this comment.
is using numpy (CPU) good enough?
do we want to leverage cupy (GPU)?
There was a problem hiding this comment.
My understanding is that vLLM engine takes care of it.
| "optional": True, | ||
| }, | ||
| # Tentative input reserved for embedding requests in OpenAI-compatible frontend. Subject to change in the future. | ||
| # WARN: Triton client should never set this input. It is reserved for embedding requests in OpenAI-compatible frontend. |
There was a problem hiding this comment.
Why limit support to only the OpenAI frontend? Maybe we should also allow deploying embedding models using only the vLLM backend?
There was a problem hiding this comment.
That's a separate issue. The input prompt format of chat/completion is different from embeddings. We need two different sets of configuration inputs for generate and embed.
…d into yinggeh/tri-49-request-for-openai-compatible-api-endpoints-for-triton
Enable vLLM to load embedding model and execute embedding requests