Error when using `/rerank` endpoint with `mixedbread-ai/mxbai-rerank-large-v2`

[jaehyeongAN]

Model description

Hi there,

I'm currently testing the recently released mixedbread-ai/mxbai-rerank-large-v2 model using the Infinity Docker container. The model loads successfully, but I'm encountering an error when making requests to the /rerank endpoint.

Docker Commands

port=8001
model=mixedbread-ai/mxbai-rerank-large-v2
volume=$PWD/data

docker run -d --gpus all \
 -v $volume:/app/.cache \
 -p $port:$port \
 michaelf34/infinity:latest \
 v2 \
 --model-id $model \
 --port $port

Here's an example of the request and response:

Request:

curl -X 'POST' \
  'http://34.60.150.157:8001/rerank' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "Where is Munich?",
  "documents": ["Munich is in Germany.", "The sky is blue."],
  "return_documents": false,
  "raw_scores": false,
  "model": "mixedbread-ai/mxbai-rerank-large-v2",
  "top_n": 1
}'

Response:

{
  "error": {
    "message": "ModelNotDeployedError: model=`mixedbread-ai/mxbai-rerank-large-v2` does not support `rerank`. Reason: the loaded moded cannot fullyfill `rerank`. Options are {'embed'}.",
    "type": null,
    "param": null,
    "code": 400
  }
}

The error message indicates that the mixedbread-ai/mxbai-rerank-large-v2 model is not currently supported for the rerank operation and only supports embed.

Is it possible to request support for the rerank functionality with this model in a future update? Thank you!

Open source status & huggingface transformers.

• The model implementation is available on transformers
• The model weights are available on huggingface-hub
• I verified that the model is currently not running in the latest version pip install infinity_emb[all] --upgrade
• I made the authors of the model aware that I want to use it with infinity_emb & check if they are aware of the issue.

[wirthual]

Possible duplicate: #552 --> Discussion here: #558