rerank endpoint not implemented

[devMls]

Hello

I have problems for access to rerank :

11:15AM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:jina-reranker-v2-base-multilingual ContextSize:512 Seed:1345567851 NBatch:512 F16Memory:false MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:2 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/espeak-ng-data RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/build/models/jina-reranker-v2-base-multilingual Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 LoadFormat: MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false ModelPath:/build/models LoraAdapters:[] LoraScales:[]}

11:15AM INF [stablediffusion] Loads OK

**11:15AM ERR Server error error="rpc error: code = Unimplemented desc = method Rerank not implemented" ip=10.0.1.19 latency=14.148126589s method=POST status=500 url=/v1/rerank**

11:16AM INF Success ip=127.0.0.1 latency="411.985µs" method=GET status=200 url=/readyz

11:17AM INF Success ip=127.0.0.1 latency="59.579µs" method=GET status=200 url=/readyz

11:18AM DBG guessDefaultsFromFile: modelPath is empty

11:18AM DBG Request for model: jina-reranker-v2-base-multilingual

11:18AM DBG Model already loaded in memory: jina-reranker-v2-base-multilingual

11:18AM DBG Checking model availability (jina-reranker-v2-base-multilingual)

11:18AM DBG Model 'jina-reranker-v2-base-multilingual' already loaded

my docker swarm config:

services:
  reranker:
    hostname: rerank_ai_server
    image: localai/localai:v2.23.0
    command: run https://gist.github.com/devMls/3e5fb2ae3fd573b29cecc0eb2a7ec822 #file://build/rerank_config.yml
    networks:
      - ragflow
      - bridge
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      interval: 1m
      timeout: 20m
      retries: 5
    environment:
      - DEBUG=true
      - HUGGING_FACE_HUB_TOKEN=secret
    configs:
      - source: rerank_config.yml
        target: /build/rerank_config.yml  
    volumes:
      - localai_rerank:/build/models:cached
networks:
  ragflow:
    external: true
  bridge:
    name: bridge
    external: true    

volumes:
  localai_rerank:
    driver: local

configs:
  rerank_config.yml:
    external: true 

my model config:

name: jina-reranker-v2-base-multilingual
backend: rerankers
parameters:
  model: huggingface://jinaai/jina-reranker-v2-base-multilingual

any clue?

[devMls]

I realised, that if I include full name of jinai, the error change:

name: jinaai/jina-reranker-v2-base-multilingual
backend: rerankers
parameters:
  model: huggingface://jinaai/jina-reranker-v2-base-multilingual

error:

11:39AM INF [/build/backend/python/sentencetransformers/run.sh] Fails: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh

11:39AM INF [/build/backend/python/vall-e-x/run.sh] Attempting to load

11:39AM INF Loading model 'jinaai/jina-reranker-v2-base-multilingual' with backend /build/backend/python/vall-e-x/run.sh

11:39AM DBG Loading model in memory from file: /build/models/jinaai/jina-reranker-v2-base-multilingual

11:39AM DBG Loading Model jinaai/jina-reranker-v2-base-multilingual with gRPC (file: /build/models/jinaai/jina-reranker-v2-base-multilingual) (backend: /build/backend/python/vall-e-x/run.sh): {backendString:/build/backend/python/vall-e-x/run.sh model:jinaai/jina-reranker-v2-base-multilingual modelID:jinaai/jina-reranker-v2-base-multilingual assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000632788 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh openvoice:/build/backend/python/openvoice/run.sh parler-tts:/build/backend/python/parler-tts/run.sh rerankers:/build/backend/python/rerankers/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}

11:39AM INF [/build/backend/python/vall-e-x/run.sh] Fails: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vall-e-x/run.sh

11:39AM ERR Server error error="could not load model - all backends returned error: [llama-cpp]: failed to load model with internal loader: could not load model: rpc error: code = Canceled desc = \n[llama-ggml]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = failed loading model\n[llama-cpp-fallback]: failed to load model with internal loader: could not load model: rpc error: code = Canceled desc = \n[whisper]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = stat /build/models/jinaai/jina-reranker-v2-base-multilingual: no such file or directory\n[piper]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = unsupported model type /build/models/jinaai/jina-reranker-v2-base-multilingual (should end with .onnx)\n[rwkv]: failed to load model with internal loader: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[stablediffusion]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = stat /build/models/jinaai/jina-reranker-v2-base-multilingual: no such file or directory\n[huggingface]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = no huggingface token provided\n[bert-embeddings]: failed to load model with internal loader: could not load model: rpc error: code = Unknown desc = failed loading model\n[/build/backend/python/exllama2/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama2/run.sh\n[/build/backend/python/transformers-musicgen/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers-musicgen/run.sh\n[/build/backend/python/diffusers/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/diffusers/run.sh\n[/build/backend/python/openvoice/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/openvoice/run.sh\n[/build/backend/python/bark/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/bark/run.sh\n[/build/backend/python/mamba/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/mamba/run.sh\n[/build/backend/python/coqui/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/coqui/run.sh\n[/build/backend/python/vllm/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vllm/run.sh\n[/build/backend/python/autogptq/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/autogptq/run.sh\n[/build/backend/python/rerankers/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/rerankers/run.sh\n[/build/backend/python/sentencetransformers/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh\n[/build/backend/python/transformers/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers/run.sh\n[/build/backend/python/parler-tts/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/parler-tts/run.sh\n[/build/backend/python/sentencetransformers/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh\n[/build/backend/python/vall-e-x/run.sh]: failed to load model with internal loader: backend not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vall-e-x/run.sh" ip=10.0.1.19 latency=20.170736557s method=POST status=500 url=/v1/rerank

[mudler]

@devMls usage of rerankers is documented here: https://localai.io/features/reranker/#usage.

Also share the full logs in debug mode please.