How to find available models to use?

[InAnYan]

Hello! I've been using Deep Java Library for embedding generation.

How can I view which models are supported? Is there an online list?

In the Criteria class I saw optModelUrls, but how these URLs are retrieved?

[frankfliu]

1. You can clone djl repository and use command line to list models, see: https://docs.djl.ai/master/docs/load_model.html#list-available-models-using-djl-command-line
2. If your model is not in our built-in model zoo, you can convert huggingface model using our convert tool: https://docs.djl.ai/master/extensions/tokenizers/index.html#use-command-line
3. If you believe this model is a popular model, you can create an issue, and can import it into our model zoo.

[InAnYan]

Thank you for the answer!

Though, I believe an online list of models in model zoo would be a convenient solution

[InAnYan]

@frankfliu Could you also advice how can I run a model from files? For example, I want to test models from https://huggingface.co/spaces/mteb/leaderboard, some of them are not in Model Zoo, and so I want to download them locally and then load with djl. How can I do that? Should I download all files or only .safetensors? I've tried to look at the documentation, but got a bit lost. Are there any differences for embedding models?

[frankfliu]

@InAnYan
Not all the models in the list can be loaded in DJL with Java engine (as of now). A couple of reason:

1. Some model is too big to fit in single GPU, you have to use Python engine for multi-GPU inference
2. Some of the model cannot be traced/converted to torchscript or onnx model
3. Our current Rust engine only support bert or distilbert architecture

You can try to use the djl-converter to convert the model:

pip install "git+https://github.com/deepjavalibrary/djl.git@9c21f6e8a53e9d12779e33afaa62db942bd45376#subdirectory=extensions/tokenizers/src/main/python"
djl-convert -m HF_MODEL_ID --output-format OnnxRuntime --device cuda --optimize O4 --trust-remote-code