Is it possible to use safetensors files?

[FrankDaze]

Hi,

there are some models that I like to use like falcon-40b, but it's only available as safetensors file. Is it possible to use these models as well? Or to convert them to a supported file format?

Thanks in advance
Frank

[thiner]

LocalAI supports vllm as backend, and vllm supports HF models(safetensors format) seamlessly. That said, LocalAI supports safetensors. You can find vllm compatible models here: https://github.com/vllm-project/vllm

[BradKML]

What about LiteLLM+Ollama, is that possible as well? And how would you set that up on a Docker instance?