bfloat16 support

[seagrine]

Feature request

Thank you for adding support for embeddinggemma-300m. I am using it on fp32 and it works great. I wonder when we might have bfloat16 support? It is especially important for embeddinggemma-300m where fp16 does not work. I was surprised not to find an existing thread to follow along so feel free to mark dupe or close + redirect.

Motivation

TEI is convenient for serving models but the lack of bfloat16 support is a real problem, especially for models like embeddinggemma-300m which don't support float16. It means we leave a great deal of performance on the table.

Your contribution

I'm not sure how I would contribute but I'm open to suggestions

[alvarobartt]

Hey @seagrine thanks for mentioning, TL;DR we're already working on that as per #809 for Metal devices and soon for CUDA too, thanks to the latest updates on candle. Not sure that's going to make it into the v1.9.0 release but rather on the next one, given that there's already a bunch of features for the coming release + we're still solving a lib linking which is preventing the candle update from 0.8.4 (current) to 0.9.2 (latest as of today).