text-embeddings-inference 1.9.0

[BrewTestBot]

Created by brew bump

---

Created with brew bump-formula-pr.

Details

release notes

What's changed?
🚨 Breaking changes

Default HiddenAct::Gelu to GeLU + tanh in favour of GeLU erf  by @vrdn-23 in Default HiddenAct::Gelu to GeLU + tanh in favour of GeLU erf  huggingface/text-embeddings-inference#753


Default GeLU implementation is now GeLU + tanh approximation instead of exact GeLU (aka. GeLU erf) to make sure that the CPU and CUDA embeddings are the same (as cuBLASlt only supports GeLU + tanh), which represents a slight misalignment from how Transformers handles it, as when hidden_act="gelu" is set in config.json, GeLU erf should be used. The numerical differences between GeLU + tanh and GeLU erf should have negligible impact on inference quality.


Set --auto-truncate to true by default by @alvarobartt in Set --auto-truncate to true by default huggingface/text-embeddings-inference#829


--auto-truncate now defaults to true, meaning that the sequences will be truncated to the lower value between the --max-batch-tokens or the maximum model length, to prevent the --max-batch-tokens from being lower than the actual maximum supported length.

🎉 Additions

Add --served-model-name for OpenAI requests via HTTP by @vrdn-23 in Add --served-model-name for OpenAI requests via HTTP huggingface/text-embeddings-inference#685
Extend download_onnx to download sharded ONNX by @alvarobartt in Extend download_onnx to download sharded ONNX huggingface/text-embeddings-inference#817
Add support for llama 2 by @michaelfeil in Add support for llama 2 huggingface/text-embeddings-inference#802
Add support for blackwell architecture (sm100, sm120) by @danielealbano in Add support for blackwell architecture (sm100, sm120) huggingface/text-embeddings-inference#735
Mf/add-support-for-llama-3-and-nemotron by @michaelfeil in Mf/add-support-for-llama-3-and-nemotron huggingface/text-embeddings-inference#805
Add support for DebertaV2 by @vrdn-23 in Add support for DebertaV2 huggingface/text-embeddings-inference#746
Add bidirectional attention and projection layer support for Qwen3-based models by @williambarberjr in Add bidirectional attention and projection layer support for Qwen3-based models huggingface/text-embeddings-inference#808

🐛 Fixes

Fix reading non-standard config for past_key_values in ONNX by @alvarobartt in Fix reading non-standard config for past_key_values in ONNX huggingface/text-embeddings-inference#751
Fix TruncationDirection to deserialize from lowercase and capitalized by @alvarobartt in Fix TruncationDirection to deserialize from lowercase and capitalized huggingface/text-embeddings-inference#755
Fix sagemaker-entrypoint* & remove SageMaker and Vertex from Dockerfile* by @alvarobartt in Fix sagemaker-entrypoint* & remove SageMaker and Vertex from Dockerfile* huggingface/text-embeddings-inference#699
Bug: Critical accuracy bugs for model_type=qwen2: no causal attention and wrong tokenizer by @michaelfeil in Bug: Critical accuracy bugs for model_type=qwen2: no causal attention and wrong tokenizer huggingface/text-embeddings-inference#762
Fix config.json reading w/ aliases for ORT by @alvarobartt in Fix config.json reading w/ aliases for ORT huggingface/text-embeddings-inference#786
Fix HTTP error code for validation by @vrdn-23 in Fix HTTP error code for validation huggingface/text-embeddings-inference#818
Fix to acquire the permit in a blocking way by @kozistr in Fix to acquire the permit in a blocking way huggingface/text-embeddings-inference#726
Read Hugging Face Hub token from cache if not provided by @alvarobartt in Read Hugging Face Hub token from cache if not provided huggingface/text-embeddings-inference#814
Align the normalize param between the gRPC and HTTP /embed interfaces by @kozistr in Align the normalize param between the gRPC and HTTP /embed interfaces huggingface/text-embeddings-inference#810

⚡ Improvements

Serialization in tokio thread instead of blocking thread, 50% reduction in latency for small models by @michaelfeil in Serialization in tokio thread instead of blocking thread, 50% reduction in latency for small models huggingface/text-embeddings-inference#767
Remove default --model-id argument by @alvarobartt in Remove default --model-id argument huggingface/text-embeddings-inference#679
feat: better Tokenization # workers heuristic by @michaelfeil in feat: better Tokenization # workers heuristic huggingface/text-embeddings-inference#766
add faster index select kernel by @michaelfeil in add faster index select kernel huggingface/text-embeddings-inference#773
feat: speedup Parallel safetensors download by @michaelfeil in feat: speedup Parallel safetensors download huggingface/text-embeddings-inference#765
feat: startup time: add cloned tokenzier fix, saves ~1-20s cold start time by @michaelfeil in feat: startup time: add cloned tokenzier fix, saves ~1-20s cold start time huggingface/text-embeddings-inference#772
Adjust the warmup phase for CPU by @kozistr in Adjust the warmup phase for CPU huggingface/text-embeddings-inference#792

📄 Other

Skip Gemma3 tests when HF_TOKEN not set by @alvarobartt in Skip Gemma3 tests when HF_TOKEN not set huggingface/text-embeddings-inference#812
Bump Rust 1.92, CUDA 12.6, Ubuntu 24.04 and add Dockerfile-cuda-blackwell-all by @alvarobartt in Bump Rust 1.92, CUDA 12.6, Ubuntu 24.04 and add Dockerfile-cuda-blackwell-all huggingface/text-embeddings-inference#823
Update rustc version to 1.92.0 by @alvarobartt in Update rustc version to 1.92.0 huggingface/text-embeddings-inference#826
Add use_flash_attn for better FA + FA2 feature gating by @alvarobartt in Add use_flash_attn for better FA + FA2 feature gating huggingface/text-embeddings-inference#825
Update CUDA to 12.9 w/ cuda-compat-12-9 by @alvarobartt in Update CUDA to 12.9 w/ cuda-compat-12-9 huggingface/text-embeddings-inference#828
Upgrade GitHub Actions for Node 24 compatibility by @salmanmkc in Upgrade GitHub Actions for Node 24 compatibility huggingface/text-embeddings-inference#782
Lint: cargo fmt and clippy fix warnings by @michaelfeil in Lint: cargo fmt and clippy fix warnings huggingface/text-embeddings-inference#776
Fix rustfmt on backend/candle/tests/*.rs files by @alvarobartt in Fix rustfmt on backend/candle/tests/*.rs files huggingface/text-embeddings-inference#800
Upgrade GitHub Actions to latest versions by @salmanmkc in Upgrade GitHub Actions to latest versions huggingface/text-embeddings-inference#783
Update version to 1.9.0 by @alvarobartt in Update version to 1.9.0 huggingface/text-embeddings-inference#830

🆕 New Contributors

@salmanmkc made their first contribution in Upgrade GitHub Actions for Node 24 compatibility huggingface/text-embeddings-inference#782
@danielealbano made their first contribution in Add support for blackwell architecture (sm100, sm120) huggingface/text-embeddings-inference#735
@williambarberjr made their first contribution in Add bidirectional attention and projection layer support for Qwen3-based models huggingface/text-embeddings-inference#808

Full Changelog: huggingface/text-embeddings-inference@v1.8.3...v1.9.0

View the full release notes at https://github.com/huggingface/text-embeddings-inference/releases/tag/v1.9.0.

---

[github-actions]

🤖 An automated task has requested bottles to be published to this PR.

Caution

Please do not push to this PR branch before the bottle commits have been pushed, as this results in a state that is difficult to recover from. If you need to resolve a merge conflict, please use a merge commit. Do not force-push to this PR branch.

[github-actions]

⚠️ Bottle publish failed.

[github-actions]

🤖 An automated task has requested bottles to be published to this PR.

Caution

Please do not push to this PR branch before the bottle commits have been pushed, as this results in a state that is difficult to recover from. If you need to resolve a merge conflict, please use a merge commit. Do not force-push to this PR branch.

[github-actions]

⚠️ Bottle publish failed.