server : pad small embedding batches

[ggerganov]

fix #13689

Temporary workaround until batching logic in libllama is improved.

[aviallon]

It works very well. There is only one issue (which may come from me): when chunking the input using HF Tokenizers to ensure we feed at most n_ubatch/n_batch/n_ctx_per_seq tokens to the embedding model, we are always around 1 to 3 tokens too big.
For instance if HF Tokenizers predicts input will be 512 tokens, it might very well be considered to be 513 tokens by llama.cpp… which will make it return an error.

For now, I worked-around this other issue by simply adding a safety margin of 3 tokens.

[ggerganov]

Most likely when you tokenize in HF transformers you don't take into account special tokens such as BOS, EOS, CLS, etc. These are model-specific and are automatically added by the llama-server.

Though it would be nice to track down the root cause - it's also possible that we are doing something wrong.

[aviallon]

@ggerganov I believe you are right on the cause. I'll experiment with that. Thank you very much for your insanely good skills and rapidity.