feature: support batched embeddings

[planetf1]

We found that loading docs into maestro knowledge (milvus backend) took around 20-25s locally (custom_embed : ollama, nomic-text-embed) when we issued a single create_document with 7 documents (resulting in 420 chunks).

This is ok, though for reporting and to avoid very long api calls, it will be broken down to per-document (a few seconds each).

HOWEVER when testing the same code against a 'remote' model server in cloud (same model), we found calls to the embedding model took highly variable times - from fairly quick to 9s seconds:

This resulted in the overall processing time taking around 14 minutes+ even for a small document.

The big factor is suspected to be latency on these many small requests. Therefore we should aim to batch embedding generation. Specifically for the milvus, custom embedding case

Additionally making the APIs async more generally would be beneficial to this scenario, though an initial change for batched embedding would be a start. See also #48

[planetf1]

After some investigation and prototyping:

• The solution environment we were working in has a nomic-embed-text service which takes 2048 tokens max
• Exceeding this limit results in failure, so limits need to be carefully set, reported on, and one could think of automatic tuning/fallback.
• Current chunking limits (for fixed) are expressed in bytes - for better matching, some chunking approaches may be better specified in tokens
• tokenization varies by model. For accurate evaluation libraries such as:
  • tiktoken - uses the openAI tokenizer for openai models
  • hugging face tokenizers/transformers - this takes a model spec & uses the appropriate tokenizer if known. The model name has to be as known to hugging face, not necessarily the deployed model name on the server being used
• using tokens=0.25* bytes is a rough heuristic if a matching algorithm cannot be found
• The configuration for embedding and batching is becoming more complex. Migrating environment settings into first-class configuration (on server, or via api) is probably desirable
• Weaviate and milvus are also in danger of behaving differently & effort is needed to ensure these are converged as much as possible
• attempting chunking exposed a few bugs in the library (and the document loader)
• chunking strategy is not saved (when used with milvus) between restarts (originally this applied to the collection itself, but this was fixed). Milvus has no collection level metadata beyond a description. Common practice is to store a JSON string here with any required collection level metadata such as chunking/embedding information

Pausing this for reconsideration whilst looking at other priorities.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 14 days. If this is still relevant, please add a comment or apply an appropriate label to keep it active. Otherwise, it will be closed in another 14 days.

[planetf1]

Still propose this as a feature for scalability -- to reduce overhead/latency of many small requests