Chunker maximum character size per embedding model (#769)

Author: Paul-Cornell

api-reference/workflow/workflows.mdx

@@ -1830,7 +1830,12 @@ Fields for `settings` include:
 
 ### Embedder node
 
-An **Embedder** node has a `type` of `embed`. 
+An **Embedder** node has a `type` of `embed`.
+
+<Warning>
+    If you add an embedder node, you must set the workflow's chunker node's `max_characters` setting to a value at or below Unstructured's recommended 
+    maximum chunk size for your specified embedding model. [Learn more](/ui/embedding#chunk-sizing-and-embedding-models).
+</Warning>
 
 [Learn about the available embedding providers and models](/ui/embedding).
 

snippets/general-shared-text/chunk-limits-embedding-models.mdx

@@ -0,0 +1,31 @@
+If your workflow has an [Embedder](/ui/embedding) node, your workflow's [Chunker](/ui/chunking) node settings must stay within the selected embedding model's token limits. 
+Exceeding these limits will cause workflow failures.
+
+Set your **Chunker** node's **Max Characters** to a value at or below Unstructured's recommended maximum chunk size for your selected embedding model, 
+as listed in the following table's last column.
+
+| Embedding model | Dimensions | Tokens | Chunker Max Characters<sup>*</sup> |
+|---|---|---|---|
+| _Amazon Bedrock_ | | | |
+| Cohere Embed English | 1024 | 512 | 1792 |
+| Cohere Embed Multilingual | 1024 | 512 | 1792 |
+| Titan Embeddings G1 - Text | 1536 | 8192 | 28672 |
+| Titan Multimodal Embeddings G1 | 1024 | 256 | 896 |
+| Titan Text Embeddings V2 | 1024 | 8192 | 28672 |
+| _Azure OpenAI_ | | | | 
+| Text Embedding 3 Large | 3072 | 8192 | 28672 |
+| Text Embedding 3 Small | 1536 | 8192 | 28672 |
+| Text Embedding Ada 002 | 1536 | 8192 | 28672 |
+| _Together AI_ | | | |
+| M2-Bert 80M 32K Retrieval | 768 | 8192 | 28672 |
+| _Voyage AI_ | | | |
+| Voyage 3 | 1024 | 32000 | 112000 |
+| Voyage 3 Large | 1024 | 32000 | 112000 |
+| Voyage 3 Lite | 512 | 32000 | 112000 |
+| Voyage Code 2 | 1536 | 16000| 56000 |
+| Voyage Code 3 | 1024 | 32000 | 112000 |
+| Voyage Finance 2 | 1024 | 32000| 112000 |
+| Voyage Law 2 | 1024 | 16000 | 56000 |
+| Voyage Multimodal 3 | 1024 | 32000 | 112000 |
+
+<sup>*</sup> This is an approximate value, determined by multiplying the embedding model's token limit by 3.5.

ui/embedding.mdx

@@ -75,3 +75,9 @@ When choosing an embedding model, be sure to pay attention to the number of dime
 embeddings field of your destination connector's table, collection, or index.
 
 <Note>You can change a workflow's preconfigured provider only through [Custom](/ui/workflows#create-a-custom-workflow) workflow settings.</Note>
+
+## Chunk sizing and embedding models
+
+import ChunkLimitsEmbeddingModels from '/snippets/general-shared-text/chunk-limits-embedding-models.mdx';
+
+<ChunkLimitsEmbeddingModels />

ui/workflows.mdx

@@ -218,6 +218,11 @@ If you did not previously set the workflow to run on a schedule, you can [run th
 
     - Click **Transform** to add a **Partitioner** or **Embedder** node. [Learn more](#custom-workflow-node-types).
 
+      <Warning>
+        If you add an **Embedder** node, you must set the **Chunker** node's **Max Characters** setting to a value at or below Unstructured's recommended 
+        maximum chunk size for your selected embedding model. [Learn more](/ui/embedding#chunk-sizing-and-embedding-models).
+      </Warning>
+
     <Tip>
         Make sure to add nodes in the correct order. If you are unsure, see the usage hints in the blue note that appears 
         in the node's settings pane.
@@ -335,6 +340,11 @@ import DeprecatedModelsUI from '/snippets/general-shared-text/deprecated-models-
     <Accordion title="Embedder node">
         For **Select Embedding Model**, select one of the available models that are shown.
 
+        <Warning>
+          If you add an **Embedder** node, you must set the **Chunker** node's **Max Characters** setting to a value at or below Unstructured's recommended 
+          maximum chunk size for your selected embedding model. [Learn more](/ui/embedding#chunk-sizing-and-embedding-models).
+        </Warning>
+
         Learn more:
 
         - [Embedding overview](/ui/embedding)