Open source: local and SaaS embedding options and settings (#675)

Author: Paul-Cornell

open-source/how-to/embedding.mdx

@@ -43,6 +43,13 @@ To use the Ingest CLI or Ingest Python library to generate embeddings, do the fo
 
 1. Choose an embedding provider that you want to use from among the following allowed providers, and note the provider's ID:
 
+   <Note>
+       The following list assumes that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS) 
+       for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), you are limited to 
+       the provider and model names that are supported by the Unstructured API. 
+       [See the list of supported provider names](/api-reference/workflow/workflows#embedder-node).
+   </Note>
+
    - The provider ID `bedrock` for [Amazon Bedrock](https://aws.amazon.com/bedrock/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/bedrock/).
    - `huggingface` for [Hugging Face](https://huggingface.co/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/huggingfacehub/).
    - `mixedbread-ai` for [Mixedbread](https://www.mixedbread.ai/). [Learn more](https://www.mixedbread.ai/docs/embeddings/overview).
@@ -65,6 +72,13 @@ To use the Ingest CLI or Ingest Python library to generate embeddings, do the fo
 
 3. For the following embedding providers, you can choose the model that you want to use. If you do choose a model, note the model's name:
 
+   <Note>
+       The following list assumes that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS) 
+       for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), you are limited to 
+       the model names that are supported by the Unstructured API. 
+       [See the list of supported model names](/api-reference/workflow/workflows#embedder-node).
+   </Note>
+
    - `bedrock`. [Choose a model](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). No default model is provided. [Learn more about the supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html).
    - `huggingface`. [Choose a model](https://huggingface.co/models?other=embeddings), or use the default model [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2).
    - `mixedbread-ai`. [Choose a model](https://www.mixedbread.ai/docs/embeddings/models), or use the default model [mixedbread-ai/mxbai-embed-large-v1](https://www.mixedbread.ai/docs/embeddings/mxbai-embed-large-v1).
@@ -76,6 +90,12 @@ To use the Ingest CLI or Ingest Python library to generate embeddings, do the fo
 
 4. Note the special settings to connect to the provider:
 
+   <Note>
+       The following special settings assume that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS) 
+       for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), do not include any of these special 
+       settings. Unstructured uses its own internal special settings when using the specified provider to generate the embeddings.
+   </Note>
+
    - For `bedrock`, you'll need an AWS access key value, the corresponding AWS secret access key value, and the corresponding AWS Region identifier. [Get an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
    - For `huggingface`, if you use a gated model (a model with special conditions that you must accept before you can use it, or a privately published model), you'll need an HF inference API key value, beginning with `hf_`. [Get an HF inference API key](https://huggingface.co/docs/api-inference/en/quicktour#get-your-api-token). To learn whether your model requires an HF inference API key, see your model provider's documentation. 
    - For `mixedbread-ai`, you'll need a Mixedbread API key value. [Get a Mixedbread API key](https://www.mixedbread.ai/dashboard?next=api-keys).
@@ -89,6 +109,18 @@ To use the Ingest CLI or Ingest Python library to generate embeddings, do the fo
 
 <AccordionGroup>
     <Accordion title="Ingest CLI">
+        <Note>
+            The following options assume that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS) 
+            for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), do not include any of the following options:
+           
+            - `--embedding-api-key`
+            - `--embedding-aws-access-key-id`
+            - `--embedding-aws-secret-access-key`
+            - `--embedding-aws-region`
+
+            Unstructured uses its own internal settings for these options when using the specified provider to generate the embeddings.
+        </Note>
+
         For the [source connector](/open-source/ingestion/source-connectors/overview) command:
 
         - Set the command's `--embedding-provider` to the provider's ID, for example `huggingface`.
@@ -101,6 +133,18 @@ To use the Ingest CLI or Ingest Python library to generate embeddings, do the fo
           - Set `--embedding-aws-region` to the corresponding AWS Region identifier.
     </Accordion>
     <Accordion title="Ingest Python library">
+         <Note>
+            The following parameters assume that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS) 
+            for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), do not include any of the following parameters:
+           
+            - `embedding_api_key`
+            - `embedding_aws_access_key_id`
+            - `embedding_aws_secret_access_key`
+            - `embedding_aws_region`
+
+            Unstructured uses its own internal settings for these parameters when using the specified provider to generate the embeddings.
+        </Note>
+
         For the [source connector's](/open-source/ingestion/source-connectors/overview) `EmbedderConfig` object:
 
         - Set the `embedding_provider` parameter to the provider's ID, for example `huggingface`.

snippets/ingest-configuration-shared/embedding-configuration.mdx

@@ -1,5 +1,21 @@
 A common embedding configuration is a critical component that allows for dynamic selection of embedders and their associated parameters to create vectors from data. This configuration provides the flexibility to choose from various embedding models and fine-tune parameters to optimize the quality and characteristics of the resulting vectors. It enables users to tailor the embedding process to the specific needs of their data and downstream applications, ensuring that the generated vectors effectively capture semantic relationships and contextual information within the dataset.
 
+The core [Unstructured](https://github.com/Unstructured-IO/unstructured) open source library does not enable the generation of embeddings by default. (However you can generate embeddings as a separate step manually. 
+[Learn how](/open-source/core-functionality/embedding).)
+
+You can configure the Unstructured CLI and Unstructured Ingest Python library to generate embeddings by specifying the 
+`embedding_provider`, `embedding_api_key`, and `embedding_model_name` options (and, for Amazon Bedrock, additional options), as follows. 
+You must provide your own API key for the specified embedding provider. To get this API key, you must first create an account with that 
+provider and set up billing directly with them. You are responsible for all costs associated with using that provider.
+
+Calls to the Unstructured CLI or Unstructured Ingest Python library that are routed to Unstructured's software-as-a-service (SaaS) 
+for processing (for example, by specifying an Unstructured API key and an Unstructured SaaS URL) require an Unstructured account for billing purposes. Unstructured's costs for generating 
+embeddings are already included in its account pricing plans. To 
+generate embeddings, you must specify the `embedding_provider` and `embedding_model_name` configuration options as follows. Unstructured uses its own internal API key 
+when using the specified provider to generate the embeddings. These 
+`embedding_provider` and `embedding_model_name` options are limited only to the provider and model names that are supported by the Unstructured API. 
+[See the list of supported provider and model names](/api-reference/workflow/workflows#embedder-node). [Learn how to specify these options](/open-source/how-to/embedding).
+
 ## Configs
 
 *   `embedding_provider`: The embedding provider to use while doing embedding. Available values include `bedrock`, `azure-openai`, `huggingface`, `mixedbread-ai`, `octoai`, `openai`, `togetherai`, `vertexai`, and `voyageai`.