feat: support native embedding generation (#106)

* Add generation type to ModelConfig

* pass tests

* added generate_text_embeddings

* tests

* remove sensitive=True old artifact no longer needed

* Slight refactor

* slight refactor

* Added embedding generator

* chunk_separator -> chunk_pattern

* update tests

* rename for consistency

* Restructure InferenceParameters -> CompletionInferenceParameters, BaseInferenceParameters, EmbeddingInferenceParameters

* Remove purpose from consolidated kwargs

* WithModelConfiguration.inference_parameters should should be typed with BaseInferenceParameters

* Type as WithModelGeneration

* Add image generation modality

* update return type for generate_kwargs

* make generation_type a field of ModelConfig as opposed to a prop resolved based on the type of InferenceParameters

* remove regex based chunking from embedding generator

* Remove image generation for now

* more tests and updates

* column_type_is_llm_generated -> column_type_is_model_generated

* change set to list: fix flaky tests

* CompletionInferenceParameters -> ChatCompletionInferenceParameters for consistency with generation_type

* Update docs

* fix deprecation warning originating from cli model settings

* update display of inference parameters in cli list

* save prog on inference parameter

* updates for the ocnfig builder

* update cli readme

* update cli for inference parmeters

* update inference parameter names

* flip order of vars

* WithCompletion -> WithChatCompletion

* specify InferenceParamsT

* Update columns.md with EmbeddingColumnConfig info

* make generation_type a descriminator field in inference params. add configuration support for max_parallel_requests and timeout

* DRY out some stuff in field.py

* Update nomenclature. prompt tokens -> input tokens, completion tokens -> output tokens in column statistics for consistency

* Add nvidia-embedding and openai-embedding to default model configs

* Fix typo in docs

* Make generate collab notebooks

* fine-tune -> adjust

Author: nabinchha

docs/colab_notebooks/1-the-basics.ipynb

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "39d7d274",
+   "id": "3599c474",
    "metadata": {},
    "source": [
     "# 🎨 Data Designer Tutorial: The Basics\n",
@@ -14,7 +14,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "60f1d002",
+   "id": "ee8bed13",
    "metadata": {},
    "source": [
     "### ⚡ Colab Setup\n",
@@ -25,7 +25,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "99c42292",
+   "id": "f43069d1",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -36,7 +36,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "2c959ca9",
+   "id": "c136bf4f",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -53,7 +53,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bc185897",
+   "id": "48739393",
    "metadata": {},
    "source": [
     "### 📦 Import the essentials\n",
@@ -64,15 +64,15 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "dc3a2d9d",
+   "id": "e459cd98",
    "metadata": {},
    "outputs": [],
    "source": [
     "from data_designer.essentials import (\n",
     "    CategorySamplerParams,\n",
+    "    ChatCompletionInferenceParams,\n",
     "    DataDesigner,\n",
     "    DataDesignerConfigBuilder,\n",
-    "    InferenceParameters,\n",
     "    LLMTextColumnConfig,\n",
     "    ModelConfig,\n",
     "    PersonFromFakerSamplerParams,\n",
@@ -85,7 +85,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "36c5f571",
+   "id": "b705d204",
    "metadata": {},
    "source": [
     "### ⚙️ Initialize the Data Designer interface\n",
@@ -98,7 +98,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "61b23c70",
+   "id": "aee62c85",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -107,7 +107,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3c9b7cb6",
+   "id": "ae65c557",
    "metadata": {},
    "source": [
     "### 🎛️ Define model configurations\n",
@@ -124,7 +124,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b86f6217",
+   "id": "1079200d",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -145,7 +145,7 @@
     "        alias=MODEL_ALIAS,\n",
     "        model=MODEL_ID,\n",
     "        provider=MODEL_PROVIDER,\n",
-    "        inference_parameters=InferenceParameters(\n",
+    "        inference_parameters=ChatCompletionInferenceParams(\n",
     "            temperature=0.5,\n",
     "            top_p=1.0,\n",
     "            max_tokens=1024,\n",
@@ -156,7 +156,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1f089871",
+   "id": "9f15426a",
    "metadata": {},
    "source": [
     "### 🏗️ Initialize the Data Designer Config Builder\n",
@@ -171,7 +171,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "3d666193",
+   "id": "79b8212c",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -180,7 +180,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e88c8881",
+   "id": "cd1d9e09",
    "metadata": {},
    "source": [
     "## 🎲 Getting started with sampler columns\n",
@@ -197,7 +197,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "79fb85c6",
+   "id": "b3f469d6",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -206,7 +206,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5106cc10",
+   "id": "e44adc6c",
    "metadata": {},
    "source": [
     "Let's start designing our product review dataset by adding product category and subcategory columns.\n"
@@ -215,7 +215,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "22b97af1",
+   "id": "82b32804",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -296,7 +296,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4857b085",
+   "id": "bd65456c",
    "metadata": {},
    "source": [
     "Next, let's add samplers to generate data related to the customer and their review.\n"
@@ -305,7 +305,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "9e90b3cb",
+   "id": "6d6d4eef",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -342,7 +342,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b36a153b",
+   "id": "eb7b415c",
    "metadata": {},
    "source": [
     "## 🦜 LLM-generated columns\n",
@@ -357,7 +357,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "4da88fe6",
+   "id": "ed811560",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -394,7 +394,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5f1b9ac8",
+   "id": "fdc0a2c8",
    "metadata": {},
    "source": [
     "### 🔁 Iteration is key – preview the dataset!\n",
@@ -411,7 +411,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "543e2f9c",
+   "id": "59987c81",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -421,7 +421,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "26136a8a",
+   "id": "0823ca7f",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -432,7 +432,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "aca4360d",
+   "id": "eca4f0bc",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -442,7 +442,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "35ca0470",
+   "id": "edd57f85",
    "metadata": {},
    "source": [
     "### 📊 Analyze the generated data\n",
@@ -455,7 +455,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "d55b402d",
+   "id": "5c681eee",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -465,7 +465,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "245b48cf",
+   "id": "14bf06f2",
    "metadata": {},
    "source": [
     "### 🆙 Scale up!\n",
@@ -478,7 +478,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "fc803eb0",
+   "id": "b7ffead1",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -488,7 +488,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "881c2043",
+   "id": "aa966388",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -501,7 +501,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "d79860d4",
+   "id": "98e1085c",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -513,7 +513,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b4b45176",
+   "id": "e0b9c65a",
    "metadata": {},
    "source": [
     "## ⏭️ Next Steps\n",