docs: add documentation on how to configure custom model settings (#124)

* Add generation type to ModelConfig

* pass tests

* added generate_text_embeddings

* tests

* remove sensitive=True old artifact no longer needed

* Slight refactor

* slight refactor

* Added embedding generator

* chunk_separator -> chunk_pattern

* update tests

* rename for consistency

* Restructure InferenceParameters -> CompletionInferenceParameters, BaseInferenceParameters, EmbeddingInferenceParameters

* Remove purpose from consolidated kwargs

* WithModelConfiguration.inference_parameters should should be typed with BaseInferenceParameters

* Type as WithModelGeneration

* Add image generation modality

* update return type for generate_kwargs

* make generation_type a field of ModelConfig as opposed to a prop resolved based on the type of InferenceParameters

* remove regex based chunking from embedding generator

* Remove image generation for now

* more tests and updates

* column_type_is_llm_generated -> column_type_is_model_generated

* change set to list: fix flaky tests

* CompletionInferenceParameters -> ChatCompletionInferenceParameters for consistency with generation_type

* Update docs

* fix deprecation warning originating from cli model settings

* update display of inference parameters in cli list

* save prog on inference parameter

* updates for the ocnfig builder

* update cli readme

* update cli for inference parmeters

* update inference parameter names

* flip order of vars

* WithCompletion -> WithChatCompletion

* specify InferenceParamsT

* Update columns.md with EmbeddingColumnConfig info

* make generation_type a descriminator field in inference params. add configuration support for max_parallel_requests and timeout

* DRY out some stuff in field.py

* docs for custom model settings

* Update nomenclature. prompt tokens -> input tokens, completion tokens -> output tokens in column statistics for consistency

* Add nvidia-embedding and openai-embedding to default model configs

* Fix typo in docs

* Make generate collab notebooks

* Address PR comments

Author: nabinchha

docs/colab_notebooks/1-the-basics.ipynb

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "3599c474",
+   "id": "709c75cf",
    "metadata": {},
    "source": [
     "# 🎨 Data Designer Tutorial: The Basics\n",
@@ -14,7 +14,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ee8bed13",
+   "id": "3cb2774e",
    "metadata": {},
    "source": [
     "### ⚡ Colab Setup\n",
@@ -25,7 +25,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "f43069d1",
+   "id": "b886272b",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -36,7 +36,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "c136bf4f",
+   "id": "f5cf20f9",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -53,7 +53,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "48739393",
+   "id": "e11a4288",
    "metadata": {},
    "source": [
     "### 📦 Import the essentials\n",
@@ -64,7 +64,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "e459cd98",
+   "id": "e8faecea",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -85,7 +85,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b705d204",
+   "id": "314d17c1",
    "metadata": {},
    "source": [
     "### ⚙️ Initialize the Data Designer interface\n",
@@ -98,7 +98,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "aee62c85",
+   "id": "be3b5c6f",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -107,7 +107,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae65c557",
+   "id": "1c2852e1",
    "metadata": {},
    "source": [
     "### 🎛️ Define model configurations\n",
@@ -124,7 +124,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "1079200d",
+   "id": "5ad52a10",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -156,7 +156,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f15426a",
+   "id": "25cce9f7",
    "metadata": {},
    "source": [
     "### 🏗️ Initialize the Data Designer Config Builder\n",
@@ -171,7 +171,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "79b8212c",
+   "id": "8ff7190c",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -180,7 +180,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cd1d9e09",
+   "id": "6bc3b23e",
    "metadata": {},
    "source": [
     "## 🎲 Getting started with sampler columns\n",
@@ -197,7 +197,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b3f469d6",
+   "id": "4cff01cb",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -206,7 +206,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e44adc6c",
+   "id": "f981ec58",
    "metadata": {},
    "source": [
     "Let's start designing our product review dataset by adding product category and subcategory columns.\n"
@@ -215,7 +215,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "82b32804",
+   "id": "70ba24a6",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -296,7 +296,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd65456c",
+   "id": "6f1a6c59",
    "metadata": {},
    "source": [
     "Next, let's add samplers to generate data related to the customer and their review.\n"
@@ -305,7 +305,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "6d6d4eef",
+   "id": "d45b925f",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -342,7 +342,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eb7b415c",
+   "id": "bf49c2b1",
    "metadata": {},
    "source": [
     "## 🦜 LLM-generated columns\n",
@@ -357,7 +357,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "ed811560",
+   "id": "669fe324",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -394,7 +394,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fdc0a2c8",
+   "id": "4d93ad9a",
    "metadata": {},
    "source": [
     "### 🔁 Iteration is key – preview the dataset!\n",
@@ -411,7 +411,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "59987c81",
+   "id": "7b2466d1",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -421,7 +421,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "0823ca7f",
+   "id": "508a2866",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -432,7 +432,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "eca4f0bc",
+   "id": "6fbdaf64",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -442,7 +442,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "edd57f85",
+   "id": "154e8e71",
    "metadata": {},
    "source": [
     "### 📊 Analyze the generated data\n",
@@ -455,7 +455,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "5c681eee",
+   "id": "7e031c7b",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -465,7 +465,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14bf06f2",
+   "id": "a60a1fab",
    "metadata": {},
    "source": [
     "### 🆙 Scale up!\n",
@@ -478,7 +478,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b7ffead1",
+   "id": "e07c6718",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -488,7 +488,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "aa966388",
+   "id": "7a5406da",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -501,7 +501,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "98e1085c",
+   "id": "f0360b0e",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -513,7 +513,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e0b9c65a",
+   "id": "d365dda0",
    "metadata": {},
    "source": [
     "## ⏭️ Next Steps\n",