Skip to content

Commit d176d70

Browse files
committed
Remove markdown table of deprecated OpenAI models
1 parent 379ae54 commit d176d70

File tree

1 file changed

+3
-10
lines changed

1 file changed

+3
-10
lines changed

learn/generation/langchain/handbook/xx-langchain-chunking.ipynb

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -185,16 +185,9 @@
185185
"cell_type": "markdown",
186186
"metadata": {},
187187
"source": [
188-
"Note that for the tokenizer we defined the encoder as `\"cl100k_base\"`. This is a specific tiktoken encoder which is used by `gpt-3.5-turbo`. Other encoders exist. At the time of writing the OpenAI specific tokenizers (using `tiktoken`) are summarized as:\n",
189-
"\n",
190-
"| Encoder | Models |\n",
191-
"| --- | --- |\n",
192-
"| `cl100k_base` | `gpt-4`, `gpt-3.5-turbo`, `text-embedding-ada-002` |\n",
193-
"| `p50k_base` | `text-davinci-003`, `code-davinci-002`, `code-cushman-002` |\n",
194-
"| `r50k_base` | `text-davinci-001`, `davinci`, `text-similarity-davinci-001` |\n",
195-
"| `gpt2` | `gpt2` |\n",
196-
"\n",
197-
"You can find these details in the [Tiktoken `model.py` script](https://github.com/openai/tiktoken/blob/main/tiktoken/model.py), or using `tiktoken.encoding_for_model`:"
188+
"Note that for the tokenizer we defined the encoder as `\"cl100k_base\"`. This is a specific tiktoken encoder which is used by `gpt-3.5-turbo`, as well as `gpt-4`, and `text-embedding-ada-002` which are models supported by OpenAI at the time of this writing. Other encoders may be available, but are used with models that are now deprecated by OpenAI.\n",
189+
"\n",
190+
"You can find more details in the [Tiktoken `model.py` script](https://github.com/openai/tiktoken/blob/main/tiktoken/model.py), or using `tiktoken.encoding_for_model`:"
198191
]
199192
},
200193
{

0 commit comments

Comments
 (0)