Skip to content

Commit 0a48a88

Browse files
authored
Update tokens.md
1 parent de5ea16 commit 0a48a88

File tree

1 file changed

+11
-2
lines changed

1 file changed

+11
-2
lines changed

articles/cosmos-db/gen-ai/tokens.md

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
2-
title: Tokens
3-
description: Overview of tokens in the context of large language models.
2+
title: LLM tokens
3+
description: Overview of tokens in large language models.
44
author: wmwxwa
55
ms.author: wangwilliam
66
ms.service: cosmos-db
@@ -11,3 +11,12 @@ ms.date: 07/01/2024
1111
# What are tokens?
1212

1313
Tokens are small chunks of text generated by splitting the input text into smaller segments. These segments can either be words or groups of characters, varying in length from a single character to an entire word. For instance, the word hamburger would be divided into tokens such as ham, bur, and ger while a short and common word like pear would be considered a single token. LLMs like GPT-3.5 or GPT-4 break words into tokens for processing.
14+
15+
## Related content
16+
- [What is a vector database?](../vector-database.md)
17+
- [Vector database in Azure Cosmos DB NoSQL](../nosql/vector-search.md)
18+
- [Vector database in Azure Cosmos DB for MongoDB](../mongodb/vcore/vector-search.md)
19+
- [What is vector search?](vector-search-overview.md)
20+
- Vector [embeddings](vector-embeddings.md)
21+
- [Distance functions](distance-functions.md)
22+
- [kNN vs ANN vector search algorithms](knn-vs-ann.md)

0 commit comments

Comments
 (0)