Create tokens.md

Author: wmwxwa

articles/cosmos-db/gen-ai/tokens.md

@@ -0,0 +1,13 @@
+---
+title: Tokens
+description: Overview of tokens in the context of large language models.
+author: wmwxwa
+ms.author: wangwilliam
+ms.service: cosmos-db
+ms.topic: conceptual
+ms.date: 07/01/2024
+---
+
+# What are tokens?
+
+Tokens are small chunks of text generated by splitting the input text into smaller segments. These segments can either be words or groups of characters, varying in length from a single character to an entire word. For instance, the word hamburger would be divided into tokens such as ham, bur, and ger while a short and common word like pear would be considered a single token. LLMs like GPT-3.5 or GPT-4 break words into tokens for processing.