Skip to content

Commit 87e56e0

Browse files
authored
Apply suggestions from code review
1 parent 1ed0023 commit 87e56e0

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

pages/generative-apis/faq.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ Note that in this example, the first line where the free tier applies will not d
5555
## What is a token and how are they counted?
5656
A token is the minimum unit of content that is seen and processed by a model. Hence, token definitions depend on input types:
5757
- For text, on average, `1` token corresponds to `~4` characters, and thus `0.75` words (as words are on average five characters long)
58-
- For images, `1` token corresponds to a square of pixels. For example, `pixtral-12b-2409 `model image tokens of `16x16` pixels (16-pixel height, and 16-pixel width, hence `256` pixels in total).
58+
- For images, `1` token corresponds to a square of pixels. For example, `pixtral-12b-2409` model image tokens of `16x16` pixels (16-pixel height, and 16-pixel width, hence `256` pixels in total).
5959

6060
The exact token count and definition depend on [tokenizers](https://huggingface.co/learn/llm-course/en/chapter2/4) used by each model. When this difference is significant (such as for image processing), you can find detailed information in each model documentation (for instance in [`pixtral-12b-2409` size limit documentation](https://www.scaleway.com/en/docs/managed-inference/reference-content/pixtral-12b-2409/#frequently-asked-questions)). Otherwise, when the model is open, you can find this information in the model files on platforms such as Hugging Face, usually in the `tokenizer_config.json` file.
6161

0 commit comments

Comments
 (0)