Skip to content

Commit 4869dc7

Browse files
celsocelso
committed
toMarkdown changelog
updates catalog
1 parent 07b9ac0 commit 4869dc7

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+2094
-429
lines changed

‎src/content/changelog/workers-ai/2025-03-19-markdown-conversion.mdx‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: Markdown conversion in Workers AI
3+
description: You can now convert documents in multiple formats to Markdown using the toMarkdown utility method in Workers AI.
4+
date: 2025-03-20T18:00:00Z
5+
---
6+
7+
Document conversion plays an important role when designing and developing AI applications and agents. Workers AI now provides the `toMarkdown` utility method that developers can use from the `env.AI` binding or the [REST API](/api/resources/ai/) for quick, easy, and convenient conversion and summary of documents in multiple formats to Markdown language.
8+
9+
In this example, we fetch a PDF document and an image from R2 and feed them both to `env.AI.toMarkdown`. The result is a list of converted documents. Workers AI models are used automatically to detect and summarize the image.
10+
11+
```typescript
12+
import { Env } from "./env";
13+
14+
export default {
15+
async fetch(request: Request, env: Env, ctx: ExecutionContext) {
16+
17+
// https://pub-979cb28270cc461d94bc8a169d8f389d.r2.dev/somatosensory.pdf
18+
const pdf = await env.R2.get('somatosensory.pdf');
19+
20+
// https://pub-979cb28270cc461d94bc8a169d8f389d.r2.dev/cat.jpeg
21+
const cat = await env.R2.get('cat.jpeg');
22+
23+
return Response.json(
24+
await env.AI.toMarkdown([
25+
{
26+
name: "somatosensory.pdf",
27+
blob: new Blob([await pdf.arrayBuffer()], { type: "application/octet-stream" }),
28+
},
29+
{
30+
name: "cat.jpeg",
31+
blob: new Blob([await cat.arrayBuffer()], { type: "application/octet-stream" }),
32+
},
33+
]),
34+
);
35+
},
36+
};
37+
```
38+
39+
This is the result:
40+
41+
```json
42+
[
43+
{
44+
"name": "somatosensory.pdf",
45+
"mimeType": "application/pdf",
46+
"format": "markdown",
47+
"tokens": 0,
48+
"data": "# somatosensory.pdf\n## Metadata\n- PDFFormatVersion=1.4\n- IsLinearized=false\n- IsAcroFormPresent=false\n- IsXFAPresent=false\n- IsCollectionPresent=false\n- IsSignaturesPresent=false\n- Producer=Prince 20150210 (www.princexml.com)\n- Title=Anatomy of the Somatosensory System\n\n## Contents\n### Page 1\nThis is a sample document to showcase..."
49+
},
50+
{
51+
"name": "cat.jpeg",
52+
"mimeType": "image/jpeg",
53+
"format": "markdown",
54+
"tokens": 0,
55+
"data": "The image is a close-up photograph of Grumpy Cat, a cat with a distinctive grumpy expression and piercing blue eyes. The cat has a brown face with a white stripe down its nose, and its ears are pointed upright. Its fur is light brown and darker around the face, with a pink nose and mouth. The cat's eyes are blue and slanted downward, giving it a perpetually grumpy appearance. The background is blurred, but it appears to be a dark brown color. Overall, the image is a humorous and iconic representation of the popular internet meme character, Grumpy Cat. The cat's facial expression and posture convey a sense of displeasure or annoyance, making it a relatable and entertaining image for many people."
56+
}
57+
]
58+
```
59+
60+
See [Markdown Conversion](/workers-ai/markdown-conversion/) for more information on supported formats, REST API and pricing.

‎src/content/docs/workers-ai/markdown-conversion.mdx‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -225,10 +225,10 @@ In addition to the Workers AI [binding](/workers-ai/configuration/bindings/), yo
225225
```bash
226226
curl https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/tomarkdown \
227227
-H 'Authorization: Bearer {API_TOKEN}' \
228-
--data-binary "@cat.jpeg" \
229-
--data-binary "@somatosensory.pdf"
228+
-F "files=@cat.jpeg" \
229+
-F "files=@somatosensory.pdf"
230230
```
231231

232232
## Pricing
233233

234-
`toMarkdown` is free for most format conversions. In some cases, like image conversion, it can use Workers AI models for object detection and summarization, which may incur additional costs if it exceeds the Workers AI free allocation limits. See the [pricing page](/workers-ai/pricing/) for more details.
234+
`toMarkdown` is free for most format conversions. In some cases, like image conversion, it can use Workers AI models for object detection and summarization, which may incur additional costs if it exceeds the Workers AI free allocation limits. See the [pricing page](/workers-ai/platform/pricing/) for more details.

‎src/content/workers-ai-models/bge-m3.json‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
{
2+
"id": "eed32bc1-8775-4985-89ce-dd1405508ad8",
3+
"source": 1,
4+
"name": "@cf/baai/bge-m3",
5+
"description": "Multi-Functionality, Multi-Linguality, and Multi-Granularity embeddings model.",
6+
"task": {
7+
"id": "0137cdcf-162a-4108-94f2-1ca59e8c65ee",
8+
"name": "Text Embeddings",
9+
"description": "Feature extraction models transform raw data into numerical features that can be processed while preserving the information in the original dataset. These models are ideal as part of building vector search applications or Retrieval Augmented Generation workflows with Large Language Models (LLM)."
10+
},
11+
"tags": [],
12+
"properties": [],
13+
"schema": {
14+
"input": {
15+
"type": "object",
16+
"properties": {
17+
"query": {
18+
"type": "string",
19+
"minLength": 1,
20+
"description": "A query you wish to perform against the provided contexts. If no query is provided the model with respond with embeddings for contexts"
21+
},
22+
"contexts": {
23+
"type": "array",
24+
"items": {
25+
"type": "object",
26+
"properties": {
27+
"text": {
28+
"type": "string",
29+
"minLength": 1,
30+
"description": "One of the provided context content"
31+
}
32+
}
33+
},
34+
"description": "List of provided contexts. Note that the index in this array is important, as the response will refer to it."
35+
}
36+
},
37+
"required": [
38+
"contexts"
39+
]
40+
},
41+
"output": {
42+
"type": "object",
43+
"contentType": "application/json",
44+
"oneOf": [
45+
{
46+
"title": "Query",
47+
"properties": {
48+
"response": {
49+
"type": "array",
50+
"items": {
51+
"type": "object",
52+
"properties": {
53+
"id": {
54+
"type": "integer",
55+
"description": "Index of the context in the request"
56+
},
57+
"score": {
58+
"type": "number",
59+
"description": "Score of the context under the index."
60+
}
61+
}
62+
}
63+
}
64+
}
65+
},
66+
{
67+
"title": "Embedding",
68+
"properties": {
69+
"response": {
70+
"type": "array",
71+
"items": {
72+
"type": "array",
73+
"items": {
74+
"type": "number"
75+
}
76+
}
77+
}
78+
}
79+
}
80+
]
81+
}
82+
}
83+
}

‎src/content/workers-ai-models/bge-reranker-base.json‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
{
2+
"id": "145337e7-cec3-4ebb-8e78-16ddfc75e580",
3+
"source": 1,
4+
"name": "@cf/baai/bge-reranker-base",
5+
"description": "Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. You can get a relevance score by inputting query and passage to the reranker. And the score can be mapped to a float value in [0,1] by sigmoid function.\n\n",
6+
"task": {
7+
"id": "19606750-23ed-4371-aab2-c20349b53a60",
8+
"name": "Text Classification",
9+
"description": "Sentiment analysis or text classification is a common NLP task that classifies a text input into labels or classes."
10+
},
11+
"tags": [],
12+
"properties": [],
13+
"schema": {
14+
"input": {
15+
"type": "object",
16+
"properties": {
17+
"query": {
18+
"type": "string",
19+
"minLength": 1,
20+
"description": "A query you wish to perform against the provided contexts."
21+
},
22+
"top_k": {
23+
"type": "integer",
24+
"default": null,
25+
"minimum": 1,
26+
"description": "Number of returned results starting with the best score."
27+
},
28+
"contexts": {
29+
"type": "array",
30+
"items": {
31+
"type": "object",
32+
"properties": {
33+
"text": {
34+
"type": "string",
35+
"minLength": 1,
36+
"description": "One of the provided context content"
37+
}
38+
}
39+
},
40+
"description": "List of provided contexts. Note that the index in this array is important, as the response will refer to it."
41+
}
42+
},
43+
"required": [
44+
"query",
45+
"contexts"
46+
]
47+
},
48+
"output": {
49+
"type": "object",
50+
"contentType": "application/json",
51+
"properties": {
52+
"response": {
53+
"type": "array",
54+
"items": {
55+
"type": "object",
56+
"properties": {
57+
"id": {
58+
"type": "integer",
59+
"description": "Index of the context in the request"
60+
},
61+
"score": {
62+
"type": "number",
63+
"description": "Score of the context under the index."
64+
}
65+
}
66+
}
67+
}
68+
}
69+
}
70+
}
71+
}

‎src/content/workers-ai-models/deepseek-coder-6.7b-base-awq.json‎

Lines changed: 43 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,26 @@
3333
"prompt": {
3434
"type": "string",
3535
"minLength": 1,
36-
"maxLength": 131072,
3736
"description": "The input text prompt for the model to generate a response."
3837
},
38+
"lora": {
39+
"type": "string",
40+
"description": "Name of the LoRA (Low-Rank Adaptation) model to fine-tune the base model."
41+
},
42+
"response_format": {
43+
"title": "JSON Mode",
44+
"type": "object",
45+
"properties": {
46+
"type": {
47+
"type": "string",
48+
"enum": [
49+
"json_object",
50+
"json_schema"
51+
]
52+
},
53+
"json_schema": {}
54+
}
55+
},
3956
"raw": {
4057
"type": "boolean",
4158
"default": false,
@@ -93,10 +110,6 @@
93110
"minimum": 0,
94111
"maximum": 2,
95112
"description": "Increases the likelihood of the model introducing new topics."
96-
},
97-
"lora": {
98-
"type": "string",
99-
"description": "Name of the LoRA (Low-Rank Adaptation) model to fine-tune the base model."
100113
}
101114
},
102115
"required": [
@@ -118,7 +131,6 @@
118131
},
119132
"content": {
120133
"type": "string",
121-
"maxLength": 131072,
122134
"description": "The content of the message as a string."
123135
}
124136
},
@@ -287,10 +299,29 @@
287299
]
288300
}
289301
},
302+
"response_format": {
303+
"title": "JSON Mode",
304+
"type": "object",
305+
"properties": {
306+
"type": {
307+
"type": "string",
308+
"enum": [
309+
"json_object",
310+
"json_schema"
311+
]
312+
},
313+
"json_schema": {}
314+
}
315+
},
316+
"raw": {
317+
"type": "boolean",
318+
"default": false,
319+
"description": "If true, a chat template is not applied and you must adhere to the specific model's expected formatting."
320+
},
290321
"stream": {
291322
"type": "boolean",
292323
"default": false,
293-
"description": "If true, the response will be streamed back incrementally."
324+
"description": "If true, the response will be streamed back incrementally using SSE, Server Sent Events."
294325
},
295326
"max_tokens": {
296327
"type": "integer",
@@ -308,7 +339,7 @@
308339
"type": "number",
309340
"minimum": 0,
310341
"maximum": 2,
311-
"description": "Controls the creativity of the AI's responses by adjusting how many possible words it considers. Lower values make outputs more predictable; higher values allow for more varied and creative responses."
342+
"description": "Adjusts the creativity of the AI's responses by controlling how many possible words it considers. Lower values make outputs more predictable; higher values allow for more varied and creative responses."
312343
},
313344
"top_k": {
314345
"type": "integer",
@@ -395,7 +426,10 @@
395426
}
396427
}
397428
}
398-
}
429+
},
430+
"required": [
431+
"response"
432+
]
399433
},
400434
{
401435
"type": "string",

0 commit comments

Comments
 (0)