Skip to content

Commit c50d9fe

Browse files
authored
docs: update document transformer (#1325)
1 parent 6a1bee1 commit c50d9fe

26 files changed

+68
-74
lines changed

content/en/docs/eino/core_modules/components/document_loader_guide/_index.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -170,9 +170,9 @@ log.Printf("doc content: %v", outDocs[0].Content)
170170

171171
## **Existing Implementations**
172172

173-
1. File Loader: Used to load documents from the local file system [Loader - local file](/docs/eino/ecosystem/document/loader_local_file)
174-
2. Web Loader: Used to load documents pointed by web URLs [Loader - web url](/docs/eino/ecosystem/document/loader_web_url)
175-
3. S3 Loader: Used to load documents stored in S3 compatible storage systems [Loader - amazon s3](/docs/eino/ecosystem/document/loader_amazon_s3)
173+
1. File Loader: Used to load documents from the local file system [Loader - local file](/docs/eino/ecosystem_integration/document/loader_local_file)
174+
2. Web Loader: Used to load documents pointed by web URLs [Loader - web url](/docs/eino/ecosystem_integration/document/loader_web_url)
175+
3. S3 Loader: Used to load documents stored in S3 compatible storage systems [Loader - amazon s3](/docs/eino/ecosystem_integration/document/loader_amazon_s3)
176176

177177
## **Reference for Self-Implementation**
178178

content/en/docs/eino/core_modules/components/document_loader_guide/document_parser_interface_guide.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -140,8 +140,8 @@ func main() {
140140

141141
### **Other Implementations**
142142

143-
- pdf parser, used for extracting and parsing PDF formatted files: [Parser - pdf](/docs/eino/ecosystem/document/parser_pdf)
144-
- html parser, used for extracting and parsing HTML formatted content: [Parser - html](/docs/eino/ecosystem/document/parser_html)
143+
- pdf parser, used for extracting and parsing PDF formatted files: [Parser - pdf](/docs/eino/ecosystem_integration/document/parser_pdf)
144+
- html parser, used for extracting and parsing HTML formatted content: [Parser - html](/docs/eino/ecosystem_integration/document/parser_html)
145145

146146
## **Using ****Document Loader**
147147

content/en/docs/eino/core_modules/components/document_transformer_guide.md

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
Description: ""
3-
date: "2025-02-11"
3+
date: "2025-05-07"
44
lastmod: ""
55
tags: []
66
title: 'Eino: Document Transformer guide'
@@ -160,9 +160,9 @@ for idx, doc := range outDocs {
160160

161161
## **Existing Implementations**
162162

163-
1. Markdown Header Splitter: Document splitting based on Markdown headers [Splitter - markdown](/docs/eino/ecosystem/document/splitter_markdown)
164-
2. Text Splitter: Document splitting based on text length or delimiters [Splitter - semantic](/docs/eino/ecosystem/document/splitter_semantic)
165-
3. Document Filter: Filtering document content based on rules [Splitter - recursive](/docs/eino/ecosystem/document/splitter_recursive)
163+
1. Markdown Header Splitter: Document splitting based on Markdown headers [Splitter - markdown](/docs/eino/ecosystem_integration/document/splitter_markdown)
164+
2. Text Splitter: Document splitting based on text length or delimiters [Splitter - semantic](/docs/eino/ecosystem_integration/document/splitter_semantic)
165+
3. Document Filter: Filtering document content based on rules [Splitter - recursive](/docs/eino/ecosystem_integration/document/splitter_recursive)
166166

167167
## **Reference Implementation**
168168

@@ -240,24 +240,21 @@ func (t *MyTransformer) Transform(ctx context.Context, src []*schema.Document, o
240240
}
241241
options = document.GetTransformerImplSpecificOptions(options, opts...)
242242

243-
// 2. Get the callback manager
244-
cm := callbacks.ManagerFromContext(ctx)
245-
246-
// 3. Trigger the pre-transformation callback
247-
ctx = cm.OnStart(ctx, info, &document.TransformerCallbackInput{
243+
// 2. Trigger the pre-transformation callback
244+
ctx = callbacks.OnStart(ctx, info, &document.TransformerCallbackInput{
248245
Input: src,
249246
})
250247

251-
// 4. Execute the transformation logic
248+
// 3. Execute the transformation logic
252249
docs, err := t.doTransform(ctx, src, options)
253250

254251
// 5. Handle errors and trigger the completion callback
255252
if err != nil {
256-
ctx = cm.OnError(ctx, info, err)
253+
ctx = callbacks.OnError(ctx, info, err)
257254
return nil, err
258255
}
259256

260-
ctx = cm.OnEnd(ctx, info, &document.TransformerCallbackOutput{
257+
ctx = callbacks.OnEnd(ctx, info, &document.TransformerCallbackOutput{
261258
Output: docs,
262259
})
263260

content/en/docs/eino/core_modules/components/embedding_guide.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,8 +137,8 @@ log.Printf("vectors in chain: %v", vectors)
137137

138138
## **Existing Implementations**
139139

140-
1. OpenAI Embedding: Generate vectors using OpenAI's text embedding model [Embedding - OpenAI](/docs/eino/ecosystem/embedding/embedding_openai)
141-
2. ARK Embedding: Generate vectors using the ARK platform's model [Embedding - ARK](/docs/eino/ecosystem/embedding/embedding_ark)
140+
1. OpenAI Embedding: Generate vectors using OpenAI's text embedding model [Embedding - OpenAI](/docs/eino/ecosystem_integration/embedding/embedding_openai)
141+
2. ARK Embedding: Generate vectors using the ARK platform's model [Embedding - ARK](/docs/eino/ecosystem_integration/embedding/embedding_ark)
142142

143143
## **Custom Implementation Reference**
144144

content/en/docs/eino/core_modules/components/indexer_guide.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@ fmt.Printf("vikingDB store success, docs=%v, resp ids=%v\n", docs, outIDs)
195195

196196
## **Existing Implementation**
197197

198-
1. Volc VikingDB Indexer: Vector database indexer based on Volcano Engine VikingDB [Indexer - volc VikingDB](/docs/eino/ecosystem/indexer/indexer_volc_vikingdb)
198+
1. Volc VikingDB Indexer: Vector database indexer based on Volcano Engine VikingDB [Indexer - volc VikingDB](/docs/eino/ecosystem_integration/indexer/indexer_volc_vikingdb)
199199

200200
## **Custom Implementation Reference**
201201

content/en/docs/eino/core_modules/components/retriever_guide.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,7 @@ log.Printf("vikingDB retrieve success, query=%v, docs=%v", query, outDocs)
223223

224224
## **Existing Implementations**
225225

226-
- Volc VikingDB Retriever: Retrieval implementation based on Volcano Engine VikingDB [Retriever - volc VikingDB](/docs/eino/ecosystem/retriever/retriever_volc_vikingdb)
226+
- Volc VikingDB Retriever: Retrieval implementation based on Volcano Engine VikingDB [Retriever - volc VikingDB](/docs/eino/ecosystem_integration/retriever/retriever_volc_vikingdb)
227227

228228
## **Reference for Custom Implementation**
229229

content/en/docs/eino/core_modules/components/tools_node_guide/_index.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -244,5 +244,5 @@ result, err := runnable.Invoke(ctx, input, compose.WithCallbacks(helper))
244244

245245
## **Existing Implementations**
246246

247-
1. Google Search Tool: Tool implementation based on Google search [Tool - Googlesearch](/docs/eino/ecosystem/tool/tool_googlesearch)
248-
2. duckduckgo search tool: Tool implementation based on duckduckgo search [Tool - DuckDuckGoSearch](/docs/eino/ecosystem/tool/tool_duckduckgo_search)
247+
1. Google Search Tool: Tool implementation based on Google search [Tool - Googlesearch](/docs/eino/ecosystem_integration/tool/tool_googlesearch)
248+
2. duckduckgo search tool: Tool implementation based on duckduckgo search [Tool - DuckDuckGoSearch](/docs/eino/ecosystem_integration/tool/tool_duckduckgo_search)

content/en/docs/eino/core_modules/components/tools_node_guide/how_to_create_a_tool.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -327,7 +327,7 @@ func useInInvoke() {
327327

328328
## Method 3 - Use the tool provided in eino-ext
329329

330-
In addition to the various custom tools that need to be implemented by yourself, there are also many general tools implemented in the eino-ext project that can be used out of the box, such [Tool - Googlesearch](/docs/eino/ecosystem/tool/tool_googlesearch)[Tool - DuckDuckGoSearch](/docs/eino/ecosystem/tool/tool_duckduckgo_search) 、wikipedia、httprequest , etc. You can refer to [https://github.com/cloudwego/eino-ext/tree/main/components/tool](https://github.com/cloudwego/eino-ext/tree/main/components/tool) for various implementations.
330+
In addition to the various custom tools that need to be implemented by yourself, there are also many general tools implemented in the eino-ext project that can be used out of the box, such [Tool - Googlesearch](/docs/eino/ecosystem_integration/tool/tool_googlesearch)[Tool - DuckDuckGoSearch](/docs/eino/ecosystem_integration/tool/tool_duckduckgo_search) 、wikipedia、httprequest , etc. You can refer to [https://github.com/cloudwego/eino-ext/tree/main/components/tool](https://github.com/cloudwego/eino-ext/tree/main/components/tool) for various implementations.
331331

332332
## Method 4 - Use the MCP protocol
333333

content/en/docs/eino/ecosystem_integration/_index.md

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -11,43 +11,43 @@ weight: 0
1111

1212
### ChatModel
1313

14-
- openai: [ChatModel - OpenAI](/docs/eino/ecosystem/chat_model/chat_model_openai)
15-
- ark: [ChatModel - ARK](/docs/eino/ecosystem/chat_model/chat_model_ark)
16-
- ollama: [ChatModel - Ollama](/docs/eino/ecosystem/chat_model/chat_model_ollama)
14+
- openai: [ChatModel - OpenAI](/docs/eino/ecosystem_integration/chat_model/chat_model_openai)
15+
- ark: [ChatModel - ARK](/docs/eino/ecosystem_integration/chat_model/chat_model_ark)
16+
- ollama: [ChatModel - Ollama](/docs/eino/ecosystem_integration/chat_model/chat_model_ollama)
1717

1818
### Document
1919

2020
#### Loader
2121

22-
- file: [Loader - local file](/docs/eino/ecosystem/document/loader_local_file)
23-
- s3: [Loader - amazon s3](/docs/eino/ecosystem/document/loader_amazon_s3)
24-
- web url: [Loader - web url](/docs/eino/ecosystem/document/loader_web_url)
22+
- file: [Loader - local file](/docs/eino/ecosystem_integration/document/loader_local_file)
23+
- s3: [Loader - amazon s3](/docs/eino/ecosystem_integration/document/loader_amazon_s3)
24+
- web url: [Loader - web url](/docs/eino/ecosystem_integration/document/loader_web_url)
2525

2626
#### Parser
2727

28-
- html: [Parser - html](/docs/eino/ecosystem/document/parser_html)
29-
- pdf: [Parser - pdf](/docs/eino/ecosystem/document/parser_pdf)
28+
- html: [Parser - html](/docs/eino/ecosystem_integration/document/parser_html)
29+
- pdf: [Parser - pdf](/docs/eino/ecosystem_integration/document/parser_pdf)
3030

3131
#### Transformer
3232

33-
- markdown splitter: [Splitter - markdown](/docs/eino/ecosystem/document/splitter_markdown)
34-
- recursive splitter: [Splitter - recursive](/docs/eino/ecosystem/document/splitter_recursive)
35-
- semantic splitter: [Splitter - semantic](/docs/eino/ecosystem/document/splitter_semantic)
33+
- markdown splitter: [Splitter - markdown](/docs/eino/ecosystem_integration/document/splitter_markdown)
34+
- recursive splitter: [Splitter - recursive](/docs/eino/ecosystem_integration/document/splitter_recursive)
35+
- semantic splitter: [Splitter - semantic](/docs/eino/ecosystem_integration/document/splitter_semantic)
3636

3737
### Embedding
3838

39-
- ark: [Embedding - ARK](/docs/eino/ecosystem/embedding/embedding_ark)
40-
- openai: [Embedding - OpenAI](/docs/eino/ecosystem/embedding/embedding_openai)
39+
- ark: [Embedding - ARK](/docs/eino/ecosystem_integration/embedding/embedding_ark)
40+
- openai: [Embedding - OpenAI](/docs/eino/ecosystem_integration/embedding/embedding_openai)
4141

4242
### Indexer
4343

44-
- volc vikingdb: [Indexer - volc VikingDB](/docs/eino/ecosystem/indexer/indexer_volc_vikingdb)
44+
- volc vikingdb: [Indexer - volc VikingDB](/docs/eino/ecosystem_integration/indexer/indexer_volc_vikingdb)
4545

4646
### Retriever
4747

48-
- volc vikingdb: [Retriever - volc VikingDB](/docs/eino/ecosystem/retriever/retriever_volc_vikingdb)
48+
- volc vikingdb: [Retriever - volc VikingDB](/docs/eino/ecosystem_integration/retriever/retriever_volc_vikingdb)
4949

5050
### Tools
5151

52-
- googlesearch: [Tool - Googlesearch](/docs/eino/ecosystem/tool/tool_googlesearch)
53-
- duckduckgo search: [Tool - DuckDuckGoSearch](/docs/eino/ecosystem/tool/tool_duckduckgo_search)
52+
- googlesearch: [Tool - Googlesearch](/docs/eino/ecosystem_integration/tool/tool_googlesearch)
53+
- duckduckgo search: [Tool - DuckDuckGoSearch](/docs/eino/ecosystem_integration/tool/tool_duckduckgo_search)

content/en/docs/eino/ecosystem_integration/chat_model/chat_model_ark.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -233,6 +233,6 @@ func main() {
233233

234234
- [Eino: ChatModel guide](/docs/eino/core_modules/components/chat_model_guide)
235235
- [Eino: ToolsNode guide](/docs/eino/core_modules/components/tools_node_guide)
236-
- [ChatModel - OpenAI](/docs/eino/ecosystem/chat_model/chat_model_openai)
237-
- [ChatModel - Ollama](/docs/eino/ecosystem/chat_model/chat_model_ollama)
236+
- [ChatModel - OpenAI](/docs/eino/ecosystem_integration/chat_model/chat_model_openai)
237+
- [ChatModel - Ollama](/docs/eino/ecosystem_integration/chat_model/chat_model_ollama)
238238
- [Volcano Engine Official Website](https://www.volcengine.com/product/doubao)

0 commit comments

Comments
 (0)