Skip to content

Commit 30761f8

Browse files
authored
docs: custom target example (#913)
1 parent 55e14ee commit 30761f8

File tree

4 files changed

+17
-22
lines changed

4 files changed

+17
-22
lines changed

docs/docs/examples/examples/custom_targets.md

Lines changed: 16 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -9,19 +9,13 @@ sidebar_custom_props:
99
tags: [custom-building-blocks]
1010
tags: [custom-building-blocks]
1111
---
12-
import { GitHubButton, YouTubeButton } from '../../../src/components/GitHubButton';
12+
import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/components/GitHubButton';
1313

1414
<GitHubButton url="https://github.com/cocoindex-io/cocoindex/tree/main/examples/custom_output_files"/>
1515

1616
## Overview
1717

18-
Let’s walk through a simple example—exporting `.md` files as `.html` using a custom file-based target. This project monitors folder changes and continuously converts markdown to HTML incrementally.
19-
Check out the full [source code](https://github.com/cocoindex-io/cocoindex/tree/main/examples/custom_output_files).
20-
21-
The overall flow is simple:
22-
This example focuses on
23-
- how to configure your custom target
24-
- the flow effortless picks up the changes in the source, recomputes only what's changed and export to the target
18+
Let’s walk through a simple example—exporting `.md` files as `.html` using a custom file-based target. This project monitors folder changes and continuously converts markdown to HTML incrementally. The overall flow is simple and primarily focuses on how to configure your custom target.
2519

2620

2721
## Ingest files
@@ -33,28 +27,26 @@ Ingest a list of markdown files:
3327
def custom_output_files(
3428
flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
3529
) -> None:
36-
"""
37-
Define an example flow that exports markdown files to HTML files.
38-
"""
3930
data_scope["documents"] = flow_builder.add_source(
4031
cocoindex.sources.LocalFile(path="data", included_patterns=["*.md"]),
4132
refresh_interval=timedelta(seconds=5),
4233
)
4334
```
4435
This ingestion creates a table with `filename` and `content` fields.
45-
36+
<DocumentationButton href="https://cocoindex.io/docs/ops/sources" text="Sources" />
4637

4738
## Process each file and collect
4839

4940
Define custom function that converts markdown to HTML
5041

5142
```python
5243
@cocoindex.op.function()
53-
5444
def markdown_to_html(text: str) -> str:
5545
return _markdown_it.render(text)
5646
```
5747

48+
<DocumentationButton href="https://cocoindex.io/docs/custom_ops/custom_functions" text="Custom Function" margin="0 0 16px 0" />
49+
5850
Define data collector and transform each document to html.
5951

6052
```python
@@ -63,21 +55,27 @@ with data_scope["documents"].row() as doc:
6355
doc["html"] = doc["content"].transform(markdown_to_html)
6456
output_html.collect(filename=doc["filename"], html=doc["html"])
6557
```
58+
![Convert markdown to html](/img/examples/custom_targets/convert.png)
6659

6760

6861
## Define the custom target
6962

7063
### Define the target spec
7164

65+
<DocumentationButton href="https://cocoindex.io/docs/custom_ops/custom_targets#target-spec" text="Target Spec" margin="0 0 16px 0" />
66+
7267
The target spec contains a directory for output files:
7368

7469
```python
7570
class LocalFileTarget(cocoindex.op.TargetSpec):
7671
directory: str
7772
```
7873

74+
7975
### Implement the connector
8076

77+
<DocumentationButton href="https://cocoindex.io/docs/custom_ops/custom_targets#target-connector" text="Target Connector" margin="0 0 16px 0" />
78+
8179
`get_persistent_key()` defines the persistent key,
8280
which uniquely identifies the target for change tracking and incremental updates. Here, we simply use the target directory as the key (e.g., `./data/output`).
8381

@@ -180,17 +178,15 @@ def mutate(
180178
### Use it in the Flow
181179

182180
```python
183-
output_html.export(
184-
"OutputHtml",
185-
LocalFileTarget(directory="output_html"),
186-
primary_key_fields=["filename"],
187-
)
181+
output_html.export(
182+
"OutputHtml",
183+
LocalFileTarget(directory="output_html"),
184+
primary_key_fields=["filename"],
185+
)
188186
```
189187

190188
## Run the example
191189

192-
Once your pipeline is set up, keeping your knowledge graph updated is simple:
193-
194190
```bash
195191
pip install -e .
196192
cocoindex update --setup main.py
45.9 KB
Loading
532 KB
Loading

examples/custom_output_files/README.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
1-
# Build text embedding and semantic search 🔍
2-
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cocoindex-io/cocoindex/blob/main/examples/text_embedding/Text_Embedding.ipynb)
1+
# Export markdown files to local Html with Custom Targets
32
[![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?color=5B5BD6)](https://github.com/cocoindex-io/cocoindex)
43

54
In this example, we will build index flow to load data from a local directory, convert them to HTML, and save the data to another local directory powered by [CocoIndex Custom Targets](https://cocoindex.io/docs/custom_ops/custom_targets).

0 commit comments

Comments
 (0)