organize targets

Author: badmonster0

docs/docs/targets/entry-oriented/lancedb.md

@@ -0,0 +1,92 @@
+---
+title: LanceDB
+description: CocoIndex LanceDB Target
+toc_max_heading_level: 4
+---
+
+import { ExampleButton } from '../../../src/components/GitHubButton';
+
+# LanceDB
+
+Exports data to a [LanceDB](https://lancedb.github.io/lancedb/) table.
+
+## Data Mapping
+
+Here's how CocoIndex data elements map to LanceDB elements during export:
+
+| CocoIndex Element | LanceDB Element |
+|-------------------|-----------------|
+| an export target  | a unique table  |
+| a collected row   | a row           |
+| a field           | a column        |
+
+
+::::info Installation and import
+
+This target is provided via an optional dependency `[lancedb]`:
+
+```sh
+pip install "cocoindex[lancedb]"
+```
+
+To use it, you need to import the submodule `cocoindex.targets.lancedb`:
+
+```python
+import cocoindex.targets.lancedb as coco_lancedb
+```
+
+::::
+
+## Spec
+
+The spec `coco_lancedb.LanceDB` takes the following fields:
+
+*   `db_uri` (`str`, required): The LanceDB database location (e.g. `./lancedb_data`).
+*   `table_name` (`str`, required): The name of the table to export the data to.
+*   `db_options` (`coco_lancedb.DatabaseOptions`, optional): Advanced database options.
+    *   `storage_options` (`dict[str, Any]`, optional): Passed through to LanceDB when connecting.
+
+Additional notes:
+
+*   Exactly one primary key field is required for LanceDB targets. We create B-Tree index on this key column.
+
+:::info
+
+LanceDB has a limitation that it cannot build a vector index on an empty table (see [LanceDB issue #4034](https://github.com/lancedb/lance/issues/4034)).
+If you want to use vector indexes, you can run the flow once to populate the target table with data, and then create the vector indexes.
+
+:::
+
+You can find an end-to-end example here: [examples/text_embedding_lancedb](https://github.com/cocoindex-io/cocoindex/tree/main/examples/text_embedding_lancedb).
+
+## `connect_async()` helper
+
+We provide a helper to obtain a shared `AsyncConnection` that is reused across your process and shared with CocoIndex's writer for strong read-after-write consistency:
+
+```python
+from cocoindex.targets import lancedb as coco_lancedb
+
+db = await coco_lancedb.connect_async("./lancedb_data")
+table = await db.open_table("TextEmbedding")
+```
+
+Signature:
+
+```python
+def connect_async(
+  db_uri: str,
+  *,
+  db_options: coco_lancedb.DatabaseOptions | None = None,
+  read_consistency_interval: datetime.timedelta | None = None
+) -> lancedb.AsyncConnection
+```
+
+Once `db_uri` matches, it automatically reuses the same connection instance without re-establishing a new connection.
+This achieves strong consistency between your indexing and querying logic, if they run in the same process.
+
+## Example
+<ExampleButton
+  href="https://github.com/cocoindex-io/cocoindex/tree/main/examples/text_embedding_lancedb"
+  text="Text Embedding LanceDB Example"
+  margin="16px 0 24px 0"
+/>

docs/docs/targets/entry-oriented/postgres.md

@@ -0,0 +1,55 @@
+---
+title: Postgres
+description: CocoIndex Postgres Target
+toc_max_heading_level: 4
+---
+
+import { ExampleButton } from '../../../src/components/GitHubButton';
+
+# Postgres
+
+Exports data to Postgres database (with pgvector extension).
+
+## Data Mapping
+
+Here's how CocoIndex data elements map to Postgres elements during export:
+
+| CocoIndex Element | Postgres Element |
+|-------------------|------------------|
+| an export target | a unique table |
+| a collected row | a row |
+| a field | a column |
+
+For example, if you have a data collector that collects rows with fields `id`, `title`, and `embedding`, it will be exported to a Postgres table with corresponding columns.
+It should be a unique table, meaning that no other export target should export to the same table.
+
+:::warning vector type mapping to Postgres
+
+Since vectors in pgvector must have fixed dimension, we only map vectors of number types with fixed dimension (i.e. *Vector[cocoindex.Float32, N]*, *Vector[cocoindex.Float64, N]*, and *Vector[cocoindex.Int64, N]*) to `vector(N)` columns.
+For all other vector types, we map them to `jsonb` columns.
+
+:::
+
+:::info U+0000 (NUL) characters in strings
+
+U+0000 (NUL) is a valid character in Unicode, but Postgres has a limitation that strings (including `text`-like types and strings in `jsonb`) cannot contain them.
+CocoIndex automatically strips U+0000 (NUL) characters from strings before exporting to Postgres. For example, if you have a string `"Hello\0World"`, it will be exported as `"HelloWorld"`.
+
+:::
+
+## Spec
+
+The spec takes the following fields:
+
+*   `database` ([auth reference](../core/flow_def#auth-registry) to `DatabaseConnectionSpec`, optional): The connection to the Postgres database.
+    See [DatabaseConnectionSpec](../core/settings#databaseconnectionspec) for its specific fields.
+    If not provided, will use the same database as the [internal storage](/docs/core/basics#internal-storage).
+
+*   `table_name` (`str`, optional): The name of the table to store to. If unspecified, will use the table name `[${AppNamespace}__]${FlowName}__${TargetName}`, e.g. `DemoFlow__doc_embeddings` or `Staging__DemoFlow__doc_embeddings`.
+
+## Example
+<ExampleButton
+  href="https://github.com/cocoindex-io/cocoindex/tree/main/examples/text_embedding"
+  text="Text Embedding Example with Postgres"
+  margin="16px 0 24px 0"
+/>

docs/docs/targets/entry-oriented/qdrant.md

@@ -0,0 +1,56 @@
+---
+title: Qdrant
+description: CocoIndex Qdrant Target
+toc_max_heading_level: 4
+---
+
+import { ExampleButton } from '../../../src/components/GitHubButton';
+
+# Qdrant
+
+Exports data to a [Qdrant](https://qdrant.tech/) collection.
+
+## Data Mapping
+
+Here's how CocoIndex data elements map to Qdrant elements during export:
+
+| CocoIndex Element | Qdrant Element |
+|-------------------|------------------|
+| an export target  | a unique collection |
+| a collected row   | a point |
+| a field           | a named vector, if fits into Qdrant vector; or a field within payload otherwise |
+
+The following vector types fit into Qdrant vector:
+*   One-dimensional vectors with fixed dimension, e.g. *Vector[Float32, N]*, *Vector[Float64, N]* and *Vector[Int64, N]*.
+    We map them to [dense vectors](https://qdrant.tech/documentation/concepts/vectors/#dense-vectors).
+*   Two-dimensional vectors whose inner layer is a one-dimensional vector with fixed dimension, e.g. *Vector[Vector[Float32, N]]*, *Vector[Vector[Int64, N]]*, *Vector[Vector[Float64, N]]*. The outer layer may or may not have a fixed dimension.
+    We map them to [multivectors](https://qdrant.tech/documentation/concepts/vectors/#multivectors).
+
+
+:::warning vector type mapping to Qdrant
+
+Since vectors in Qdrant must have fixed dimension, we only map vectors of number types with fixed dimension to Qdrant vectors.
+For all other vector types, we map to Qdrant payload as JSON arrays.
+
+:::
+
+## Spec
+
+The spec takes the following fields:
+
+*   `connection` ([auth reference](../core/flow_def#auth-registry) to `QdrantConnection`, optional): The connection to the Qdrant instance. `QdrantConnection` has the following fields:
+    *   `grpc_url` (`str`): The [gRPC URL](https://qdrant.tech/documentation/interfaces/#grpc-interface) of the Qdrant instance, e.g. `http://localhost:6334/`.
+    *   `api_key` (`str`, optional). API key to authenticate requests with.
+
+    If `connection` is not provided, will use local Qdrant instance at `http://localhost:6334/` by default.
+
+*   `collection_name` (`str`, required): The name of the collection to export the data to.
+
+You can find an end-to-end example [here](https://github.com/cocoindex-io/cocoindex/tree/main/examples/text_embedding_qdrant).
+
+## Example
+<ExampleButton
+  href="https://github.com/cocoindex-io/cocoindex/tree/main/examples/text_embedding_qdrant"
+  text="Text Embedding Qdrant Example"
+  margin="16px 0 24px 0"
+/>

docs/docs/targets/index.md

@@ -0,0 +1,35 @@
+---
+title: Targets
+description: CocoIndex Built-in Targets
+toc_max_heading_level: 4
+---
+
+# CocoIndex Built-in Targets
+
+For each target, data are exported from a data collector, containing data of multiple entries, each with multiple fields.
+The way to map data from a data collector to a target depends on data model of the target.
+
+## Entry-Oriented Targets
+
+An entry-oriented target organizes data into independent entries, such as rows, key-value pairs, or documents.
+Each entry is self-contained and does not explicitly link to others.
+There is usually a straightforward mapping from data collector rows to entries.
+
+| Target   | Link |
+|----------|------|
+| Postgres | [Postgres](./targets/entry-oriented/postgres) |
+| Qdrant   | [Qdrant](./targets/entry-oriented/qdrant)     |
+| LanceDB  | [LanceDB](./targets/entry-oriented/lancedb)   |
+
+
+## Property Graph Targets
+
+Property graph is a widely-adopted model for knowledge graphs, where both nodes and relationships can have properties.
+[Graph database concepts](https://neo4j.com/docs/getting-started/appendix/graphdb-concepts/) has a good introduction to basic concepts of property graphs.
+
+
+| Target   | Link |
+|----------|------|
+| Neo4j | [Neo4j](./targets/property-graph/neo4j) |
+| Kuzu   | [Kuzu](./targets/property-graph/kuzu)     |
+