diff --git a/docs/docs/core/basics.md b/docs/docs/core/basics.md index 466dc2ccb..4c8fb2437 100644 --- a/docs/docs/core/basics.md +++ b/docs/docs/core/basics.md @@ -1,17 +1,17 @@ --- -title: Basics -description: "CocoIndex basic concepts: indexing flow, data, operations, data updates, etc." +title: Indexing Basics +description: "CocoIndex basic concepts for indexing: indexing flow, data, operations, data updates, etc." --- -# CocoIndex Basics +# CocoIndex Indexing Basics An **index** is a collection of data stored in a way that is easy for retrieval. -CocoIndex is an ETL framework for building indexes from specified data sources, a.k.a. indexing. It also offers utilities for users to retrieve data from the indexes. +CocoIndex is an ETL framework for building indexes from specified data sources, a.k.a. **indexing**. It also offers utilities for users to retrieve data from the indexes. -## Indexing flow +An **indexing flow** extracts data from specified data sources, upon specified transformations, and puts the transformed data into specified storage for later retrieval. -An indexing flow extracts data from specified data sources, upon specified transformations, and puts the transformed data into specified storage for later retrieval. +## Indexing flow elements An indexing flow has two aspects: data and operations on data. @@ -42,7 +42,7 @@ An **operation** in an indexing flow defines a step in the flow. An operation is "import" and "transform" operations produce output data, whose data type is determined based on the operation spec and data types of input data (for "transform" operation only). -### Example +## An indexing flow example For the example shown in the [Quickstart](../getting_started/quickstart) section, the indexing flow is as follows: @@ -60,7 +60,7 @@ This shows schema and example data for the indexing flow: ![Data Example](data_example.svg) -### Life cycle of an indexing flow +## Life cycle of an indexing flow An indexing flow, once set up, maintains a long-lived relationship between data source and data in target storage. This means: @@ -95,19 +95,10 @@ CocoIndex works the same way, but with more powerful capabilities: This means when writing your flow operations, you can treat source data as if it were static - focusing purely on defining the transformation logic. CocoIndex takes care of maintaining the dynamic relationship between sources and target data behind the scenes. -### Internal storage +## Internal storage As an indexing flow is long-lived, it needs to store intermediate data to keep track of the states. CocoIndex uses internal storage for this purpose. Currently, CocoIndex uses Postgres database as the internal storage. -See [Initialization](initialization) for configuring its location, and `cocoindex setup` CLI command (see [CocoIndex CLI](cli)) creates tables for the internal storage. - -## Retrieval - -There are two ways to retrieve data from target storage built by an indexing flow: - -* Query the underlying target storage directly for maximum flexibility. -* Use CocoIndex *query handlers* for a more convenient experience with built-in tooling support (e.g. CocoInsight) to understand query performance against the target data. - -Query handlers are tied to specific indexing flows. They accept query inputs, transform them by defined operations, and retrieve matching data from the target storage that was created by the flow. \ No newline at end of file +See [Initialization](initialization) for configuring its location, and `cocoindex setup` CLI command (see [CocoIndex CLI](cli)) creates tables for the internal storage. \ No newline at end of file diff --git a/docs/docs/core/flow_def.mdx b/docs/docs/core/flow_def.mdx index 88b0cee0e..45bd7aba5 100644 --- a/docs/docs/core/flow_def.mdx +++ b/docs/docs/core/flow_def.mdx @@ -1,7 +1,6 @@ --- title: Flow Definition description: Define a CocoIndex flow, by specifying source, transformations and storages, and connect input/output data of them. -toc_max_heading_level: 4 --- import Tabs from '@theme/Tabs'; diff --git a/docs/docs/getting_started/quickstart.md b/docs/docs/getting_started/quickstart.md index 79087cdbe..7fb4a550f 100644 --- a/docs/docs/getting_started/quickstart.md +++ b/docs/docs/getting_started/quickstart.md @@ -132,7 +132,7 @@ if __name__ == "__main__": The `@cocoindex.main_fn` declares a function as the main function for an indexing application. This achieves the following effects: -* Initialize the CocoIndex librart states. Settings (e.g. database URL) are loaded from environment variables by default. +* Initialize the CocoIndex library states. Settings (e.g. database URL) are loaded from environment variables by default. * When the CLI is invoked with `cocoindex` subcommand, `cocoindex CLI` takes over the control, which provides convenient ways to manage the index. See the next step for more details. ## Step 3: Run the indexing pipeline and queries diff --git a/docs/docs/query.mdx b/docs/docs/query.mdx new file mode 100644 index 000000000..31a954e51 --- /dev/null +++ b/docs/docs/query.mdx @@ -0,0 +1,96 @@ +--- +title: Query Support +description: CocoIndex supports vector search and text search. +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# CocoIndex Query Support + +The main functionality of CocoIndex is indexing. +The goal of indexing is to enable efficient querying against your data. +You can use any libraries or frameworks of your choice to perform queries. +At the same time, CocoIndex provides seamless integration between indexing and querying workflows. +For example, you can share transformations between indexing and querying, and easily retrieve table names when using CocoIndex's default naming conventions. + +## Transform Flow + +Sometimes a part of the transformation logic needs to be shared between indexing and querying, +e.g. when we build a vector index and query against it, the embedding computation needs to be consistent between indexing and querying. + +In this case, you can: + +1. Extract a sub-flow with the shared transformation logic into a standalone function. + * It takes one or more data slices as input. + * It returns one data slice as output. + * You need to annotate data types for both inputs and outputs as type parameter for `cocoindex.DataSlice[T]`. See [data types](./core/data_types.mdx) for more details about supported data types. + +2. When you're defining your indexing flow, you can directly call the function. + The body will be executed, so that the transformation logic will be added as part of the indexing flow. + +3. At query time, you usually want to directly run the function with specific input data, instead of letting it called as part of a long-lived indexing flow. + To do this, declare the function as a *transform flow*, by decorating it with `@cocoindex.transform_flow()`. + This will add a `eval()` method to the function, so that you can directly call with specific input data. + + + + + +The [quickstart](getting_started/quickstart#step-41-extract-common-transformations) shows an example: + +```python +@cocoindex.transform_flow() +def text_to_embedding(text: cocoindex.DataSlice[str]) -> cocoindex.DataSlice[list[float]]: + return text.transform( + cocoindex.functions.SentenceTransformerEmbed( + model="sentence-transformers/all-MiniLM-L6-v2")) +``` + +When you're defining your indexing flow, you can directly call the function: + +```python +with doc["chunks"].row() as chunk: + chunk["embedding"] = text_to_embedding(chunk["text"]) +``` + +or, using the `call()` method of the transform flow on the first argument, to make operations chainable: + +```python +with doc["chunks"].row() as chunk: + chunk["embedding"] = chunk["text"].call(text_to_embedding) +``` + +Any time, you can call the `eval()` method with specific string, which will return a `list[float]`: + +```python +print(text_to_embedding.eval("Hello, world!")) +``` + + + + +## Get Target Native Names + +In your indexing flow, when you export data to a target, you can specify the target name (e.g. a database table name, a collection name, the node label in property graph databases, etc.) explicitly, +or for some backends you can also omit it and let CocoIndex generate a default name for you. +For the latter case, CocoIndex provides a utility function `cocoindex.utils.get_target_storage_default_name()` to get the default name. +It takes the following arguments: + +* `flow` (type: `cocoindex.Flow`): The flow to get the default name for. +* `target_name` (type: `str`): The export target name, appeared in the `export()` call. + +For example: + + + + +```python +table_name = cocoindex.utils.get_target_storage_default_name(text_embedding_flow, "doc_embeddings") +query = f"SELECT filename, text FROM {table_name} ORDER BY embedding <=> %s::vector DESC LIMIT 5" +... +``` + + + + diff --git a/docs/sidebars.ts b/docs/sidebars.ts index 0d56aca90..497c2c430 100644 --- a/docs/sidebars.ts +++ b/docs/sidebars.ts @@ -44,6 +44,11 @@ const sidebars: SidebarsConfig = { 'ai/llm', ], }, + { + type: 'doc', + id: 'query', + label: 'Query Support', + }, { type: 'category', label: 'About',