Merge branch 'cocoindex-io:main' into feat-anthropic-dataflow

Author: par4m

.github/ISSUE_TEMPLATE/💡-feature-request.md

@@ -11,14 +11,10 @@ assignees: ''
 **What is the use case?**
 
 **Describe the solution you'd like**
-A clear and concise description of what you want to happen.
-
-**Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
 
 **Additional context**
-Add any other context or screenshots about the feature request here.
+
 
 ---
-[Contributing Guide](https://cocoindex.io/docs/about/contributing)
-For changes that takes more than a day, we recommend you to leave a comment on the issue like **`I'm working on it`**  or **`Can I work on this issue?`** to avoid duplicating work.
+❤️ Contributors, please refer to 📙[Contributing Guide](https://cocoindex.io/docs/about/contributing).
+Unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like **`I'm working on it`**  or **`Can I work on this issue?`** to avoid duplicating work. Our [Discord server](https://discord.com/invite/zpA9S2DR7s) is always open and friendly.

.vscode/settings.json

@@ -0,0 +1,7 @@
+{
+    "cSpell.words": [
+        "cocoindex",
+        "reindexing",
+        "timedelta"
+    ]
+}

docs/docs/about/contributing.md

@@ -23,11 +23,20 @@ We tag issues with the ["good first issue"](https://github.com/cocoindex-io/coco
 ## Start hacking! Setting Up Development Environment 
 Following the steps below to get cocoindex build on latest codebase locally - if you are making changes to cocoindex funcionality and want to test it out.
 
--   Install Rust toolchain: [docs](https://rust-lang.org/tools/install)
+-   🦀 [Install Rust](https://rust-lang.org/tools/install)
+    
+    If you don't have Rust installed, run
+    ```bash
+    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+    ```
+    Already have Rust? Make sure it's up to date 
+    ```bash 
+    rustup update
+    ```
 
--   (Optional) Setup Python virtual environment:
+-   (Recommended) Setup Python virtual environment:
     ```bash
-    virtualenv --python=$(which python3.12) .venv
+    python3 -m venv .venv
     ```
     Activate the virtual environment, before any installings / buildings / runnings:
 
@@ -51,6 +60,7 @@ Following the steps below to get cocoindex build on latest codebase locally - if
     ```
 
 ## Submit Your Code
+CocoIndex is committed to the highest standards of code quality. Please ensure your code is thoroughly tested before submitting a PR.
 
 To submit your code:
 

docs/docs/core/flow_def.mdx

@@ -49,12 +49,55 @@ See [Flow Running](/docs/core/flow_methods) for more details on it.
 </TabItem>
 </Tabs>
 
-## Flow Builder
+## Data Scope
+
+A **data scope** represents data for a certain unit, e.g. the top level scope (involving all data for a flow), for a document, or for a chunk.
+A data scope has a bunch of fields and collectors, and users can add new fields and collectors to it.
+
+### Get or Add a Field
+
+You can get or add a field of a data scope (which is a data slice). 
+
+:::note
 
-The `FlowBuilder` object is the starting point to construct a flow.
+You cannot override an existing field.
+
+:::
+
+<Tabs>
+<TabItem value="python" label="Python" default>
+
+Getting and setting a field of a data scope is done by the `[]` operator with a field name:
+
+```python
+@cocoindex.flow_def(name="DemoFlow")
+def demo_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
+
+    # Add "documents" to the top-level data scope.
+    data_scope["documents"] = flow_builder.add_source(DemoSourceSpec(...))
+
+    # Each row of "documents" is a child scope.
+    with data_scope["documents"].row() as document:
+
+        # Get "content" from the document scope, transform, and add "summary" to scope.
+        document["summary"] = field1_row["content"].transform(DemoFunctionSpec(...))
+```
+
+</TabItem>
+</Tabs>
+
+### Add a collector
+
+See [Data Collector](#data-collector) below for more details.
+
+## Data Slice
+
+A **data slice** references a subset of data belonging to a data scope, e.g. a specific field from a data scope.
+A data slice has a certain data type, and it's the input for most operations.
 
 ### Import from source
 
+To get the initial data slice, we need to start from importing data from a source.
 `FlowBuilder` provides a `add_source()` method to import data from external sources.
 A *source spec* needs to be provided for any import operation, to describe the source and parameters related to the source.
 Import must happen at the top level, and the field created by import must be in the top-level struct.
@@ -72,10 +115,6 @@ def demo_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataSco
 </TabItem>
 </Tabs>
 
-`add_source()` returns a `DataSlice`. Once external data sources are imported, you can further transform them using methods exposed by these data objects, as discussed in the following sections.
-
-We'll describe different data objects in next few sections.
-
 :::note
 
 The actual value of data is not available at the time when we define the flow: it's only available at runtime.
@@ -111,51 +150,6 @@ and only perform transformations on changed source keys.
 
 :::
 
-## Data Scope
-
-A **data scope** represents data for a certain unit, e.g. the top level scope (involving all data for a flow), for a document, or for a chunk.
-A data scope has a bunch of fields and collectors, and users can add new fields and collectors to it.
-
-### Get or Add a Field
-
-You can get or add a field of a data scope (which is a data slice). 
-
-:::note
-
-You cannot override an existing field.
-
-:::
-
-<Tabs>
-<TabItem value="python" label="Python" default>
-
-Getting and setting a field of a data scope is done by the `[]` operator with a field name:
-
-```python
-@cocoindex.flow_def(name="DemoFlow")
-def demo_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
-
-    # Add "documents" to the top-level data scope.
-    data_scope["documents"] = flow_builder.add_source(DemoSourceSpec(...))
-
-    # Each row of "documents" is a child scope.
-    with data_scope["documents"].row() as document:
-
-        # Get "content" from the document scope, transform, and add "summary" to scope.
-        document["summary"] = field1_row["content"].transform(DemoFunctionSpec(...))
-```
-
-</TabItem>
-</Tabs>
-
-### Add a collector
-
-See [Data Collector](#data-collector) below for more details.
-
-## Data Slice
-
-A **data slice** references a subset of data belonging to a data scope, e.g. a specific field from a data scope.
-A data slice has a certain data type, and it's the input for most operations.
 
 ### Transform
 
@@ -164,7 +158,7 @@ A *function spec* needs to be provided for any transform operation, to describe
 
 The function takes one or multiple data arguments.
 The first argument is the data slice to be transformed, and the `transform()` method is applied from it.
-Other arguments can be passed in as positional arguments or keyword arguments, aftert the function spec.
+Other arguments can be passed in as positional arguments or keyword arguments, after the function spec.
 
 <Tabs>
 <TabItem value="python" label="Python" default>
@@ -300,6 +294,29 @@ CocoIndex provides a common way to configure indexes for various storages.
 
 ## Miscellaneous
 
+### Target Declarations
+
+Most time a target storage is created by calling `export()` method on a collector, and this `export()` call comes with configurations needed for the target storage, e.g. options for storage indexes.
+Occasionally, you may need to specify some configurations for target storage out of the context of any specific data collector.
+
+For example, for graph database targets like `Neo4j`, you may have a data collector to export data to Neo4j relationships, which will create nodes referenced by various relationships in turn.
+These nodes don't directly come from any specific data collector (consider relationships from different data collectors may share the same nodes).
+To specify configurations for these nodes, you can *declare* spec for related node labels.
+
+`FlowBuilder` provides `declare()` method for this purpose, which takes the spec to declare, as provided by various target types.
+
+<Tabs>
+<TabItem value="python" label="Python" default>
+
+```python
+flow_builder.declare(
+    cocoindex.storages.Neo4jDeclarations(...)
+)
+```
+
+</TabItem>
+</Tabs>
+
 ### Auth Registry
 
 CocoIndex manages an auth registry. It's an in-memory key-value store, mainly to store authentication information for a backend.
@@ -310,11 +327,10 @@ Operation spec is the default way to configure a backend. But it has the followi
 *   Once an operation is removed after flow definition code change, the spec is also gone.
     But we still need to be able to drop the backend (e.g. a table) by `cocoindex setup` or `cocoindex drop`.
 
-
 Auth registry is introduced to solve the problems above. It works as follows:
 
 *   You can create new **auth entry** by a key and a value.
-*   You can references the entry by the key, and pass it as part of spec for certain operations. e.g. `Neo4jRelationship` takes `connection` field in the form of auth entry reference.
+*   You can references the entry by the key, and pass it as part of spec for certain operations. e.g. `Neo4j` takes `connection` field in the form of auth entry reference.
 
 <Tabs>
 <TabItem value="python" label="Python" default>

docs/docs/core/flow_methods.mdx

@@ -62,7 +62,7 @@ This action has two modes:
 :::info
 
 For both modes, CocoIndex is performing *incremental processing*,
-i.e. we only performs computations and storage mutations on source data that are changed, or the flow has changed.
+i.e. we only perform computations and storage mutations on source data that are changed, or the flow has changed.
 This is to achieve best efficiency.
 
 :::