Merge branch 'main' into qdrant

Author: Anush008

Cargo.toml

@@ -87,4 +87,7 @@ hyper-rustls = { version = "0.27.5" }
 yup-oauth2 = "12.1.0"
 rustls = { version = "0.23.25" }
 http-body-util = "0.1.3"
+yaml-rust2 = "0.10.0"
+urlencoding = "2.1.3"
 qdrant-client = "1.13.0"
+

docs/docs/about/community.md

@@ -7,7 +7,7 @@ description: Join the CocoIndex community
 
 Welcome with a huge coconut hug 🥥⋆｡˚🤗.
 
-We are super excited for community contributions of all kinds - whether it's code improvements, documentation updates, issue reports, feature requests on [GitHub](https://github.com/cocoIndex/cocoindex), and discussions in our [Discord](https://discord.com/invite/zpA9S2DR7s).
+We are super excited for community contributions of all kinds - whether it's code improvements, documentation updates, issue reports, feature requests on [GitHub](https://github.com/cocoindex-io/cocoindex), and discussions in our [Discord](https://discord.com/invite/zpA9S2DR7s).
 
 We would love to fostering an inclusive, welcoming, and supportive environment. Contributing to CocoIndex should feel collaborative, friendly and enjoyable for everyone. Together, we can build better AI applications through robust data infrastructure.
 

docs/docs/about/contributing.md

@@ -36,7 +36,7 @@ We love contributions from our community! This guide explains how to get involve
 
 To submit your code:
 
-1. Fork the [CocoIndex repository](https://github.com/cocoIndex/cocoindex)
+1. Fork the [CocoIndex repository](https://github.com/cocoindex-io/cocoindex)
 2. [Create a new branch](https://docs.github.com/en/desktop/making-changes-in-a-branch/managing-branches-in-github-desktop) on your fork
 3. Make your changes
 4. [Open a Pull Request (PR)](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) when your work is ready for review

docs/docs/core/cli.mdx

@@ -65,6 +65,7 @@ The following subcommands are available:
 | `setup` | Check and apply setup changes for flows, including the internal and target storage (to export). |
 | `show` | Show the spec for a specific flow. |
 | `update` | Update the index defined by the flow. |
+| `evaluate` | Evaluate the flow and dump flow outputs to files.  Instead of updating the index, it dumps what should be indexed to files. Mainly used for evaluation purpose. |
 
 Use `--help` to see the full list of subcommands, and `subcommand --help` to see the usage of a specific one.
 

docs/docs/core/flow_methods.mdx

@@ -12,7 +12,7 @@ After a flow is defined as discussed in [Flow Definition](/docs/core/flow_def),
 
 ## update
 
-The `update()` method will update will update the index defined by the flow.
+The `update()` method will update the index defined by the flow.
 
 Once the function returns, the indice is fresh up to the moment when the function is called.
 
@@ -23,5 +23,25 @@ Once the function returns, the indice is fresh up to the moment when the functio
 flow.update()
 ```
 
+</TabItem>
+</Tabs>
+
+## evaluate_and_dump
+
+The `evaluate_and_dump()` method evaluates the flow and dump flow outputs to files.
+
+It takes a `EvaluateAndDumpOptions` dataclass as input to configure, with the following fields:
+
+*   `output_dir` (type: `str`, required): The directory to dump the result to.
+*   `use_cache` (type: `bool`, default: `True`): Use already-cached intermediate data if available.
+    Note that we only reuse existing cached data without updating the cache even if it's turned on.
+
+<Tabs>
+<TabItem value="python" label="Python" default>
+
+```python
+flow.evaluate_and_dump(EvaluateAndDumpOptions(output_dir="./eval_output"))
+```
+
 </TabItem>
 </Tabs>

docs/docs/getting_started/quickstart.md

@@ -217,6 +217,6 @@ It will ask you to enter a query and it will return the top 10 results.
 Next, you may want to:
 
 *   Learn about [CocoIndex Basics](../core/basics.md).
-*   Learn about other examples in the [examples](https://github.com/cocoIndex/cocoindex/tree/main/examples) directory.
+*   Learn about other examples in the [examples](https://github.com/cocoindex-io/cocoindex/tree/main/examples) directory.
     *    The `text_embedding` example is this quickstart with some polishing (loading environment variables from `.env` file, extract pieces shared by the indexing flow and query handler into a function).
     *    Pick other examples to learn upon your interest.

docs/docs/ops/functions.md

@@ -49,6 +49,17 @@ Return type: `vector[float32; N]`, where `N` is determined by the model
 *   `output_type` (type: `type`, required): The type of the output. e.g. a dataclass type name. See [Data Types](/docs/core/data_types) for all supported data types. The LLM will output values that match the schema of the type.
 *   `instruction` (type: `str`, optional): Additional instruction for the LLM.
 
+:::tip Clear type definitions
+
+Definitions of the `output_type` is fed into LLM as guidance to generate the output.
+To improve the quality of the extracted information, giving clear definitions for your dataclasses is especially important, e.g.
+
+*   Provide readable field names for your dataclasses.
+*   Provide reasonable docstrings for your dataclasses.
+*   For any optional fields, clearly annotate that they are optional, by `SomeType | None` or `typing.Optional[SomeType]`.
+
+:::
+
 Input data:
 
 *   `text` (type: `str`, required): The text to extract information from.

examples/code_embedding/README.md

@@ -1,4 +1,14 @@
-Simple example for cocoindex: build embedding index based on local files.
+# Build embedding index for codebase
+
+![Build embedding index for codebase](https://cocoindex.io/blogs/assets/images/cover-9bf0a7cff69b66a40918ab2fc1cea0c7.png)
+
+In this example, we will build an embedding index for a codebase using CocoIndex. CocoIndex provides built-in support for code base chunking, with native Tree-sitter support. [Tree-sitter](https://en.wikipedia.org/wiki/Tree-sitter_%28parser_generator%29) is a parser generator tool and an incremental parsing library, it is available in Rust 🦀 - [GitHub](https://github.com/tree-sitter/tree-sitter). CocoIndex has built-in Rust integration with Tree-sitter to efficiently parse code and extract syntax trees for various programming languages.
+
+
+Please give [Cocoindex on Github](https://github.com/cocoindex-io/cocoindex) a star to support us if you like our work. Thank you so much with a warm coconut hug 🥥🤗. [![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?color=5B5BD6)](https://github.com/cocoindex-io/cocoindex)
+
+You can find a detailed blog post with step by step tutorial and explanations [here](https://cocoindex.io/blogs/index-code-base-for-rag).
+
 
 ## Prerequisite
 [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one.