ops: add pre-commit and hooks for code checking (#641)

* ops: add pre-commit and hooks for checking

* chore: pass pre-commit checks

* docs: add pre-commit setup instructions

* fix: run maturin before other hooks on relevant file changes

Author: lemorage

.github/ISSUE_TEMPLATE/💡-feature-request.md

@@ -17,4 +17,4 @@ assignees: ''
 
 ---
 ❤️ Contributors, please refer to 📙[Contributing Guide](https://cocoindex.io/docs/about/contributing).
-Unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like **`I'm working on it`**  or **`Can I work on this issue?`** to avoid duplicating work. Our [Discord server](https://discord.com/invite/zpA9S2DR7s) is always open and friendly.
+Unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like **`I'm working on it`**  or **`Can I work on this issue?`** to avoid duplicating work. Our [Discord server](https://discord.com/invite/zpA9S2DR7s) is always open and friendly.

.github/scripts/update_version.sh

@@ -19,4 +19,4 @@ else
 fi
 
 # Update Cargo.toml
-sed "${SED_INLINE[@]}" "s/^version = .*/version = \"$VERSION\"/" Cargo.toml
+sed "${SED_INLINE[@]}" "s/^version = .*/version = \"$VERSION\"/" Cargo.toml

.pre-commit-config.yaml

@@ -0,0 +1,71 @@
+ci:
+  autofix_prs: false
+  autoupdate_schedule: 'monthly'
+
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: check-case-conflict
+        # Check for files with names that would conflict on a case-insensitive
+        # filesystem like MacOS HFS+ or Windows FAT.
+      - id: check-merge-conflict
+        # Check for files that contain merge conflict strings.
+      - id: check-symlinks
+        # Checks for symlinks which do not point to anything.
+        exclude: ".*(.github.*)$"
+      - id: detect-private-key
+        # Checks for the existence of private keys.
+      - id: end-of-file-fixer
+        # Makes sure files end in a newline and only a newline.
+        exclude: ".*(data.*|licenses.*|_static.*|\\.ya?ml|\\.jpe?g|\\.png|\\.svg|\\.webp)$"
+      - id: trailing-whitespace
+        # Trims trailing whitespace.
+        exclude_types: [python]  # Covered by Ruff W291.
+        exclude: ".*(data.*|licenses.*|_static.*|\\.ya?ml|\\.jpe?g|\\.png|\\.svg|\\.webp)$"
+
+  - repo: local
+    hooks:
+        - id: maturin-develop
+          name: maturin develop
+          entry: maturin develop
+          language: system
+          files: ^(python/|src/|Cargo\.toml|pyproject\.toml)
+          pass_filenames: false
+
+        - id: cargo-fmt
+          name: cargo fmt
+          entry: cargo fmt
+          language: system
+          types: [rust]
+          pass_filenames: false
+
+        - id: cargo-test
+          name: cargo test
+          entry: cargo test
+          language: system
+          types: [rust]
+          pass_filenames: false
+
+        - id: mypy-check
+          name: mypy type check
+          entry: mypy
+          language: system
+          types: [python]
+          pass_filenames: false
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.12.0
+    hooks:
+      - id: ruff-format
+        types: [python]
+        pass_filenames: true
+  
+  - repo: https://github.com/christophmeissner/pytest-pre-commit
+    rev: 1.0.0
+    hooks:
+      - id: pytest
+        language: system
+        types: [python]
+        pass_filenames: false
+        always_run: false

.vscode/settings.json

@@ -6,4 +6,4 @@
     ],
     "editor.formatOnSave": true,
     "python.formatting.provider": "ruff"
-}
+}

CONTRIBUTING.md

@@ -1 +1 @@
-We love contributions from our community ❤️. Please check out our [contributing guide](https://cocoindex.io/docs/about/contributing).
+We love contributions from our community ❤️. Please check out our [contributing guide](https://cocoindex.io/docs/about/contributing).

README.md

@@ -32,10 +32,10 @@ Unlike a workflow orchestration framework where data is usually opaque, in CocoI
 
 ```python
 # import
-data['content'] = flow_builder.add_source(...) 
+data['content'] = flow_builder.add_source(...)
 
 # transform
-data['out'] = data['content'] 
+data['out'] = data['content']
     .transform(...)
     .transform(...)
 
@@ -56,17 +56,17 @@ As a data framework, CocoIndex takes it to the next level on data freshness. **I
 The frameworks takes care of
 - Change data capture.
 - Figure out what exactly needs to be updated, and only updating that without having to recompute everything.
-  
+
 This makes it fast to reflect any source updates to the target store. If you have concerns with surfacing stale data to AI agents and are spending lots of efforts working on infra piece to optimize the latency, the framework actually handles it for you.
 
 
 ## Quick Start:
-If you're new to CocoIndex, we recommend checking out 
+If you're new to CocoIndex, we recommend checking out
 - 📖 [Documentation](https://cocoindex.io/docs)
 - ⚡  [Quick Start Guide](https://cocoindex.io/docs/getting_started/quickstart)
-- 🎬 [Quick Start Video Tutorial](https://youtu.be/gv5R8nOXsWU?si=9ioeKYkMEnYevTXT) 
+- 🎬 [Quick Start Video Tutorial](https://youtu.be/gv5R8nOXsWU?si=9ioeKYkMEnYevTXT)
 
-### Setup 
+### Setup
 
 1. Install CocoIndex Python library
 
@@ -136,8 +136,8 @@ It defines an index flow like this:
 | [Google Drive Text Embedding](examples/gdrive_text_embedding) | Index text documents from Google Drive |
 | [Docs to Knowledge Graph](examples/docs_to_knowledge_graph) | Extract relationships from Markdown documents and build a knowledge graph |
 | [Embeddings to Qdrant](examples/text_embedding_qdrant) | Index documents in a Qdrant collection for semantic search |
-| [FastAPI Server with Docker](examples/fastapi_server_docker) | Run the semantic search server in a Dockerized FastAPI setup | 
-| [Product Recommendation](examples/product_recommendation) | Build real-time product recommendations with LLM and graph database| 
+| [FastAPI Server with Docker](examples/fastapi_server_docker) | Run the semantic search server in a Dockerized FastAPI setup |
+| [Product Recommendation](examples/product_recommendation) | Build real-time product recommendations with LLM and graph database|
 | [Image Search with Vision API](examples/image_search) | Generates detailed captions for images using a vision model, embeds them, enables live-updating semantic search via FastAPI and served on a React frontend|
 
 More coming and stay tuned 👀!
@@ -159,7 +159,7 @@ Join our community here:
 - 📜 [Read our blog posts](https://cocoindex.io/blogs/)
 
 ## Support us:
-We are constantly improving, and more features and examples are coming soon. If you love this project, please drop us a star ⭐ at GitHub repo [![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?color=5B5BD6)](https://github.com/cocoindex-io/cocoindex) to stay tuned and help us grow. 
+We are constantly improving, and more features and examples are coming soon. If you love this project, please drop us a star ⭐ at GitHub repo [![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?color=5B5BD6)](https://github.com/cocoindex-io/cocoindex) to stay tuned and help us grow.
 
 ## License
 CocoIndex is Apache 2.0 licensed.

check.sh

@@ -15,22 +15,22 @@ We use [GitHub Issues](https://github.com/cocoindex-io/cocoindex/issues) to trac
 
 We tag issues with the ["good first issue"](https://github.com/cocoindex-io/cocoindex/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) label for beginner contributors.
 
-## How to Contribute 
+## How to Contribute
 - If you decide to work on an issue, unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like **`I'm working on it`**  or **`Can I work on this issue?`** to avoid duplicating work.
 - For larger features, we recommend you to discuss with us first in our [Discord server](https://discord.com/invite/zpA9S2DR7s) to coordinate the design and work.
 - Our [Discord server](https://discord.com/invite/zpA9S2DR7s) are constantly open. If you are unsure about anything, it is a good place to discuss! We'd love to collaborate and will always be friendly.
 
-## Start hacking! Setting Up Development Environment 
+## Start hacking! Setting Up Development Environment
 Following the steps below to get cocoindex build on latest codebase locally - if you are making changes to cocoindex funcionality and want to test it out.
 
 -   🦀 [Install Rust](https://rust-lang.org/tools/install)
-    
+
     If you don't have Rust installed, run
     ```sh
     curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
     ```
-    Already have Rust? Make sure it's up to date 
-    ```sh 
+    Already have Rust? Make sure it's up to date
+    ```sh
     rustup update
     ```
 
@@ -46,14 +46,19 @@ Following the steps below to get cocoindex build on latest codebase locally - if
 
 -   Install required tools:
     ```sh
-    pip install maturin mypy ruff
+    pip install maturin mypy pre-commit
     ```
 
 -   Build the library. Run at the root of cocoindex directory:
     ```sh
     maturin develop
     ```
 
+-   Install and enable pre-commit hooks. This ensures all checks run automatically before each commit:
+    ```sh
+    pre-commit install
+    ```
+
 -   Before running a specific example, set extra environment variables, for exposing extra traces, allowing dev UI, etc.
     ```sh
     . ./.env.lib_debug
@@ -67,10 +72,14 @@ To submit your code:
 1. Fork the [CocoIndex repository](https://github.com/cocoindex-io/cocoindex)
 2. [Create a new branch](https://docs.github.com/en/desktop/making-changes-in-a-branch/managing-branches-in-github-desktop) on your fork
 3. Make your changes
-4. Make sure all tests and linting pass by running
-   ```sh
-   ./check.sh
-   ```
+4. Run the pre-commit checks (automatically triggered on `git commit`)
+
+    :::tip
+    To run them manually (same as CI):
+        ```sh
+        pre-commit run --all-files
+        ```
+    :::
 
 5. [Open a Pull Request (PR)](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) when your work is ready for review
 

docs/docs/about/contributing.md

@@ -136,9 +136,9 @@ pip install 'litellm[proxy]'
 **Example for OpenAI:**
 ```yaml
 model_list:
-  - model_name: "*"             
+  - model_name: "*"
     litellm_params:
-      model: openai/*           
+      model: openai/*
       api_key: os.environ/LITELLM_API_KEY
 ```
 
@@ -176,7 +176,7 @@ litellm --config config.yml
 ```python
 cocoindex.LlmSpec(
     api_type=cocoindex.LlmApiType.LITE_LLM,
-    model="deepseek-r1", 
+    model="deepseek-r1",
     address="http://127.0.0.1:4000", # default url of LiteLLM
 )
 ```

docs/docs/ai/llm.mdx

@@ -71,7 +71,7 @@ An indexing flow, once set up, maintains a long-lived relationship between data
 
     *   **One time update**: Once triggered, CocoIndex updates the target data to reflect the version of source data up to the current moment.
     *   **Live update**: CocoIndex continuously reacts to changes of source data and updates the target data accordingly, based on various **change capture mechanisms** for the source.
-    
+
     See more details in the [build / update target data](flow_methods#build--update-target-data) section.
 
 3.  CocoIndex intelligently reprocesses to propagate source changes to target by:
@@ -101,4 +101,4 @@ As an indexing flow is long-lived, it needs to store intermediate data to keep t
 CocoIndex uses internal storage for this purpose.
 
 Currently, CocoIndex uses Postgres database as the internal storage.
-See [Settings](settings#databaseconnectionspec) for configuring its location, and `cocoindex setup` CLI command (see [CocoIndex CLI](cli)) creates tables for the internal storage.
+See [Settings](settings#databaseconnectionspec) for configuring its location, and `cocoindex setup` CLI command (see [CocoIndex CLI](cli)) creates tables for the internal storage.