feat: enhance pre commit hook (#66)

gaurpulkit · web-flow · commit 780052f05fd9 · 2025-08-21T13:05:16.000+05:30
* feat: enhance pre-commit hook with validation and detailed setup instructions

- Updated the pre-commit hook description for clarity.
- Added validation for the configuration file in the executor hook to ensure it exists and is not empty.
- Expanded README and documentation to include detailed setup instructions for the pre-commit hook, including configuration options and best practices.

* feat: enhance pre-commit hook with improved config handling and logging

- Added functionality to load configuration from both file and API, with detailed error handling.
- Introduced logging statements to provide feedback during the execution of the pre-commit hook.
- Improved handling of changed files and insights generation process, ensuring better visibility into operations.

* Fix manifest object handling in executor hook logging

* Enhance logging in executor hook to include config ID when using API

* Update logging in executor hook to clarify config ID output

* Fix executor hook to only fail when actual issues are found

* Fix pre-commit hook: load manifest/catalog from files instead of generating partial ones

* Update pre-commit hook configuration to version v0.0.27 and enhance documentation

- Updated the pre-commit hook version to v0.0.27 in configuration files.
- Added new optional arguments for manifest and catalog file paths in the README and documentation.
- Clarified requirements for DBT artifacts in the documentation to improve user guidance.

* Refactor executor hook for improved configuration handling and logging

- Introduced functions to load configuration from both file and API, enhancing error handling and user feedback.
- Added new command-line arguments for better flexibility in specifying paths and configurations.
- Streamlined the process of loading manifest and catalog files, with improved logging for insights generation.
- Enhanced the handling of changed files for selective model testing, ensuring clarity in output and error messages.

* Update argument extraction in executor hook to include an additional return value

- Modified the `extract_arguments` function to return an extra string, enhancing the argument handling capabilities.
- This change supports future extensions and improves flexibility in argument parsing.
diff --git a/.pre-commit-hooks.yaml b/.pre-commit-hooks.yaml
@@ -1,7 +1,16 @@
 - id: datapilot_run_dbt_checks
   name: datapilot run dbt checks
-  description: datapilot run dbt checks
+  description: Run DataPilot dbt project health checks on changed files
   entry: datapilot_run_dbt_checks
   language: python
   types_or: [yaml, sql]
   require_serial: true
+  # Optional arguments that can be passed to the hook:
+  # --config-path: Path to configuration file
+  # --token: API token for authentication
+  # --instance-name: Tenant/instance name
+  # --backend-url: Backend URL (defaults to https://api.myaltimate.com)
+  # --config-name: Name of config to use from API
+  # --manifest-path: Path to DBT manifest file (defaults to ./target/manifest.json)
+  # --catalog-path: Path to DBT catalog file (defaults to ./target/catalog.json)
+  # --base-path: Base path of the dbt project (defaults to current directory)
diff --git a/README.md b/README.md
@@ -42,6 +42,40 @@ The [--config-path] is an optional argument. You can provide a yaml file with ov
 
 Note: The dbt docs generate requires an active database connection and may take a long time for projects with large number of models.
 
+### Pre-commit Hook Integration
+
+DataPilot provides a pre-commit hook that automatically runs health checks on changed files before each commit. This ensures code quality and catches issues early in the development process.
+
+#### Quick Setup
+
+1. Install pre-commit:
+```bash
+pip install pre-commit
+```
+
+2. Add to your `.pre-commit-config.yaml`:
+```yaml
+repos:
+  - repo: https://github.com/AltimateAI/datapilot-cli
+    rev: v0.0.27  # Always use a specific version tag
+    hooks:
+      - id: datapilot_run_dbt_checks
+        args: [
+          "--config-path", "./datapilot-config.yaml",
+          "--token", "${DATAPILOT_TOKEN}",
+          "--instance-name", "${DATAPILOT_INSTANCE}",
+          "--manifest-path", "./target/manifest.json",
+          "--catalog-path", "./target/catalog.json"
+        ]
+```
+
+3. Install the hook:
+```bash
+pre-commit install
+```
+
+For detailed setup instructions, see the [Pre-commit Hook Setup Guide](docs/pre-commit-setup.md).
+
 ### Checks
 
 The following checks are available:
diff --git a/docs/hooks.rst b/docs/hooks.rst
@@ -11,37 +11,171 @@ To use the DataPilot pre-commit hook, follow these steps:
 
 1. Install the `pre-commit` package if you haven't already:
 
-```
-pip install pre-commit
-```
+.. code-block:: shell
+
+    pip install pre-commit
 
 2. Add the following configuration to your .pre-commit-config.yaml file in the root of your repository:
 
-```
+.. code-block:: yaml
+
     repos:
-  - repo: https://github.com/AltimateAI/datapilot-cli
-    rev: <revision>
-    hooks:
-      - id: datapilot_run_dbt_checks
-        args: ["--config-path", "path/to/your/config/file"]
-```
+      - repo: https://github.com/AltimateAI/datapilot-cli
+        rev: v0.0.27  # Use a specific version tag, not 'main'
+        hooks:
+          - id: datapilot_run_dbt_checks
+            args: [
+              "--config-path", "./datapilot-config.yaml",
+              "--token", "${DATAPILOT_TOKEN}",
+              "--instance-name", "${DATAPILOT_INSTANCE}"
+            ]
+
+Configuration Options
+---------------------
+
+The DataPilot pre-commit hook supports several configuration options:
+
+**Required Configuration:**
+
+- ``rev``: Always use a specific version tag (e.g., ``v0.0.27``) instead of ``main`` for production stability
+
+**Optional Arguments:**
+
+- ``--config-path``: Path to your DataPilot configuration file
+- ``--token``: Your API token for authentication (can use environment variables)
+- ``--instance-name``: Your tenant/instance name (can use environment variables)
+- ``--backend-url``: Backend URL (defaults to https://api.myaltimate.com)
+- ``--config-name``: Name of config to use from API
+- ``--base-path``: Base path of the dbt project (defaults to current directory)
+- ``--manifest-path``: Path to the DBT manifest file (defaults to {base_path}/target/manifest.json)
+- ``--catalog-path``: Path to the DBT catalog file (defaults to {base_path}/target/catalog.json)
 
-Replace <revision> with the desired revision of the DataPilot repository and "path/to/your/config/file" with the path to your configuration file.
+**Environment Variables:**
+
+You can use environment variables for sensitive information:
+
+.. code-block:: yaml
+
+    repos:
+      - repo: https://github.com/AltimateAI/datapilot-cli
+        rev: v0.0.27
+        hooks:
+          - id: datapilot_run_dbt_checks
+            args: [
+              "--config-path", "./datapilot-config.yaml",
+              "--token", "${DATAPILOT_TOKEN}",
+              "--instance-name", "${DATAPILOT_INSTANCE}",
+              "--manifest-path", "./target/manifest.json",
+              "--catalog-path", "./target/catalog.json"
+            ]
+
+**Configuration File Example:**
+
+Create a ``datapilot-config.yaml`` file in your project root:
+
+.. code-block:: yaml
+
+    # DataPilot Configuration
+    disabled_insights:
+      - "hard_coded_references"
+      - "duplicate_sources"
+
+    # Custom settings for your project
+    project_settings:
+      max_fanout: 10
+      require_tests: true
 
 3. Install the pre-commit hook:
 
-```
-pre-commit install
-```
+.. code-block:: shell
+
+    pre-commit install
 
 Usage
 -----
 
-Once the hook is installed, it will run automatically before each commit. If any issues are detected, the commit will be aborted, and you will be prompted to fix the issues before retrying the commit.
+Once the hook is installed, it will run automatically before each commit. The hook will:
+
+1. **Validate Configuration**: Check that your config file exists and is valid
+2. **Authenticate**: Use your provided token and instance name to authenticate
+3. **Load DBT Artifacts**: Load manifest and catalog files for analysis
+4. **Analyze Changes**: Only analyze files that have changed in the commit
+5. **Report Issues**: Display any issues found and prevent the commit if problems are detected
+
+**Required DBT Artifacts:**
+
+The pre-commit hook requires DBT manifest and catalog files to function properly:
 
+- **Manifest File**: Generated by running `dbt compile` or `dbt run`. Default location: `./target/manifest.json`
+- **Catalog File**: Generated by running `dbt docs generate`. Default location: `./target/catalog.json`
 
-If you want to manually run all pre-commit hooks on a repository, run `pre-commit run --all-files`. To run individual hooks use `pre-commit run <hook_id>`.
+**Note**: The catalog file is optional but recommended for comprehensive analysis. If not available, the hook will continue without catalog information.
 
+**Manual Execution:**
+
+To manually run all pre-commit hooks on a repository:
+
+.. code-block:: shell
+
+    pre-commit run --all-files
+
+To run individual hooks:
+
+.. code-block:: shell
+
+    pre-commit run datapilot_run_dbt_checks
+
+**Troubleshooting:**
+
+- **Authentication Issues**: Ensure your token and instance name are correctly set
+- **Empty Config Files**: The hook will fail if your config file is empty or invalid
+- **Missing Manifest File**: Ensure you have run `dbt compile` or `dbt run` to generate the manifest.json file
+- **Missing Catalog File**: Run `dbt docs generate` to create the catalog.json file (optional but recommended)
+- **No Changes**: If no relevant files have changed, the hook will skip execution
+- **Network Issues**: Ensure you have access to the DataPilot API
+
+Best Practices
+-------------
+
+1. **Use Version Tags**: Always specify a version tag in the ``rev`` field, never use ``main``
+2. **Environment Variables**: Use environment variables for sensitive information like tokens
+3. **Configuration Files**: Create a dedicated config file for your project settings
+4. **Regular Updates**: Update to new versions when they become available
+5. **Team Coordination**: Ensure all team members use the same configuration
+
+Example Complete Setup
+---------------------
+
+Here's a complete example of a ``.pre-commit-config.yaml`` file:
+
+.. code-block:: yaml
+
+    # .pre-commit-config.yaml
+    exclude: '^(\.tox|ci/templates|\.bumpversion\.cfg)(/|$)'
+
+    repos:
+      - repo: https://github.com/astral-sh/ruff-pre-commit
+        rev: v0.1.14
+        hooks:
+          - id: ruff
+            args: [--fix, --exit-non-zero-on-fix, --show-fixes]
+
+      - repo: https://github.com/psf/black
+        rev: 23.12.1
+        hooks:
+          - id: black
+
+      - repo: https://github.com/AltimateAI/datapilot-cli
+        rev: v0.0.27
+        hooks:
+          - id: datapilot_run_dbt_checks
+            args: [
+              "--config-path", "./datapilot-config.yaml",
+              "--token", "${DATAPILOT_TOKEN}",
+              "--instance-name", "${DATAPILOT_INSTANCE}",
+              "--manifest-path", "./target/manifest.json",
+              "--catalog-path", "./target/catalog.json"
+            ]
 
 Feedback and Contributions
 --------------------------
diff --git a/src/datapilot/core/platforms/dbt/hooks/executor_hook.py b/src/datapilot/core/platforms/dbt/hooks/executor_hook.py
