Merge branch 'main' into fix/remove-pypy310-from-tests

Author: jontsai

CONTRIBUTING.rst

@@ -73,8 +73,80 @@ For merging, you should:
 3. Add a note to ``CHANGELOG.rst`` about the changes.
 4. Add yourself to ``AUTHORS.rst``.
 
-Tips
-----
+Running Tests
+-------------
+
+Quick Start (Using Make)
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The easiest way to run tests::
+
+    # First time setup - create virtual environment
+    make venv
+    source .venv/bin/activate
+
+    # Install dependencies
+    make install
+
+    # Run all tests
+    make test
+
+    # Run just the catalog vendor tests
+    make test-vendor
+
+    # Run tests with coverage
+    make test-cov
+
+Manual Test Commands
+~~~~~~~~~~~~~~~~~~~~
+
+If you prefer to run pytest directly::
+
+    # Activate virtual environment
+    source .venv/bin/activate
+
+    # Run catalog extra fields tests
+    python -m pytest tests/test_vendor/test_catalog_v1.py -v
+
+All Test Commands
+~~~~~~~~~~~~~~~~~
+
+::
+
+    # Run all catalog vendor tests
+    python -m pytest tests/test_vendor/ -v
+
+    # Run specific test file
+    python -m pytest tests/test_vendor/test_catalog_v1.py -v
+
+    # Run specific test class
+    python -m pytest tests/test_vendor/test_catalog_v1.py::TestMetadataExtraFields -v
+
+    # Run specific test method
+    python -m pytest tests/test_vendor/test_catalog_v1.py::TestMetadataExtraFields::test_metadata_accepts_extra_fields -v
+
+    # Run with more verbose output
+    python -m pytest tests/test_vendor/test_catalog_v1.py -vv
+
+    # Run and show print statements
+    python -m pytest tests/test_vendor/test_catalog_v1.py -v -s
+
+    # Run all tests in the project
+    python -m pytest tests/ -v
+
+Using tox
+~~~~~~~~~
+
+The GitHub Actions CI uses tox to run tests across multiple Python and Pydantic versions::
+
+    # Run tests with Python 3.10 and Pydantic 2.10 (no coverage)
+    python3 -m tox -e py310-pydantic210-nocov
+
+    # Run tests with coverage
+    python3 -m tox -e py310-pydantic210-cover
+
+    # Run specific tests with tox
+    python3 -m tox -e py310-pydantic210-nocov -- tests/test_vendor/test_catalog_v1.py
 
 To run a subset of tests::
 
@@ -83,3 +155,14 @@ To run a subset of tests::
 To run all the test environments in *parallel*::
 
     tox -p auto
+
+Continuous Integration
+~~~~~~~~~~~~~~~~~~~~~~
+
+Tests run automatically on every push and pull request via GitHub Actions (``.github/workflows/github-actions.yml``).
+
+The CI runs tests across:
+
+* Python versions: 3.10, 3.11, 3.12, PyPy 3.9
+* Pydantic versions: 2.8, 2.10
+* With and without coverage reports

Makefile

@@ -0,0 +1,54 @@
+.PHONY: help venv install test test-vendor test-cov test-all clean lint format
+
+## help - Display help about make targets for this Makefile
+help:
+	@cat Makefile | grep '^## ' --color=never | cut -c4- | sed -e "`printf 's/ - /\t- /;'`" | column -s "`printf '\t'`" -t
+
+## venv - Create virtual environment
+venv:
+	python3 -m venv .venv
+	.venv/bin/pip install --upgrade pip
+	@echo ""
+	@echo "Virtual environment created. Activate with:"
+	@echo "  source .venv/bin/activate"
+
+## install - Install package and dependencies in development mode
+install:
+	pip install -e .
+	pip install pytest pytest-cov tox pre-commit ruff
+
+## test - Run tests quickly
+test:
+	python -m pytest tests/ -v
+
+## test-vendor - Run catalog vendor tests
+test-vendor:
+	python -m pytest tests/test_vendor/test_catalog_v1.py -v
+
+## test-cov - Run tests with coverage report
+test-cov:
+	python -m pytest --cov=src --cov-report=term-missing --cov-report=html tests/ -v
+
+## test-all - Run full test suite with tox (all Python/Pydantic versions)
+test-all:
+	tox
+
+## lint - Run code quality checks
+lint:
+	pre-commit run --all-files
+
+## format - Format code with ruff
+format:
+	ruff format src/ tests/
+
+## clean - Remove build artifacts and cache
+clean:
+	rm -rf build/
+	rm -rf dist/
+	rm -rf *.egg-info
+	rm -rf .pytest_cache/
+	rm -rf .tox/
+	rm -rf htmlcov/
+	rm -rf .coverage
+	find . -type d -name __pycache__ -exec rm -rf {} +
+	find . -type f -name '*.pyc' -delete

README.md

@@ -1,3 +1,11 @@
+# DataPilot CLI
+
+[![Build Status](https://github.com/AltimateAI/datapilot-cli/workflows/build/badge.svg)](https://github.com/AltimateAI/datapilot-cli/actions)
+[![PyPI version](https://badge.fury.io/py/altimate-datapilot-cli.svg)](https://pypi.org/project/altimate-datapilot-cli/)
+[![Python Version](https://img.shields.io/pypi/pyversions/altimate-datapilot-cli.svg)](https://pypi.org/project/altimate-datapilot-cli/)
+[![License](https://img.shields.io/github/license/AltimateAI/datapilot-cli.svg)](https://github.com/AltimateAI/datapilot-cli/blob/main/LICENSE)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![Maintained](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/AltimateAI/datapilot-cli/graphs/commit-activity)
 
 ## Introduction
 

README.rst

@@ -4,32 +4,39 @@ Overview
 
 .. start-badges
 
-.. list-table::
-    :stub-columns: 1
-
-    * - docs
-      - |docs|
-    * - tests
-      - | |github-actions|
-        | |codecov|
-        | |scrutinizer|
-    * - package
-      - | |version| |wheel| |supported-versions| |supported-implementations|
-        | |commits-since|
 .. |docs| image:: https://readthedocs.org/projects/datapilot/badge/?style=flat
     :target: https://datapilot.readthedocs.io/
     :alt: Documentation Status
 
-.. |github-actions| image:: https://github.com/AltimateAI/datapilot-cli/actions/workflows/github-actions.yml/badge.svg
-    :alt: GitHub Actions Build Status
-    :target: https://github.com/AltimateAI/datapilot/actions
+.. |build| image:: https://github.com/AltimateAI/datapilot-cli/workflows/build/badge.svg
+    :target: https://github.com/AltimateAI/datapilot-cli/actions
+    :alt: Build Status
 
 .. |codecov| image:: https://codecov.io/gh/anandgupta42/datapilot/branch/main/graphs/badge.svg?branch=main
     :alt: Coverage Status
     :target: https://app.codecov.io/github/anandgupta42/datapilot
 
+.. |pypi| image:: https://badge.fury.io/py/altimate-datapilot-cli.svg
+    :target: https://pypi.org/project/altimate-datapilot-cli/
+    :alt: PyPI Package
+
+.. |pyversion| image:: https://img.shields.io/pypi/pyversions/altimate-datapilot-cli.svg
+    :target: https://pypi.org/project/altimate-datapilot-cli/
+    :alt: Python Versions
+
+.. |license| image:: https://img.shields.io/github/license/AltimateAI/datapilot-cli.svg
+    :target: https://github.com/AltimateAI/datapilot-cli/blob/main/LICENSE
+    :alt: License
+
+.. |ruff| image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json
+    :target: https://github.com/astral-sh/ruff
+    :alt: Ruff
 
+.. |maintained| image:: https://img.shields.io/badge/Maintained%3F-yes-green.svg
+    :target: https://github.com/AltimateAI/datapilot-cli/graphs/commit-activity
+    :alt: Maintained
 
+|docs| |build| |codecov| |pypi| |pyversion| |license| |ruff| |maintained|
 
 .. end-badges
 

tests/test_vendor/__init__.py

@@ -0,0 +1,113 @@
+"""Tests for catalog v1 parser, specifically testing extra fields handling."""
+import pytest
+
+from vendor.dbt_artifacts_parser.parsers.catalog.catalog_v1 import Metadata
+
+
+class TestMetadataExtraFields:
+    """Test that Metadata class accepts extra fields from dbt."""
+
+    def test_metadata_accepts_extra_fields(self):
+        """Test that metadata accepts fields not explicitly defined in the model."""
+        # Test with a new field that dbt might add in the future
+        data = {
+            "dbt_schema_version": "https://schemas.getdbt.com/dbt/catalog/v1.json",
+            "dbt_version": "1.9.0",
+            "generated_at": "2025-11-05T10:00:00Z",
+            "invocation_id": "test-invocation-123",
+            "invocation_started_at": "2025-11-05T09:59:00Z",  # New field
+            "new_future_field": "some_value",  # Another potential future field
+        }
+
+        # This should not raise a validation error
+        metadata = Metadata(**data)
+
+        # Verify that known fields are accessible normally
+        assert metadata.dbt_schema_version == "https://schemas.getdbt.com/dbt/catalog/v1.json"
+        assert metadata.dbt_version == "1.9.0"
+        assert metadata.generated_at == "2025-11-05T10:00:00Z"
+        assert metadata.invocation_id == "test-invocation-123"
+
+    def test_metadata_extra_fields_in_pydantic_extra(self):
+        """Test that extra fields are stored in __pydantic_extra__."""
+        data = {
+            "dbt_version": "1.9.0",
+            "invocation_started_at": "2025-11-05T09:59:00Z",
+            "new_field_1": "value1",
+            "new_field_2": 123,
+        }
+
+        metadata = Metadata(**data)
+
+        # Extra fields should be stored in __pydantic_extra__
+        assert metadata.__pydantic_extra__ is not None
+        assert "invocation_started_at" in metadata.__pydantic_extra__
+        assert "new_field_1" in metadata.__pydantic_extra__
+        assert "new_field_2" in metadata.__pydantic_extra__
+        assert metadata.__pydantic_extra__["invocation_started_at"] == "2025-11-05T09:59:00Z"
+        assert metadata.__pydantic_extra__["new_field_1"] == "value1"
+        assert metadata.__pydantic_extra__["new_field_2"] == 123
+
+    def test_metadata_model_dump_includes_extra_fields(self):
+        """Test that model_dump() includes extra fields."""
+        data = {
+            "dbt_version": "1.9.0",
+            "invocation_id": "test-123",
+            "invocation_started_at": "2025-11-05T09:59:00Z",
+            "future_field": "future_value",
+        }
+
+        metadata = Metadata(**data)
+        dumped = metadata.model_dump()
+
+        # All fields including extra should be in the dump
+        assert dumped["dbt_version"] == "1.9.0"
+        assert dumped["invocation_id"] == "test-123"
+        assert dumped["invocation_started_at"] == "2025-11-05T09:59:00Z"
+        assert dumped["future_field"] == "future_value"
+
+    def test_metadata_with_no_extra_fields(self):
+        """Test that metadata works normally when no extra fields are provided."""
+        data = {
+            "dbt_version": "1.9.0",
+            "generated_at": "2025-11-05T10:00:00Z",
+        }
+
+        metadata = Metadata(**data)
+
+        assert metadata.dbt_version == "1.9.0"
+        assert metadata.generated_at == "2025-11-05T10:00:00Z"
+
+    def test_metadata_with_only_extra_fields(self):
+        """Test that metadata accepts data with only extra fields (all known fields are Optional)."""
+        data = {
+            "some_new_field": "value",
+            "another_new_field": 42,
+        }
+
+        # This should work since all defined fields are Optional
+        metadata = Metadata(**data)
+
+        assert metadata.__pydantic_extra__["some_new_field"] == "value"
+        assert metadata.__pydantic_extra__["another_new_field"] == 42
+
+    def test_invocation_started_at_as_extra_field(self):
+        """Test the specific case of invocation_started_at being handled as an extra field."""
+        # This is the real-world scenario: dbt adds invocation_started_at
+        data = {
+            "dbt_schema_version": "https://schemas.getdbt.com/dbt/catalog/v1.json",
+            "dbt_version": "1.9.0",
+            "generated_at": "2025-11-05T10:00:00Z",
+            "invocation_id": "abc-123-def-456",
+            "invocation_started_at": "2025-11-05T09:55:30.123456Z",
+        }
+
+        # Should not raise ValidationError
+        metadata = Metadata(**data)
+
+        # The field should be accessible via __pydantic_extra__
+        assert metadata.__pydantic_extra__["invocation_started_at"] == "2025-11-05T09:55:30.123456Z"
+
+        # And should be included in model_dump()
+        dumped = metadata.model_dump()
+        assert dumped["invocation_started_at"] == "2025-11-05T09:55:30.123456Z"