Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/gemini-dispatch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ jobs:
review:
needs: "dispatch"
if: |-
${{ needs.dispatch.outputs.command == 'review' }}
${{ needs.dispatch.outputs.command == 'review' && secrets.GEMINI_API_KEY != '' }}
uses: "./.github/workflows/gemini-review.yml"
permissions:
contents: "read"
Expand All @@ -144,7 +144,7 @@ jobs:
triage:
needs: "dispatch"
if: |-
${{ needs.dispatch.outputs.command == 'triage' }}
${{ needs.dispatch.outputs.command == 'triage' && secrets.GEMINI_API_KEY != '' }}
uses: "./.github/workflows/gemini-triage.yml"
permissions:
contents: "read"
Expand All @@ -158,7 +158,7 @@ jobs:
invoke:
needs: "dispatch"
if: |-
${{ needs.dispatch.outputs.command == 'invoke' }}
${{ needs.dispatch.outputs.command == 'invoke' && secrets.GEMINI_API_KEY != '' }}
uses: "./.github/workflows/gemini-invoke.yml"
permissions:
contents: "read"
Expand Down
69 changes: 69 additions & 0 deletions remaining_test_issues.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
<!--
SPDX-FileCopyrightText: 2025 Knitli Inc.

SPDX-License-Identifier: MIT OR Apache-2.0
-->

# Remaining Test Issues & Proposed Solutions

This document outlines the current state of the test suite after initial fixes, including specific issues that are blocking tests from passing and proposed solutions for each.

## 1. The `AsyncMock` Attribute Issue in `test_config_validation_flow.py`

**Issue:**
Tests in `test_config_validation_flow.py` were failing with:
`AttributeError: 'coroutine' object has no attribute 'is_compatible_with'` or `RuntimeWarning: coroutine 'AsyncMockMixin._execute_mock_call' was never awaited`.

**Root Cause:**
`mock_checkpoint_manager` was an `AsyncMock()`. Python's `AsyncMock` recursively returns new `AsyncMock` instances for any unconfigured method calls.
When `ConfigChangeAnalyzer` calls `self.checkpoint_manager._extract_fingerprint()`, it gets an `AsyncMock` back (which acts as a coroutine), rather than a `CheckpointSettingsFingerprint` or a standard `Mock`. Later, when it attempts to call `.is_compatible_with()` on this returned object, it throws an `AttributeError` because the returned object is a coroutine, not a fingerprint instance.

**Resolution (applied):**
Changed `mock_checkpoint_manager` base type from `AsyncMock()` to `Mock()`, with async methods (`load`, `validate_checkpoint_compatibility`) explicitly configured as `AsyncMock`. This ensures synchronous methods like `_extract_fingerprint` return regular `Mock` objects.

## 2. DI Container and `actual_vector_store` Warnings

**Issue:**
There are several `RuntimeWarning: coroutine 'actual_vector_store' was never awaited` warnings in pytest output.

**Root Cause:**
The `actual_vector_store` fixture is defined as an `async def` but is likely being injected into synchronous tests or other synchronous fixtures without being awaited properly, or the test runner isn't fully recognizing the fixture dependencies correctly due to a mix of pytest-asyncio versions or missing `@pytest.mark.asyncio` markers on the tests that consume it indirectly.

**Proposed Solutions:**
1. Ensure that any test or fixture consuming `actual_vector_store` is an `async` function and properly marked with `@pytest.mark.asyncio`.
2. Evaluate if `actual_vector_store` genuinely needs to be an async fixture. If it only creates a synchronous object (like an in-memory dictionary), it can be converted to a normal `def` fixture.

## 3. General Pytest Version Compatibility

**Issue:**
During the run, there were warnings related to `PytestRemovedIn9Warning`, and the environment required explicitly pinning `pytest-asyncio` to avoid crashes.

**Proposed Solutions:**
1. **Standardize Pytest Async Markers:** Ensure all asynchronous tests are consistently decorated with `@pytest.mark.asyncio` (not `@pytest.mark.async_test` which is a custom or typoed mark).
2. **Async Fixture Decoration:** This repo sets `asyncio_mode = "auto"` in `pyproject.toml`, which means plain `@pytest.fixture` works for `async def` fixtures — they do **not** need to be re-decorated with `@pytest_asyncio.fixture`. The `asyncio_mode = "auto"` setting handles them automatically.

## 4. Unimplemented and Skipped Tests

**Issue:**
A large number of tests (around 95) are skipped due to missing infrastructure (e.g., Docker Qdrant instances) or pending DI integration implementations.

**Proposed Solutions:**
1. **CI Infrastructure:** For tests requiring Qdrant or real external APIs, either use mock containers (like testcontainers) during CI or ensure the environment variables/services are spun up correctly before running the suite.
2. **DI Updates:** Complete the integration of the new Dependency Injection system into the legacy tests that are marked with "DI integration not yet implemented". Note the distinction between two different settings paths:
- **DI-resolved services**: use `container.override(CodeWeaverSettingsType, test_settings)` to inject test settings.
- **Direct `get_settings()` calls**: `codeweaver.core.get_settings()` constructs settings from the installed package environment and does *not* consult the container. Code paths that call it directly (e.g. `CheckpointManager._extract_fingerprint`, `_create_fingerprint`) must be patched at the call site — either via `unittest.mock.patch("codeweaver.engine.managers.checkpoint_manager.get_settings", ...)` or by explicitly mocking the methods themselves (as done in the `mock_checkpoint_manager` fixture).

## 5. `CheckpointManager` and `get_settings()` — Important Note

`CheckpointManager` calls `get_settings()` directly (imported from `codeweaver.core`). This function constructs a new settings instance by detecting the installed package; it does **not** read from the DI container. Therefore, `container.override(CodeWeaverSettingsType, test_settings)` alone will **not** influence what `_extract_fingerprint()` or `_create_fingerprint()` receive when they call `get_settings()` internally.

**Resolution (applied):** The `mock_checkpoint_manager` fixture now explicitly mocks `_extract_fingerprint` and `_create_fingerprint` with `side_effect` functions that return real `CheckpointSettingsFingerprint` instances built from the mock checkpoint/config data. This avoids the need to either patch `get_settings` or instantiate a real `CheckpointManager` in tests.

For future tests that need to exercise the *real* fingerprint extraction logic, patch `get_settings` at its call site:

```python
from unittest.mock import patch

with patch("codeweaver.engine.managers.checkpoint_manager.get_settings", return_value=test_settings):
result = checkpoint_manager._extract_fingerprint(checkpoint)
```
1 change: 1 addition & 0 deletions scripts/code-quality/update-licenses.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: 2025 Knitli Inc.
# SPDX-FileContributor: Adam Poulemanos <adam@knit.li>
#
Expand Down
Empty file modified scripts/language-support/export-classifications.py
100644 → 100755
Empty file.
Empty file modified scripts/language-support/generate-overrides.py
100644 → 100755
Empty file.
Empty file modified scripts/language-support/holdout-evaluation.py
100644 → 100755
Empty file.
Empty file modified scripts/model_data/hf-models.json.license
100644 → 100755
Empty file.
Empty file modified scripts/model_data/secondary_providers.json
100644 → 100755
Empty file.
Empty file modified scripts/model_data/secondary_providers.json.license
100644 → 100755
Empty file.
Empty file modified scripts/performance_baseline.py
100644 → 100755
Empty file.
Loading