Skip to content

Commit 19bcd1a

Browse files
fordNfordN
authored
Add workflows: formatting/linting, unit tests, integration tests (#2)
* .github/workflows: Add Action to check linting and formatting - Runs on PRs; if linting and formatting fixes found it writes a comment describing how to apply and detailing the diff * .github/workflows: Add Action to run unit tests - Runs on PRs and on push to main branch - Badge on README displaying results * tests: use testcontainers for running integration tests - only required for testing the Redis and PostgreSQL loaders, the other data systems use the file system and don't require a binary to be run - also updated testing documentation in README * .github/workflows: Add Action to run integration tests
1 parent 55ba565 commit 19bcd1a

File tree

7 files changed

+476
-77
lines changed

7 files changed

+476
-77
lines changed

‎.github/workflows/integration-tests.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: Integration Tests
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
paths:
7+
- 'src/**/*.py'
8+
- 'tests/**/*.py'
9+
- 'pyproject.toml'
10+
- '.github/workflows/integration-tests.yml'
11+
pull_request:
12+
branches: [ main ]
13+
paths:
14+
- 'src/**/*.py'
15+
- 'tests/**/*.py'
16+
- 'pyproject.toml'
17+
- '.github/workflows/integration-tests.yml'
18+
19+
jobs:
20+
integration-tests:
21+
runs-on: ubuntu-latest
22+
23+
steps:
24+
- uses: actions/checkout@v4
25+
26+
- name: Install uv
27+
uses: astral-sh/setup-uv@v4
28+
with:
29+
enable-cache: true
30+
cache-dependency-glob: "**/pyproject.toml"
31+
32+
- name: Set up Python 3.12
33+
run: uv python install 3.12
34+
35+
- name: Install all dependencies
36+
run: uv sync --group all_loaders --group test --group dev
37+
38+
- name: Run integration tests with coverage
39+
env:
40+
USE_TESTCONTAINERS: "true"
41+
TESTCONTAINERS_RYUK_DISABLED: "true"
42+
run: |
43+
uv run pytest tests/integration/ -v --tb=short -m "integration" \
44+
-k "not snowflake" \
45+
--cov=src/amp/loaders --cov-report=xml --cov-report=term-missing
46+
47+
- name: Upload coverage reports
48+
uses: codecov/codecov-action@v4
49+
if: always()
50+
with:
51+
file: ./coverage.xml
52+
flags: integration
53+
name: codecov-integration
54+
fail_ci_if_error: false

‎.github/workflows/ruff.yml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
name: Ruff
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
paths:
7+
- '**.py'
8+
- 'pyproject.toml'
9+
- '.github/workflows/ruff.yml'
10+
pull_request:
11+
branches: [ main ]
12+
paths:
13+
- '**.py'
14+
- 'pyproject.toml'
15+
- '.github/workflows/ruff.yml'
16+
17+
jobs:
18+
ruff:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- uses: actions/checkout@v4
22+
23+
- name: Set up Python
24+
uses: actions/setup-python@v5
25+
with:
26+
python-version: '3.11'
27+
28+
- name: Install Ruff
29+
run: pip install ruff
30+
31+
- name: Run Ruff format check
32+
run: ruff format --check . --diff
33+
34+
- name: Run Ruff linter
35+
run: ruff check . --output-format=github
36+
37+
# Optional: Create a comment on PR with formatting suggestions
38+
ruff-suggestions:
39+
if: github.event_name == 'pull_request' && failure()
40+
needs: ruff
41+
runs-on: ubuntu-latest
42+
permissions:
43+
pull-requests: write
44+
steps:
45+
- uses: actions/checkout@v4
46+
47+
- name: Set up Python
48+
uses: actions/setup-python@v5
49+
with:
50+
python-version: '3.11'
51+
52+
- name: Install Ruff
53+
run: pip install ruff
54+
55+
- name: Generate fix suggestions
56+
run: |
57+
echo "## 🎨 Ruff Formatting & Linting Report" >> suggestions.md
58+
echo "" >> suggestions.md
59+
echo "Run the following commands locally to fix issues:" >> suggestions.md
60+
echo "" >> suggestions.md
61+
echo '```bash' >> suggestions.md
62+
echo "ruff format ." >> suggestions.md
63+
echo "ruff check . --fix" >> suggestions.md
64+
echo '```' >> suggestions.md
65+
echo "" >> suggestions.md
66+
67+
# Show what would be changed
68+
echo "### Formatting changes needed:" >> suggestions.md
69+
echo '```diff' >> suggestions.md
70+
ruff format --check . --diff >> suggestions.md 2>&1 || true
71+
echo '```' >> suggestions.md
72+
echo "" >> suggestions.md
73+
74+
# Show linting issues
75+
echo "### Linting issues:" >> suggestions.md
76+
echo '```' >> suggestions.md
77+
ruff check . >> suggestions.md 2>&1 || true
78+
echo '```' >> suggestions.md
79+
80+
- name: Comment PR
81+
uses: actions/github-script@v7
82+
with:
83+
script: |
84+
const fs = require('fs');
85+
const suggestions = fs.readFileSync('suggestions.md', 'utf8');
86+
87+
// Find existing comment
88+
const { data: comments } = await github.rest.issues.listComments({
89+
owner: context.repo.owner,
90+
repo: context.repo.repo,
91+
issue_number: context.issue.number,
92+
});
93+
94+
const botComment = comments.find(comment =>
95+
comment.user.type === 'Bot' &&
96+
comment.body.includes('🎨 Ruff Formatting & Linting Report')
97+
);
98+
99+
if (botComment) {
100+
// Update existing comment
101+
await github.rest.issues.updateComment({
102+
owner: context.repo.owner,
103+
repo: context.repo.repo,
104+
comment_id: botComment.id,
105+
body: suggestions
106+
});
107+
} else {
108+
// Create new comment
109+
await github.rest.issues.createComment({
110+
owner: context.repo.owner,
111+
repo: context.repo.repo,
112+
issue_number: context.issue.number,
113+
body: suggestions
114+
});
115+
}

‎.github/workflows/unit-tests.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: Unit Tests
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
paths:
7+
- 'src/**/*.py'
8+
- 'tests/**/*.py'
9+
- 'pyproject.toml'
10+
- '.github/workflows/unit-tests.yml'
11+
pull_request:
12+
branches: [ main ]
13+
paths:
14+
- 'src/**/*.py'
15+
- 'tests/**/*.py'
16+
- 'pyproject.toml'
17+
- '.github/workflows/unit-tests.yml'
18+
19+
jobs:
20+
unit-tests:
21+
runs-on: ubuntu-latest
22+
strategy:
23+
matrix:
24+
python-version: ['3.12']
25+
26+
steps:
27+
- uses: actions/checkout@v4
28+
29+
- name: Install uv
30+
uses: astral-sh/setup-uv@v4
31+
32+
- name: Set up Python ${{ matrix.python-version }}
33+
run: uv python install ${{ matrix.python-version }}
34+
35+
- name: Install dependencies
36+
run: |
37+
uv sync --all-groups
38+
39+
- name: Run unit tests
40+
run: |
41+
uv run pytest tests/unit/ -m "unit" -v --tb=short
42+
43+
- name: Run unit tests with coverage
44+
run: |
45+
uv run pytest tests/unit/ -m "unit" --cov=src/amp --cov-report=xml --cov-report=term-missing
46+
47+
- name: Upload coverage reports
48+
uses: codecov/codecov-action@v4
49+
if: always()
50+
with:
51+
file: ./coverage.xml
52+
flags: unittests
53+
name: codecov-umbrella
54+
fail_ci_if_error: false

‎README.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,12 @@
11
# Python Amp Client
22

3+
[![Unit tests status](https://github.com/edgeandnode/amp-python/actions/workflows/unit-tests.yml/badge.svg?event=push)](https://github.com/edgeandnode/amp-python/actions/workflows/unit-tests.yml)
4+
[![Integration tests status](https://github.com/edgeandnode/amp-python/actions/workflows/integration-tests.yml/badge.svg?event=push)](https://github.com/edgeandnode/amp-python/actions/workflows/integration-tests.yml)
5+
[![Formatting status](https://github.com/edgeandnode/amp-python/actions/workflows/ruff.yml/badge.svg?event=push)](https://github.com/edgeandnode/amp-python/actions/workflows/ruff.yml)
6+
7+
8+
## Overview
9+
310
Client for issuing queries to an Amp server and working with the returned data.
411

512
## Installation
@@ -45,11 +52,87 @@ You can then use it in your python scripts, apps or notebooks.
4552
The project is set up to use the [`pytest`](https://docs.pytest.org/en/stable/) testing framework.
4653
It follows [standard python test discovery rules](https://docs.pytest.org/en/stable/explanation/goodpractices.html#test-discovery).
4754

55+
## Quick Test Commands
56+
4857
Run all tests
4958
```bash
5059
uv run pytest
5160
```
5261

62+
Run only unit tests (fast, no external dependencies)
63+
```bash
64+
make test-unit
65+
```
66+
67+
Run integration tests with automatic container setup
68+
```bash
69+
make test-integration
70+
```
71+
72+
Run all tests with coverage
73+
```bash
74+
make test-all
75+
```
76+
77+
## Integration Testing
78+
79+
Integration tests can run in two modes:
80+
81+
### 1. Automatic Container Mode (Default)
82+
The integration tests will automatically spin up PostgreSQL and Redis containers using testcontainers. This is the default mode and requires Docker to be installed and running.
83+
84+
```bash
85+
# Run integration tests with automatic containers
86+
uv run pytest tests/integration/ -m integration
87+
```
88+
89+
**Note**: The configuration automatically disables Ryuk (testcontainers cleanup container) to avoid Docker connectivity issues. If you need Ryuk enabled, set `TESTCONTAINERS_RYUK_DISABLED=false`.
90+
91+
### 2. Manual Setup Mode
92+
If you prefer to use your own database instances, you can disable testcontainers:
93+
94+
```bash
95+
# Disable testcontainers and use manual configuration
96+
export USE_TESTCONTAINERS=false
97+
98+
# Configure your database connections
99+
export POSTGRES_HOST=localhost
100+
export POSTGRES_PORT=5432
101+
export POSTGRES_DB=test_amp
102+
export POSTGRES_USER=postgres
103+
export POSTGRES_PASSWORD=yourpassword
104+
105+
export REDIS_HOST=localhost
106+
export REDIS_PORT=6379
107+
export REDIS_PASSWORD=yourpassword # Optional
108+
109+
# Run tests
110+
uv run pytest tests/integration/ -m integration
111+
```
112+
113+
For manual setup, you can use the provided Makefile commands:
114+
```bash
115+
# Start test databases manually
116+
make test-setup
117+
118+
# Run tests
119+
make test-integration
120+
121+
# Clean up databases
122+
make test-cleanup
123+
```
124+
125+
## Loader-Specific Tests
126+
127+
Run tests for specific loaders:
128+
```bash
129+
make test-postgresql # PostgreSQL tests
130+
make test-redis # Redis tests
131+
make test-deltalake # Delta Lake tests
132+
make test-iceberg # Iceberg tests
133+
make test-lmdb # LMDB tests
134+
```
135+
53136
# Linting and formatting
54137

55138
Ruff is configured to be used for linting and formatting of this project.

0 commit comments

Comments
 (0)