Skip to content

Commit 74e495a

Browse files
fordNfordN
committed
tests: use testcontainers for running integration tests
- only required for testing the Redis and PostgreSQL loaders, the other data systems use the file system and don't require a binary to be run - also updated testing documentation in README
1 parent a41e9c4 commit 74e495a

File tree

4 files changed

+251
-77
lines changed

4 files changed

+251
-77
lines changed

‎README.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# Python Amp Client
22

33
[![Unit tests status](https://github.com/edgeandnode/amp-python/actions/workflows/unit-tests.yml/badge.svg?event=push)](https://github.com/edgeandnode/amp-python/actions/workflows/unit-tests.yml)
4+
[![Integration tests status](https://github.com/edgeandnode/amp-python/actions/workflows/integration-tests.yml/badge.svg?event=push)](https://github.com/edgeandnode/amp-python/actions/workflows/integration-tests.yml)
45
[![Formatting status](https://github.com/edgeandnode/amp-python/actions/workflows/ruff.yml/badge.svg?event=push)](https://github.com/edgeandnode/amp-python/actions/workflows/ruff.yml)
56

67

@@ -51,11 +52,87 @@ You can then use it in your python scripts, apps or notebooks.
5152
The project is set up to use the [`pytest`](https://docs.pytest.org/en/stable/) testing framework.
5253
It follows [standard python test discovery rules](https://docs.pytest.org/en/stable/explanation/goodpractices.html#test-discovery).
5354

55+
## Quick Test Commands
56+
5457
Run all tests
5558
```bash
5659
uv run pytest
5760
```
5861

62+
Run only unit tests (fast, no external dependencies)
63+
```bash
64+
make test-unit
65+
```
66+
67+
Run integration tests with automatic container setup
68+
```bash
69+
make test-integration
70+
```
71+
72+
Run all tests with coverage
73+
```bash
74+
make test-all
75+
```
76+
77+
## Integration Testing
78+
79+
Integration tests can run in two modes:
80+
81+
### 1. Automatic Container Mode (Default)
82+
The integration tests will automatically spin up PostgreSQL and Redis containers using testcontainers. This is the default mode and requires Docker to be installed and running.
83+
84+
```bash
85+
# Run integration tests with automatic containers
86+
uv run pytest tests/integration/ -m integration
87+
```
88+
89+
**Note**: The configuration automatically disables Ryuk (testcontainers cleanup container) to avoid Docker connectivity issues. If you need Ryuk enabled, set `TESTCONTAINERS_RYUK_DISABLED=false`.
90+
91+
### 2. Manual Setup Mode
92+
If you prefer to use your own database instances, you can disable testcontainers:
93+
94+
```bash
95+
# Disable testcontainers and use manual configuration
96+
export USE_TESTCONTAINERS=false
97+
98+
# Configure your database connections
99+
export POSTGRES_HOST=localhost
100+
export POSTGRES_PORT=5432
101+
export POSTGRES_DB=test_amp
102+
export POSTGRES_USER=postgres
103+
export POSTGRES_PASSWORD=yourpassword
104+
105+
export REDIS_HOST=localhost
106+
export REDIS_PORT=6379
107+
export REDIS_PASSWORD=yourpassword # Optional
108+
109+
# Run tests
110+
uv run pytest tests/integration/ -m integration
111+
```
112+
113+
For manual setup, you can use the provided Makefile commands:
114+
```bash
115+
# Start test databases manually
116+
make test-setup
117+
118+
# Run tests
119+
make test-integration
120+
121+
# Clean up databases
122+
make test-cleanup
123+
```
124+
125+
## Loader-Specific Tests
126+
127+
Run tests for specific loaders:
128+
```bash
129+
make test-postgresql # PostgreSQL tests
130+
make test-redis # Redis tests
131+
make test-deltalake # Delta Lake tests
132+
make test-iceberg # Iceberg tests
133+
make test-lmdb # LMDB tests
134+
```
135+
59136
# Linting and formatting
60137

61138
Ruff is configured to be used for linting and formatting of this project.

‎tests/conftest.py‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,25 @@
1616

1717
logging.basicConfig(level=logging.INFO)
1818

19+
# Control whether to use testcontainers
20+
USE_TESTCONTAINERS = os.getenv('USE_TESTCONTAINERS', 'true').lower() == 'true'
21+
22+
# Disable Ryuk if not explicitly enabled (solves Docker connectivity issues)
23+
if 'TESTCONTAINERS_RYUK_DISABLED' not in os.environ:
24+
os.environ['TESTCONTAINERS_RYUK_DISABLED'] = 'true'
25+
26+
# Import testcontainers conditionally
27+
if USE_TESTCONTAINERS:
28+
try:
29+
from testcontainers.postgres import PostgresContainer
30+
from testcontainers.redis import RedisContainer
31+
TESTCONTAINERS_AVAILABLE = True
32+
except ImportError:
33+
TESTCONTAINERS_AVAILABLE = False
34+
logging.warning("Testcontainers not available. Falling back to manual configuration.")
35+
else:
36+
TESTCONTAINERS_AVAILABLE = False
37+
1938

2039
# Shared configuration fixtures
2140
@pytest.fixture(scope='session')
@@ -91,6 +110,80 @@ def test_config():
91110
}
92111

93112

113+
# Testcontainers fixtures
114+
@pytest.fixture(scope='session')
115+
def postgres_container():
116+
"""PostgreSQL container for integration tests"""
117+
if not TESTCONTAINERS_AVAILABLE:
118+
pytest.skip("Testcontainers not available")
119+
120+
container = PostgresContainer(
121+
image="postgres:13",
122+
username="test_user",
123+
password="test_pass",
124+
dbname="test_db"
125+
)
126+
container.start()
127+
128+
yield container
129+
130+
container.stop()
131+
132+
133+
@pytest.fixture(scope='session')
134+
def redis_container():
135+
"""Redis container for integration tests"""
136+
if not TESTCONTAINERS_AVAILABLE:
137+
pytest.skip("Testcontainers not available")
138+
139+
container = RedisContainer(image="redis:7-alpine")
140+
container.start()
141+
142+
yield container
143+
144+
container.stop()
145+
146+
147+
@pytest.fixture(scope='session')
148+
def postgresql_test_config(request):
149+
"""PostgreSQL configuration from testcontainer or environment"""
150+
if TESTCONTAINERS_AVAILABLE and USE_TESTCONTAINERS:
151+
# Get the postgres_container fixture
152+
postgres_container = request.getfixturevalue('postgres_container')
153+
return {
154+
'host': postgres_container.get_container_host_ip(),
155+
'port': postgres_container.get_exposed_port(5432),
156+
'database': 'test_db',
157+
'user': 'test_user',
158+
'password': 'test_pass',
159+
'max_connections': 10,
160+
'batch_size': 10000,
161+
}
162+
else:
163+
# Fall back to manual config from environment
164+
return request.getfixturevalue('postgresql_config')
165+
166+
167+
@pytest.fixture(scope='session')
168+
def redis_test_config(request):
169+
"""Redis configuration from testcontainer or environment"""
170+
if TESTCONTAINERS_AVAILABLE and USE_TESTCONTAINERS:
171+
# Get the redis_container fixture
172+
redis_container = request.getfixturevalue('redis_container')
173+
return {
174+
'host': redis_container.get_container_host_ip(),
175+
'port': redis_container.get_exposed_port(6379),
176+
'db': 0,
177+
'password': None, # Default Redis container has no password
178+
'max_connections': 10,
179+
'batch_size': 100,
180+
'pipeline_size': 500,
181+
}
182+
else:
183+
# Fall back to manual config from environment
184+
return request.getfixturevalue('redis_config')
185+
186+
94187
@pytest.fixture(scope='session')
95188
def delta_test_env():
96189
"""Create Delta Lake test environment for the session"""

0 commit comments

Comments
 (0)