Skip to content

Commit b436b46

Browse files
fordNfordN
committed
tests: use testcontainers for running integration tests
- only required for testing the Redis and PostgreSQL loaders, the other data systems use the file system and don't require a binary to be run - also updated testing documentation in README
1 parent 6df54ac commit b436b46

File tree

4 files changed

+247
-77
lines changed

4 files changed

+247
-77
lines changed

‎README.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# Python Amp Client
22

33
[![Unit tests status](https://github.com/edgeandnode/amp-python/actions/workflows/unit-tests.yml/badge.svg?event=push)](https://github.com/edgeandnode/amp-python/actions/workflows/unit-tests.yml)
4+
[![Integration tests status](https://github.com/edgeandnode/amp-python/actions/workflows/integration-tests.yml/badge.svg?event=push)](https://github.com/edgeandnode/amp-python/actions/workflows/integration-tests.yml)
45
[![Formatting status](https://github.com/edgeandnode/amp-python/actions/workflows/ruff.yml/badge.svg?event=push)](https://github.com/edgeandnode/amp-python/actions/workflows/ruff.yml)
56

67

@@ -51,11 +52,87 @@ You can then use it in your python scripts, apps or notebooks.
5152
The project is set up to use the [`pytest`](https://docs.pytest.org/en/stable/) testing framework.
5253
It follows [standard python test discovery rules](https://docs.pytest.org/en/stable/explanation/goodpractices.html#test-discovery).
5354

55+
## Quick Test Commands
56+
5457
Run all tests
5558
```bash
5659
uv run pytest
5760
```
5861

62+
Run only unit tests (fast, no external dependencies)
63+
```bash
64+
make test-unit
65+
```
66+
67+
Run integration tests with automatic container setup
68+
```bash
69+
make test-integration
70+
```
71+
72+
Run all tests with coverage
73+
```bash
74+
make test-all
75+
```
76+
77+
## Integration Testing
78+
79+
Integration tests can run in two modes:
80+
81+
### 1. Automatic Container Mode (Default)
82+
The integration tests will automatically spin up PostgreSQL and Redis containers using testcontainers. This is the default mode and requires Docker to be installed and running.
83+
84+
```bash
85+
# Run integration tests with automatic containers
86+
uv run pytest tests/integration/ -m integration
87+
```
88+
89+
**Note**: The configuration automatically disables Ryuk (testcontainers cleanup container) to avoid Docker connectivity issues. If you need Ryuk enabled, set `TESTCONTAINERS_RYUK_DISABLED=false`.
90+
91+
### 2. Manual Setup Mode
92+
If you prefer to use your own database instances, you can disable testcontainers:
93+
94+
```bash
95+
# Disable testcontainers and use manual configuration
96+
export USE_TESTCONTAINERS=false
97+
98+
# Configure your database connections
99+
export POSTGRES_HOST=localhost
100+
export POSTGRES_PORT=5432
101+
export POSTGRES_DB=test_amp
102+
export POSTGRES_USER=postgres
103+
export POSTGRES_PASSWORD=yourpassword
104+
105+
export REDIS_HOST=localhost
106+
export REDIS_PORT=6379
107+
export REDIS_PASSWORD=yourpassword # Optional
108+
109+
# Run tests
110+
uv run pytest tests/integration/ -m integration
111+
```
112+
113+
For manual setup, you can use the provided Makefile commands:
114+
```bash
115+
# Start test databases manually
116+
make test-setup
117+
118+
# Run tests
119+
make test-integration
120+
121+
# Clean up databases
122+
make test-cleanup
123+
```
124+
125+
## Loader-Specific Tests
126+
127+
Run tests for specific loaders:
128+
```bash
129+
make test-postgresql # PostgreSQL tests
130+
make test-redis # Redis tests
131+
make test-deltalake # Delta Lake tests
132+
make test-iceberg # Iceberg tests
133+
make test-lmdb # LMDB tests
134+
```
135+
59136
# Linting and formatting
60137

61138
Ruff is configured to be used for linting and formatting of this project.

‎tests/conftest.py‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,26 @@
1616

1717
logging.basicConfig(level=logging.INFO)
1818

19+
# Control whether to use testcontainers
20+
USE_TESTCONTAINERS = os.getenv('USE_TESTCONTAINERS', 'true').lower() == 'true'
21+
22+
# Disable Ryuk if not explicitly enabled (solves Docker connectivity issues)
23+
if 'TESTCONTAINERS_RYUK_DISABLED' not in os.environ:
24+
os.environ['TESTCONTAINERS_RYUK_DISABLED'] = 'true'
25+
26+
# Import testcontainers conditionally
27+
if USE_TESTCONTAINERS:
28+
try:
29+
from testcontainers.postgres import PostgresContainer
30+
from testcontainers.redis import RedisContainer
31+
32+
TESTCONTAINERS_AVAILABLE = True
33+
except ImportError:
34+
TESTCONTAINERS_AVAILABLE = False
35+
logging.warning('Testcontainers not available. Falling back to manual configuration.')
36+
else:
37+
TESTCONTAINERS_AVAILABLE = False
38+
1939

2040
# Shared configuration fixtures
2141
@pytest.fixture(scope='session')
@@ -91,6 +111,75 @@ def test_config():
91111
}
92112

93113

114+
# Testcontainers fixtures
115+
@pytest.fixture(scope='session')
116+
def postgres_container():
117+
"""PostgreSQL container for integration tests"""
118+
if not TESTCONTAINERS_AVAILABLE:
119+
pytest.skip('Testcontainers not available')
120+
121+
container = PostgresContainer(image='postgres:13', username='test_user', password='test_pass', dbname='test_db')
122+
container.start()
123+
124+
yield container
125+
126+
container.stop()
127+
128+
129+
@pytest.fixture(scope='session')
130+
def redis_container():
131+
"""Redis container for integration tests"""
132+
if not TESTCONTAINERS_AVAILABLE:
133+
pytest.skip('Testcontainers not available')
134+
135+
container = RedisContainer(image='redis:7-alpine')
136+
container.start()
137+
138+
yield container
139+
140+
container.stop()
141+
142+
143+
@pytest.fixture(scope='session')
144+
def postgresql_test_config(request):
145+
"""PostgreSQL configuration from testcontainer or environment"""
146+
if TESTCONTAINERS_AVAILABLE and USE_TESTCONTAINERS:
147+
# Get the postgres_container fixture
148+
postgres_container = request.getfixturevalue('postgres_container')
149+
return {
150+
'host': postgres_container.get_container_host_ip(),
151+
'port': postgres_container.get_exposed_port(5432),
152+
'database': 'test_db',
153+
'user': 'test_user',
154+
'password': 'test_pass',
155+
'max_connections': 10,
156+
'batch_size': 10000,
157+
}
158+
else:
159+
# Fall back to manual config from environment
160+
return request.getfixturevalue('postgresql_config')
161+
162+
163+
@pytest.fixture(scope='session')
164+
def redis_test_config(request):
165+
"""Redis configuration from testcontainer or environment"""
166+
if TESTCONTAINERS_AVAILABLE and USE_TESTCONTAINERS:
167+
# Get the redis_container fixture
168+
redis_container = request.getfixturevalue('redis_container')
169+
return {
170+
'host': redis_container.get_container_host_ip(),
171+
'port': redis_container.get_exposed_port(6379),
172+
'db': 0,
173+
'password': None, # Default Redis container has no password
174+
'max_connections': 10,
175+
'batch_size': 100,
176+
'pipeline_size': 500,
177+
}
178+
else:
179+
# Fall back to manual config from environment
180+
return request.getfixturevalue('redis_config')
181+
182+
94183
@pytest.fixture(scope='session')
95184
def delta_test_env():
96185
"""Create Delta Lake test environment for the session"""

0 commit comments

Comments
 (0)