Skip to content

Commit 6b9ecf7

Browse files
GSMLG-BOTGSMLG-BOT
committed
feat: add comprehensive e2e test suite with multi-node distributed testing
## Summary Introduces a completely separate end-to-end testing infrastructure independent from unit tests: - **Multi-node distributed tests**: 15 tests using LocalCluster for real Erlang clusters - Leader election and failover scenarios (3 tests) - Network partition handling (4 tests) - Data consistency and replication (5 tests) - Node failure and recovery (3 tests) - **Complete separation from unit tests**: Uses MIX_ENV=e2e_test with isolated dependencies - Separate configuration in config/e2e_test.exs - Independent data directories - Dedicated Mix aliases (mix test.e2e, mix test.e2e.distributed) - **Comprehensive helper utilities**: ClusterHelper module (500+ lines) - Cluster lifecycle management (start, stop, restart) - Network partition simulation and healing - Node failure injection - Raft leader detection and waiting - **GitHub Actions CI/CD integration**: Dedicated e2e-test.yml workflow - Runs distributed tests on every push/PR (~5 min) - Nightly runs with full test suite - Manual workflow dispatch support - **Extensive documentation** (600+ lines total) - Quick start guide (2-minute setup) - Comprehensive README with examples - Setup summary document - Updated CLAUDE.md project documentation ## Dependencies Added - local_cluster ~> 2.0 (only e2e_test environment) - httpoison ~> 2.0 (only e2e_test environment) ## Files Created - e2e_test/test_helper.exs - e2e_test/support/e2e_cluster_helper.ex - e2e_test/distributed/leader_election_test.exs - e2e_test/distributed/network_partition_test.exs - e2e_test/distributed/data_consistency_test.exs - e2e_test/distributed/node_failure_test.exs - e2e_test/README.md - e2e_test/QUICKSTART.md - config/e2e_test.exs - .github/workflows/e2e-test.yml - E2E_SETUP_SUMMARY.md ## Files Modified - mix.exs: Added dependencies, elixirc_paths, and Mix aliases - CLAUDE.md: Added e2e testing documentation - .gitignore: Added e2e test artifacts ## Quick Start ```bash # One-time setup epmd -daemon MIX_ENV=e2e_test mix deps.get MIX_ENV=e2e_test mix compile # Run tests mix test.e2e.distributed ``` ## Testing Run e2e tests locally: ```bash # All distributed tests mix test.e2e.distributed # Specific test file MIX_ENV=e2e_test mix test e2e_test/distributed/leader_election_test.exs # Verbose output MIX_ENV=e2e_test mix test e2e_test/ --trace ``` GitHub Actions will run automatically on: - Every push/PR to main/develop - Nightly at 2 AM UTC - Manual workflow dispatch
1 parent 6a9edb6 commit 6b9ecf7

File tree

15 files changed

+1742
-5
lines changed

15 files changed

+1742
-5
lines changed

‎.github/workflows/e2e-test.yml‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
name: E2E Tests
2+
3+
on:
4+
push:
5+
branches: [ main, develop ]
6+
pull_request:
7+
branches: [ main ]
8+
# Allow manual trigger
9+
workflow_dispatch:
10+
# Run nightly
11+
schedule:
12+
- cron: '0 2 * * *' # Run at 2 AM UTC daily
13+
14+
jobs:
15+
e2e-distributed:
16+
name: E2E Distributed Tests
17+
runs-on: ubuntu-latest
18+
timeout-minutes: 30
19+
20+
strategy:
21+
fail-fast: false
22+
matrix:
23+
otp: ['28']
24+
elixir: ['1.18']
25+
26+
steps:
27+
- name: Checkout code
28+
uses: actions/checkout@v5
29+
30+
- name: Set up Elixir
31+
uses: erlef/setup-beam@v1
32+
with:
33+
elixir-version: ${{ matrix.elixir }}
34+
otp-version: ${{ matrix.otp }}
35+
36+
- name: Cache dependencies
37+
uses: actions/cache@v4
38+
with:
39+
path: |
40+
deps
41+
_build
42+
key: ${{ runner.os }}-e2e-${{ matrix.otp }}-${{ matrix.elixir }}-mix-${{ hashFiles('**/mix.lock') }}
43+
restore-keys: |
44+
${{ runner.os }}-e2e-${{ matrix.otp }}-${{ matrix.elixir }}-mix-
45+
46+
- name: Install dependencies
47+
run: |
48+
mix local.hex --force
49+
mix local.rebar --force
50+
MIX_ENV=e2e_test mix deps.get
51+
52+
- name: Compile e2e tests
53+
run: MIX_ENV=e2e_test mix compile
54+
env:
55+
MIX_ENV: e2e_test
56+
57+
- name: Start EPMD (Erlang Port Mapper Daemon)
58+
run: epmd -daemon
59+
60+
- name: Run E2E Distributed Tests
61+
run: MIX_ENV=e2e_test mix test e2e_test/distributed/ --trace
62+
env:
63+
MIX_ENV: e2e_test
64+
65+
- name: Upload test results
66+
if: failure()
67+
uses: actions/upload-artifact@v4
68+
with:
69+
name: e2e-test-results-${{ matrix.otp }}-${{ matrix.elixir }}
70+
path: |
71+
_build/e2e_test/test-junit-report.xml
72+
data/e2e_test/
73+
74+
e2e-docker:
75+
name: E2E Docker Tests
76+
runs-on: ubuntu-latest
77+
timeout-minutes: 45
78+
# Only run Docker tests on schedule or manual trigger (slower)
79+
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
80+
81+
strategy:
82+
fail-fast: false
83+
matrix:
84+
otp: ['28']
85+
elixir: ['1.18']
86+
87+
steps:
88+
- name: Checkout code
89+
uses: actions/checkout@v5
90+
91+
- name: Set up Docker Buildx
92+
uses: docker/setup-buildx-action@v3
93+
94+
- name: Set up Elixir
95+
uses: erlef/setup-beam@v1
96+
with:
97+
elixir-version: ${{ matrix.elixir }}
98+
otp-version: ${{ matrix.otp }}
99+
100+
- name: Cache dependencies
101+
uses: actions/cache@v4
102+
with:
103+
path: |
104+
deps
105+
_build
106+
key: ${{ runner.os }}-e2e-docker-${{ matrix.otp }}-${{ matrix.elixir }}-mix-${{ hashFiles('**/mix.lock') }}
107+
restore-keys: |
108+
${{ runner.os }}-e2e-docker-${{ matrix.otp }}-${{ matrix.elixir }}-mix-
109+
110+
- name: Install dependencies
111+
run: |
112+
mix local.hex --force
113+
mix local.rebar --force
114+
MIX_ENV=e2e_test mix deps.get
115+
116+
- name: Run E2E Docker Tests
117+
run: MIX_ENV=e2e_test mix test e2e_test/docker/ --trace
118+
env:
119+
MIX_ENV: e2e_test
120+
121+
- name: Upload Docker test results
122+
if: failure()
123+
uses: actions/upload-artifact@v4
124+
with:
125+
name: e2e-docker-test-results-${{ matrix.otp }}-${{ matrix.elixir }}
126+
path: |
127+
_build/e2e_test/test-junit-report.xml
128+
129+
e2e-summary:
130+
name: E2E Test Summary
131+
runs-on: ubuntu-latest
132+
needs: [e2e-distributed]
133+
if: always()
134+
135+
steps:
136+
- name: Check test results
137+
run: |
138+
echo "E2E Distributed Tests: ${{ needs.e2e-distributed.result }}"
139+
if [ "${{ needs.e2e-distributed.result }}" != "success" ]; then
140+
echo "❌ E2E distributed tests failed"
141+
exit 1
142+
else
143+
echo "✅ E2E distributed tests passed"
144+
fi

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ concord-*.tar
2323
/tmp/
2424
plts/
2525
nonode@nohost/
26+
concord_e2e_*
27+
/data/
2628

2729
# devenv
2830
.devenv/

‎CLAUDE.md‎

Lines changed: 52 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Concord is a distributed, strongly-consistent **embedded** key-value store built
1515
# Compile the project
1616
mix compile
1717

18-
# Run all tests
18+
# Run unit tests (fast, isolated)
1919
mix test
2020

2121
# Run specific test file
@@ -27,6 +27,15 @@ mix test test/concord_test.exs:42
2727
# Run with coverage
2828
mix test --cover
2929

30+
# Run e2e tests (multi-node, distributed scenarios)
31+
mix test.e2e
32+
33+
# Run only distributed e2e tests
34+
mix test.e2e.distributed
35+
36+
# Run specific e2e test file
37+
MIX_ENV=e2e_test mix test e2e_test/distributed/leader_election_test.exs
38+
3039
# Run linting
3140
mix credo
3241

@@ -192,15 +201,16 @@ The configuration follows standard Elixir patterns with environment-specific fil
192201

193202
## Testing Strategy
194203

195-
Test categories:
204+
### Unit Tests (`test/`)
205+
206+
Fast, isolated tests for individual components:
207+
196208
- **Unit Tests**: Basic CRUD operations, validation (e.g., `test/concord_test.exs`)
197209
- **Feature Tests**: TTL, compression, bulk operations, queries, indexes
198210
- **Auth Tests**: Token management, authorization flows
199211
- **RBAC Tests**: Role management, ACL rules, permission checking (`test/concord/rbac_test.exs` - 34 tests)
200212
- **Multi-Tenancy Tests**: Tenant lifecycle, quotas, usage tracking (`test/concord/multi_tenancy_test.exs` - 41 tests)
201213
- **Telemetry Tests**: Event emission verification
202-
- **Integration Tests**: Multi-node scenarios, HTTP API
203-
- **Performance Tests**: Benchmarks in `test/performance/`
204214

205215
**Important Testing Notes:**
206216
- Tests use `Concord.TestHelper.start_test_cluster()` to initialize Ra cluster
@@ -210,6 +220,39 @@ Test categories:
210220
- State machine version changes require cluster restart or data cleanup
211221
- Tests run with `async: false` to avoid Ra cluster conflicts
212222

223+
### E2E Tests (`e2e_test/`)
224+
225+
**Separate test suite** for distributed, multi-node scenarios:
226+
227+
- **Leader Election**: Raft leader election and failover (`e2e_test/distributed/leader_election_test.exs`)
228+
- **Network Partitions**: Split-brain, quorum behavior, partition healing (`e2e_test/distributed/network_partition_test.exs`)
229+
- **Data Consistency**: Replication, concurrent writes, TTL consistency (`e2e_test/distributed/data_consistency_test.exs`)
230+
- **Node Failures**: Crash tolerance, recovery, log replay (`e2e_test/distributed/node_failure_test.exs`)
231+
232+
**E2E Testing Environment:**
233+
- Uses `MIX_ENV=e2e_test` (separate from unit tests)
234+
- Spawns actual Erlang nodes with LocalCluster (3-5 nodes per test)
235+
- Longer execution time: ~5 minutes for full suite
236+
- Resource intensive: ~1GB RAM, requires EPMD running
237+
- See `e2e_test/README.md` for detailed documentation
238+
239+
**Running E2E Tests:**
240+
```bash
241+
# Run all e2e tests
242+
mix test.e2e
243+
244+
# Run only distributed tests
245+
mix test.e2e.distributed
246+
247+
# Run specific e2e test
248+
MIX_ENV=e2e_test mix test e2e_test/distributed/leader_election_test.exs
249+
250+
# Run with verbose output
251+
MIX_ENV=e2e_test mix test e2e_test/ --trace
252+
```
253+
254+
### Performance Tests
255+
213256
**Running Performance Benchmarks:**
214257
```bash
215258
# Run all benchmarks
@@ -235,7 +278,7 @@ mix run test/performance/kv_operations_benchmark.exs
235278
- Track leader changes via telemetry events
236279

237280
### Important File Locations
238-
- Raft logs and snapshots: `{data_dir}/` (default: `./data/dev` in development)
281+
- Raft logs and snapshots: `{data_dir}/` (default: `./data/dev` in development, `./data/e2e_test` in e2e tests)
239282
- Ra data directory: `nonode@nohost/` (gitignored - test artifacts)
240283
- ETS tables:
241284
- `:concord_store` - Main KV storage
@@ -249,6 +292,10 @@ mix run test/performance/kv_operations_benchmark.exs
249292
- Audit logs: `audit_logs/` directory (JSONL format)
250293
- RBAC module: `lib/concord/rbac.ex`
251294
- Multi-tenancy: `lib/concord/multi_tenancy.ex`, `lib/concord/multi_tenancy/rate_limiter.ex`
295+
- **E2E Tests**: `e2e_test/` directory (separate from `test/`)
296+
- `e2e_test/support/e2e_cluster_helper.ex` - Multi-node cluster utilities
297+
- `e2e_test/distributed/` - Distributed system tests
298+
- `config/e2e_test.exs` - E2E test configuration
252299

253300
## Feature-Specific Guidance
254301

0 commit comments

Comments
 (0)