Skip to content

Commit f86fcb9

Browse files
josecelanojosecelano
committed
docs: [#146] add specification for updating pre-commit script for GitHub runner-compatible E2E tests
- Add comprehensive specification document for issue #146 - Update EPIC #112 with new sub-issue 1-6 and time estimate - Add 'Preinstalling' to project dictionary for spell checking - Total EPIC estimate updated: 22-33.5 hours (was 21-32 hours) Related to #112
1 parent 256d674 commit f86fcb9

File tree

3 files changed

+305
-1
lines changed

3 files changed

+305
-1
lines changed

‎docs/issues/112-epic-refactor-and-improve-e2e-test-execution.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,10 @@ This epic is broken down into sub-issues:
1717
1-3. #119 - Update CI Workflows and Remove Bash Scripts (2-4 hours)
1818
1-4. #120 - Configure GitHub Copilot Agent Environment (2-3 hours)
1919
1-5. #121 - Install Git Pre-Commit Hooks for Copilot Agent (2-3 hours)
20+
1-6. #146 - Update Pre-Commit Script for GitHub Runner-Compatible E2E Tests (1-1.5 hours)
2021
```
2122

22-
**Total Estimated Time**: 21-32 hours split across multiple focused sub-issues
23+
**Total Estimated Time**: 22-33.5 hours split across multiple focused sub-issues
2324

2425
## Overview
2526

‎docs/issues/146-1-6-update-precommit-script-for-github-runner-compatible-e2e-tests.md‎

Lines changed: 302 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,302 @@
1+
# Update Pre-Commit Script for GitHub Runner-Compatible E2E Tests
2+
3+
**Issue**: [#146](https://github.com/torrust/torrust-tracker-deployer/issues/146)
4+
**Parent Epic**: [#112](https://github.com/torrust/torrust-tracker-deployer/issues/112) - Refactor and Improve E2E Test Execution
5+
**Related**:
6+
7+
- [#121](https://github.com/torrust/torrust-tracker-deployer/issues/121) - Install Git Pre-Commit Hooks for Copilot Agent
8+
- [#120](https://github.com/torrust/torrust-tracker-deployer/issues/120) - Configure GitHub Copilot Agent Environment
9+
- [docs/e2e-testing.md](../e2e-testing.md) - E2E Testing Documentation
10+
- [GitHub Docs: Preinstalling tools in Copilot's environment](https://docs.github.com/en/enterprise-cloud@latest/copilot/how-tos/use-copilot-agents/coding-agent/customize-the-agent-environment#preinstalling-tools-or-dependencies-in-copilots-environment)
11+
12+
## Overview
13+
14+
Update the pre-commit verification script (`scripts/pre-commit.sh`) to run GitHub runner-compatible E2E tests instead of the full E2E test suite. This enables GitHub Copilot agents to successfully execute pre-commit checks in GitHub Actions environments where LXD VM network connectivity is limited.
15+
16+
## Problem Statement
17+
18+
### Current Situation
19+
20+
The pre-commit script currently runs the E2E full test suite:
21+
22+
```bash
23+
"Running comprehensive E2E tests|All E2E tests passed|(Filtering logs to WARNING level and above - this may take a few minutes)|RUST_LOG=warn|cargo run --bin e2e-tests-full"
24+
```
25+
26+
**Issues**:
27+
28+
- `e2e-tests-full` requires LXD VMs with full network connectivity
29+
- GitHub Actions runners experience network connectivity issues inside LXD VMs
30+
- Cannot download Docker GPG keys, package repositories timeout
31+
- GitHub Copilot agents run in GitHub Actions infrastructure and **cannot execute** this test
32+
33+
### Why E2E Full Tests Can't Run on GitHub Runners
34+
35+
**Root Cause**: Known GitHub Actions networking limitations with nested virtualization:
36+
37+
- [GitHub Issue 13003](https://github.com/actions/runner-images/issues/13003) - Network connectivity issues with LXD VMs
38+
- [GitHub Issue 1187](https://github.com/actions/runner-images/issues/1187) - Original networking issue
39+
- [GitHub Issue 2890](https://github.com/actions/runner-images/issues/2890) - Specific apt repository timeout issues
40+
41+
### Why We Keep E2E Full Tests
42+
43+
The E2E full test suite (`src/bin/e2e_tests_full.rs`) is valuable for **human developers** with local environments where:
44+
45+
- All dependencies are installed (LXD, OpenTofu, Ansible)
46+
- LXD VM networking works correctly
47+
- Can validate the complete deployment pipeline in a single run
48+
49+
## Goals
50+
51+
- [ ] Replace `e2e-tests-full` with GitHub runner-compatible E2E tests in pre-commit script
52+
- [ ] Maintain comprehensive E2E coverage through split test execution
53+
- [ ] Enable GitHub Copilot agents to run pre-commit checks successfully
54+
- [ ] Keep clear documentation about test suite purposes and limitations
55+
56+
## 🏗️ Architecture Requirements
57+
58+
**Script Location**: `scripts/pre-commit.sh`
59+
**Pattern**: Shell script configuration with step definitions
60+
**Integration**: Used by Git pre-commit hooks (see [#121](https://github.com/torrust/torrust-tracker-deployer/issues/121))
61+
62+
### Testing Strategy
63+
64+
**Current Approach**: Single E2E full test (local-only)
65+
**New Approach**: Split E2E test execution (GitHub runner-compatible)
66+
67+
**Test Suite Compatibility**:
68+
69+
- ✅ `e2e-provision-and-destroy-tests` - Uses LXD VMs but no nested networking
70+
- ✅ `e2e-config-tests` - Uses Docker containers with reliable networking
71+
- ❌ `e2e-tests-full` - Requires LXD VM networking (local development only)
72+
73+
## Specifications
74+
75+
### Test Execution Split
76+
77+
The pre-commit script should run **two separate E2E test commands** instead of one:
78+
79+
**Before** (current):
80+
81+
```bash
82+
"Running comprehensive E2E tests|All E2E tests passed|(Filtering logs to WARNING level and above - this may take a few minutes)|RUST_LOG=warn|cargo run --bin e2e-tests-full"
83+
```
84+
85+
**After** (proposed):
86+
87+
```bash
88+
"Running E2E provision and destroy tests|Provision and destroy tests passed|(Testing infrastructure lifecycle - this may take a few minutes)|RUST_LOG=warn|cargo run --bin e2e-provision-and-destroy-tests"
89+
"Running E2E configuration tests|Configuration tests passed|(Testing software installation and configuration)|RUST_LOG=warn|cargo run --bin e2e-config-tests"
90+
```
91+
92+
### Step Configuration Format
93+
94+
Each step follows the format:
95+
96+
```text
97+
"description|success_message|special_note|env_vars|command"
98+
```
99+
100+
**Key Points**:
101+
102+
- Two separate steps for better granularity
103+
- Clear descriptions matching test purposes
104+
- Appropriate timing notes for each test type
105+
- Same log level filtering (`RUST_LOG=warn`)
106+
107+
### Coverage Comparison
108+
109+
**E2E Full Tests Coverage**:
110+
111+
- Infrastructure provisioning (LXD VMs)
112+
- Cloud-init completion
113+
- Docker installation
114+
- Docker Compose installation
115+
- Infrastructure destruction
116+
117+
**Split Tests Coverage** (combined):
118+
119+
- ✅ Infrastructure provisioning (LXD VMs) - `e2e-provision-and-destroy-tests`
120+
- ✅ Cloud-init completion - `e2e-provision-and-destroy-tests`
121+
- ✅ Docker installation - `e2e-config-tests`
122+
- ✅ Docker Compose installation - `e2e-config-tests`
123+
- ✅ Infrastructure destruction - `e2e-provision-and-destroy-tests`
124+
125+
**Result**: Same coverage, GitHub runner-compatible execution
126+
127+
## Implementation Plan
128+
129+
### Phase 1: Update Pre-Commit Script (30 minutes)
130+
131+
- [ ] Task 1.1: Replace the E2E full test step with provision and destroy test step
132+
- [ ] Task 1.2: Add the configuration test step after provision and destroy test step
133+
- [ ] Task 1.3: Verify step numbering and total step count are correct
134+
- [ ] Task 1.4: Test the script locally to ensure both tests run successfully
135+
136+
### Phase 2: Documentation Updates (15 minutes)
137+
138+
- [ ] Task 2.1: Update `docs/contributing/commit-process.md` to document the split E2E test execution
139+
- [ ] Task 2.2: Update `.github/copilot-instructions.md` if needed
140+
- [ ] Task 2.3: Ensure documentation clarifies when to use E2E full tests vs split tests
141+
142+
### Phase 3: Validation (15 minutes)
143+
144+
- [ ] Task 3.1: Run `./scripts/pre-commit.sh` locally to verify both E2E tests execute
145+
- [ ] Task 3.2: Verify pre-commit script passes when both E2E tests succeed
146+
- [ ] Task 3.3: Verify pre-commit script fails when either E2E test fails
147+
- [ ] Task 3.4: Run shellcheck on the modified script
148+
149+
## Acceptance Criteria
150+
151+
> **Note for Contributors**: These criteria define what the PR reviewer will check. Use this as your pre-review checklist before submitting the PR to minimize back-and-forth iterations.
152+
153+
**Quality Checks**:
154+
155+
- [ ] Pre-commit checks pass: `./scripts/pre-commit.sh`
156+
- [ ] Shellcheck passes for `scripts/pre-commit.sh`
157+
158+
**Task-Specific Criteria**:
159+
160+
- [ ] Pre-commit script runs provision and destroy E2E tests
161+
- [ ] Pre-commit script runs configuration E2E tests
162+
- [ ] Both E2E test steps execute in sequence
163+
- [ ] Script succeeds when both tests pass
164+
- [ ] Script fails appropriately when either test fails
165+
- [ ] Step numbering and count are correct
166+
- [ ] Timing notes accurately describe each test's duration expectations
167+
- [ ] Documentation reflects the split E2E test approach
168+
169+
**Testing Criteria**:
170+
171+
- [ ] Verify locally: `./scripts/pre-commit.sh` completes successfully
172+
- [ ] Verify provision test step runs: Check for "Running E2E provision and destroy tests" message
173+
- [ ] Verify config test step runs: Check for "Running E2E configuration tests" message
174+
- [ ] Both test steps show success messages with timing
175+
176+
## Related Documentation
177+
178+
- [docs/e2e-testing.md](../e2e-testing.md) - E2E Testing Guide (documents test suite split strategy)
179+
- [docs/contributing/commit-process.md](../contributing/commit-process.md) - Commit Process (documents pre-commit requirements)
180+
- [.github/workflows/test-e2e-provision.yml](../../.github/workflows/test-e2e-provision.yml) - CI workflow for provision tests
181+
- [.github/workflows/test-e2e-config.yml](../../.github/workflows/test-e2e-config.yml) - CI workflow for config tests
182+
- [#120](https://github.com/torrust/torrust-tracker-deployer/issues/120) - Configure GitHub Copilot Agent Environment
183+
- [#121](https://github.com/torrust/torrust-tracker-deployer/issues/121) - Git pre-commit hook installation (depends on this issue)
184+
- [GitHub Actions Runner Images: Ubuntu 24.04](https://github.com/actions/runner-images/blob/ubuntu24/20251030.96/images/ubuntu/Ubuntu2404-Readme.md) - Pre-installed software on runners
185+
- [GitHub Docs: Preinstalling tools in Copilot's environment](https://docs.github.com/en/enterprise-cloud@latest/copilot/how-tos/use-copilot-agents/coding-agent/customize-the-agent-environment#preinstalling-tools-or-dependencies-in-copilots-environment) - Official documentation on customizing Copilot agent environment
186+
187+
## Implementation Details
188+
189+
### Script Modification
190+
191+
The `STEPS` array in `scripts/pre-commit.sh` currently contains 6 steps. After the modification, it will still contain 6 steps but with step 5 split into two separate steps:
192+
193+
**Current Steps**:
194+
195+
1. Checking for unused dependencies (cargo machete)
196+
2. Running linters
197+
3. Running tests
198+
4. Testing cargo documentation
199+
5. Running comprehensive E2E tests ← **This will be replaced**
200+
6. Running code coverage check
201+
202+
**New Steps**:
203+
204+
1. Checking for unused dependencies (cargo machete)
205+
2. Running linters
206+
3. Running tests
207+
4. Testing cargo documentation
208+
5. Running E2E provision and destroy tests ← **New**
209+
6. Running E2E configuration tests ← **New**
210+
7. Running code coverage check
211+
212+
**Total Steps**: 7 (was 6)
213+
214+
### Code Changes
215+
216+
Replace this line in the `STEPS` array:
217+
218+
```bash
219+
"Running comprehensive E2E tests|All E2E tests passed|(Filtering logs to WARNING level and above - this may take a few minutes)|RUST_LOG=warn|cargo run --bin e2e-tests-full"
220+
```
221+
222+
With these two lines:
223+
224+
```bash
225+
"Running E2E provision and destroy tests|Provision and destroy tests passed|(Testing infrastructure lifecycle - this may take a few minutes)|RUST_LOG=warn|cargo run --bin e2e-provision-and-destroy-tests"
226+
"Running E2E configuration tests|Configuration tests passed|(Testing software installation and configuration)|RUST_LOG=warn|cargo run --bin e2e-config-tests"
227+
```
228+
229+
### Human Developer Workflow
230+
231+
**For Local Development** (human developers):
232+
233+
- Can still run `cargo run --bin e2e-tests-full` manually for comprehensive single-run testing
234+
- Pre-commit hook will run split tests (same coverage, GitHub compatible)
235+
- Choice between convenience (manual full test) vs compatibility (automated split tests)
236+
237+
**For GitHub Copilot Agents**:
238+
239+
- Pre-commit hook always runs split tests
240+
- Both tests execute successfully in GitHub Actions environment
241+
- Comprehensive coverage without networking limitations
242+
243+
## Notes
244+
245+
### GitHub Actions Runner Pre-installed Dependencies
246+
247+
The GitHub Actions Ubuntu 24.04 runners come with many dependencies **already installed**:
248+
249+
- ✅ **Ansible 2.19.3** - Pre-installed ([runner image docs](https://github.com/actions/runner-images/blob/ubuntu24/20251030.96/images/ubuntu/Ubuntu2404-Readme.md))
250+
- ✅ **Rust 1.90.0 + Cargo 1.90.0** - Pre-installed
251+
- ✅ **LXD** - Pre-installed (verified working in CI)
252+
- ✅ **Docker** - Pre-installed (Docker 28.0.4, Docker Compose v2 2.38.2)
253+
- ✅ **cargo-machete** - Installed via dependency-installer package (issue #113-#117)
254+
- ✅ **OpenTofu** - Installed via dependency-installer package (issue #113-#117)
255+
256+
This means the split E2E tests (`e2e-provision-and-destroy-tests` and `e2e-config-tests`) have **all required dependencies available** in the GitHub Actions environment.
257+
258+
### Rationale for This Change
259+
260+
**Problem**: GitHub Copilot agents need to run pre-commit checks but cannot execute `e2e-tests-full` due to GitHub Actions networking limitations.
261+
262+
**Solution**: Use split E2E tests that provide the same coverage but are compatible with GitHub Actions infrastructure.
263+
264+
**Benefits**:
265+
266+
- ✅ Enables Copilot agents to run pre-commit checks
267+
- ✅ Maintains comprehensive E2E coverage
268+
- ✅ Aligns pre-commit script with CI workflows
269+
- ✅ Human developers can still use `e2e-tests-full` manually for convenience
270+
271+
### Why Not Remove E2E Full Tests?
272+
273+
The E2E full test binary (`src/bin/e2e_tests_full.rs`) provides value for human developers:
274+
275+
- **Convenience**: Single command to validate complete deployment pipeline
276+
- **Speed**: Faster than running two separate commands manually
277+
- **Local Development**: Works perfectly in local environments with proper networking
278+
- **Debugging**: Easier to debug full flow in one execution
279+
280+
It's kept as a **local development convenience tool**, not removed from the codebase.
281+
282+
### Alternative Considered: Conditional Execution
283+
284+
**Alternative**: Detect environment (local vs GitHub) and conditionally run different tests.
285+
286+
**Rejected Because**:
287+
288+
- ❌ Adds complexity to the script
289+
- ❌ Different behavior in different environments (confusing)
290+
- ❌ Split tests provide same coverage regardless of environment
291+
- ❌ Harder to maintain and reason about
292+
293+
**Better Approach**: Always run split tests in pre-commit hook, let developers manually run full tests when desired.
294+
295+
## Time Estimate
296+
297+
**Total Estimated Time**: 1-1.5 hours
298+
299+
- Script modification: 30 minutes
300+
- Documentation updates: 15 minutes
301+
- Testing and validation: 15 minutes
302+
- Buffer for edge cases: 15 minutes

‎project-words.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ pids
121121
pipefail
122122
postconditions
123123
preconfigured
124+
Preinstalling
124125
preinstalls
125126
prereq
126127
println

0 commit comments

Comments
 (0)