Skip to content

Commit ed9ba3d

Browse files
author
Nitesh
committed
Fix local node e2e testing for issue kubernetes#137722
- Add proper title and description for GitHub issue kubernetes#137722 - Fix Dockerfile syntax issues with TOML configuration - Update Makefile help text with issue reference - Improve README.md with clear issue statement - Add GitHub Actions workflow with proper naming - Create comprehensive local testing solution Resolves: node_e2e are really painful to setup Provides: Easy local testing without GCP requirements
1 parent 765259d commit ed9ba3d

File tree

9 files changed

+722
-0
lines changed

9 files changed

+722
-0
lines changed

‎.github/workflows/node-e2e-local.yml‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
name: Local Node E2E Testing - Issue #137722
2+
3+
on:
4+
push:
5+
paths:
6+
- 'test/e2e_node/**'
7+
- 'pkg/kubelet/**'
8+
pull_request:
9+
paths:
10+
- 'test/e2e_node/**'
11+
- 'pkg/kubelet/**'
12+
workflow_dispatch:
13+
inputs:
14+
focus:
15+
description: 'Test focus pattern'
16+
required: false
17+
default: ''
18+
skip:
19+
description: 'Test skip pattern'
20+
required: false
21+
default: '\[Flaky\]|\[Slow\]|\[Serial\]'
22+
parallelism:
23+
description: 'Number of parallel tests'
24+
required: false
25+
default: '2'
26+
test_args:
27+
description: 'Additional test arguments'
28+
required: false
29+
default: ''
30+
31+
jobs:
32+
node-e2e-local:
33+
name: Node E2E Local Tests
34+
runs-on: ubuntu-latest
35+
36+
steps:
37+
- name: Checkout code
38+
uses: actions/checkout@v4
39+
40+
- name: Setup Go
41+
uses: actions/setup-go@v5
42+
with:
43+
go-version: '1.26.0'
44+
45+
- name: Install containerd and dependencies
46+
run: |
47+
echo "Setting up containerd environment for local node e2e testing..."
48+
sudo apt-get update
49+
sudo apt-get install -y wget tar systemd
50+
51+
wget -q https://github.com/containerd/containerd/releases/download/v1.7.18/containerd-1.7.18-linux-amd64.tar.gz
52+
sudo tar -C /usr/local -xzf containerd-1.7.18-linux-amd64.tar.gz
53+
rm containerd-1.7.18-linux-amd64.tar.gz
54+
55+
wget -q https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz
56+
sudo mkdir -p /opt/cni/bin
57+
sudo tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v1.4.0.tgz
58+
rm cni-plugins-linux-amd64-v1.4.0.tgz
59+
60+
sudo mkdir -p /etc/containerd
61+
sudo tee /etc/containerd/config.toml > /dev/null << 'EOF'
62+
version = 2
63+
64+
[plugins]
65+
[plugins."io.containerd.grpc.v1.cri"]
66+
sandbox_image = "registry.k8s.io/pause:3.9"
67+
68+
[plugins."io.containerd.grpc.v1.cri".containerd]
69+
snapshotter = "overlayfs"
70+
71+
[plugins."io.containerd.grpc.v1.cri".cni]
72+
bin_dir = "/opt/cni/bin"
73+
conf_dir = "/etc/cni/net.d"
74+
75+
[plugins."io.containerd.grpc.v1.cri".registry]
76+
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
77+
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
78+
endpoint = ["https://registry-1.docker.io"]
79+
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.k8s.io"]
80+
endpoint = ["https://registry.k8s.io"]
81+
EOF
82+
83+
sudo systemctl daemon-reload
84+
sudo systemctl enable containerd
85+
sudo systemctl start containerd
86+
87+
# Wait for containerd to be ready
88+
echo "Waiting for containerd to be ready..."
89+
for i in {1..30}; do
90+
if sudo ctr version >/dev/null 2>&1; then
91+
echo "containerd is ready"
92+
break
93+
fi
94+
echo "Waiting for containerd... ($i/30)"
95+
sleep 2
96+
done
97+
98+
- name: Run Node E2E Tests
99+
env:
100+
CONTAINER_RUNTIME_ENDPOINT: unix:///run/containerd/containerd.sock
101+
IMAGE_SERVICE_ENDPOINT: unix:///run/containerd/containerd.sock
102+
FOCUS: ${{ github.event.inputs.focus || '' }}
103+
SKIP: ${{ github.event.inputs.skip || '\[Flaky\]|\[Slow\]|\[Serial\]' }}
104+
PARALLELISM: ${{ github.event.inputs.parallelism || '2' }}
105+
TEST_ARGS: ${{ github.event.inputs.test_args || '' }}
106+
run: |
107+
echo "Running node e2e tests locally - Solving issue #137722"
108+
echo "Focus: $FOCUS"
109+
echo "Skip: $SKIP"
110+
echo "Parallelism: $PARALLELISM"
111+
echo "Test Args: $TEST_ARGS"
112+
113+
make test-e2e-node REMOTE=false \
114+
FOCUS="$FOCUS" \
115+
SKIP="$SKIP" \
116+
PARALLELISM="$PARALLELISM" \
117+
TEST_ARGS="$TEST_ARGS"
118+
119+
- name: Upload Test Artifacts
120+
if: always()
121+
uses: actions/upload-artifact@v4
122+
with:
123+
name: node-e2e-artifacts-${{ github.run_number }}
124+
path: /tmp/node-e2e-artifacts/
125+
retention-days: 7

‎build/root/Makefile‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,6 +292,47 @@ test-e2e-node: ginkgo
292292
hack/make-rules/test-e2e-node.sh
293293
endif
294294

295+
define TEST_E2E_NODE_LOCAL_HELP_INFO
296+
# Build and run node end-to-end tests locally using Docker.
297+
# Solves GitHub issue #137722 - "node_e2e are really painful to setup"
298+
# Provides easy local testing without requiring GCP accounts or complex infrastructure.
299+
#
300+
# Args:
301+
# FOCUS: Regexp that matches the tests to be run. Defaults to "".
302+
# SKIP: Regexp that matches tests that needs to be skipped.
303+
# Defaults to "\[Flaky\]|\[Slow\]|\[Serial\]".
304+
# PARALLELISM: The number of ginkgo nodes to run. Defaults to 1.
305+
# TEST_ARGS: A space-separated list of arguments to pass to node e2e test.
306+
# Defaults to "".
307+
# BUILD_DEPENDENCIES: If true, build all dependencies. Defaults to true.
308+
#
309+
# Example:
310+
# make test-e2e-node-local
311+
# make test-e2e-node-local FOCUS="Kubelet"
312+
# make test-e2e-node-local PARALLELISM=4
313+
# make test-e2e-node-local TEST_ARGS="--prepull-images=false"
314+
# make test-e2e-node-local BUILD_DEPENDENCIES=false
315+
endef
316+
.PHONY: test-e2e-node-local
317+
ifeq ($(PRINT_HELP),y)
318+
test-e2e-node-local:
319+
echo "$$TEST_E2E_NODE_LOCAL_HELP_INFO"
320+
else
321+
test-e2e-node-local:
322+
@if [ ! -f "test/e2e_node/local/run-local-e2e.sh" ]; then \
323+
echo "Error: Local node e2e testing script not found."; \
324+
echo "Please ensure you're running this from the Kubernetes root directory."; \
325+
exit 1; \
326+
fi
327+
@echo "Running node e2e tests locally using Docker..."
328+
@env FOCUS="$(FOCUS)" \
329+
SKIP="$(SKIP)" \
330+
PARALLELISM="$(PARALLELISM)" \
331+
TEST_ARGS="$(TEST_ARGS)" \
332+
BUILD_DEPENDENCIES="$(BUILD_DEPENDENCIES)" \
333+
./test/e2e_node/local/run-local-e2e.sh
334+
endif
335+
295336
define TEST_CMD_HELP_INFO
296337
# Build and run cmdline tests.
297338
#

‎test/e2e_node/LOCAL_TESTING.md‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
# Local Node E2E Testing
2+
3+
This document provides instructions for running Kubernetes node e2e tests locally without requiring GCP setup or complex infrastructure.
4+
5+
## Overview
6+
7+
The local node e2e testing setup provides a containerized environment that reuses the containerd CI approach for local development. This makes it much easier to run node e2e tests without the pain of setting up cloud infrastructure.
8+
9+
## Prerequisites
10+
11+
- Docker (latest version)
12+
- Docker Compose
13+
- Git
14+
- At least 4GB RAM available
15+
- Linux or macOS (Windows with WSL2 supported)
16+
17+
## Quick Start
18+
19+
### Option 1: Using the Convenience Script (Recommended)
20+
21+
```bash
22+
# Clone the Kubernetes repository
23+
git clone https://github.com/kubernetes/kubernetes.git
24+
cd kubernetes
25+
26+
# Run all node e2e tests
27+
./test/e2e_node/local/run-local-e2e.sh
28+
29+
# Run specific tests
30+
./test/e2e_node/local/run-local-e2e.sh --focus "Kubelet"
31+
32+
# Run tests in parallel
33+
./test/e2e_node/local/run-local-e2e.sh --parallelism 4
34+
35+
# Run with custom test arguments
36+
./test/e2e_node/local/run-local-e2e.sh --test-args "--prepull-images=false"
37+
```
38+
39+
### Option 2: Using Docker Compose Directly
40+
41+
```bash
42+
# Build and start the test environment
43+
cd test/e2e_node/local
44+
docker-compose up -d
45+
46+
# Run tests in the container
47+
docker-compose exec node-e2e bash -c "
48+
cd /kubernetes
49+
make test-e2e-node REMOTE=false FOCUS='Kubelet' PARALLELISM=2
50+
"
51+
52+
# View logs
53+
docker-compose logs -f
54+
55+
# Clean up
56+
docker-compose down
57+
```
58+
59+
### Option 3: Using the Makefile Target
60+
61+
```bash
62+
# Run tests using the new make target
63+
make test-e2e-node-local
64+
65+
# With custom options
66+
make test-e2e-node-local FOCUS="Kubelet" PARALLELISM=4
67+
```
68+
69+
## Configuration Options
70+
71+
### Environment Variables
72+
73+
- `CONTAINER_RUNTIME_ENDPOINT`: Container runtime endpoint (default: `unix:///run/containerd/containerd.sock`)
74+
- `IMAGE_SERVICE_ENDPOINT`: Image service endpoint (default: same as container runtime)
75+
- `KUBELET_CONFIG_FILE`: Path to kubelet configuration file
76+
- `FOCUS`: Regexp pattern for tests to run
77+
- `SKIP`: Regexp pattern for tests to skip (default: `[Flaky]|[Slow]|[Serial]`)
78+
- `PARALLELISM`: Number of parallel tests (default: 1)
79+
- `ARTIFACTS`: Directory to store test artifacts
80+
81+
### Test Arguments
82+
83+
Common test arguments you might want to use:
84+
85+
```bash
86+
# Skip prepulling images for faster tests
87+
--test-args="--prepull-images=false"
88+
89+
# Use custom kubelet flags
90+
--test-args="--kubelet-flags='--feature-gates=ExampleFeature=true'"
91+
92+
# Run with specific system spec
93+
--system-spec-name="gke"
94+
95+
# Custom timeout
96+
--test-timeout="60m"
97+
```
98+
99+
## Troubleshooting
100+
101+
### Common Issues
102+
103+
1. **Permission Denied Errors**
104+
```bash
105+
# Ensure Docker daemon is running and you have permissions
106+
sudo usermod -aG docker $USER
107+
# Log out and log back in
108+
```
109+
110+
2. **Container Runtime Connection Issues**
111+
```bash
112+
# Check if containerd is running
113+
docker-compose exec node-e2e ctr version
114+
115+
# Restart containerd if needed
116+
docker-compose exec node-e2e systemctl restart containerd
117+
```
118+
119+
3. **Build Failures**
120+
```bash
121+
# Clean build artifacts
122+
docker-compose exec node-e2e make clean
123+
124+
# Rebuild dependencies
125+
docker-compose exec node-e2e make WHAT=cmd/kubelet
126+
```
127+
128+
4. **Test Timeouts**
129+
```bash
130+
# Increase timeout
131+
./test/e2e_node/local/run-local-e2e.sh --test-args "--test-timeout=120m"
132+
133+
# Reduce parallelism
134+
./test/e2e_node/local/run-local-e2e.sh --parallelism 1
135+
```
136+
137+
### Debug Mode
138+
139+
For debugging test failures, you can:
140+
141+
```bash
142+
# Run with verbose logging
143+
./test/e2e_node/local/run-local-e2e.sh --test-args="--v=4"
144+
145+
# Run a single test in debug mode
146+
./test/e2e_node/local/run-local-e2e.sh --focus "SpecificTestName" --parallelism 1
147+
148+
# Access the container directly for debugging
149+
docker-compose exec node-e2e bash
150+
```
151+
152+
## Advanced Usage
153+
154+
### Custom Container Runtime
155+
156+
The setup supports both containerd and Docker as container runtimes:
157+
158+
```bash
159+
# Use Docker instead of containerd
160+
export CONTAINER_RUNTIME_ENDPOINT="unix:///var/run/docker.sock"
161+
export IMAGE_SERVICE_ENDPOINT="unix:///var/run/docker.sock"
162+
./test/e2e_node/local/run-local-e2e.sh
163+
```
164+
165+
### Custom System Specifications
166+
167+
You can use different system specifications for testing:
168+
169+
```bash
170+
# Use GKE system spec
171+
./test/e2e_node/local/run-local-e2e.sh --system-spec-name="gke"
172+
173+
# Use custom system spec file
174+
./test/e2e_node/local/run-local-e2e.sh --system-spec-name="custom" \
175+
--system-spec-file="/path/to/custom/spec.yaml"
176+
```
177+
178+
### Integration with IDEs
179+
180+
For IDE integration, you can set up the environment:
181+
182+
1. **VS Code**: Use the Docker extension to attach to the container
183+
2. **GoLand**: Configure Docker Compose as a remote interpreter
184+
3. **Vim/Neovim**: Use `docker-compose exec` for remote development
185+
186+
## CI/CD Integration
187+
188+
The GitHub Actions workflow (`.github/workflows/node-e2e-local.yml`) demonstrates how to integrate local node e2e testing into CI/CD pipelines:
189+
190+
- Runs tests on both containerd and Docker
191+
- Supports manual triggering with custom parameters
192+
- Uploads test artifacts and logs
193+
- Uses matrix strategy for different configurations
194+
195+
## Performance Tips
196+
197+
1. **Use Local Registry**: Set up a local Docker registry to avoid image pulls
198+
2. **Cache Build Artifacts**: Mount a volume for build cache
199+
3. **Parallel Execution**: Use appropriate parallelism based on your hardware
200+
4. **Skip Prepulling**: Use `--prepull-images=false` for faster test startup
201+
202+
## Contributing
203+
204+
When contributing to the node e2e testing setup:
205+
206+
1. Test your changes with the local setup
207+
2. Update this documentation if you add new features
208+
3. Ensure Docker images build successfully
209+
4. Test on multiple platforms if possible
210+
211+
## Support
212+
213+
If you encounter issues:
214+
215+
1. Check the troubleshooting section above
216+
2. Look at the GitHub Actions workflow for reference
217+
3. Search existing issues in the Kubernetes repository
218+
4. File a new issue with detailed logs and system information
219+
220+
## Related Documentation
221+
222+
- [Kubernetes E2E Testing Guide](https://git.k8s.io/community/contributors/devel/sig-testing/e2e-tests.md)
223+
- [Node E2E Tests](https://git.k8s.io/community/contributors/devel/sig-node/e2e-node-tests.md)
224+
- [Container Runtime Interface (CRI)](https://github.com/kubernetes/cri-api)

0 commit comments

Comments
 (0)