Testing & Integration

This document consolidates all information about running tests, integration tests, and development workflows for the Reaper project.

Quick Reference

All common tasks are available via make. Run make help for the full list.

Task	Command
Full CI check (recommended before push)	`make ci`
Unit tests	`make test`
Clippy (macOS)	`make clippy`
Clippy (Linux cross-check)	`make check-linux`
Coverage (Docker, CI-parity)	`make coverage`
Integration tests (full suite)	`make integration`
Integration tests (K8s only, skip cargo)	`make integration-quick`

Unit Tests

Run Rust tests natively on your machine:

cargo test

Tests run in a few seconds and provide immediate feedback. Use this for development iteration.

Test Modules

integration_basic_binary - Basic runtime functionality
integration_user_management - User/group handling (UID/GID switching, privilege dropping, umask)
integration_shim - Shim-specific tests
integration_io - FIFO stdout/stderr redirection
integration_exec - Exec into running containers
integration_overlay - Overlay filesystem tests

Run a specific test:

cargo test --test integration_basic_binary

Integration Tests (Kubernetes)

The main integration test suite runs against a kind (Kubernetes in Docker) cluster. It validates:

✓ DNS resolution in container
✓ Basic command execution (echo)
✓ Overlay filesystem sharing across pods
✓ Host filesystem protection (no leakage to host)
✓ UID/GID switching with securityContext
✓ Privilege drop to non-root user
✓ Shim cleanup after pod deletion
✓ No defunct (zombie) processes
✓ kubectl exec support

Full Suite (Recommended)

Runs cargo tests, builds binaries, creates a kind cluster, and runs all integration tests:

./scripts/run-integration-tests.sh

Options:

--skip-cargo — Skip Rust unit tests (useful for rapid K8s-only reruns)
--no-cleanup — Keep the kind cluster running after tests (for debugging)
--verbose — Also print debug output to stdout (in addition to log file)
--agent-only — Only run agent tests (skip cargo, integration, and controller tests)
--crd-only — Only run CRD controller tests (skip cargo, integration, and agent tests)
--help — Show usage

Examples

Rerun K8s tests against an existing cluster:

./scripts/run-integration-tests.sh --skip-cargo --no-cleanup

Run only CRD controller tests (fast iteration on ReaperPod CRD):

./scripts/run-integration-tests.sh --crd-only --no-cleanup

Run only agent tests:

./scripts/run-integration-tests.sh --agent-only --no-cleanup

Keep cluster for interactive debugging:

./scripts/run-integration-tests.sh --no-cleanup

Then interact with the cluster:

kubectl get pods
kubectl logs <pod-name>
kubectl describe pod <pod-name>

Test Output & Logs

Console output: Test results with pass/fail badges
Log file: /tmp/reaper-integration-logs/integration-test.log (detailed diagnostics)
GitHub Actions: Results posted to job summary when run in CI

How It Works

The test harness orchestrates:

Phase 1: Rust cargo tests (integration_* tests)
Phase 2: Kubernetes infrastructure setup
- Create or reuse kind cluster
- Build static musl binaries (matches node architecture)
- Deploy shim and runtime binaries to cluster node
- Configure containerd with the Reaper runtime
Phase 3: Kubernetes readiness
- Wait for API server and nodes
- Create RuntimeClass
- Wait for default ServiceAccount
Phase 4: Integration tests
- DNS, echo, overlay, host protection, UID/GID switching, privilege drop, exec, zombie check
Phase 4b: Controller tests (ReaperPod CRD)
- CRD installation, controller deployment, ReaperPod lifecycle, status mirroring, exit code propagation, annotations, custom printer columns, garbage collection
Phase 5: Summary & reporting

Coverage

Generate code coverage report using Docker:

./scripts/docker-coverage.sh

This runs cargo-tarpaulin (Linux-first tool) in a container with appropriate capabilities.

Containerd Configuration

Configure a containerd instance to use the Reaper shim runtime:

./scripts/configure-containerd.sh <context> <node-id>

<context>: kind or minikube (determines config locations)
<node-id>: Docker container ID (e.g., from docker ps)

This script is automatically run by run-integration-tests.sh.

Development Workflow

Before Pushing (CI-parity on macOS)

Run the full CI-equivalent check locally:

make ci

This runs, in order: fmt check, clippy (macOS), clippy (Linux cross-check), cargo test, and coverage (Docker + tarpaulin). If this passes, CI will pass.

Quick Iteration

For fast feedback during development:

make test              # Unit tests only (seconds)
make clippy            # macOS clippy
make check-linux       # Catches #[cfg(linux)] compilation issues

Linux Cross-Check

The overlay module (overlay.rs) is gated by #[cfg(target_os = "linux")] and doesn't compile on macOS. make check-linux cross-checks clippy against the x86_64-unknown-linux-gnu target to catch compilation errors in Linux-only code without leaving macOS.

Requires the target (one-time setup):

rustup target add x86_64-unknown-linux-gnu

Coverage (CI-parity)

Coverage runs tarpaulin inside Docker to match CI exactly:

make coverage

Configuration lives in tarpaulin.toml. Functions requiring root + Linux namespaces (tested by kind-integration) are excluded via #[cfg(not(tarpaulin_include))] so coverage reflects what unit tests can actually reach.

Integration Test Iteration

If you're iterating on overlay or shim logic:

# First run (build cluster, binaries, tests)
./scripts/run-integration-tests.sh --no-cleanup

# Make code changes...

# Rebuild and test (skip cargo, reuse cluster)
cargo build --release --bin containerd-shim-reaper-v2 --bin reaper-runtime --target x86_64-unknown-linux-musl
./scripts/run-integration-tests.sh --skip-cargo --no-cleanup

# Repeat until satisfied...

# Final cleanup run
./scripts/run-integration-tests.sh --skip-cargo

Troubleshooting

No kind cluster available

The test harness automatically creates one. If it fails, check:

Docker is running: docker ps
kind is installed: kind --version
Sufficient disk space: df -h

Pod stuck in Pending

Check containerd logs on the node:

docker exec <node-id> journalctl -u containerd -n 50 --no-pager

Check Kubelet logs:

docker exec <node-id> journalctl -u kubelet -n 50 --no-pager

Test times out

Increase timeout in test function or check node resources:

docker exec <node-id> top -b -n 1
docker exec <node-id> df -h

RuntimeClass not found

Wait a few seconds after applying the RuntimeClass, as it takes time to propagate.

Directory Structure

reaper/
├── scripts/
│   ├── run-integration-tests.sh      [MAIN] Orchestrates all integration tests
│   ├── install-reaper.sh             Ansible-based installation (DEPRECATED)
│   ├── build-node-image.sh           Build reaper-node installer image for Kind
│   ├── build-controller-image.sh     Build reaper-controller image for Kind
│   ├── install-node.sh               Init container script for node DaemonSet
│   ├── generate-kind-inventory.sh    Auto-generate Kind inventory for Ansible (DEPRECATED)
│   ├── configure-containerd.sh       Helper to configure containerd
│   ├── install-hooks.sh              Setup git hooks (optional)
│   └── docker-coverage.sh            Run coverage in Docker
├── tests/
│   ├── integration_basic_binary.rs
│   ├── integration_user_management.rs
│   ├── integration_shim.rs
│   ├── integration_io.rs
│   ├── integration_exec.rs
│   └── integration_overlay.rs
├── deploy/
│   └── kubernetes/                   [K8s cluster config examples]
├── examples/                         [Runnable Kind-based demos]
└── docs/
    └── TESTING.md                    [This file]

CI Integration

The CI pipeline (.github/workflows/ci.yml) runs automatically on:

Push to main or fix/** branches
Pull requests targeting main

Changes to documentation (*.md, docs/**), LICENSE*, and .gitignore are excluded from triggering runs.

Jobs

The pipeline runs these jobs:

Job	Description
Format	`cargo fmt -- --check`
Clippy	`cargo clippy --workspace --all-targets -- -D warnings`
Security Audit	`cargo audit`
Tests	`cargo test --verbose`
Coverage	`cargo tarpaulin` → Codecov upload
Build and Cache	Cross-compile static musl binaries (all 4 binaries)
kind-integration	Full integration test suite (`run-integration-tests.sh --skip-cargo`)
Example Validation	`test-examples.sh --skip-cluster`

Results

Results are posted to the GitHub Actions job summary. If any test fails, the workflow reports the failure with diagnostics.

Archived / Deprecated Scripts

The following scripts have been consolidated into run-integration-tests.sh and are no longer maintained:

kind-integration.sh — Replaced by run-integration-tests.sh (more features, better test reporting)
minikube-setup-runtime.sh — Minikube support deprecated
minikube-test.sh — Minikube support deprecated
test-k8s-integration.sh — Replaced by run-integration-tests.sh
docker-test.sh — Optional helper; use cargo test for speed or docker-coverage.sh for coverage

Next Steps

Read the Architecture documentation for deeper understanding
Check Overlay Design for filesystem isolation details
See SHIMV2 Design for runtime internals

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing & Integration

Quick Reference

Unit Tests

Test Modules

Integration Tests (Kubernetes)

Full Suite (Recommended)

Examples

Test Output & Logs

How It Works

Coverage

Containerd Configuration

Development Workflow

Before Pushing (CI-parity on macOS)

Quick Iteration

Linux Cross-Check

Coverage (CI-parity)

Integration Test Iteration

Troubleshooting

No kind cluster available

Pod stuck in Pending

Test times out

RuntimeClass not found

Directory Structure

CI Integration

Jobs

Results

Archived / Deprecated Scripts

Next Steps

FilesExpand file tree

TESTING.md

Latest commit

History

TESTING.md

File metadata and controls

Testing & Integration

Quick Reference

Unit Tests

Test Modules

Integration Tests (Kubernetes)

Full Suite (Recommended)

Examples

Test Output & Logs

How It Works

Coverage

Containerd Configuration

Development Workflow

Before Pushing (CI-parity on macOS)

Quick Iteration

Linux Cross-Check

Coverage (CI-parity)

Integration Test Iteration

Troubleshooting

No kind cluster available

Pod stuck in Pending

Test times out

RuntimeClass not found

Directory Structure

CI Integration

Jobs

Results

Archived / Deprecated Scripts

Next Steps