Skip to content

Commit ade37a2

Browse files
committed
feat: cgroups v2 support for stress command with dual-mode injection
Implements native cgroups v2 support for stress command, replacing legacy dockhack+cgexec. Highlights: - Dual mode: default (child cgroup) and --inject-cgroup (same cgroup) - New cg-inject binary (pure Go) for precise cgroup placement - Works on cgroups v1 and v2, cgroupfs and systemd drivers - Kubernetes-aware cgroup path resolution via ContainerInspect - Eliminates privileged mode requirement (no SYS_ADMIN, no AppArmor) - Fixes goroutine leak in stress container error path - Extensive unit and integration tests
1 parent ae2d218 commit ade37a2

23 files changed

+2123
-249
lines changed

CLAUDE.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
## Build & Test Commands
44

55
- **Build:** `make build` (builds for current TARGETOS/TARGETARCH)
6+
- **Build cg-inject:** `make build-cg-inject` (builds static cg-inject binary)
67
- **Full pipeline:** `make all` (format → lint → test → build)
78
- **Unit tests:** `make test` (requires `CGO_ENABLED=1` for race detector)
89
- **Test with coverage:** `make test-coverage`
@@ -39,6 +40,7 @@ Integration tests use [bats](https://github.com/bats-core/bats-core) and run ins
3940

4041
```
4142
cmd/main.go — CLI entry point, all command/flag definitions
43+
cmd/cg-inject/ — cg-inject binary: moves process into target container's cgroup
4244
pkg/
4345
chaos/
4446
command.go — ChaosCommand interface, scheduling/interval runner
@@ -54,7 +56,7 @@ pkg/
5456
util/ — Shared utilities (IP/port parsing)
5557
mocks/ — Generated mock files (mockery)
5658
tests/ — Bats integration tests
57-
docker/ — Dockerfiles (main, alpine-nettools, debian-nettools)
59+
docker/ — Dockerfiles (main, alpine-nettools, debian-nettools, stress)
5860
deploy/ — K8s/OpenShift deployment manifests
5961
examples/ — Demo scripts
6062
```
@@ -65,7 +67,7 @@ examples/ — Demo scripts
6567
- **Docker client** (`pkg/container/docker_client.go`): Concrete implementation using Docker SDK
6668
- **Chaos commands**: Each action implements `ChaosCommand` interface with `Run(ctx, random)` method
6769
- **Network emulation**: Executes `tc` commands inside a sidecar container via Docker exec
68-
- **Stress testing**: Runs `stress-ng` via Docker exec in a sidecar container
70+
- **Stress testing**: Two modes — (1) default child-cgroup mode places stress-ng sidecar in target's cgroup via Docker's `--cgroup-parent`; (2) inject-cgroup mode (`--inject-cgroup`) uses `cg-inject` binary to write sidecar PID into target's `cgroup.procs` for shared resource accounting
6971
- **Target selection**: Container names (exact), comma-separated lists, or `re2:` prefixed regex patterns
7072
- **Label filtering**: `--label key=value` flags for container selection
7173
- **Interval mode**: `--interval` flag for recurring chaos on a schedule

Makefile

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,12 @@ build: dependency | ; $(info $(M) building $(GOOS)/$(GOARCH) binary...) @ ## Bui
5151
-ldflags "$(LDFLAGS_VERSION)" \
5252
-o $(BIN)/$(basename $(MODULE)) ./cmd/main.go
5353

54+
.PHONY: build-cg-inject
55+
build-cg-inject: dependency | ; $(info $(M) building cg-inject $(GOOS)/$(GOARCH) binary...) @ ## Build cg-inject static binary
56+
$Q CGO_ENABLED=0 $(GO) build \
57+
-ldflags '-s -w' \
58+
-o $(BIN)/cg-inject ./cmd/cg-inject/
59+
5460
.PHONY: release
5561
release: clean ; $(info $(M) building binaries for multiple os/arch...) @ ## Build program binary for paltforms and os
5662
$(foreach GOOS, $(PLATFORMS),\

README.md

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,9 @@ Pumba is a chaos testing and network emulation tool for Docker containers. Inspi
3131
graph LR
3232
A[Pumba CLI] -->|Docker API| B[Docker Engine]
3333
B -->|List & Filter| C[Target Containers]
34-
34+
3535
A -->|kill / stop / pause / rm| C
36-
36+
3737
A -->|netem / iptables| D[Helper Container]
3838
D -->|Shares network namespace| C
3939
D -->|Runs tc / iptables| E[Network Chaos]
@@ -44,16 +44,16 @@ For **network chaos** (netem, iptables), Pumba creates a helper container that s
4444

4545
## Features
4646

47-
| Category | Commands | Description |
48-
|----------|----------|-------------|
49-
| **Container Chaos** | `kill`, `stop`, `pause`, `rm`, `restart` | Disrupt container lifecycle |
50-
| **Execute** | `exec` | Run commands inside containers |
51-
| **Network Delay** | `netem delay` | Add latency to egress traffic |
52-
| **Packet Loss** | `netem loss`, `iptables loss` | Drop packets (egress and ingress) |
53-
| **Network Effects** | `netem duplicate`, `corrupt`, `rate` | Duplicate, corrupt, or rate-limit packets |
54-
| **Stress Testing** | `stress` | CPU, memory, I/O stress via stress-ng |
55-
| **Targeting** | names, regex (`re2:`), labels, `--random` | Flexible container selection |
56-
| **Scheduling** | `--interval` | Recurring chaos at fixed intervals |
47+
| Category | Commands | Description |
48+
| ------------------- | ----------------------------------------- | ----------------------------------------------------------------------------- |
49+
| **Container Chaos** | `kill`, `stop`, `pause`, `rm`, `restart` | Disrupt container lifecycle |
50+
| **Execute** | `exec` | Run commands inside containers |
51+
| **Network Delay** | `netem delay` | Add latency to egress traffic |
52+
| **Packet Loss** | `netem loss`, `iptables loss` | Drop packets (egress and ingress) |
53+
| **Network Effects** | `netem duplicate`, `corrupt`, `rate` | Duplicate, corrupt, or rate-limit packets |
54+
| **Stress Testing** | `stress` | CPU, memory, I/O stress via stress-ng (child cgroup or same-cgroup injection) |
55+
| **Targeting** | names, regex (`re2:`), labels, `--random` | Flexible container selection |
56+
| **Scheduling** | `--interval` | Recurring chaos at fixed intervals |
5757

5858
## Quick Start
5959

@@ -96,22 +96,22 @@ docker run -it --rm \
9696

9797
## Docker Images
9898

99-
| Registry | Image | Status |
100-
|----------|-------|--------|
101-
| **GitHub Container Registry** | `ghcr.io/alexei-led/pumba` | ✅ Primary |
102-
| Docker Hub | `alexeiled/pumba` | ⚠️ Deprecated |
99+
| Registry | Image | Status |
100+
| ----------------------------- | -------------------------- | ------------- |
101+
| **GitHub Container Registry** | `ghcr.io/alexei-led/pumba` | ✅ Primary |
102+
| Docker Hub | `alexeiled/pumba` | ⚠️ Deprecated |
103103

104104
Images are built natively for **linux/amd64** and **linux/arm64** (no QEMU).
105105

106106
## Documentation
107107

108-
| Document | Description |
109-
|----------|-------------|
110-
| **[User Guide](docs/guide.md)** | Container chaos commands, targeting, scheduling, configuration |
111-
| **[Network Chaos](docs/network-chaos.md)** | netem, iptables, advanced scenarios, architecture diagrams |
112-
| **[Stress Testing](docs/stress-testing.md)** | CPU/memory/IO stress testing with stress-ng |
113-
| **[Deployment](docs/deployment.md)** | Docker, Kubernetes DaemonSets, OpenShift |
114-
| **[Contributing](CONTRIBUTING.md)** | Build from source, run tests, project structure |
108+
| Document | Description |
109+
| -------------------------------------------- | -------------------------------------------------------------- |
110+
| **[User Guide](docs/guide.md)** | Container chaos commands, targeting, scheduling, configuration |
111+
| **[Network Chaos](docs/network-chaos.md)** | netem, iptables, advanced scenarios, architecture diagrams |
112+
| **[Stress Testing](docs/stress-testing.md)** | CPU/memory/IO stress testing with stress-ng |
113+
| **[Deployment](docs/deployment.md)** | Docker, Kubernetes DaemonSets, OpenShift |
114+
| **[Contributing](CONTRIBUTING.md)** | Build from source, run tests, project structure |
115115

116116
## Demo
117117

cmd/cg-inject/main.go

Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
// cg-inject is a minimal binary that moves itself into a target container's cgroup
2+
// and then exec's stress-ng (or any other command). This enables same-cgroup stress
3+
// testing where stress-ng shares resource accounting with the target container.
4+
//
5+
// Usage:
6+
//
7+
// cg-inject --target-id <containerID> [--cgroup-driver auto|cgroupfs|systemd] -- /stress-ng <args...>
8+
// cg-inject --cgroup-path <path> -- /stress-ng <args...>
9+
package main
10+
11+
import (
12+
"errors"
13+
"fmt"
14+
"os"
15+
"path"
16+
"strings"
17+
"syscall"
18+
)
19+
20+
// cgroupVersion represents the cgroup hierarchy version.
21+
type cgroupVersion int
22+
23+
const (
24+
cgroupV1 cgroupVersion = 1
25+
cgroupV2 cgroupVersion = 2
26+
)
27+
28+
// cgroupDriver represents the Docker cgroup driver.
29+
type cgroupDriver string
30+
31+
const (
32+
driverAuto cgroupDriver = "auto"
33+
driverCgroupfs cgroupDriver = "cgroupfs"
34+
driverSystemd cgroupDriver = "systemd"
35+
)
36+
37+
// config holds the parsed command-line configuration.
38+
type config struct {
39+
targetID string
40+
cgroupPath string // explicit cgroup base path (e.g., /kubepods/burstable/pod-uid/container-id)
41+
driver cgroupDriver
42+
commandArgs []string
43+
}
44+
45+
// for testing: override cgroup filesystem root and syscall.Exec
46+
var (
47+
cgroupRoot = "/sys/fs/cgroup"
48+
execCommand = syscall.Exec
49+
)
50+
51+
func main() {
52+
cfg, err := parseArgs(os.Args[1:])
53+
if err != nil {
54+
fmt.Fprintf(os.Stderr, "cg-inject: %v\n", err)
55+
os.Exit(1)
56+
}
57+
58+
if err := run(cfg); err != nil {
59+
fmt.Fprintf(os.Stderr, "cg-inject: %v\n", err)
60+
os.Exit(1)
61+
}
62+
}
63+
64+
func parseArgs(args []string) (config, error) {
65+
cfg := config{driver: driverAuto}
66+
67+
// find the -- separator
68+
dashIdx := -1
69+
for i, a := range args {
70+
if a == "--" {
71+
dashIdx = i
72+
break
73+
}
74+
}
75+
if dashIdx < 0 {
76+
return cfg, errors.New("missing '--' separator before command")
77+
}
78+
79+
cfg.commandArgs = args[dashIdx+1:]
80+
if len(cfg.commandArgs) == 0 {
81+
return cfg, errors.New("no command specified after '--'")
82+
}
83+
84+
if err := parseFlags(&cfg, args[:dashIdx]); err != nil {
85+
return cfg, err
86+
}
87+
if err := validateConfig(&cfg); err != nil {
88+
return cfg, err
89+
}
90+
91+
return cfg, nil
92+
}
93+
94+
func parseFlags(cfg *config, flagArgs []string) error {
95+
for i := 0; i < len(flagArgs); i++ {
96+
switch flagArgs[i] {
97+
case "--target-id":
98+
i++
99+
if i >= len(flagArgs) {
100+
return errors.New("--target-id requires a value")
101+
}
102+
cfg.targetID = flagArgs[i]
103+
case "--cgroup-driver":
104+
i++
105+
if i >= len(flagArgs) {
106+
return errors.New("--cgroup-driver requires a value")
107+
}
108+
d := cgroupDriver(flagArgs[i])
109+
if d != driverAuto && d != driverCgroupfs && d != driverSystemd {
110+
return fmt.Errorf("unknown cgroup driver %q (expected auto, cgroupfs, or systemd)", flagArgs[i])
111+
}
112+
cfg.driver = d
113+
case "--cgroup-path":
114+
i++
115+
if i >= len(flagArgs) {
116+
return errors.New("--cgroup-path requires a value")
117+
}
118+
cfg.cgroupPath = flagArgs[i]
119+
default:
120+
return fmt.Errorf("unknown flag %q", flagArgs[i])
121+
}
122+
}
123+
return nil
124+
}
125+
126+
func validateConfig(cfg *config) error {
127+
if cfg.cgroupPath != "" {
128+
if cfg.targetID != "" {
129+
return errors.New("--cgroup-path and --target-id are mutually exclusive")
130+
}
131+
if strings.Contains(cfg.cgroupPath, "..") {
132+
return errors.New("--cgroup-path must not contain '..' path components")
133+
}
134+
return nil
135+
}
136+
if cfg.targetID == "" {
137+
return errors.New("--target-id is required (or use --cgroup-path)")
138+
}
139+
if !isValidContainerID(cfg.targetID) {
140+
return fmt.Errorf("invalid container ID %q: must be 12-64 hex characters", cfg.targetID)
141+
}
142+
return nil
143+
}
144+
145+
const (
146+
minContainerIDLen = 12
147+
maxContainerIDLen = 64
148+
)
149+
150+
// isValidContainerID checks that the string is a valid Docker container ID (12-64 hex chars).
151+
func isValidContainerID(id string) bool {
152+
if len(id) < minContainerIDLen || len(id) > maxContainerIDLen {
153+
return false
154+
}
155+
for _, c := range id {
156+
if (c < '0' || c > '9') && (c < 'a' || c > 'f') {
157+
return false
158+
}
159+
}
160+
return true
161+
}
162+
163+
func run(cfg config) error {
164+
version := detectCgroupVersion()
165+
166+
var paths []string
167+
if cfg.cgroupPath != "" {
168+
paths = cgroupProcsPathsFromBase(cfg.cgroupPath, version)
169+
} else {
170+
driver := cfg.driver
171+
if driver == driverAuto {
172+
driver = detectDriver(version)
173+
}
174+
paths = cgroupProcsPaths(cfg.targetID, version, driver)
175+
}
176+
177+
pid := os.Getpid()
178+
for _, procsPath := range paths {
179+
if err := writePID(procsPath, pid); err != nil {
180+
return fmt.Errorf("writing PID %d to %s: %w", pid, procsPath, err)
181+
}
182+
}
183+
184+
// Resolve the full path to the command
185+
cmdPath := cfg.commandArgs[0]
186+
return execCommand(cmdPath, cfg.commandArgs, os.Environ())
187+
}
188+
189+
// detectCgroupVersion checks whether the system uses cgroups v2 or v1.
190+
// If /sys/fs/cgroup/cgroup.controllers exists, it's v2.
191+
func detectCgroupVersion() cgroupVersion {
192+
if _, err := os.Stat(cgroupRoot + "/cgroup.controllers"); err == nil {
193+
return cgroupV2
194+
}
195+
return cgroupV1
196+
}
197+
198+
// detectDriver infers the Docker cgroup driver by looking for Docker-specific
199+
// cgroup directories. The /docker/ directory is created by the cgroupfs driver,
200+
// while docker-*.scope entries under system.slice are created by the systemd driver.
201+
// Falls back to cgroupfs (Docker's default) if neither is found.
202+
func detectDriver(version cgroupVersion) cgroupDriver {
203+
if version == cgroupV2 {
204+
// Check for cgroupfs driver's /docker/ directory first (Docker's default)
205+
if _, err := os.Stat(cgroupRoot + "/docker"); err == nil {
206+
return driverCgroupfs
207+
}
208+
// Check for systemd driver's scope entries
209+
if _, err := os.Stat(cgroupRoot + "/system.slice"); err == nil {
210+
return driverSystemd
211+
}
212+
} else {
213+
// v1: check under the cpu controller hierarchy
214+
if _, err := os.Stat(cgroupRoot + "/cpu/docker"); err == nil {
215+
return driverCgroupfs
216+
}
217+
if _, err := os.Stat(cgroupRoot + "/cpu/system.slice"); err == nil {
218+
return driverSystemd
219+
}
220+
}
221+
return driverCgroupfs
222+
}
223+
224+
// v1Controllers lists the cgroup v1 controller hierarchies that stress-ng needs
225+
// for accurate resource accounting (CPU, memory, block I/O, CPU accounting, PIDs).
226+
var v1Controllers = []string{"cpu", "memory", "blkio", "cpuacct", "pids"}
227+
228+
// cgroupProcsPaths returns the cgroup.procs paths to write the PID into.
229+
// On cgroups v2 (unified hierarchy), a single write covers all controllers.
230+
// On cgroups v1, each controller has a separate hierarchy requiring its own write.
231+
func cgroupProcsPaths(targetID string, version cgroupVersion, driver cgroupDriver) []string {
232+
if version == cgroupV2 {
233+
if driver == driverSystemd {
234+
return []string{cgroupRoot + "/system.slice/docker-" + targetID + ".scope/cgroup.procs"}
235+
}
236+
return []string{cgroupRoot + "/docker/" + targetID + "/cgroup.procs"}
237+
}
238+
// cgroups v1: write to each controller hierarchy
239+
paths := make([]string, 0, len(v1Controllers))
240+
for _, ctrl := range v1Controllers {
241+
if driver == driverSystemd {
242+
paths = append(paths, cgroupRoot+"/"+ctrl+"/system.slice/docker-"+targetID+".scope/cgroup.procs")
243+
} else {
244+
paths = append(paths, cgroupRoot+"/"+ctrl+"/docker/"+targetID+"/cgroup.procs")
245+
}
246+
}
247+
return paths
248+
}
249+
250+
// cgroupProcsPathsFromBase returns cgroup.procs paths using an explicit base cgroup path.
251+
// On cgroups v2, a single path suffices. On v1, each controller hierarchy is addressed.
252+
// path.Join is used to normalize double slashes when basePath has a leading slash.
253+
func cgroupProcsPathsFromBase(basePath string, version cgroupVersion) []string {
254+
if version == cgroupV2 {
255+
return []string{path.Join(cgroupRoot, basePath, "cgroup.procs")}
256+
}
257+
paths := make([]string, 0, len(v1Controllers))
258+
for _, ctrl := range v1Controllers {
259+
paths = append(paths, path.Join(cgroupRoot, ctrl, basePath, "cgroup.procs"))
260+
}
261+
return paths
262+
}
263+
264+
// writePID writes a PID to an existing cgroup.procs file.
265+
// Uses O_WRONLY without O_CREATE to fail if the cgroup path doesn't exist.
266+
func writePID(procsPath string, pid int) error {
267+
f, err := os.OpenFile(procsPath, os.O_WRONLY, 0)
268+
if err != nil {
269+
return err
270+
}
271+
_, err = fmt.Fprintf(f, "%d\n", pid)
272+
if closeErr := f.Close(); err == nil {
273+
err = closeErr
274+
}
275+
return err
276+
}

0 commit comments

Comments
 (0)