Skip to content

Commit e177cb2

Browse files
daisy-ycguodaisy-ycguo
committed
Add user guide for RestartAllContainers
Signed-off-by: Daisy Guo <daiguo@nvidia.com>
1 parent 3457783 commit e177cb2

File tree

10 files changed

+664
-0
lines changed

10 files changed

+664
-0
lines changed

‎docs/user-guide/04_restart-all-containers-for-podcliqueset/Dockerfile‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Build stage
2+
FROM golang:1.21-alpine AS builder
3+
WORKDIR /app
4+
COPY src/ ./
5+
RUN go mod tidy && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /restart-watcher .
6+
7+
# Runtime stage
8+
FROM alpine:3.19
9+
RUN apk --no-cache add ca-certificates
10+
COPY --from=builder /restart-watcher /restart-watcher
11+
ENTRYPOINT ["/restart-watcher"]

‎docs/user-guide/04_restart-all-containers-for-podcliqueset/Makefile‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# restart-watcher image build and push
2+
# Override for your registry: make push REGISTRY=myreg.io/myuser
3+
REGISTRY ?= localhost:5000
4+
IMAGE_NAME ?= restart-watcher
5+
IMAGE_TAG ?= latest
6+
FULL_IMAGE := $(REGISTRY)/$(IMAGE_NAME):$(IMAGE_TAG)
7+
8+
.PHONY: build image push all tidy
9+
10+
# Build Go binary locally (for development)
11+
build:
12+
cd src && go build -o ../bin/restart-watcher .
13+
14+
# Build Docker image (binary built inside container)
15+
image:
16+
docker build -t $(FULL_IMAGE) -f Dockerfile .
17+
@echo "Built $(FULL_IMAGE)"
18+
19+
# Push image to registry
20+
push: image
21+
docker push $(FULL_IMAGE)
22+
@echo "Pushed $(FULL_IMAGE)"
23+
24+
# Tidy go.mod (run from repo root)
25+
tidy:
26+
cd src && go mod tidy
27+
28+
all: image

‎docs/user-guide/04_restart-all-containers-for-podcliqueset/README.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# PodCliqueSet In-Place Restart Guide
2+
3+
This guide uses Kubernetes 1.35+ **RestartAllContainers** to trigger in-place restarts of all Pods in a Grove **PodCliqueSet** via a single **ConfigMap** field `restartGeneration`. Pod names, UIDs, and IPs stay the same (no rescheduling).
4+
5+
## Use case: restart without rescheduling
6+
7+
Sometimes we want a **PodCliqueSet** (or all its Pods) to restart **without going through rescheduling**—for example when **upgrading container image versions** or when we need a clean re-run of init containers and main containers while keeping the same Pod identity and placement.
8+
9+
Deleting and recreating Pods is costly: it involves the scheduler, node allocation, and re-initialization of networking and storage. Kubernetes 1.35’s [Restart All Containers](https://kubernetes.io/blog/2026/01/02/kubernetes-v1-35-restart-all-containers/) feature provides an **in-place** restart instead: the kubelet restarts all containers in the Pod while preserving the Pod’s UID, IP address, volumes, and node assignment. Init containers run again in order, then all main containers start with a fresh state—so an image update or configuration change can take effect without any rescheduling. This guide shows how to trigger that in-place restart for an entire Grove PodCliqueSet at once via a ConfigMap.
10+
11+
### Limitations
12+
13+
This guide applies only to **restarting PodCliqueSets** (in-place restart of all Pods belonging to a Grove PodCliqueSet). It does not cover other workload types or cluster-wide restart scenarios.
14+
15+
## Idea
16+
17+
- Each Pod runs a **restart-watcher** sidecar (Go). It uses in-cluster config to poll the ConfigMap `grove-restart-control` in the same namespace for the key `restartGeneration`.
18+
- When it sees `restartGeneration` **increase**, the watcher exits with a configured code (default 88), which triggers **RestartAllContainers** for that Pod.
19+
- To trigger a batch in-place restart, **kubectl patch** the ConfigMap to increment `restartGeneration`; all Pods with the watcher will see the new value on the next poll and restart in place.
20+
21+
## Directory layout
22+
23+
```
24+
04_restart-all-containers-for-podcliqueset/
25+
├── src/
26+
│ ├── main.go # restart-watcher sidecar source
27+
│ └── go.mod
28+
├── Dockerfile # build watcher image
29+
├── Makefile # build and push image
30+
├── manifests/
31+
│ ├── namespace.yaml
32+
│ ├── rbac.yaml # SA + Role + RoleBinding (read ConfigMap)
33+
│ ├── configmap.yaml
34+
│ └── podcliqueset.yaml
35+
└── README.md
36+
```
37+
38+
## Prerequisites
39+
40+
1. **Cluster**: Kubernetes **1.35+** with **RestartAllContainersOnContainerExits** and **NodeDeclaredFeatures** enabled. Both feature gates must be enabled on **both** the API server and the kubelet. **RestartAllContainersOnContainerExits** depends on **NodeDeclaredFeatures**, so enable them together. See your cluster or distribution docs for how to set feature gates.
41+
2. **Grove**: CRD and Operator installed, **v0.1.0-alpha.4 or later**.
42+
3. **Registry**: A Docker registry you can push the `restart-watcher` image to and that cluster nodes can pull from.
43+
44+
## Steps
45+
46+
### 1. Build and push the restart-watcher image
47+
48+
From this guide's directory:
49+
50+
```bash
51+
# Set your registry (required)
52+
export REGISTRY=your-registry.io/your-user
53+
export IMAGE_TAG=latest
54+
55+
make push
56+
```
57+
58+
Note the image name, e.g. `$(REGISTRY)/restart-watcher:$(IMAGE_TAG)`.
59+
60+
### 2. Set the watcher image in the PodCliqueSet
61+
62+
Edit `manifests/podcliqueset.yaml` and replace both `WATCHER_IMAGE` with the image you pushed, e.g.:
63+
64+
```bash
65+
sed -i "s|WATCHER_IMAGE|${REGISTRY}/restart-watcher:${IMAGE_TAG}|g" manifests/podcliqueset.yaml
66+
```
67+
68+
Or change `image: WATCHER_IMAGE` to e.g. `image: your-registry.io/your-user/restart-watcher:latest` by hand.
69+
70+
### 3. Deploy PodCliqueSet and ConfigMap
71+
72+
```bash
73+
kubectl apply -f manifests/namespace.yaml
74+
kubectl apply -f manifests/rbac.yaml
75+
kubectl apply -f manifests/configmap.yaml
76+
kubectl apply -f manifests/podcliqueset.yaml
77+
```
78+
79+
The [example PodCliqueSet](manifests/podcliqueset.yaml) has two PodCliques, **pca** and **pcb**, and both include the **restart-watcher** sidecar so that incrementing `restartGeneration` restarts all 6 Pods. If you only want to restart one PodClique, add the restart-watcher sidecar only to that clique’s `podSpec` in the manifest; Pods without the sidecar will not react to the ConfigMap.
80+
81+
### 4. Wait for Pods to be ready
82+
83+
The PodCliqueSet has **pca** (replicas=2) and **pcb** (replicas=4), 6 Pods in total:
84+
85+
```bash
86+
kubectl get podcliqueset -n grove-restart-demo
87+
kubectl get pods -n grove-restart-demo -l app.kubernetes.io/part-of=grove-restart-demo-pcs -o wide
88+
```
89+
90+
Confirm all 6 Pods are `Running`.
91+
92+
### 5. (Optional) Record Pod name, Pod ID, and IP
93+
94+
To compare before and after the trigger (names, UIDs, and IPs should stay the same):
95+
96+
```bash
97+
kubectl get pods -n grove-restart-demo -l app.kubernetes.io/part-of=grove-restart-demo-pcs \
98+
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.uid}{"\t"}{.status.podIP}{"\n"}{end}'
99+
```
100+
101+
### 6. Trigger one “full PodCliqueSet in-place restart”
102+
103+
Increment the ConfigMap `grove-restart-control` key `restartGeneration`:
104+
105+
```bash
106+
# Get current value
107+
current=$(kubectl get configmap grove-restart-control -n grove-restart-demo -o jsonpath='{.data.restartGeneration}')
108+
next=$((current + 1))
109+
110+
# Patch with next value
111+
kubectl patch configmap grove-restart-control -n grove-restart-demo \
112+
--type merge \
113+
-p "{\"data\":{\"restartGeneration\":\"${next}\"}}"
114+
```
115+
116+
Within the next poll interval (default 5 seconds), all Pods with the restart-watcher will see the new value, exit with 88, and trigger RestartAllContainers for their Pod.
117+
118+
### 7. Observe
119+
120+
After about 10–20 seconds:
121+
122+
```bash
123+
kubectl get pods -n grove-restart-demo -l app.kubernetes.io/part-of=grove-restart-demo-pcs -o wide
124+
```
125+
126+
**Expected**:
127+
128+
- Pod **names, UIDs and IPs unchanged** (no new Pods created or deleted).
129+
- **Restart counts** increased (e.g. `kubectl get pod <name> -n grove-restart-demo -o jsonpath='{range .status.containerStatuses[*]}{.name} restarts={.restartCount}{"\n"}{end}'`).
130+
131+
To trigger again, repeat step 6 (increment `restartGeneration` again).
132+
133+
### 8. Cleanup
134+
135+
```bash
136+
kubectl delete -f manifests/podcliqueset.yaml
137+
kubectl delete -f manifests/configmap.yaml
138+
kubectl delete -f manifests/rbac.yaml
139+
kubectl delete -f manifests/namespace.yaml
140+
```
141+
142+
## Environment variables (restart-watcher)
143+
144+
| Variable | Meaning | Default |
145+
|----------|---------|--------|
146+
| `CM_NAMESPACE` | ConfigMap namespace | Prefer `metadata.namespace` via fieldRef |
147+
| `CM_NAME` | ConfigMap name | `grove-restart-control` |
148+
| `KEY_NAME` | Key name | `restartGeneration` |
149+
| `POLL_INTERVAL_SECONDS` | Poll interval (seconds) | `5` |
150+
| `TRIGGER_EXIT_CODE` | Exit code that triggers RestartAllContainers | `88` |
151+
152+
## References
153+
154+
- Kubernetes 1.35: [Restart All Containers](https://kubernetes.io/blog/2026/01/02/kubernetes-v1-35-restart-all-containers/), [KEP-5532](https://kep.k8s.io/5532)

‎docs/user-guide/04_restart-all-containers-for-podcliqueset/manifests/configmap.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# ConfigMap watched by restart-watcher sidecars.
2+
# Increment data.restartGeneration to trigger in-place RestartAllContainers on all Pods
3+
# that have the watcher sidecar.
4+
apiVersion: v1
5+
kind: ConfigMap
6+
metadata:
7+
name: grove-restart-control
8+
namespace: grove-restart-demo
9+
data:
10+
restartGeneration: "0"

‎docs/user-guide/04_restart-all-containers-for-podcliqueset/manifests/namespace.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: grove-restart-demo
5+
labels:
6+
app.kubernetes.io/name: grove-restart-demo

‎docs/user-guide/04_restart-all-containers-for-podcliqueset/manifests/podcliqueset.yaml‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Grove PodCliqueSet: pca (nginx:1.25, replicas=2), pcb (nginx:1.25, replicas=4).
2+
# Each Pod has a restart-watcher sidecar that watches ConfigMap grove-restart-control.
3+
# When restartGeneration is incremented, watchers exit with code 88 and trigger
4+
# RestartAllContainers (K8s 1.35+). Replace WATCHER_IMAGE with your built image.
5+
#
6+
# Prerequisites:
7+
# - Kubernetes 1.35+ with RestartAllContainersOnContainerExits and NodeDeclaredFeatures
8+
# - Grove CRD + Operator installed
9+
---
10+
apiVersion: grove.io/v1alpha1
11+
kind: PodCliqueSet
12+
metadata:
13+
name: grove-restart-demo-pcs
14+
namespace: grove-restart-demo
15+
labels:
16+
app: grove-restart-demo
17+
spec:
18+
replicas: 1
19+
template:
20+
terminationDelay: 1m
21+
cliqueStartupType: CliqueStartupTypeExplicit
22+
cliques:
23+
- name: pca
24+
spec:
25+
roleName: rolea
26+
replicas: 2
27+
podSpec:
28+
restartPolicy: Always
29+
serviceAccountName: grove-restart-demo
30+
containers:
31+
- name: main
32+
image: nginx:1.25
33+
ports:
34+
- containerPort: 80
35+
resources:
36+
requests:
37+
cpu: 10m
38+
- name: restart-watcher
39+
image: WATCHER_IMAGE
40+
imagePullPolicy: IfNotPresent
41+
env:
42+
- name: CM_NAMESPACE
43+
valueFrom:
44+
fieldRef:
45+
fieldPath: metadata.namespace
46+
- name: CM_NAME
47+
value: "grove-restart-control"
48+
- name: KEY_NAME
49+
value: "restartGeneration"
50+
- name: POLL_INTERVAL_SECONDS
51+
value: "5"
52+
- name: TRIGGER_EXIT_CODE
53+
value: "88"
54+
restartPolicy: Always
55+
restartPolicyRules:
56+
- action: RestartAllContainers
57+
exitCodes:
58+
operator: In
59+
values: [88]
60+
- name: pcb
61+
spec:
62+
roleName: roleb
63+
replicas: 4
64+
podSpec:
65+
restartPolicy: Always
66+
serviceAccountName: grove-restart-demo
67+
containers:
68+
- name: main
69+
image: nginx:1.25
70+
ports:
71+
- containerPort: 80
72+
resources:
73+
requests:
74+
cpu: 10m
75+
- name: restart-watcher
76+
image: WATCHER_IMAGE
77+
imagePullPolicy: IfNotPresent
78+
env:
79+
- name: CM_NAMESPACE
80+
valueFrom:
81+
fieldRef:
82+
fieldPath: metadata.namespace
83+
- name: CM_NAME
84+
value: "grove-restart-control"
85+
- name: KEY_NAME
86+
value: "restartGeneration"
87+
- name: POLL_INTERVAL_SECONDS
88+
value: "5"
89+
- name: TRIGGER_EXIT_CODE
90+
value: "88"
91+
restartPolicy: Always
92+
restartPolicyRules:
93+
- action: RestartAllContainers
94+
exitCodes:
95+
operator: In
96+
values: [88]

‎docs/user-guide/04_restart-all-containers-for-podcliqueset/manifests/rbac.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# ServiceAccount and RBAC for Pods that run the restart-watcher sidecar.
2+
# The watcher needs to read the ConfigMap "grove-restart-control" in the same namespace.
3+
---
4+
apiVersion: v1
5+
kind: ServiceAccount
6+
metadata:
7+
name: grove-restart-demo
8+
namespace: grove-restart-demo
9+
---
10+
apiVersion: rbac.authorization.k8s.io/v1
11+
kind: Role
12+
metadata:
13+
name: grove-restart-configmap-reader
14+
namespace: grove-restart-demo
15+
rules:
16+
- apiGroups: [""]
17+
resources: ["configmaps"]
18+
resourceNames: ["grove-restart-control"]
19+
verbs: ["get", "list", "watch"]
20+
---
21+
apiVersion: rbac.authorization.k8s.io/v1
22+
kind: RoleBinding
23+
metadata:
24+
name: grove-restart-demo-read-configmap
25+
namespace: grove-restart-demo
26+
roleRef:
27+
apiGroup: rbac.authorization.k8s.io
28+
kind: Role
29+
name: grove-restart-configmap-reader
30+
subjects:
31+
- kind: ServiceAccount
32+
name: grove-restart-demo
33+
namespace: grove-restart-demo

‎docs/user-guide/04_restart-all-containers-for-podcliqueset/src/go.mod‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
module github.com/daisy/restart-watcher
2+
3+
go 1.21
4+
5+
require (
6+
k8s.io/api v0.28.4
7+
k8s.io/apimachinery v0.28.4
8+
k8s.io/client-go v0.28.4
9+
)
10+
11+
require (
12+
github.com/davecgh/go-spew v1.1.1 // indirect
13+
github.com/emicklei/go-restful/v3 v3.9.0 // indirect
14+
github.com/go-logr/logr v1.2.4 // indirect
15+
github.com/go-openapi/jsonpointer v0.19.6 // indirect
16+
github.com/go-openapi/jsonreference v0.20.2 // indirect
17+
github.com/go-openapi/swag v0.22.3 // indirect
18+
github.com/gogo/protobuf v1.3.2 // indirect
19+
github.com/golang/protobuf v1.5.3 // indirect
20+
github.com/google/gnostic-models v0.6.8 // indirect
21+
github.com/google/go-cmp v0.5.9 // indirect
22+
github.com/google/gofuzz v1.2.0 // indirect
23+
github.com/google/uuid v1.3.0 // indirect
24+
github.com/josharian/intern v1.0.0 // indirect
25+
github.com/json-iterator/go v1.1.12 // indirect
26+
github.com/mailru/easyjson v0.7.7 // indirect
27+
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
28+
github.com/modern-go/reflect2 v1.0.2 // indirect
29+
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
30+
golang.org/x/net v0.17.0 // indirect
31+
golang.org/x/oauth2 v0.8.0 // indirect
32+
golang.org/x/sys v0.13.0 // indirect
33+
golang.org/x/term v0.13.0 // indirect
34+
golang.org/x/text v0.13.0 // indirect
35+
golang.org/x/time v0.3.0 // indirect
36+
google.golang.org/appengine v1.6.7 // indirect
37+
google.golang.org/protobuf v1.31.0 // indirect
38+
gopkg.in/inf.v0 v0.9.1 // indirect
39+
gopkg.in/yaml.v2 v2.4.0 // indirect
40+
gopkg.in/yaml.v3 v3.0.1 // indirect
41+
k8s.io/klog/v2 v2.100.1 // indirect
42+
k8s.io/kube-openapi v0.0.0-20230717233707-2695361300d9 // indirect
43+
k8s.io/utils v0.0.0-20230406110748-d93618cff8a2 // indirect
44+
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
45+
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
46+
sigs.k8s.io/yaml v1.3.0 // indirect
47+
)

0 commit comments

Comments
 (0)