Skip to content

Commit 14b6391

Browse files
authored
chore(docs): Create runbook to update Chart dependency (#2400)
1 parent f95da2e commit 14b6391

File tree

2 files changed

+354
-0
lines changed

2 files changed

+354
-0
lines changed

.github/workflows/build_external_container_images.yaml.yml renamed to .github/workflows/build_external_container_images.yaml

File renamed without changes.
Lines changed: 354 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,354 @@
1+
# Chainloop Chart Dependencies Upgrade Runbook
2+
3+
## Overview
4+
5+
This runbook provides step-by-step instructions for upgrading Helm chart dependencies in the Chainloop project. Follow this process to ensure safe, consistent upgrades while maintaining system stability.
6+
7+
### Dependencies Managed
8+
- **PostgreSQL**: Database backend (Bitnami chart)
9+
- **Vault**: Secret storage (Bitnami chart)
10+
- **Dex**: OIDC provider *(self-managed, separate process)*
11+
12+
---
13+
14+
## CRITICAL RESTRICTIONS
15+
16+
### Version Upgrade Rules
17+
| Upgrade Type | Example | Status |
18+
|--------------|---------|--------|
19+
| Patch upgrade | `1.2.3``1.2.4` | ALLOWED |
20+
| Minor upgrade | `1.2.x``1.3.x` | ALLOWED |
21+
| Major upgrade | `1.x.x``2.x.x` | **FORBIDDEN - STOP IMMEDIATELY** |
22+
23+
**MANDATORY CHECK**: Any major version upgrade attempt must **STOP** the process and escalate for manual review.
24+
25+
---
26+
27+
## Upgrade Types
28+
29+
### Chart Version Upgrade Only
30+
- **Purpose**: Upgrade chart to latest minor version
31+
- **Scope**: Update chart version and potentially images
32+
- **Use Case**: Feature updates, security patches, regular maintenance
33+
- **Rule**: Container images are **ONLY** updated as part of chart upgrades, never independently
34+
35+
---
36+
37+
## Pre-Upgrade Validation
38+
39+
### Step 1: Identify Current State
40+
```bash
41+
# Check current chart version
42+
cat deployment/chainloop/charts/<chart-name>/Chart.yaml | grep "^version:"
43+
44+
# Check current app version
45+
cat deployment/chainloop/charts/<chart-name>/Chart.yaml | grep "^appVersion:"
46+
```
47+
48+
### Step 2: Version Compatibility Check
49+
```bash
50+
# Example validation
51+
CURRENT_MAJOR=$(echo "$CURRENT_VERSION" | cut -d. -f1)
52+
TARGET_MAJOR=$(echo "$TARGET_VERSION" | cut -d. -f1)
53+
54+
if [ "$CURRENT_MAJOR" != "$TARGET_MAJOR" ]; then
55+
echo "FORBIDDEN: Major version upgrade detected"
56+
echo "Current: $CURRENT_VERSION → Target: $TARGET_VERSION"
57+
exit 1
58+
fi
59+
```
60+
61+
### Step 3: Stop Conditions
62+
**STOP IMMEDIATELY if**:
63+
- Major version change detected
64+
- Breaking changes require manual intervention
65+
- Dependencies conflict
66+
67+
---
68+
69+
## Reference Resources
70+
71+
### Bitnami Charts Repository Structure
72+
```
73+
bitnami/charts/
74+
└── <chart-name>/
75+
├── Chart.yaml # Chart metadata and version
76+
├── CHANGELOG.md # Version history and changes
77+
├── values.yaml # Default configuration
78+
└── templates/ # Helm templates
79+
```
80+
81+
### Bitnami Containers Repository Structure
82+
```
83+
bitnami/containers/
84+
└── <image-name>/
85+
└── <major-version>/
86+
└── <distro>-<distro-version>/
87+
├── Dockerfile # Contains APP_VERSION
88+
└── ...
89+
```
90+
91+
### Finding Resources
92+
| Resource | Location | Purpose |
93+
|----------|----------|---------|
94+
| Chart versions | [Bitnami Charts](https://github.com/bitnami/charts) + `CHANGELOG.md` | Find available chart versions |
95+
| Container images | [Bitnami Containers](https://github.com/bitnami/containers) + commit history | Find image versions and commit hashes |
96+
| Release tags | Commit messages: `Release <name>-<version>-<distro>-<distro-version>-r<revision>` | Identify specific releases |
97+
98+
---
99+
100+
## Type 1: Specific Image Upgrade Process
101+
102+
### Step 1: Locate Target Container Image
103+
1. Navigate to [Bitnami Containers](https://github.com/bitnami/containers)
104+
2. Find image folder: `bitnami/<image-name>`
105+
3. Check commit history: `https://github.com/bitnami/containers/commits/main/bitnami/<image-name>`
106+
4. Find commit with message: `Release <image>-<version>-<distro>-<distro-version>-r<revision>`
107+
5. Note the commit hash and APP_VERSION from Dockerfile
108+
109+
### Step 2: Update Image References
110+
```bash
111+
# Edit Chart.yaml - update appVersion only
112+
vi deployment/charts/<chart-name>/Chart.yaml
113+
114+
# Update appVersion field to match container's APP_VERSION
115+
# Keep chart version unchanged
116+
```
117+
118+
### Step 3: Update Build Configuration
119+
```bash
120+
# Edit build workflow to reference the correct commit
121+
vi .github/workflows/build_external_container_images.yaml
122+
123+
# Update commit hash for the specific image
124+
```
125+
126+
---
127+
128+
## Type 2: Latest Minor Version Chart Upgrade Process
129+
130+
### Step 1: Locate Target Chart Version
131+
1. Navigate to [Bitnami Charts](https://github.com/bitnami/charts)
132+
2. Open `bitnami/<chart-name>/CHANGELOG.md`
133+
3. Find latest minor version (ensure no major version change)
134+
4. Note target chart version
135+
136+
### Step 2: Version Validation
137+
```bash
138+
# MANDATORY: Verify minor upgrade only
139+
CURRENT_CHART_VERSION="<current>"
140+
TARGET_CHART_VERSION="<target>"
141+
142+
# Validate major version compatibility
143+
# If major version differs, STOP PROCESS
144+
```
145+
146+
### Step 3: Download and Extract Target Chart
147+
```bash
148+
# Pull chart to temporary location
149+
helm pull bitnami/<chart-name> --version <target-version> --untar --untardir /tmp
150+
151+
# Examine the new chart structure
152+
ls -la /tmp/<chart-name>/
153+
```
154+
155+
### Step 4: Check for Image Changes
156+
```bash
157+
# Compare current vs target chart images
158+
diff deployment/charts/<chart-name>/Chart.yaml /tmp/<chart-name>/Chart.yaml
159+
160+
# Look for changes in:
161+
# - appVersion field
162+
# - images section (if present)
163+
# - dependencies
164+
```
165+
166+
### Step 5: Update Container Images (if changed)
167+
**Execute only if images changed in target chart**:
168+
169+
1. **Locate new image versions**:
170+
```bash
171+
# For each changed image, find in Bitnami Containers
172+
# Pattern: <app-version>-<distro>-<distro-version>-r<revision>
173+
# Example: 15.3.0-debian-12-r1
174+
```
175+
176+
2. **Get APP_VERSION from Dockerfile**:
177+
```bash
178+
# Navigate to bitnami/containers/<image>/<major-version>/<distro>-<version>/
179+
# Extract APP_VERSION from Dockerfile
180+
```
181+
182+
3. **Update build configuration**:
183+
```bash
184+
# Update commit hash in build workflow
185+
vi .github/workflows/build_external_container_images.yaml
186+
```
187+
188+
### Step 6: Vendorize Chart Update
189+
```bash
190+
# Replace vendorized chart with new version
191+
cp -r /tmp/<chart-name>/* deployment/charts/<chart-name>/
192+
193+
# Update Chart.yaml if images changed
194+
vi deployment/charts/<chart-name>/Chart.yaml
195+
# Set appVersion to APP_VERSION from Bitnami Containers
196+
197+
# Update values.yaml if needed
198+
# Replace docker.io/bitnami/* with Chainloop registry paths
199+
vi deployment/charts/<chart-name>/values.yaml
200+
```
201+
202+
### Step 7: Update Dependencies
203+
```bash
204+
# CRITICAL: Update dependencies in correct order
205+
206+
# 1. Update vendorized chart dependencies first
207+
cd deployment/charts/<chart-name>
208+
helm dependency update
209+
helm dependency build
210+
211+
# 2. Update main chart dependency version
212+
cd ../../chainloop
213+
vi Chart.yaml # Update dependency version to match vendorized chart
214+
215+
# 3. Update main chart dependencies
216+
helm dependency update
217+
helm dependency build
218+
219+
cd ../..
220+
```
221+
222+
### Step 8: Clean Up
223+
```bash
224+
# Remove temporary files
225+
rm -rf /tmp/<chart-name>
226+
227+
# Verify working directory is clean
228+
git status
229+
```
230+
231+
---
232+
233+
## Verification & Testing
234+
235+
### Local Verification
236+
```bash
237+
# 1. Lint charts
238+
helm lint deployment/charts/<chart-name>
239+
helm lint deployment/chainloop
240+
241+
# 2. Template validation
242+
helm template deployment/charts/<chart-name>
243+
helm template deployment/chainloop
244+
245+
# 3. Local testing
246+
cd devel && docker compose up
247+
248+
# 4. Integration testing
249+
# Run your specific integration test suite
250+
```
251+
252+
### Image Consistency Checks
253+
```bash
254+
# Verify consistency across:
255+
# - Chart.yaml appVersion
256+
# - values.yaml image tags
257+
# - Build workflow commit references
258+
259+
grep -r "appVersion\|image.*tag" deployment/charts/<chart-name>/
260+
grep -r "<chart-name>" .github/workflows/build_external_container_images.yaml
261+
```
262+
263+
---
264+
265+
## Files Modified
266+
267+
### Chart Files
268+
- `deployment/charts/<chart-name>/Chart.yaml` - Chart version, appVersion
269+
- `deployment/charts/<chart-name>/values.yaml` - Image references
270+
- `deployment/charts/<chart-name>/Chart.lock` - Dependency lock
271+
- `deployment/chainloop/Chart.yaml` - Main chart dependencies
272+
- `deployment/chainloop/Chart.lock` - Main dependency lock
273+
274+
### CI/CD Configuration
275+
- `.github/workflows/build_external_container_images.yaml` - Image build references
276+
277+
---
278+
279+
## Troubleshooting
280+
281+
| Issue | Symptoms | Solution |
282+
|-------|----------|----------|
283+
| **Image Version Mismatch** | Services fail to start | Verify APP_VERSION matches Chart.yaml appVersion |
284+
| **Build Failures** | CI/CD fails to build images | Check commit reference contains required image versions |
285+
| **Image Pull Failures** | Deployment fails | Ensure all image tags are consistent and updated |
286+
| **Dependency Conflicts** | Helm dependency errors | Check compatibility between chart versions |
287+
| **Missing Container Images** | Image not found errors | Check Bitnami Containers history for renamed/removed images |
288+
289+
### Debug Commands
290+
```bash
291+
# Check image references
292+
grep -r "image:" deployment/charts/<chart-name>/
293+
294+
# Verify build configuration
295+
grep -A5 -B5 "<chart-name>" .github/workflows/build_external_container_images.yaml
296+
297+
# Check dependency status
298+
helm dependency list deployment/chainloop
299+
300+
# Validate chart syntax
301+
helm lint deployment/charts/<chart-name> --strict
302+
```
303+
304+
---
305+
306+
## Emergency Procedures
307+
308+
### Rollback Steps
309+
1. **Immediate**: Revert git changes
310+
```bash
311+
git checkout HEAD -- deployment/
312+
```
313+
314+
2. **Clean state**: Remove lock files and rebuild
315+
```bash
316+
find deployment/ -name "Chart.lock" -delete
317+
cd deployment/chainloop && helm dependency build
318+
```
319+
320+
3. **Verify**: Test rolled-back version
321+
```bash
322+
cd devel && docker compose down && docker compose up
323+
```
324+
325+
### Escalation Criteria
326+
**Contact team lead when**:
327+
- Major version upgrade is required
328+
- Breaking changes affect core functionality
329+
- Multiple dependency conflicts arise
330+
- Data migration is required
331+
332+
---
333+
334+
## Process Checklist
335+
336+
### Pre-Upgrade
337+
- [ ] Version compatibility verified (no major version change)
338+
- [ ] Current state documented
339+
- [ ] Backup/rollback plan confirmed
340+
341+
### During Upgrade
342+
- [ ] Target version located and validated
343+
- [ ] Image changes identified and updated
344+
- [ ] Charts vendorized correctly
345+
- [ ] Dependencies updated in correct order
346+
- [ ] Build configuration updated (if needed)
347+
348+
### Post-Upgrade
349+
- [ ] Local testing passed
350+
- [ ] Chart linting clean
351+
- [ ] Image consistency verified
352+
- [ ] Integration tests passed
353+
- [ ] Documentation updated
354+
- [ ] Temporary files cleaned up

0 commit comments

Comments
 (0)