|
| 1 | +# Chainloop Chart Dependencies Upgrade Runbook |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +This runbook provides step-by-step instructions for upgrading Helm chart dependencies in the Chainloop project. Follow this process to ensure safe, consistent upgrades while maintaining system stability. |
| 6 | + |
| 7 | +### Dependencies Managed |
| 8 | +- **PostgreSQL**: Database backend (Bitnami chart) |
| 9 | +- **Vault**: Secret storage (Bitnami chart) |
| 10 | +- **Dex**: OIDC provider *(self-managed, separate process)* |
| 11 | + |
| 12 | +--- |
| 13 | + |
| 14 | +## CRITICAL RESTRICTIONS |
| 15 | + |
| 16 | +### Version Upgrade Rules |
| 17 | +| Upgrade Type | Example | Status | |
| 18 | +|--------------|---------|--------| |
| 19 | +| Patch upgrade | `1.2.3` → `1.2.4` | ALLOWED | |
| 20 | +| Minor upgrade | `1.2.x` → `1.3.x` | ALLOWED | |
| 21 | +| Major upgrade | `1.x.x` → `2.x.x` | **FORBIDDEN - STOP IMMEDIATELY** | |
| 22 | + |
| 23 | +**MANDATORY CHECK**: Any major version upgrade attempt must **STOP** the process and escalate for manual review. |
| 24 | + |
| 25 | +--- |
| 26 | + |
| 27 | +## Upgrade Types |
| 28 | + |
| 29 | +### Chart Version Upgrade Only |
| 30 | +- **Purpose**: Upgrade chart to latest minor version |
| 31 | +- **Scope**: Update chart version and potentially images |
| 32 | +- **Use Case**: Feature updates, security patches, regular maintenance |
| 33 | +- **Rule**: Container images are **ONLY** updated as part of chart upgrades, never independently |
| 34 | + |
| 35 | +--- |
| 36 | + |
| 37 | +## Pre-Upgrade Validation |
| 38 | + |
| 39 | +### Step 1: Identify Current State |
| 40 | +```bash |
| 41 | +# Check current chart version |
| 42 | +cat deployment/chainloop/charts/<chart-name>/Chart.yaml | grep "^version:" |
| 43 | + |
| 44 | +# Check current app version |
| 45 | +cat deployment/chainloop/charts/<chart-name>/Chart.yaml | grep "^appVersion:" |
| 46 | +``` |
| 47 | + |
| 48 | +### Step 2: Version Compatibility Check |
| 49 | +```bash |
| 50 | +# Example validation |
| 51 | +CURRENT_MAJOR=$(echo "$CURRENT_VERSION" | cut -d. -f1) |
| 52 | +TARGET_MAJOR=$(echo "$TARGET_VERSION" | cut -d. -f1) |
| 53 | + |
| 54 | +if [ "$CURRENT_MAJOR" != "$TARGET_MAJOR" ]; then |
| 55 | + echo "FORBIDDEN: Major version upgrade detected" |
| 56 | + echo "Current: $CURRENT_VERSION → Target: $TARGET_VERSION" |
| 57 | + exit 1 |
| 58 | +fi |
| 59 | +``` |
| 60 | + |
| 61 | +### Step 3: Stop Conditions |
| 62 | +**STOP IMMEDIATELY if**: |
| 63 | +- Major version change detected |
| 64 | +- Breaking changes require manual intervention |
| 65 | +- Dependencies conflict |
| 66 | + |
| 67 | +--- |
| 68 | + |
| 69 | +## Reference Resources |
| 70 | + |
| 71 | +### Bitnami Charts Repository Structure |
| 72 | +``` |
| 73 | +bitnami/charts/ |
| 74 | +└── <chart-name>/ |
| 75 | + ├── Chart.yaml # Chart metadata and version |
| 76 | + ├── CHANGELOG.md # Version history and changes |
| 77 | + ├── values.yaml # Default configuration |
| 78 | + └── templates/ # Helm templates |
| 79 | +``` |
| 80 | + |
| 81 | +### Bitnami Containers Repository Structure |
| 82 | +``` |
| 83 | +bitnami/containers/ |
| 84 | +└── <image-name>/ |
| 85 | + └── <major-version>/ |
| 86 | + └── <distro>-<distro-version>/ |
| 87 | + ├── Dockerfile # Contains APP_VERSION |
| 88 | + └── ... |
| 89 | +``` |
| 90 | + |
| 91 | +### Finding Resources |
| 92 | +| Resource | Location | Purpose | |
| 93 | +|----------|----------|---------| |
| 94 | +| Chart versions | [Bitnami Charts](https://github.com/bitnami/charts) + `CHANGELOG.md` | Find available chart versions | |
| 95 | +| Container images | [Bitnami Containers](https://github.com/bitnami/containers) + commit history | Find image versions and commit hashes | |
| 96 | +| Release tags | Commit messages: `Release <name>-<version>-<distro>-<distro-version>-r<revision>` | Identify specific releases | |
| 97 | + |
| 98 | +--- |
| 99 | + |
| 100 | +## Type 1: Specific Image Upgrade Process |
| 101 | + |
| 102 | +### Step 1: Locate Target Container Image |
| 103 | +1. Navigate to [Bitnami Containers](https://github.com/bitnami/containers) |
| 104 | +2. Find image folder: `bitnami/<image-name>` |
| 105 | +3. Check commit history: `https://github.com/bitnami/containers/commits/main/bitnami/<image-name>` |
| 106 | +4. Find commit with message: `Release <image>-<version>-<distro>-<distro-version>-r<revision>` |
| 107 | +5. Note the commit hash and APP_VERSION from Dockerfile |
| 108 | + |
| 109 | +### Step 2: Update Image References |
| 110 | +```bash |
| 111 | +# Edit Chart.yaml - update appVersion only |
| 112 | +vi deployment/charts/<chart-name>/Chart.yaml |
| 113 | + |
| 114 | +# Update appVersion field to match container's APP_VERSION |
| 115 | +# Keep chart version unchanged |
| 116 | +``` |
| 117 | + |
| 118 | +### Step 3: Update Build Configuration |
| 119 | +```bash |
| 120 | +# Edit build workflow to reference the correct commit |
| 121 | +vi .github/workflows/build_external_container_images.yaml |
| 122 | + |
| 123 | +# Update commit hash for the specific image |
| 124 | +``` |
| 125 | + |
| 126 | +--- |
| 127 | + |
| 128 | +## Type 2: Latest Minor Version Chart Upgrade Process |
| 129 | + |
| 130 | +### Step 1: Locate Target Chart Version |
| 131 | +1. Navigate to [Bitnami Charts](https://github.com/bitnami/charts) |
| 132 | +2. Open `bitnami/<chart-name>/CHANGELOG.md` |
| 133 | +3. Find latest minor version (ensure no major version change) |
| 134 | +4. Note target chart version |
| 135 | + |
| 136 | +### Step 2: Version Validation |
| 137 | +```bash |
| 138 | +# MANDATORY: Verify minor upgrade only |
| 139 | +CURRENT_CHART_VERSION="<current>" |
| 140 | +TARGET_CHART_VERSION="<target>" |
| 141 | + |
| 142 | +# Validate major version compatibility |
| 143 | +# If major version differs, STOP PROCESS |
| 144 | +``` |
| 145 | + |
| 146 | +### Step 3: Download and Extract Target Chart |
| 147 | +```bash |
| 148 | +# Pull chart to temporary location |
| 149 | +helm pull bitnami/<chart-name> --version <target-version> --untar --untardir /tmp |
| 150 | + |
| 151 | +# Examine the new chart structure |
| 152 | +ls -la /tmp/<chart-name>/ |
| 153 | +``` |
| 154 | + |
| 155 | +### Step 4: Check for Image Changes |
| 156 | +```bash |
| 157 | +# Compare current vs target chart images |
| 158 | +diff deployment/charts/<chart-name>/Chart.yaml /tmp/<chart-name>/Chart.yaml |
| 159 | + |
| 160 | +# Look for changes in: |
| 161 | +# - appVersion field |
| 162 | +# - images section (if present) |
| 163 | +# - dependencies |
| 164 | +``` |
| 165 | + |
| 166 | +### Step 5: Update Container Images (if changed) |
| 167 | +**Execute only if images changed in target chart**: |
| 168 | + |
| 169 | +1. **Locate new image versions**: |
| 170 | + ```bash |
| 171 | + # For each changed image, find in Bitnami Containers |
| 172 | + # Pattern: <app-version>-<distro>-<distro-version>-r<revision> |
| 173 | + # Example: 15.3.0-debian-12-r1 |
| 174 | + ``` |
| 175 | + |
| 176 | +2. **Get APP_VERSION from Dockerfile**: |
| 177 | + ```bash |
| 178 | + # Navigate to bitnami/containers/<image>/<major-version>/<distro>-<version>/ |
| 179 | + # Extract APP_VERSION from Dockerfile |
| 180 | + ``` |
| 181 | + |
| 182 | +3. **Update build configuration**: |
| 183 | + ```bash |
| 184 | + # Update commit hash in build workflow |
| 185 | + vi .github/workflows/build_external_container_images.yaml |
| 186 | + ``` |
| 187 | + |
| 188 | +### Step 6: Vendorize Chart Update |
| 189 | +```bash |
| 190 | +# Replace vendorized chart with new version |
| 191 | +cp -r /tmp/<chart-name>/* deployment/charts/<chart-name>/ |
| 192 | + |
| 193 | +# Update Chart.yaml if images changed |
| 194 | +vi deployment/charts/<chart-name>/Chart.yaml |
| 195 | +# Set appVersion to APP_VERSION from Bitnami Containers |
| 196 | + |
| 197 | +# Update values.yaml if needed |
| 198 | +# Replace docker.io/bitnami/* with Chainloop registry paths |
| 199 | +vi deployment/charts/<chart-name>/values.yaml |
| 200 | +``` |
| 201 | + |
| 202 | +### Step 7: Update Dependencies |
| 203 | +```bash |
| 204 | +# CRITICAL: Update dependencies in correct order |
| 205 | + |
| 206 | +# 1. Update vendorized chart dependencies first |
| 207 | +cd deployment/charts/<chart-name> |
| 208 | +helm dependency update |
| 209 | +helm dependency build |
| 210 | + |
| 211 | +# 2. Update main chart dependency version |
| 212 | +cd ../../chainloop |
| 213 | +vi Chart.yaml # Update dependency version to match vendorized chart |
| 214 | + |
| 215 | +# 3. Update main chart dependencies |
| 216 | +helm dependency update |
| 217 | +helm dependency build |
| 218 | + |
| 219 | +cd ../.. |
| 220 | +``` |
| 221 | + |
| 222 | +### Step 8: Clean Up |
| 223 | +```bash |
| 224 | +# Remove temporary files |
| 225 | +rm -rf /tmp/<chart-name> |
| 226 | + |
| 227 | +# Verify working directory is clean |
| 228 | +git status |
| 229 | +``` |
| 230 | + |
| 231 | +--- |
| 232 | + |
| 233 | +## Verification & Testing |
| 234 | + |
| 235 | +### Local Verification |
| 236 | +```bash |
| 237 | +# 1. Lint charts |
| 238 | +helm lint deployment/charts/<chart-name> |
| 239 | +helm lint deployment/chainloop |
| 240 | + |
| 241 | +# 2. Template validation |
| 242 | +helm template deployment/charts/<chart-name> |
| 243 | +helm template deployment/chainloop |
| 244 | + |
| 245 | +# 3. Local testing |
| 246 | +cd devel && docker compose up |
| 247 | + |
| 248 | +# 4. Integration testing |
| 249 | +# Run your specific integration test suite |
| 250 | +``` |
| 251 | + |
| 252 | +### Image Consistency Checks |
| 253 | +```bash |
| 254 | +# Verify consistency across: |
| 255 | +# - Chart.yaml appVersion |
| 256 | +# - values.yaml image tags |
| 257 | +# - Build workflow commit references |
| 258 | + |
| 259 | +grep -r "appVersion\|image.*tag" deployment/charts/<chart-name>/ |
| 260 | +grep -r "<chart-name>" .github/workflows/build_external_container_images.yaml |
| 261 | +``` |
| 262 | + |
| 263 | +--- |
| 264 | + |
| 265 | +## Files Modified |
| 266 | + |
| 267 | +### Chart Files |
| 268 | +- `deployment/charts/<chart-name>/Chart.yaml` - Chart version, appVersion |
| 269 | +- `deployment/charts/<chart-name>/values.yaml` - Image references |
| 270 | +- `deployment/charts/<chart-name>/Chart.lock` - Dependency lock |
| 271 | +- `deployment/chainloop/Chart.yaml` - Main chart dependencies |
| 272 | +- `deployment/chainloop/Chart.lock` - Main dependency lock |
| 273 | + |
| 274 | +### CI/CD Configuration |
| 275 | +- `.github/workflows/build_external_container_images.yaml` - Image build references |
| 276 | + |
| 277 | +--- |
| 278 | + |
| 279 | +## Troubleshooting |
| 280 | + |
| 281 | +| Issue | Symptoms | Solution | |
| 282 | +|-------|----------|----------| |
| 283 | +| **Image Version Mismatch** | Services fail to start | Verify APP_VERSION matches Chart.yaml appVersion | |
| 284 | +| **Build Failures** | CI/CD fails to build images | Check commit reference contains required image versions | |
| 285 | +| **Image Pull Failures** | Deployment fails | Ensure all image tags are consistent and updated | |
| 286 | +| **Dependency Conflicts** | Helm dependency errors | Check compatibility between chart versions | |
| 287 | +| **Missing Container Images** | Image not found errors | Check Bitnami Containers history for renamed/removed images | |
| 288 | + |
| 289 | +### Debug Commands |
| 290 | +```bash |
| 291 | +# Check image references |
| 292 | +grep -r "image:" deployment/charts/<chart-name>/ |
| 293 | + |
| 294 | +# Verify build configuration |
| 295 | +grep -A5 -B5 "<chart-name>" .github/workflows/build_external_container_images.yaml |
| 296 | + |
| 297 | +# Check dependency status |
| 298 | +helm dependency list deployment/chainloop |
| 299 | + |
| 300 | +# Validate chart syntax |
| 301 | +helm lint deployment/charts/<chart-name> --strict |
| 302 | +``` |
| 303 | + |
| 304 | +--- |
| 305 | + |
| 306 | +## Emergency Procedures |
| 307 | + |
| 308 | +### Rollback Steps |
| 309 | +1. **Immediate**: Revert git changes |
| 310 | + ```bash |
| 311 | + git checkout HEAD -- deployment/ |
| 312 | + ``` |
| 313 | + |
| 314 | +2. **Clean state**: Remove lock files and rebuild |
| 315 | + ```bash |
| 316 | + find deployment/ -name "Chart.lock" -delete |
| 317 | + cd deployment/chainloop && helm dependency build |
| 318 | + ``` |
| 319 | + |
| 320 | +3. **Verify**: Test rolled-back version |
| 321 | + ```bash |
| 322 | + cd devel && docker compose down && docker compose up |
| 323 | + ``` |
| 324 | + |
| 325 | +### Escalation Criteria |
| 326 | +**Contact team lead when**: |
| 327 | +- Major version upgrade is required |
| 328 | +- Breaking changes affect core functionality |
| 329 | +- Multiple dependency conflicts arise |
| 330 | +- Data migration is required |
| 331 | + |
| 332 | +--- |
| 333 | + |
| 334 | +## Process Checklist |
| 335 | + |
| 336 | +### Pre-Upgrade |
| 337 | +- [ ] Version compatibility verified (no major version change) |
| 338 | +- [ ] Current state documented |
| 339 | +- [ ] Backup/rollback plan confirmed |
| 340 | + |
| 341 | +### During Upgrade |
| 342 | +- [ ] Target version located and validated |
| 343 | +- [ ] Image changes identified and updated |
| 344 | +- [ ] Charts vendorized correctly |
| 345 | +- [ ] Dependencies updated in correct order |
| 346 | +- [ ] Build configuration updated (if needed) |
| 347 | + |
| 348 | +### Post-Upgrade |
| 349 | +- [ ] Local testing passed |
| 350 | +- [ ] Chart linting clean |
| 351 | +- [ ] Image consistency verified |
| 352 | +- [ ] Integration tests passed |
| 353 | +- [ ] Documentation updated |
| 354 | +- [ ] Temporary files cleaned up |
0 commit comments