Skip to content

Commit 9fb25bd

Browse files
cfsmp3claude
cfsmp3
and
claude
committed
feat: Add safe deployment scripts with automatic rollback
Add deployment scripts that implement a safe deployment flow: - pre_deploy.sh: Validates environment, saves state for rollback - deploy.sh: Pulls code, installs deps, runs migrations, reloads service - post_deploy.sh: Health check with retries and fallback - rollback.sh: Restores previous commit/migration on failure Features: - Concurrent deployment protection (lock file) - Database connection validation before deploy - Commit and migration version saved for rollback - Health check with 6 retries, 5s delay - Fallback to root endpoint if /health doesn't exist - Automatic database migration downgrade on rollback - Comprehensive logging and error messages These scripts are designed to be used with GitHub Actions but can also be run manually for testing or emergency deployments. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
1 parent 53b3d7f commit 9fb25bd

File tree

5 files changed

+589
-0
lines changed

5 files changed

+589
-0
lines changed

‎install/deploy/README.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# Deployment Scripts
2+
3+
These scripts implement safe, automated deployment with automatic rollback on failure.
4+
5+
## Overview
6+
7+
```
8+
┌─────────────────────────────────────────────────────────────┐
9+
│ DEPLOYMENT FLOW │
10+
├─────────────────────────────────────────────────────────────┤
11+
│ pre_deploy.sh → deploy.sh → post_deploy.sh │
12+
│ ↓ ↓ ↓ │
13+
│ Validate env Update code Health check │
14+
│ Save state Run migrations Verify app │
15+
│ ↓ ↓ ↓ │
16+
│ ┌────┴────┐ │
17+
│ ↓ ↓ │
18+
│ SUCCESS FAILURE │
19+
│ ↓ ↓ │
20+
│ Done rollback.sh │
21+
└─────────────────────────────────────────────────────────────┘
22+
```
23+
24+
## Scripts
25+
26+
### pre_deploy.sh
27+
Validates the environment before deployment:
28+
- Checks for concurrent deployments (lock file)
29+
- Verifies install folder and config.py exist
30+
- Tests database connection
31+
- Saves current commit and migration version for rollback
32+
- Checks disk space
33+
34+
### deploy.sh
35+
Performs the actual deployment:
36+
- Fetches and updates code from git
37+
- Installs/updates Python dependencies
38+
- Runs database migrations
39+
- Copies CI scripts to test data directory
40+
- Reloads the platform service
41+
42+
### post_deploy.sh
43+
Verifies deployment was successful:
44+
- Waits for application to start
45+
- Checks `/health` endpoint (with fallback to `/`)
46+
- Retries up to 6 times with 5-second delay
47+
- Returns success (0) or failure (1)
48+
49+
### rollback.sh
50+
Restores previous working state:
51+
- Restores previous git commit
52+
- Downgrades database migrations if needed
53+
- Reinstalls dependencies
54+
- Reloads service
55+
- Verifies rollback was successful
56+
57+
## Environment Variables
58+
59+
| Variable | Default | Description |
60+
|----------|---------|-------------|
61+
| `INSTALL_FOLDER` | `/var/www/sample-platform` | Application directory |
62+
| `SAMPLE_REPOSITORY` | `/repository` | Test data repository |
63+
| `DEPLOY_BRANCH` | `master` | Git branch to deploy |
64+
| `HEALTH_URL` | `http://127.0.0.1/health` | Health check endpoint |
65+
| `FALLBACK_URL` | `http://127.0.0.1/` | Fallback check URL |
66+
| `MAX_RETRIES` | `6` | Health check retry count |
67+
| `RETRY_DELAY` | `5` | Seconds between retries |
68+
69+
## Manual Usage
70+
71+
```bash
72+
# Set environment
73+
export INSTALL_FOLDER="/var/www/sample-platform"
74+
export SAMPLE_REPOSITORY="/repository"
75+
76+
# Run deployment
77+
cd $INSTALL_FOLDER
78+
sudo bash install/deploy/pre_deploy.sh && \
79+
sudo bash install/deploy/deploy.sh && \
80+
sudo bash install/deploy/post_deploy.sh || \
81+
sudo bash install/deploy/rollback.sh
82+
```
83+
84+
## GitHub Actions Integration
85+
86+
These scripts are designed to be called from the GitHub Actions workflow:
87+
88+
```yaml
89+
- name: Pre-deployment checks
90+
run: sudo bash install/deploy/pre_deploy.sh
91+
92+
- name: Deploy
93+
run: sudo bash install/deploy/deploy.sh
94+
95+
- name: Verify deployment
96+
run: sudo bash install/deploy/post_deploy.sh
97+
98+
- name: Rollback on failure
99+
if: failure()
100+
run: sudo bash install/deploy/rollback.sh
101+
```
102+
103+
## Files Created During Deployment
104+
105+
| File | Purpose |
106+
|------|---------|
107+
| `/tmp/sp-deploy.lock` | Prevents concurrent deployments |
108+
| `/tmp/sp-deploy-backup-dir.txt` | Points to backup directory |
109+
| `/tmp/sp-deploy-YYYYMMDD-HHMMSS/` | Backup directory with rollback info |
110+
| `/tmp/health_response.json` | Last health check response |
111+
112+
## Troubleshooting
113+
114+
### Deployment stuck
115+
Check for stale lock file:
116+
```bash
117+
ls -la /tmp/sp-deploy.lock
118+
rm /tmp/sp-deploy.lock # If stale (>10 minutes old)
119+
```
120+
121+
### Health check failing
122+
```bash
123+
# Check application logs
124+
tail -f /var/www/sample-platform/logs/error.log
125+
126+
# Check service status
127+
systemctl status platform
128+
129+
# Test health endpoint manually
130+
curl -v http://127.0.0.1/health
131+
```
132+
133+
### Rollback failed
134+
```bash
135+
# Check current state
136+
cd /var/www/sample-platform
137+
git log --oneline -5
138+
FLASK_APP=./run.py flask db current
139+
140+
# Manual rollback
141+
git checkout <previous-commit>
142+
FLASK_APP=./run.py flask db downgrade <migration-id>
143+
systemctl restart platform
144+
```

‎install/deploy/deploy.sh‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
#!/bin/bash
2+
# Main deployment script
3+
# Exit codes: 0 = success, 1 = failure (triggers rollback)
4+
#
5+
# This script performs the actual deployment steps:
6+
# 1. Pull latest code from git
7+
# 2. Install/update Python dependencies
8+
# 3. Run database migrations
9+
# 4. Copy CI scripts
10+
# 5. Reload the application service
11+
12+
set -e
13+
14+
INSTALL_FOLDER="${INSTALL_FOLDER:-/var/www/sample-platform}"
15+
SAMPLE_REPOSITORY="${SAMPLE_REPOSITORY:-/repository}"
16+
DEPLOY_BRANCH="${DEPLOY_BRANCH:-master}"
17+
18+
echo "=== Starting deployment ==="
19+
echo "Timestamp: $(date -Iseconds)"
20+
echo "Branch: $DEPLOY_BRANCH"
21+
echo "Install folder: $INSTALL_FOLDER"
22+
23+
cd "$INSTALL_FOLDER"
24+
25+
# Step 1: Fetch latest code
26+
echo ""
27+
echo "--- Step 1: Fetching latest code ---"
28+
git fetch origin "$DEPLOY_BRANCH"
29+
echo "✓ Fetched latest from origin/$DEPLOY_BRANCH"
30+
31+
# Step 2: Check for local changes and handle them
32+
if ! git diff --quiet; then
33+
echo "WARNING: Local changes detected, stashing..."
34+
git stash push -m "pre-deploy-$(date +%Y%m%d-%H%M%S)" || true
35+
fi
36+
37+
# Step 3: Update to latest code
38+
echo ""
39+
echo "--- Step 2: Updating code ---"
40+
git checkout "$DEPLOY_BRANCH"
41+
git reset --hard "origin/$DEPLOY_BRANCH"
42+
NEW_COMMIT=$(git rev-parse HEAD)
43+
echo "✓ Updated to commit: $NEW_COMMIT"
44+
45+
# Step 4: Update dependencies
46+
echo ""
47+
echo "--- Step 3: Updating dependencies ---"
48+
python3 -m pip install -r requirements.txt --quiet --disable-pip-version-check
49+
echo "✓ Dependencies updated"
50+
51+
# Step 5: Run database migrations
52+
echo ""
53+
echo "--- Step 4: Running database migrations ---"
54+
FLASK_APP=./run.py python3 -m flask db upgrade
55+
echo "✓ Database migrations complete"
56+
57+
# Step 6: Copy CI scripts (if directories exist)
58+
echo ""
59+
echo "--- Step 5: Updating CI scripts ---"
60+
if [ -d "${SAMPLE_REPOSITORY}/TestData" ]; then
61+
if [ -f "install/ci-vm/ci-linux/ci/bootstrap" ]; then
62+
mkdir -p "${SAMPLE_REPOSITORY}/TestData/ci-linux"
63+
cp "install/ci-vm/ci-linux/ci/bootstrap" "${SAMPLE_REPOSITORY}/TestData/ci-linux/bootstrap"
64+
echo "✓ Copied ci-linux/bootstrap"
65+
fi
66+
if [ -f "install/ci-vm/ci-linux/ci/runCI" ]; then
67+
mkdir -p "${SAMPLE_REPOSITORY}/TestData/ci-linux"
68+
cp "install/ci-vm/ci-linux/ci/runCI" "${SAMPLE_REPOSITORY}/TestData/ci-linux/runCI"
69+
echo "✓ Copied ci-linux/runCI"
70+
fi
71+
if [ -f "install/ci-vm/ci-windows/ci/runCI.bat" ]; then
72+
mkdir -p "${SAMPLE_REPOSITORY}/TestData/ci-windows"
73+
cp "install/ci-vm/ci-windows/ci/runCI.bat" "${SAMPLE_REPOSITORY}/TestData/ci-windows/runCI.bat"
74+
echo "✓ Copied ci-windows/runCI.bat"
75+
fi
76+
else
77+
echo "⚠ TestData directory not found, skipping CI script copy"
78+
fi
79+
80+
# Step 7: Reload application
81+
echo ""
82+
echo "--- Step 6: Reloading application ---"
83+
if systemctl is-active --quiet platform; then
84+
systemctl reload platform
85+
echo "✓ Platform service reloaded"
86+
else
87+
echo "⚠ Platform service not running, attempting start..."
88+
systemctl start platform
89+
echo "✓ Platform service started"
90+
fi
91+
92+
echo ""
93+
echo "=== Deployment complete ==="
94+
echo "Deployed commit: $NEW_COMMIT"
95+
echo "Waiting for application to start..."
96+
exit 0

‎install/deploy/post_deploy.sh‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/bin/bash
2+
# Post-deployment health check
3+
# Exit codes: 0 = healthy, 1 = unhealthy (triggers rollback)
4+
#
5+
# This script verifies the deployment was successful by checking:
6+
# 1. The /health endpoint (if available)
7+
# 2. Fallback: the root endpoint responds with 2xx/3xx
8+
9+
set -e
10+
11+
# Configuration
12+
HEALTH_URL="${HEALTH_URL:-http://127.0.0.1/health}"
13+
FALLBACK_URL="${FALLBACK_URL:-http://127.0.0.1/}"
14+
MAX_RETRIES="${MAX_RETRIES:-6}"
15+
RETRY_DELAY="${RETRY_DELAY:-5}"
16+
17+
echo "=== Post-deployment health check ==="
18+
echo "Timestamp: $(date -Iseconds)"
19+
echo "Health URL: $HEALTH_URL"
20+
echo "Max retries: $MAX_RETRIES"
21+
echo "Retry delay: ${RETRY_DELAY}s"
22+
23+
# Wait for application to start
24+
echo ""
25+
echo "Waiting for application to initialize..."
26+
sleep 3
27+
28+
# Check health endpoint with retries
29+
echo ""
30+
echo "--- Health check ---"
31+
for i in $(seq 1 $MAX_RETRIES); do
32+
echo "Attempt $i/$MAX_RETRIES..."
33+
34+
# Try the /health endpoint first
35+
HTTP_CODE=$(curl -s -o /tmp/health_response.json -w "%{http_code}" "$HEALTH_URL" 2>/dev/null || echo "000")
36+
37+
if [ "$HTTP_CODE" = "200" ]; then
38+
echo "✓ Health check passed (HTTP $HTTP_CODE)"
39+
echo ""
40+
echo "Response:"
41+
cat /tmp/health_response.json 2>/dev/null | python3 -m json.tool 2>/dev/null || cat /tmp/health_response.json
42+
echo ""
43+
echo "=== Deployment verified successfully ==="
44+
exit 0
45+
elif [ "$HTTP_CODE" = "404" ]; then
46+
# Health endpoint doesn't exist, try fallback
47+
echo "Health endpoint not found, trying fallback URL..."
48+
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$FALLBACK_URL" 2>/dev/null || echo "000")
49+
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 400 ]; then
50+
echo "✓ Fallback check passed (HTTP $HTTP_CODE)"
51+
echo ""
52+
echo "=== Deployment verified successfully (using fallback) ==="
53+
exit 0
54+
fi
55+
elif [ "$HTTP_CODE" = "503" ]; then
56+
echo "Health check returned unhealthy (HTTP 503)"
57+
echo "Response:"
58+
cat /tmp/health_response.json 2>/dev/null || echo "(no response body)"
59+
fi
60+
61+
if [ "$i" -lt "$MAX_RETRIES" ]; then
62+
echo "Health check returned HTTP $HTTP_CODE, retrying in ${RETRY_DELAY}s..."
63+
sleep "$RETRY_DELAY"
64+
fi
65+
done
66+
67+
echo ""
68+
echo "=== HEALTH CHECK FAILED ==="
69+
echo "Health check failed after $MAX_RETRIES attempts"
70+
echo "Last HTTP code: $HTTP_CODE"
71+
echo "Last response:"
72+
cat /tmp/health_response.json 2>/dev/null || echo "(no response)"
73+
echo ""
74+
echo "Deployment verification FAILED - rollback recommended"
75+
exit 1

0 commit comments

Comments
 (0)