Skip to content

Commit 4af53ec

Browse files
sohel2020sohel2020
authored
feat: Db migration script (#54)
* BA PSQL backup and restore script * update precommit
1 parent 79ba441 commit 4af53ec

File tree

4 files changed

+290
-0
lines changed

4 files changed

+290
-0
lines changed

‎.pre-commit-config.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,10 @@ repos:
1010
- id: check-added-large-files
1111
- id: check-merge-conflict
1212
- id: mixed-line-ending
13+
14+
- repo: https://github.com/compilerla/conventional-pre-commit
15+
rev: v4.3.0
16+
hooks:
17+
- id: conventional-pre-commit
18+
stages: [commit-msg]
19+
args: [--strict, --force-scope, feat, fix, docs, test, ci, refactor, perf, chore, revert"]

‎scripts/db-migration/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.sql

‎scripts/db-migration/README.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
2+
# PostgreSQL Backup & Restore Migration Guide
3+
4+
This guide describes the process for migrating PostgreSQL databases from Bitnami pods (per-app DB pods in Kubernetes) to a single PG Cloud Native cluster using the `backup_restore.sh` script.
5+
6+
## Overview
7+
8+
**Migration Path:** Bitnami PostgreSQL (individual pods) → PG Cloud Native (shared cluster)
9+
- **Source:** Per-application database pods in Kubernetes
10+
- **Target:** Single PostgreSQL cluster with multiple databases
11+
- **Method:** Dump and restore using custom backup script
12+
13+
## Prerequisites
14+
- Downtime is required during backup and restore. Block access to the database service (scale down application pods, disable ingress, or update NetworkPolicy) to prevent new data insertion.
15+
- The script requires a `.env` file with DB credentials. For PG Cloud Native, credentials are stored in Kubernetes secrets with prefix `qs-postgresql-cluster-access-`.
16+
- The script must be run in an environment with `psql` (version 17) and `pg_dump` installed.
17+
18+
## Migration Steps
19+
20+
### 1. Backup from Bitnami PostgreSQL
21+
22+
**Option A: kubectl port-forward**
23+
```sh
24+
kubectl port-forward svc/<bitnami-db-service> 5432:5432
25+
# Example:
26+
kubectl port-forward svc/bitnami-postgresql-pharia 5432:5432
27+
```
28+
Create a `.env` file with DB credentials (host: `localhost`, port: `5432`).
29+
Run the backup script:
30+
```sh
31+
./pg_backup_restore.sh backup <output_file.sql>
32+
```
33+
34+
**Option B: Pod with psql 17**
35+
1. Launch a pod with psql 17 installed.
36+
2. Inside the pod, create the `.env` file and copy the `backup_restore.sh` script.
37+
3. In `.env`, set `DB_HOST` to the Bitnami database service name and `DB_PORT` to `5432`.
38+
4. Run:
39+
```sh
40+
./pg_backup_restore.sh backup <output_file.sql>
41+
```
42+
43+
### 2. Restore to PG Cloud Native
44+
45+
PG Cloud Native exposes a single service endpoint for all databases. The endpoint will be either `qs-postgresql-cluster-pharia-rw` or `qs-postgresql-cluster-temporal-rw` based on the database.
46+
Retrieve DB credentials from the Kubernetes secret (prefix: `qs-postgresql-cluster-access-`).
47+
Create a `.env` file with the new credentials.
48+
49+
If needed, port-forward the PG Cloud Native service:
50+
```sh
51+
kubectl port-forward svc/qs-postgresql-cluster-pharia-rw 5432:5432
52+
# Or for temporal:
53+
kubectl port-forward svc/qs-postgresql-cluster-temporal-rw 5432:5432
54+
```
55+
Run the restore script:
56+
```sh
57+
./pg_backup_restore.sh restore <input_file.sql>
58+
# Example:
59+
./pg_backup_restore.sh restore backup-pharia.sql
60+
```
61+
62+
### 3. Switch Application Configuration
63+
- Update application config to use the new PG Cloud Native database endpoint and credentials.
64+
- Redeploy applications if needed.
65+
66+
### 4. Validate Migration
67+
- Test application connectivity and data integrity.
68+
- Resume normal operations.
69+
70+
## Example .env File
71+
```
72+
DB_HOST=your_host
73+
DB_PORT=5432
74+
DB_USER=your_user
75+
DB_PASSWORD=your_password
76+
DB_NAME=your_dbname
77+
```
78+
79+
## Notes
80+
- The script reads credentials from `.env` in the same directory.
81+
- For PG Cloud Native, all databases share one cluster and endpoint through the PG pooler.
82+
- Always use psql 17 for compatibility.
83+
- Ensure backups are stored securely and tested before deleting old databases.
84+
85+
## Troubleshooting
86+
- If you see connection errors, verify port-forwarding, credentials, and network access.
87+
- Ensure downtime is enforced to prevent data loss.
88+
- For restore, make sure the target database exists and is empty or ready for import.

‎scripts/db-migration/pg_backup_restore.sh‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
#!/bin/bash
2+
# pg_backup_restore.sh
3+
# Backup and restore PostgreSQL databases using static credentials from a .env file
4+
# Backup will exclude ownership information
5+
6+
set -e
7+
8+
# Check for psql version 17
9+
check_psql_version() {
10+
echo "🔧 Checking PostgreSQL client version..."
11+
12+
if ! command -v psql >/dev/null 2>&1; then
13+
echo "❌ Error: PostgreSQL client (psql) is not installed." >&2
14+
echo "💡 Please install PostgreSQL 17 client tools." >&2
15+
exit 2
16+
fi
17+
18+
local version
19+
version=$(psql --version | awk '{print $3}')
20+
major_version=$(echo "$version" | cut -d. -f1)
21+
22+
echo "📋 Found PostgreSQL client version: $version"
23+
24+
if [ "$major_version" != "17" ]; then
25+
echo "❌ Error: PostgreSQL client version 17 is required. Found version $version." >&2
26+
echo "💡 Please install PostgreSQL 17 client tools for compatibility." >&2
27+
exit 3
28+
fi
29+
30+
echo "✅ PostgreSQL client version 17 verified!"
31+
echo ""
32+
}
33+
34+
# Load environment variables from .env file in the same directory as the script
35+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
36+
ENV_FILE="$SCRIPT_DIR/.env"
37+
38+
echo "🗂️ Loading database configuration from .env file..."
39+
40+
if [ -f "$ENV_FILE" ]; then
41+
echo "✓ Found .env file at: $ENV_FILE"
42+
set -a
43+
# shellcheck source=.env
44+
source "$ENV_FILE"
45+
set +a
46+
echo "✓ Environment variables loaded successfully"
47+
echo ""
48+
else
49+
echo "❌ Error: .env file not found at $ENV_FILE" >&2
50+
echo "💡 Please create a .env file with the required database configuration." >&2
51+
exit 4
52+
fi
53+
54+
# Validate required environment variables
55+
require_env_vars() {
56+
local missing=0
57+
echo "🔍 Validating database configuration..."
58+
59+
for var in DB_HOST DB_PORT DB_USER DB_PASSWORD DB_NAME; do
60+
if [ -z "${!var}" ]; then
61+
echo "❌ Error: $var is not set in .env file." >&2
62+
missing=1
63+
else
64+
if [ "$var" != "DB_PASSWORD" ]; then
65+
echo "✓ $var: ${!var}"
66+
else
67+
echo "✓ $var: [HIDDEN]"
68+
fi
69+
fi
70+
done
71+
72+
if [ "$missing" -eq 1 ]; then
73+
echo ""
74+
echo "❌ Configuration validation failed! Please check your .env file."
75+
exit 5
76+
fi
77+
78+
echo "✅ Database configuration validated successfully!"
79+
echo ""
80+
}
81+
82+
usage() {
83+
echo ""
84+
echo "PostgreSQL Backup & Restore Script"
85+
echo "----------------------------------"
86+
echo "Usage:"
87+
echo " $0 backup <output_file> # Create a backup of the database to <output_file>"
88+
echo " $0 restore <input_file> # Restore the database from <input_file>"
89+
echo ""
90+
echo "Environment:"
91+
echo " Reads DB credentials from .env file in the same directory as this script."
92+
echo " .env file must contain:"
93+
echo " DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME"
94+
echo ""
95+
echo "Examples:"
96+
echo " $0 backup backup.sql"
97+
echo " $0 restore backup.sql"
98+
echo ""
99+
exit 1
100+
}
101+
102+
run_backup() {
103+
local output_file="$1"
104+
echo "🔄 Starting database backup operation..."
105+
echo "📊 Database: $DB_NAME"
106+
echo "🖥️ Host: $DB_HOST:$DB_PORT"
107+
echo "👤 User: $DB_USER"
108+
echo "📁 Output file: $output_file"
109+
echo ""
110+
echo "⏳ Running pg_dump (this may take a while for large databases)..."
111+
112+
PGPASSWORD="$DB_PASSWORD" pg_dump --host="$DB_HOST" --port="$DB_PORT" --username="$DB_USER" --no-owner --format=plain --file="$output_file" "$DB_NAME"
113+
114+
if [ $? -eq 0 ]; then
115+
echo "✅ Backup completed successfully!"
116+
echo "📄 Backup file: $output_file"
117+
echo "📊 File size: $(du -h "$output_file" | cut -f1)"
118+
else
119+
echo "❌ Backup failed! Please check your database connection and permissions." >&2
120+
exit 6
121+
fi
122+
}
123+
124+
run_restore() {
125+
local input_file="$1"
126+
127+
# Check if input file exists
128+
if [ ! -f "$input_file" ]; then
129+
echo "❌ Error: Backup file '$input_file' does not exist!" >&2
130+
exit 8
131+
fi
132+
133+
echo "🔄 Database restore operation"
134+
echo "📊 Target database: $DB_NAME"
135+
echo "🖥️ Host: $DB_HOST:$DB_PORT"
136+
echo "👤 User: $DB_USER"
137+
echo "📁 Source file: $input_file"
138+
echo "📊 File size: $(du -h "$input_file" | cut -f1)"
139+
echo ""
140+
echo "⚠️ WARNING: This will overwrite all data in the target database!"
141+
echo ""
142+
143+
read -p "🤔 Are you sure you want to restore the database '$DB_NAME' from '$input_file'? [y/N]: " confirm
144+
if [[ "$confirm" =~ ^[Yy]$ ]]; then
145+
echo ""
146+
echo "⏳ Running database restore (this may take a while for large backups)..."
147+
148+
PGPASSWORD="$DB_PASSWORD" psql --host="$DB_HOST" --port="$DB_PORT" --username="$DB_USER" --dbname="$DB_NAME" -f "$input_file"
149+
150+
if [ $? -eq 0 ]; then
151+
echo ""
152+
echo "✅ Database restore completed successfully!"
153+
echo "📊 Database '$DB_NAME' has been restored from '$input_file'"
154+
else
155+
echo ""
156+
echo "❌ Restore failed! Please check the backup file format and database permissions." >&2
157+
exit 7
158+
fi
159+
else
160+
echo ""
161+
echo "🚫 Restore operation cancelled by user."
162+
exit 0
163+
fi
164+
}
165+
166+
if [ $# -lt 2 ]; then
167+
usage
168+
fi
169+
170+
echo "🚀 PostgreSQL Backup & Restore Script Starting..."
171+
echo ""
172+
173+
check_psql_version
174+
require_env_vars
175+
176+
COMMAND="$1"
177+
FILE="$2"
178+
179+
echo "📋 Operation requested: $COMMAND"
180+
echo "📁 Target file: $FILE"
181+
echo ""
182+
183+
case "$COMMAND" in
184+
backup)
185+
run_backup "$FILE"
186+
;;
187+
restore)
188+
run_restore "$FILE"
189+
;;
190+
*)
191+
echo "❌ Error: Unknown command '$COMMAND'" >&2
192+
usage
193+
;;
194+
esac

0 commit comments

Comments
 (0)