rclone added

Author: chaos-dotcom

.env.example

@@ -25,4 +25,16 @@ PRIMARY_DB=your_laterbase_primary_db_name
 PGADMIN_EMAIL=your_pgadmin_email@example.com
 
 # Password for the default pgAdmin user login
-PGADMIN_PASSWORD=your_pgadmin_password
+PGADMIN_PASSWORD=your_pgadmin_password
+
+# --- Backup Agent Configuration (Rclone & Retention) ---
+# Name of the rclone remote configured in your rclone.conf (e.g., "mygdrive", "s3remote")
+# Leave blank to disable cloud upload.
+RCLONE_REMOTE_NAME=
+
+# Path within the rclone remote where backups should be stored (e.g., "resolve_backups/production")
+# Leave blank to disable cloud upload.
+RCLONE_REMOTE_PATH=
+
+# Number of days to keep local backups in the ./backups volume (default is 7 if not set)
+BACKUP_RETENTION_DAYS=7

README.md

@@ -3,7 +3,7 @@
 This project, nicknamed "Laterbase", sets up a Docker-based environment specifically designed for **DaVinci Resolve PostgreSQL databases**. It consists of:
 
 1.  **PostgreSQL Standby Server:** Creates a hot standby replica of your primary DaVinci Resolve PostgreSQL database using streaming replication.
-2.  **Hourly Backup Agent:** Performs hourly logical backups (`pg_dump`) of your primary DaVinci Resolve database.
+2.  **Hourly Backup Agent:** Performs hourly logical backups (`pg_dump`) of your primary DaVinci Resolve database, optionally uploads them to cloud storage using `rclone`, and manages local backup retention.
 3.  **pgAdmin 4 UI:** Provides a web-based graphical interface for managing and monitoring both the primary DaVinci Resolve database and the standby replica.
 
 <p align="center">
@@ -24,14 +24,21 @@ graph TD
 
     subgraph "Laterbase (Docker Host)"
         StandbyDB[("laterbase-standby<br>PostgreSQL Standby")]
-        BackupAgent["laterbase-backup-agent<br>(Hourly pg_dump)"]
+        BackupAgent["laterbase-backup-agent<br>(Hourly pg_dump, rclone upload, retention)"]
         PgAdminUI[("laterbase-pgadmin<br>pgAdmin 4 UI")]
-        BackupVolume[/backups Volume/]
+        BackupVolume[/Local Backups<br>(./backups)/]
+        RcloneConfigVolume[/Rclone Config<br>(./rclone_config)/]
+    end
+
+    subgraph "Cloud Storage (Optional)"
+        CloudBackup{{Rclone Remote<br>e.g., S3, Google Drive}}
     end
 
     PrimaryDB -- Streaming Replication --> StandbyDB
     PrimaryDB -- pg_dump --> BackupAgent
     BackupAgent -- Writes .sql.gz --> BackupVolume
+    BackupAgent -- Copies .sql.gz --> CloudBackup
+    BackupAgent -- Reads Config --> RcloneConfigVolume
     PgAdminUI -- User Connects --> PrimaryDB
     PgAdminUI -- User Connects --> StandbyDB
     StandbyDB -- Uses --> ReplicationSlot
@@ -42,6 +49,8 @@ graph TD
     style PgAdminUI fill:#0b9e92,stroke:#333,stroke-width:2px
     style ReplicationSlot fill:#359e0b,stroke:#333,stroke-width:1px
     style BackupVolume fill:#9e8f0b,stroke:#333,stroke-width:1px
+    style RcloneConfigVolume fill:#b0a01c,stroke:#333,stroke-width:1px
+    style CloudBackup fill:#1caab0,stroke:#333,stroke-width:2px
 ```
 
 **Note on Physical Replication Slot:** A physical replication slot (`laterbase_standby_slot` in this setup) is a feature on the primary PostgreSQL server. It ensures that the primary server retains the necessary transaction logs (WAL segments) required by the standby server, even if the standby disconnects temporarily. This prevents the standby from falling too far behind and needing a full resynchronization.
@@ -56,6 +65,11 @@ graph TD
     *   Replace `YOUR_PGADMIN_PASSWORD_HERE` with the password you want for the pgAdmin login.
     *   Adjust `PRIMARY_PORT` or `PRIMARY_USER` if they differ from the defaults (5432, postgres).
     *   Optionally, uncomment and set `POSTGRES_USER`, `POSTGRES_DB`, or `PGDATA` under the "Standby Server Configuration" section to override the defaults used by the standby service.
+    *   **Rclone Upload (Optional):**
+        *   Set `RCLONE_REMOTE_NAME` to the name of the remote you have configured in your `rclone.conf` file (e.g., `mygdrive`, `s3remote`). Leave blank to disable uploads.
+        *   Set `RCLONE_REMOTE_PATH` to the directory path within your rclone remote where backups should be stored (e.g., `resolve_backups/production`). Leave blank to disable uploads.
+    *   **Backup Retention (Optional):**
+        *   Set `BACKUP_RETENTION_DAYS` to the number of days you want to keep local backups in the `./backups` directory. Defaults to 7 if not set.
 
 2.  **Primary PostgreSQL Server Preparation (`PRIMARY_HOST`):**
     * Ensure the target database is marked as shared or accessible on the resolve box
@@ -77,6 +91,16 @@ graph TD
         mkdir backups
         ```
 
+4.  **Rclone Configuration (Optional):**
+    *   If you want to use the cloud upload feature:
+        *   Create a directory named `rclone_config` in the same directory as `docker-compose.yml`:
+            ```bash
+            mkdir rclone_config
+            ```
+        *   Place your configured `rclone.conf` file inside the `./rclone_config/` directory. The backup container will mount this directory read-only at `/config/rclone.conf`.
+        *   Ensure the remote name you use in `rclone.conf` matches the `RCLONE_REMOTE_NAME` set in your `.env` file.
+        *   **Security Note:** The `rclone.conf` file contains sensitive credentials. Ensure appropriate file permissions are set on the host machine (`chmod 600 ./rclone_config/rclone.conf`).
+
 ## Usage
 
 1.  **Build and Start Containers:**
@@ -109,5 +133,6 @@ graph TD
 *   `setup_standby.sh`: Script run inside the standby container on first start to perform the initial base backup and configure replication.
 *   `backup.sh`: Script run by cron inside the backup agent container to perform hourly `pg_dump` backups.
 *   `crontab.txt`: Defines the cron schedule for `backup.sh`.
-*   `./backups/` (Directory to be created): Host directory where backup files (`.sql.gz`) will be stored by the backup agent.
+*   `./backups/` (Directory to be created): Host directory where local backup files (`.sql.gz`) will be stored by the backup agent.
+*   `./rclone_config/` (Directory to be created, optional): Host directory containing the `rclone.conf` file for cloud uploads.
 

backup/Dockerfile.backup

@@ -5,10 +5,12 @@ FROM debian:bullseye-slim
 RUN apt-get update && apt-get install -y --no-install-recommends \
     cron \
     postgresql-client \
+    rclone \
+    ca-certificates \
     && rm -rf /var/lib/apt/lists/*
 
 # Create directory for backup scripts and backups
-RUN mkdir -p /etc/cron.d /app /backups
+RUN mkdir -p /etc/cron.d /app /backups /config
 
 # Copy the backup script and crontab
 COPY backup.sh /app/backup.sh

backup/backup.sh

@@ -12,6 +12,13 @@ DB_USER="${PRIMARY_USER:-postgres}"
 DB_NAME="${PRIMARY_DB}" # This needs to be the actual Resolve DB name
 # PGPASSWORD should be set as an environment variable in docker-compose
 
+# Rclone Configuration (Set these in docker-compose/.env)
+RCLONE_REMOTE_NAME="${RCLONE_REMOTE_NAME}" # e.g., mygdrive
+RCLONE_REMOTE_PATH="${RCLONE_REMOTE_PATH}" # e.g., resolve_backups
+
+# Backup Retention (Set in docker-compose/.env, defaults to 7 days)
+BACKUP_RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-7}"
+
 BACKUP_DIR="/backups"
 TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
 BACKUP_FILENAME="${DB_NAME}_backup_${TIMESTAMP}.sql.gz"
@@ -30,6 +37,14 @@ if [ -z "$PGPASSWORD" ]; then
   echo "$(date): Error: PGPASSWORD environment variable is not set." >> /var/log/cron.log
   exit 1
 fi
+if [ -z "$RCLONE_REMOTE_NAME" ]; then
+  echo "$(date): Warning: RCLONE_REMOTE_NAME is not set. Backup will not be uploaded." >> /var/log/cron.log
+  # Decide if this should be a fatal error (exit 1) or just a warning
+fi
+if [ -z "$RCLONE_REMOTE_PATH" ]; then
+  echo "$(date): Warning: RCLONE_REMOTE_PATH is not set. Backup will not be uploaded." >> /var/log/cron.log
+  # Decide if this should be a fatal error (exit 1) or just a warning
+fi
 
 # --- Backup Execution ---
 echo "$(date): Starting backup of database '${DB_NAME}' from host '${DB_HOST}:${DB_PORT}' to ${BACKUP_PATH}" >> /var/log/cron.log
@@ -52,9 +67,24 @@ else
   exit 1
 fi
 
-# --- Optional: Cleanup old backups (e.g., keep last 7 days) ---
-# echo "$(date): Cleaning up backups older than 7 days..." >> /var/log/cron.log
-# find "$BACKUP_DIR" -name "${DB_NAME}_backup_*.sql.gz" -type f -mtime +7 -delete
+# --- Rclone Upload ---
+if [ -n "$RCLONE_REMOTE_NAME" ] && [ -n "$RCLONE_REMOTE_PATH" ]; then
+  RCLONE_DESTINATION="${RCLONE_REMOTE_NAME}:${RCLONE_REMOTE_PATH}"
+  echo "$(date): Uploading backup to ${RCLONE_DESTINATION}..." >> /var/log/cron.log
+  rclone copy "$BACKUP_PATH" "$RCLONE_DESTINATION" --config /config/rclone.conf --log-file /var/log/cron.log --log-level INFO
+  if [ $? -eq 0 ]; then
+    echo "$(date): Upload successful: ${BACKUP_FILENAME} to ${RCLONE_DESTINATION}" >> /var/log/cron.log
+  else
+    echo "$(date): Error: Upload failed for ${BACKUP_FILENAME} to ${RCLONE_DESTINATION}." >> /var/log/cron.log
+    # Decide if upload failure should be fatal (exit 1) or just logged
+  fi
+else
+   echo "$(date): Skipping rclone upload as RCLONE_REMOTE_NAME or RCLONE_REMOTE_PATH is not set." >> /var/log/cron.log
+fi
+
+# --- Local Cleanup ---
+echo "$(date): Cleaning up local backups older than ${BACKUP_RETENTION_DAYS} days in ${BACKUP_DIR}..." >> /var/log/cron.log
+find "$BACKUP_DIR" -name "${DB_NAME}_backup_*.sql.gz" -type f -mtime "+${BACKUP_RETENTION_DAYS}" -print -delete >> /var/log/cron.log 2>&1
 
 echo "$(date): Backup process finished." >> /var/log/cron.log
 

docker-compose.yml

@@ -51,9 +51,16 @@ services:
       # Optional: Define these in .env if different from defaults in backup.sh
       # PRIMARY_PORT: ${PRIMARY_PORT:-5432}
       # PRIMARY_USER: ${PRIMARY_USER:-postgres}
+      # --- Rclone & Retention Configuration (Define in .env) ---
+      RCLONE_REMOTE_NAME: ${RCLONE_REMOTE_NAME:-} # e.g., mygdrive
+      RCLONE_REMOTE_PATH: ${RCLONE_REMOTE_PATH:-} # e.g., resolve_backups
+      BACKUP_RETENTION_DAYS: ${BACKUP_RETENTION_DAYS:-7} # Default 7 days
     volumes:
       # Mount a local directory on the host to store the backups
       - ./backups:/backups
+      # Mount a local directory containing rclone.conf
+      # Create ./rclone_config directory and place your rclone.conf file there
+      - ./rclone_config:/config:ro # Mount read-only for security
     restart: unless-stopped
     networks:
       - laterbase_net