Merge pull request #1130 from NASA-IMPACT/database_import_bug_fixes

Updated database restore command

Author: CarsonDavis

README.md

@@ -18,7 +18,6 @@ $ docker-compose -f local.yml build
 ```bash
 $ docker-compose -f local.yml up
 ```
-
 ### Non-Docker Local Setup
 
 If you prefer to run the project without Docker, follow these steps:
@@ -69,12 +68,23 @@ $ docker-compose -f local.yml run --rm django python manage.py createsuperuser
 #### Creating Additional Users
 
 Create additional users through the admin interface (/admin).
+## Database Backup and Restore
+
+COSMOS provides dedicated management commands for backing up and restoring your PostgreSQL database. These commands handle both compressed and uncompressed backups and work seamlessly in both local and production environments using Docker.
+
+### Backup Directory Structure
 
-### Database Backup and Restore
+All backups are stored in the `/backups` directory at the root of your project. This directory is mounted as a volume in both local and production Docker configurations, making it easy to manage backups across different environments.
 
-COSMOS provides dedicated management commands for backing up and restoring your PostgreSQL database. These commands handle both compressed and uncompressed backups and automatically detect your server environment from your configuration.
+- Local development: `./backups/`
+- Production server: `/path/to/project/backups/`
 
-#### Creating a Database Backup
+If the directory doesn't exist, create it:
+```bash
+mkdir backups
+```
+
+### Creating a Database Backup
 
 To create a backup of your database:
 
@@ -85,23 +95,24 @@ docker-compose -f local.yml run --rm django python manage.py database_backup
 # Create an uncompressed backup
 docker-compose -f local.yml run --rm django python manage.py database_backup --no-compress
 
-# Specify custom output location
-docker-compose -f local.yml run --rm django python manage.py database_backup --output /path/to/output.sql
+# Specify custom output location within backups directory
+docker-compose -f local.yml run --rm django python manage.py database_backup --output my_custom_backup.sql
 ```
 
 The backup command will automatically:
 - Detect your server environment (Production/Staging/Local)
 - Use database credentials from your environment settings
 - Generate a dated filename if no output path is specified
+- Save the backup to the mounted `/backups` directory
 - Compress the backup by default (can be disabled with --no-compress)
 
-#### Restoring from a Database Backup
+### Restoring from a Database Backup
 
-To restore your database from a backup:
+To restore your database from a backup, it will need to be in the `/backups` directory. You can then run the following command:
 
 ```bash
 # Restore from a backup (handles both .sql and .sql.gz files)
-docker-compose -f local.yml run --rm django python manage.py database_restore path/to/backup.sql[.gz]
+docker-compose -f local.yml run --rm django python manage.py database_restore backups/backup_file_name.sql.gz
 ```
 
 The restore command will:
@@ -111,7 +122,7 @@ The restore command will:
 - Restore all data from the backup
 - Handle all database credentials from your environment settings
 
-#### Working with Remote Servers
+### Working with Remote Servers
 
 When working with production or staging servers:
 
@@ -120,44 +131,37 @@ When working with production or staging servers:
 # For production
 ssh user@production-server
 cd /path/to/project
-
-# For staging
-ssh user@staging-server
-cd /path/to/project
 ```
 
-2. Then run the backup command with the production configuration:
+2. Create a backup on the remote server:
 ```bash
 docker-compose -f production.yml run --rm django python manage.py database_backup
 ```
 
-3. Copy the backup to your local machine:
+3. Copy the backup from the remote server's backup directory to your local machine:
 ```bash
-scp user@remote-server:/path/to/backup.sql.gz ./local-backup.sql.gz
+scp user@remote-server:/path/to/project/backups/backup_name.sql.gz ./backups/
 ```
 
-4. Finally, restore locally:
+4. Restore locally:
 ```bash
-docker-compose -f local.yml run --rm django python manage.py database_restore local-backup.sql.gz
+docker-compose -f local.yml run --rm django python manage.py database_restore backups/backup_name.sql.gz
 ```
 
-#### Alternative Methods
-
-While the database_backup and database_restore commands are the recommended approach, there are alternative methods available:
+### Alternative Methods
 
-##### Using JSON Fixtures (for smaller datasets)
-If you're working with a smaller dataset, you can use Django's built-in fixtures:
+While the database_backup and database_restore commands are the recommended approach, you can also use Django's built-in fixtures for smaller datasets:
 
 ```bash
 # Create a backup excluding content types
-docker-compose -f production.yml run --rm --user root django python manage.py dumpdata \
+docker-compose -f production.yml run --rm django python manage.py dumpdata \
     --natural-foreign --natural-primary \
     --exclude=contenttypes --exclude=auth.Permission \
     --indent 2 \
-    --output /app/backups/prod_backup-$(date +%Y%m%d).json
+    --output backups/prod_backup-$(date +%Y%m%d).json
 
 # Restore from a fixture
-docker-compose -f local.yml run --rm django python manage.py loaddata /path/to/backup.json
+docker-compose -f local.yml run --rm django python manage.py loaddata backups/backup_name.json
 ```
 
 Note: For large databases (>1.5GB), the database_backup and database_restore commands are strongly recommended over JSON fixtures as they handle large datasets more efficiently.

sde_collections/management/commands/database_backup.py

@@ -4,15 +4,16 @@
 Usage:
     docker-compose -f local.yml run --rm django python manage.py database_backup
     docker-compose -f local.yml run --rm django python manage.py database_backup --no-compress
-    docker-compose -f local.yml run --rm django python manage.py database_backup --output /path/to/output.sql
+    docker-compose -f local.yml run --rm django python manage.py database_backup --output my_backup.sql
     docker-compose -f production.yml run --rm django python manage.py database_backup
+
+All backups are stored in the /backups directory, which is mounted as a volume in both local
+and production environments. If specifying a custom output path, it will be relative to this directory.
 """
 
-import enum
 import gzip
 import os
 import shutil
-import socket
 import subprocess
 from contextlib import contextmanager
 from datetime import datetime
@@ -21,21 +22,6 @@
 from django.core.management.base import BaseCommand
 
 
-class Server(enum.Enum):
-    PRODUCTION = "PRODUCTION"
-    STAGING = "STAGING"
-    UNKNOWN = "UNKNOWN"
-
-
-def detect_server() -> Server:
-    hostname = socket.gethostname().upper()
-    if "PRODUCTION" in hostname:
-        return Server.PRODUCTION
-    elif "STAGING" in hostname:
-        return Server.STAGING
-    return Server.UNKNOWN
-
-
 @contextmanager
 def temp_file_handler(filename: str):
     """Context manager to handle temporary files, ensuring cleanup."""
@@ -58,36 +44,45 @@ def add_arguments(self, parser):
         parser.add_argument(
             "--output",
             type=str,
-            help="Output file path (default: auto-generated based on server name and date)",
+            help="Output file path (default: auto-generated in /app/backups directory)",
         )
 
-    def get_backup_filename(self, server: Server, compress: bool, custom_output: str = None) -> tuple[str, str]:
+    def get_backup_filename(self, compress: bool, custom_output: str = None) -> tuple[str, str]:
         """Generate backup filename and actual dump path.
 
         Args:
-            server: Server enum indicating the environment
             compress: Whether the output should be compressed
             custom_output: Optional custom output path
 
         Returns:
-            tuple[str, str]: A tuple containing (final_filename, temp_filename)
-                - final_filename: The name of the final backup file (with .gz if compressed)
-                - temp_filename: The name of the temporary dump file (always without .gz)
+            tuple[str, str]: A tuple containing:
+                - final_filename: Full path for the final backup file (with .gz if compressed)
+                - temp_filename: Full path for the temporary dump file (without .gz)
         """
+        backup_dir = "/app/backups"
+        os.makedirs(backup_dir, exist_ok=True)
+
         if custom_output:
+            # If custom_output is relative, make it relative to backup_dir
+            if not custom_output.startswith("/"):
+                custom_output = os.path.join(backup_dir, custom_output)
+
             # Ensure the output directory exists
             output_dir = os.path.dirname(custom_output)
             if output_dir:
                 os.makedirs(output_dir, exist_ok=True)
 
             if compress:
-                return custom_output + (".gz" if not custom_output.endswith(".gz") else ""), custom_output.removesuffix(
+                return custom_output + (
+                    ".gz" if not custom_output.endswith(".gz") else ""
+                ), custom_output.removesuffix(  # noqa
                     ".gz"
                 )
             return custom_output, custom_output
         else:
             date_str = datetime.now().strftime("%Y%m%d")
-            temp_filename = f"{server.value.lower()}_backup_{date_str}.sql"
+            env_name = os.getenv("BACKUP_ENVIRONMENT", "unknown")
+            temp_filename = os.path.join(backup_dir, f"{env_name}_backup_{date_str}.sql")
             final_filename = f"{temp_filename}.gz" if compress else temp_filename
             return final_filename, temp_filename
 
@@ -116,9 +111,15 @@ def compress_file(self, input_file: str, output_file: str) -> None:
                 shutil.copyfileobj(f_in, f_out)
 
     def handle(self, *args, **options):
-        server = detect_server()
+        if not os.getenv("BACKUP_ENVIRONMENT"):
+            self.stdout.write(
+                self.style.WARNING(
+                    "Note: Set BACKUP_ENVIRONMENT in your env if you want automatic environment-based filenames"
+                )
+            )
+
         compress = not options["no_compress"]
-        backup_file, dump_file = self.get_backup_filename(server, compress, options.get("output"))
+        backup_file, dump_file = self.get_backup_filename(compress, options.get("output"))
 
         env = os.environ.copy()
         env["PGPASSWORD"] = settings.DATABASES["default"]["PASSWORD"]
@@ -133,10 +134,10 @@ def handle(self, *args, **options):
 
             self.stdout.write(
                 self.style.SUCCESS(
-                    f"Successfully created {'compressed ' if compress else ''}backup for {server.value}: {backup_file}"
+                    f"Successfully created {'compressed ' if compress else ''}backup at: backups/{os.path.basename(backup_file)}"  # noqa
                 )
             )
         except subprocess.CalledProcessError as e:
-            self.stdout.write(self.style.ERROR(f"Backup failed on {server.value}: {str(e)}"))
+            self.stdout.write(self.style.ERROR(f"Backup failed: {str(e)}"))
         except Exception as e:
             self.stdout.write(self.style.ERROR(f"Error during backup process: {str(e)}"))

sde_collections/management/commands/database_restore.py

@@ -2,8 +2,11 @@
 Management command to restore PostgreSQL database from backup.
 
 Usage:
-    docker-compose -f local.yml run --rm django python manage.py database_restore path/to/backup.sql[.gz]
-    docker-compose -f production.yml run --rm django python manage.py database_restore path/to/backup.sql[.gz]
+    docker-compose -f local.yml run --rm django python manage.py database_restore backups/backup.sql[.gz]
+    docker-compose -f production.yml run --rm django python manage.py database_restore backups/backup.sql[.gz]
+
+The backup file should be located in the /backups directory, which is mounted as a volume in both
+local and production environments.
 """
 
 import enum