Merge pull request #5 from NeotomaDB/feature/singlecontainer

Containerized and set up to deploy to AWS.

Author: SimonGoring

.gitignore

@@ -1,5 +1,6 @@
 
 .env
+infrastructure/parameters.json
 
 *.gz
 

CITATION.cff

@@ -0,0 +1,10 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+authors:
+- family-names: "Goring"
+  given-names: "Simon James"
+  orcid: "https://orcid.org/0000-0002-2700-4605"
+title: "Neotoma Snapshot Sanitizer"
+version: 1.0.0
+date-released: 2025-07-28
+url: "https://github.com/NeotomaDB/clean_backup"

CONTRIBUTING.md

@@ -0,0 +1,23 @@
+# Introduction
+
+Thank you for considering contributing to Neotoma's development. It's people like you that make Neotoma such a great community.
+
+Following these guidelines helps to communicate that you respect the time of the developers managing and developing this open source project. In return, they should reciprocate that respect in addressing your issue, assessing changes, and helping you finalize your pull requests.
+
+Improving documentation, bug triaging, or writing tutorials are all examples of helpful contributions that mean less work for the developers.  If you wish to make any of these contributions please read on.
+
+## Submitting changes
+
+Send GitHub Pull Requests to [NeotomaDB/clean_backup](https://github.com/NeotomaDB/clean_backup), adding a clear list of the contributions of the pull request. In particular we'd appreciate test coverage where possible, and more documentation.
+
+Always write a clear log message for your commits. One-line messages are fine for small changes, but bigger changes should look like this:
+
+$ git commit -m "A brief summary of the commit
+>
+> A paragraph describing what changed and its impact."
+
+Please try to use some form of linting, or code style with your contributions. This includes proper indentation (generally, 2 spaces), proper spacing around commas and operators, no trailing whitespace and clear variable naming conventions.
+
+## Asking for Help
+
+Please feel free to contact us through this repositories Issues, by contacting us directly, or through our Slack channel, to ask any questions.

README.md

@@ -1,59 +1,69 @@
+[![NSF-1948926](https://img.shields.io/badge/NSF-1948926-blue.svg)](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1948926)
+[![NSF-2410961](https://img.shields.io/badge/NSF-2410961-blue.svg)](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2410961)
+
 [![lifecycle](https://img.shields.io/badge/lifecycle-stable-green.svg)](https://www.tidyverse.org/lifecycle/#stable)
 
 
 # Neotoma Anonymized Backups
 
-This repository generates a container service for Neotoma that copies the [Neotoma database](https://neotomadb.org) into a container and overwrites sensitive data using a random `md5` hash. The container then uploads the data to a Neotoma [AWS S3 bucket]() where the snapshot is made publically available.
+This repository generates a container service for Neotoma that copies the [Neotoma Paleoecology Database](https://neotomadb.org) into a Docker container and overwrites sensitive data using a random `md5` hash. The bash script running in the container then uploads the data to a Neotoma AWS S3 bucket where the snapshot is made publically available through a URL that is shared on the Neotoma website.
+
+The compressed file (`neotoma_clean_{DATETIME}.tr.gz`) includes a [bash script](archives/regenbash.sh) that will re-build the database in a user's local Postgres instance. Currently the bash script only runs for Mac and Linux. There is an experimental [Windows batch script](archives/experimental_win_restore.bat) that can be used with caution.
+
+We welcome any user contributions see the [contributors guide](CONTRIBUTING.md).
+
+## Restoring the Database
 
-The compressed file (XXXX) includes a small README and a script to re-build the database in a local Postgres instance.
+The most recent snapshot of the Neotoma Database will always be tagged as `neotoma_clean_latest` in the compressed file, but the actual SQL file used to restore the database will be named with the date the snapshot was taken. Generally, the snapshots will be taken every month. If there is a need for a more recent snapshot, please contact the database administrators to request a newer snapshot.
 
-The following installation instructions were tested on PostgreSQL version 16, using script regenbash.sh (mac and linux).
-Alternatively, the commands can be entered directly in the command line. PostgreSQL must already be installed.
+### Postgres Extensions Used
 
-## Postgres Extensions Used
+The Docker container uses Postgres 15, and the current RDS database version is PostgreSQL v15.14. The local database requires the following extensions to be installed before you can restore Neotoma locally:
 
-* [pg_trgm](https://www.postgresql.org/docs/current/pgtrgm.html)
+* [pg_trgm](https://www.postgresql.org/docs/current/pgtrgm.html): Helps with full-text searching of publications.
 * [intarray](https://www.postgresql.org/docs/9.1/intarray.html)
-* [unaccent](https://www.postgresql.org/docs/current/unaccent.html)
-* External: [postgis](https://postgis.net/)
-* External: [vector/pgvector](https://github.com/pgvector/pgvector)
+* [unaccent](https://www.postgresql.org/docs/current/unaccent.html): Helps with searches for terms that may include accents (sitenames, contact names).
+* External: [postgis](https://postgis.net/): Helps manage spatial data.
 
-These extensions are used to improve functionality within the Neotoma Database. External tools such as `postgis` and `pgvector` must be installed prior to creation within the Postgres server. We include the bash script in an effort to help users make the restoration process as simple as possible.
+These extensions are used to improve functionality within the Neotoma Database. The `pg_grgm`, `intarray`, and `unaccent` extensions are included with PostgreSQL. External tools such as `postgis` must be installed prior to creation within the Postgres server.
 
-## Restoring the Database
+The [regenbash.sh](archives/regenbash.sh) script automates some of the creation of the extensions within the restored database.
+
+### Restoring from the Cloud
+
+The *most recent* version of the clean database is always uploaded as a `.tar.gz` file to Neotoma S3 cloud storage. You can download it directly by clicking the badge below. Note that this download is over 2 Gigs in size.
+
+[![Download Snapshot](https://img.shields.io/badge/Download-Neotoma--Snapshot-orange.svg)](https://neotoma-remote-store.s3.us-east-2.amazonaws.com/neotoma_clean_latest.tar.gz)
+
+Once the file is downloaded, you can extract it locally. The file archive contains the following files (the terminal date for the sql file may differ):
 
-1. If you haven't already, [download the backup](https://neotomaprimarybackup.s3.us-east-2.amazonaws.com/clean_dump.tar.gz) to your local drive.
+* dbsetup.sql
+* experimental_win_restore.bat
+* regenbash.sh
+* neotoma_clean_2025-07-01.sql
 
-2. Unzip the snapshot file (with commandline, or a tool):
+Once you execute `regenbash.sh` (Mac/Linux) or `experimental_win_restore.bat` (Windows) the database will be restored from the text file to your local database within a database `neotoma` at which point you can use the database from whichever database management system you'd like to use.
 
-	`gunzip clean_dump.tar.gz`
+## AWS Infrastructure
 
-2. Enter the folder and restore database using the command `bash regenbash.sh`.  For help, use: `bash regenbash.sh --help`
+The backup itself is generated through AWS. There are two steps, the first is packaging the Docker image and sending it to ECR, the second is initiating the Batch job, which will run the scripts in the Docker container.
 
-	`bash regenbash.sh`
+![AWS Configuration](/assets/AWS_scrub_database_infrastructure.svg)
 
-	The script performs the following actions. A password prompt will appear at each step:
-	
-		The database "neotoma" is first dropped if it exists;
-		The new "neotoma" is created;
-		Extenesions are installled;
-		The snapshot file (neotoma_ndb_only_2024-03-18.sql) is loaded into the new database.
+All files (with the exception of files that directly expose secrets) are available in this repository. All secrets are contained in a `parameters.yaml` file in the `./infrastructure` folder. We provide a [`parameters-template.yaml`](./infrastructure/parameters-template.json) file for convenience, so that users can see which key-value pairs are needed for full implementation of the workflow.
 
-3. Alternatively, instead of using the script, the commands can be entered directly via command line:
+### Docker Configuration
 
-	dropdb neotoma -h localhost -U username
-	createdb neotoma -h localhost -U username
-	psql -h localhost -d neotoma -U username -c "CREATE EXTENSION postgis;"
-	psql -h localhost -d neotoma -U username -c "CREATE EXTENSION pg_trgm;"
-	psql -h localhost -d neotoma -U username -f neotoma_ndb_only_2024-03-18.sql
+The Docker [configuration file](batch.Dockerfile) sets up a container with PostgreSQL 15 and PostGIS. The Docker container sets up the system, creates a connection to a containerized Postgres database, and then uses `pg_dump` to create a plaintext SQL dump of the remote Neotoma database that is restored within the container. To sanitize the database of sensitive data we execute the script [`app/scrubbed_database.sh`](app/scrubbed_database.sh). The SQL statements write over rows in the Data Stewards tables as well as the Contacts tables.
 
+The Docker container is built and deployed to the AWS ECR using the script [`build-and-push.sh`](build-and-push.sh). For this script to work, the user must have the AWS CLI installed, and have permissions to access Neotoma AWS services.
 
+### AWS Infrastructure Builder
 
-4. To view database using command line interactive terminal:
+The scripts [`deploy.sh`](deploy.sh) and [`update.sh`](update.sh) are used to deploy the [Batch Infrastructure](infrastructure/batch-infrastructure.yaml) configuration to CloudFormation, which will then be used to define the AWS Batch run when a job is submitted.
 
-	psql neotoma username
+Within the infrastructure file there is a defined `ScheduleRule`, which uses the EventBridge [`cron()`](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-scheduled-rule-pattern.html) scheduler to execute the backup snapshot at 2am on the first day of each month.  Single instances of the job can also be executed using [`test_job.sh`](test_job.sh).
 
-	Meta-command \d ("describe") will list all the tables in the publice schema. To view the schema (ndb) and tables in the database,
-	expand the search path by entering the command:
+## Final Overview
 
-		SET search_path TO 'ndb', public;
+With this repository, we implement a monthly backup system using AWS infrastructure to provide Neotoma users with a sanitized version of the database for local use on their personal systems.

README.txt

@@ -0,0 +1,13 @@
+#!/bin/bash
+# connect_database.sh - Simple database connection
+
+set -e
+
+echo "Setting up database connection..."
+
+# Use direct RDS connection
+echo "Using direct RDS connection"
+export DB_HOST=${RDS_ENDPOINT:-"neotomaprivate.cxkwxkjpj8zi.us-east-2.rds.amazonaws.com"}
+export DB_PORT=${RDS_PORT:-"5432"}
+
+echo "Database connection configured: ${DB_HOST}:${DB_PORT}"

app/app.Dockerfile

@@ -1,31 +1,59 @@
 #!/bin/bash
-
 set -e
 
 # by: Simon Goring
 # Documentation for pg_dump is available at https://www.postgresql.org/docs/current/app-pgdump.html
 # This script uses pg_dump to duplicate the whole database to a local version.
 
-bash /home/app/connect_cloud.sh
+# We're logging out to a proper log file here.
+exec > >(tee -a /var/log/db-sanitize.log)
+exec 2>&1
+
+echo "💡 Starting database sanitization at $(date)"
+DATESTAMP=$(date +"%Y-%m-%d")
+
+echo "🔌 Connecting to the primary Neotoma database..."
+source /home/app/connect_database.sh
 
-echo Dumping:
+echo "⛁ Dumping the primary database from ${DB_HOST}:${DB_PORT}:"
 export PGPASSWORD=$REMOTE_PASSWORD
+pg_dump -v -O -C -c --no-owner -x -U $REMOTE_USER -h ${DB_HOST} -p ${DB_PORT} \
+    --no-subscriptions -T ap.globalmammals -T ap.icesheets -N cron -Fp -d neotoma > /home/archives/tempdump.dump
 
-pg_dump -C -v -O --no-owner -x -U $REMOTE_USER -h localhost -p 5454 -T ap.globalmammals -N cron -Fc -d neotoma > /home/archives/tempdump.dump
+echo "Checking to ensure the dump is stable:"
+pg_restore --list /home/archives/tempdump.dump | head -20
 
+echo "🛠 Restoring the database locally"
 export PGPASSWORD=$POSTGRES_PASSWORD
-psql -U postgres -h pgneotoma -d postgres -c "DROP DATABASE IF EXISTS neotoma;"
-psql -U postgres -h pgneotoma -d postgres -c "CREATE DATABASE neotoma;"
-psql -U postgres -h pgneotoma -d postgres -c "CREATE EXTENSION IF NOT EXISTS postgis;"
-psql -U postgres -h pgneotoma -d postgres -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
-psql -U postgres -h pgneotoma -d postgres -c "CREATE EXTENSION IF NOT EXISTS vector;"
-psql -U postgres -h pgneotoma -d postgres -c "CREATE EXTENSION IF NOT EXISTS intarray;"
-psql -U postgres -h pgneotoma -d postgres -c "CREATE EXTENSION IF NOT EXISTS unaccent;"
-pg_restore -U postgres -h pgneotoma -c --no-owner --no-privileges -d neotoma /home/archives/tempdump.dump
-psql -U postgres -h pgneotoma -d neotoma -c "UPDATE ti.stewards SET username=SUBSTRING(md5(random()::text) from 1 for 10), pwd=SUBSTRING(md5(random()::text) from 1 for 10);"
-psql -U postgres -h pgneotoma -d neotoma -c "UPDATE ndb.contacts SET address=SUBSTRING(md5(random()::text) from 1 for 10), phone=SUBSTRING(md5(random()::text) from 1 for 10), fax=SUBSTRING(md5(random()::text) from 1 for 10), email=SUBSTRING(md5(random()::text) from 1 for 10);"
-PGPASSWORD=postgres pg_dump -C -v -O --no-owner -x  -Fc -p 5432 -h pgneotoma -d neotoma -U postgres > /home/archives/neotoma_clean.dump
+psql -U postgres -h localhost -d postgres -c "DROP DATABASE IF EXISTS neotoma;"
+psql -U postgres -h localhost -d postgres -c "CREATE DATABASE neotoma;"
+psql -U postgres -h localhost -d postgres -c "CREATE EXTENSION IF NOT EXISTS postgis;"
+psql -U postgres -h localhost -d postgres -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
+psql -U postgres -h localhost -d postgres -c "CREATE EXTENSION IF NOT EXISTS intarray;"
+psql -U postgres -h localhost -d postgres -c "CREATE EXTENSION IF NOT EXISTS unaccent;"
+
+echo "Restoring database in container. Checking for errors."
+
+psql -U postgres -h localhost -d neotoma < /home/archives/tempdump.dump
+
+
+echo "🧹 Cleaning up all sensitive data from the database."
+psql -U postgres -h localhost -d neotoma -c "UPDATE ti.stewards SET username=SUBSTRING(md5(random()::text) from 1 for 10), pwd=SUBSTRING(md5(random()::text) from 1 for 10);"
+psql -U postgres -h localhost -d neotoma -c "UPDATE ndb.contacts SET address=SUBSTRING(md5(random()::text) from 1 for 10), phone=SUBSTRING(md5(random()::text) from 1 for 10), fax=SUBSTRING(md5(random()::text) from 1 for 10), email=SUBSTRING(md5(random()::text) from 1 for 10);"
+
+echo "✍🏼 Creating the final cleaned dump."
+PGPASSWORD=postgres pg_dump -C -v -O --no-owner -x  -Fc -p 5432 -h localhost -d neotoma -U postgres > /home/archives/neotoma_clean_${DATESTAMP}.dump
+
+echo "📦 Compressing the dumped database."
+tar -zcvf /home/archives/neotoma_clean_${DATESTAMP}.tar.gz -C /home/archives/ .
+
+echo "💾 Uploading the archive to S3."
+aws s3 cp /home/archives/neotoma_clean_${DATESTAMP}.tar.gz s3://neotoma-remote-store/ --content-encoding "application/x-compressed-tar"
+aws s3 cp s3://neotoma-remote-store/neotoma_clean_${DATESTAMP}.tar.gz s3://neotoma-remote-store/neotoma_clean_latest.tar.gz --content-encoding "application/x-compressed-tar"
+
+echo "🗑️ Removing temporary files..."
 rm /home/archives/tempdump.dump
-tar -zcvf /home/archives/clean_dump.tar.gz /home/archives/
+rm /home/archives/neotoma_clean_${DATESTAMP}.dump
+rm /home/archives/neotoma_clean_${DATESTAMP}.tar.gz
 
-aws s3 cp --content-encoding "application/x-compressed-tar" /home/archives/clean_dump.tar.gz s3://neotomaprimarybackup
+echo "✔ Database sanitization completed successfully at $(date)"

app/connect_cloud.sh

@@ -1,7 +1,5 @@
 DROP DATABASE IF EXISTS neotoma;
-CREATE DATABASE neotoma;
 CREATE EXTENSION IF NOT EXISTS postgis;
 CREATE EXTENSION IF NOT EXISTS pg_trgm;
-CREATE EXTENSION IF NOT EXISTS vector;
 CREATE EXTENSION IF NOT EXISTS intarray;
-CREATE EXTENSION IF NOT EXISTS unaccent;
+CREATE EXTENSION IF NOT EXISTS unaccent;