Migration Tooling / Release Process (#27)

* use poetry

* add push to github docker registry

* fix release.yml formatting error

* add .dockerignore file

* remove docker/make files that are not used anymore

* split pypgstac code to migrate.py load.py to help organize, add migration path logic for multi-step incremental migrations

* add pydocstyle comments

* add new scripts to enable a staging process for sql migrations

* use scripts/stageversion to add steps between all versions

* use scripts/stageversion to add steps between all versions

* Add documentation on release process to README

Author: bitner

.dockerignore

@@ -0,0 +1,7 @@
+.envrc*
+*/dist/*
+*.pyc
+*.egg-info
+*.eggs
+venv/*
+*/.direnv/*

Makefile

@@ -19,31 +19,46 @@ cd pypgstac
 poetry build pypgstac
 pip install dist/pypgstac-[version]-py3-none-any.whl
 ```
+PyPGStac will get the database connection settings from the standard PG environment variables. It can also take a DSN database url "postgresql://..." via the --dsn flag.
+ - PGHOST
+ - PGPORT
+ - PGUSER
+ - PGDATABASE
+ - PGPASSWORD
+
+## Making Changes to SQL
+All changes to SQL should only be made in the sql directory. SQL Files will be run in alphabetical order.
+
+## Adding Tests
+PGStac uses PGTap to test SQL. Tests can be found in tests/pgtap.sql and are run using `scripts/test`
+
 
 ## Migrations
-To install the latest version of pgstac on an empty database, you can directly load the primary source code.
-```
-psql -f pgstac.sql
-```
-Which calls the main SQL files in the sql directory.
+PyPGStac has a utility to help apply migrations to an existing PGStac instance to bring it up to date.
 
-All development should take place on these files.
+Migrations are stored in pypgstac/pypgstac/migrations and are distributed with the PyPGStac package. There are two types of migrations.
+ - Base migrations install PGStac into a database with no current PGStac installation. These migrations follow the file pattern "pgstac.[version].sql"
+ - Incremental migrations are used to move PGStac from one version to the next. These migrations follow the file pattern "pgstac.[version].[fromversion].sql"
 
-For each new version of PGStac, two migrations should be added to pypgstac/pypgstac/migrations/
- - pgstac.[version].sql (equivalent to `cat sql/*.sql > migration.sql && echo "insert into migrations (versions) VALUES ('[version]')" >> migration.sql)
- - pgstac.[version].[fromversion].sql (Migration to move from existing version to new version, can be created by hand or using the makemigration.sh tool below)
+These migrations can be created when releasing a new version of PGStac using"
+```
+scripts/stageversion [version or increment type]
+```
+Examples:
+```
+scripts/stageversion 0.2.8
+scripts/stageversion patch # if current version is 0.2.7 will bump to 0.2.8
+scripts/stageversion minor # if current version is 0.2.7 will bump to 0.3.0
+scripts/stageversion major # if current version is 0.2.7 will bump to 1.0.0
+```
+This will create a base migration for the new version and will create incremental migrations between any existing base migrations. The incremental migrations that are automatically generated by this script will have the extension ".staged" on the file. You must manually review (and make any modifications necessary) this file and remove the ".staged" extension to enable the migration.
 
 ### Running Migrations
-Migrations can be installed by either directly running the appropriate migration sql file for from and target PGStac versions:
-`psql -f pypgstac/pypgstac/migrations/pgstac.0.1.8.sql`
-
-Or by using pypgstac:
-`pypgstac migrate`
+PyPGStac has a utility for checking the version of an existing PGStac database and applying the appropriate migrations in the correct order. It can also be used to setup a database from scratch.
 
-### Creating Migrations Using Schema Diff
-To create a migration from a previous version of pgstac you can calculate the migration from the running instance of pgstac using the makemigration.sh command. This will use docker to copy the schema of the existing database and the new sql into new docker databases and create/test the migration between the two.
+To create an initial PGStac database or bring an existing one up to date:
 ```
-makemigration.sh postgresql://myuser:mypassword@myhost:myport/mydatabase
+pypgstac migrate
 ```
 
 ## Bulk Data Loading
@@ -100,4 +115,21 @@ scripts/console --db
 To run migrations on the development database, use
 ```bash
 scripts/migrate
-```
+```
+
+To stage code and configurations and create template migrations for a version release, use
+```bash
+scripts/stageversion
+```
+
+## Release Process
+1) Make sure all your code is added and committed
+2) Create a PR against the main branch
+3) Once the PR has been merged, start the release process.
+4) Use `scripts/stagerelease` as documented in migrations section above making sure to rename any files ending in ".staged" in the migrations section
+5) Add details for release to the CHANGELOG
+6) Add/Commit any changes
+7) Run tests `scripts/test`
+8) Create a git tag `git tag v0.2.8` using new version number
+9) Push the git tag `git push origin v0.2.8`
+10) The CI process will push pypgstac to PyPi, create a docker image on ghcr.io, and create a release on github.

README.md

@@ -1 +1,2 @@
+"""PyPGStac Version."""
 __version__ = "0.2.8"