Adding changes to allow Postgres-to-Postgres migrations with schema, trigger, and index migration support via mirrors #3684
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
|
|
jpruiz114
force-pushed
the
feature/postgres-schema-object-migration
branch
from
d3a2970 to
a6f3672
Compare
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
force-pushed
the
feature/postgres-schema-object-migration
branch
from
a6f3672 to
5b114ae
Compare
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
force-pushed
the
feature/postgres-schema-object-migration
branch
from
5b114ae to
396879b
Compare
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
force-pushed
the
feature/postgres-schema-object-migration
branch
from
396879b to
8a3fdb5
Compare
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
Author
|
Some notes to self and for visibility: I've created this repository with some additional files to run PeerDB locally and a few test PostgreSQL databases for the Postgres-to-Postgres migrations supporting schema, trigger, and index migration: https://github.com/jpruiz114/peerdb-postgres-improvement-testing |
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
force-pushed
the
feature/postgres-schema-object-migration
branch
from
9a91fe4 to
ddc34db
Compare
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
force-pushed
the
feature/postgres-schema-object-migration
branch
from
ddc34db to
9074841
Compare
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
force-pushed
the
feature/postgres-schema-object-migration
branch
from
9074841 to
8844f2a
Compare
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
jpruiz114
had a problem deploying
to
external-contributor
GitHub Actions
Failure
…trigger, and index migration support via mirrors Adding linting change to remediate ST1005: error strings should not be capitalized (staticcheck) Make schema migration test independent of catalog schema Allow test to run against any PostgreSQL database by using direct PostgresConfig from env vars instead of catalog config. Fix prealloc lint warning by pre-allocating schemaDeltas slice Fix perfsprint lint warning by replacing fmt.Sprintf with string concatenation Fix perfsprint lint warnings: replace fmt.Sprintf with string concatenation Fix line length lint warnings: break long lines to meet 144 char limit Fix dupArg lint warning: remove no-op strings.ReplaceAll call Fix gci lint warning: sort imports alphabetically fix: reorder imports in schema.go to satisfy gci linter fix: resolve linting issues in postgres connector - Fix import ordering in schema.go (gci) - Replace fmt.Sprintf with string concatenation for schema migration logs (perfsprint) - Split long line in constraint drop statement (lll) fix: address linting issues in postgres connector Fix gci import ordering, replace fmt.Sprintf with string concatenation, and split long lines to meet linting requirements. fix schema migration issues so triggers and constraints migrate correctly Removing test files Adding changes to attempt and remediate behavior where some artifacts are not replicated Add existence checks and savepoints for triggers and constraints migration - Check if triggers exist before creating to prevent transaction abort - Check if constraints exist before creating to prevent transaction abort - Use savepoints to isolate failures per table - Follow SQL script order: Functions -> Triggers -> Constraints - Continue processing other tables if one fails Move constraint existence check before drop to prevent transaction abort - Check if constraint exists BEFORE attempting to drop it - This prevents transaction abort when constraint was created inline with table - Fixes issue where users_email_key constraint causes rollback Skip UNIQUE constraints that are already enforced by indexes - When a UNIQUE constraint has a corresponding index with the same name, skip creating the constraint - PostgreSQL automatically creates indexes for UNIQUE constraints - Since we create indexes before constraints, the index already exists and enforces uniqueness - This fixes the issue where UNIQUE constraints fail because their index was already created Skip UNIQUE indexes during extraction if corresponding constraint exists - Prevent duplicate extraction of UNIQUE indexes that have corresponding UNIQUE constraints - PostgreSQL automatically creates indexes for UNIQUE constraints - By skipping the index during extraction, we only migrate the constraint - The constraint creation will automatically create the index - This prevents conflicts and duplicate work - Update constraint application comment to reflect this is now a safety check Workaround: simplify constraint and trigger creation logic - Remove DROP CONSTRAINT logic that could cause transaction aborts - Check if constraint/trigger exists and skip if it does - Create constraint/trigger and handle 'already exists' error gracefully - Never abort transaction on errors - always continue with other tables - This prevents transaction rollbacks from failed DROP operations - Simpler and more robust approach Support PostgreSQL 16+ NOT NULL constraints - Extract NOT NULL constraints (type 'n') from pg_constraint - Skip them during migration as they're handled by column definitions - PostgreSQL 16+ tracks NOT NULL as explicit constraints - Column definitions already include NOT NULL, so no need to re-apply fix(postgres): replicate UNIQUE and NOT NULL constraints properly fix: update test calls to match ReplayTableSchemaDeltas signature fix: correct NOT NULL constraint syntax and relax UNIQUE index skip test fix: don't pass indexes in UNIQUE constraint test - constraints create their own indexes fix: filter out auto-created NOT NULL constraints in UNIQUE constraint test fix: use unique table names in UNIQUE constraint test to avoid conflicts fix: use simple static table names instead of random suffixes to avoid transaction issues fix: add explicit DROP TABLE before creating test tables to handle previous failed runs fix: simplify UNIQUE constraint test to avoid connection pool issues minor format change to schema migration test
jpruiz114
force-pushed
the
feature/postgres-schema-object-migration
branch
from
8844f2a to
b8d8d44
Compare
jpruiz114
requested a deployment
to
external-contributor
GitHub Actions
Waiting
jpruiz114
requested a deployment
to
external-contributor
GitHub Actions
Waiting
jpruiz114
requested a deployment
to
external-contributor
GitHub Actions
Waiting
jpruiz114
requested a deployment
to
external-contributor
GitHub Actions
Waiting
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.