Fix pg_dump for hash partitioning on enum columns #1176
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Author
|
Automatic partition handling: Auto-enables --load-via-partition-root when dumping partitioned tables with hash partitioning on enum types to prevent partition constraint violations after dump/reload cycles |
yjhjstz
reviewed
Jul 1, 2025
yjhjstz
reviewed
Jul 1, 2025
yjhjstz
reviewed
Jul 1, 2025
tuhaihe
reviewed
Jul 22, 2025
tuhaihe
requested changes
Jul 22, 2025
Member
tuhaihe
left a comment
tuhaihe
left a comment
There was a problem hiding this comment.
Updates:
This PR is cherry-picked from the PG upstream: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=bc8cd50fefd369b217f80078585c486505aafb62#patch4.
When cherry-picking commits, we need to preserve the original commit and message. If changes are necessary, create a new commit to apply them, thereby respecting the original author's contributions, as described here.
I would like to request changes to this PR again.
tuhaihe
force-pushed
the
fix_pg_dump
branch
2 times, most recently
from
9a71425 to
515c029
Compare
tuhaihe
approved these changes
Jul 31, 2025
yjhjstz
approved these changes
Aug 1, 2025
gongxun0928
reviewed
Aug 4, 2025
Hash partitioning on an enum is problematic because the hash codes are derived from the OIDs assigned to the enum values, which will almost certainly be different after a dump-and-reload than they were before. This means that some rows probably end up in different partitions than before, causing restore to fail because of partition constraint violations. (pg_upgrade dodges this problem by using hacks to force the enum values to keep the same OIDs, but that's not possible nor desirable for pg_dump.) Users can work around that by specifying --load-via-partition-root, but since that's a dump-time not restore-time decision, one might find out the need for it far too late. Instead, teach pg_dump to apply that option automatically when dealing with a partitioned table that has hash-on-enum partitioning. Also deal with a pre-existing issue for --load-via-partition-root mode: in a parallel restore, we try to TRUNCATE target tables just before loading them, in order to enable some backend optimizations. This is bad when using --load-via-partition-root because (a) we're likely to suffer deadlocks from restore jobs trying to restore rows into other partitions than they came from, and (b) if we miss getting a deadlock we might still lose data due to a TRUNCATE removing rows from some already-completed restore job. The fix for this is conceptually simple: just don't TRUNCATE if we're dealing with a --load-via-partition-root case. The tricky bit is for pg_restore to identify those cases. In dumps using COPY commands we can inspect each COPY command to see if it targets the nominal target table or some ancestor. However, in dumps using INSERT commands it's pretty impractical to examine the INSERTs in advance. To provide a solution for that going forward, modify pg_dump to mark TABLE DATA items that are using --load-via-partition-root with a comment. (This change also responds to a complaint from Robert Haas that the dump output for --load-via-partition-root is pretty confusing.) pg_restore checks for the special comment as well as checking the COPY command if present. This will fail to identify the combination of --load-via-partition-root and --inserts in pre-existing dump files, but that should be a pretty rare case in the field. If it does happen you will probably get a deadlock failure that you can work around by not using parallel restore, which is the same as before this bug fix. Having done this, there seems no remaining reason for the alarmism in the pg_dump man page about combining --load-via-partition-root with parallel restore, so remove that warning. Patch by me; thanks to Julien Rouhaud for review. Back-patch to v11 where hash partitioning was introduced. Discussion: https://postgr.es/m/[email protected]
Since we cherry-picked the Postgres pg_dump commit to Cloudberry, we need to update the files to support Cloudberry CI to let me pass.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #ENUM_HASH_PARTITION_ISSUE
Fixes #PARALLEL_RESTORE_DEADLOCK
What does this PR do?
--load-via-partition-rootwhen dumping partitioned tables with hash partitioning on enum types to prevent partition constraint violations after dump/reload cycles--load-via-partition-rootwith parallel restore from pg_dump documentationType of Change
Breaking Changes
None
Test Plan
Impact
Performance:
User-facing changes:
No significant change in dump performance for partitioned tables
Parallel restore stability improves overall throughput by eliminating deadlock points
Dependencies:
Users no longer need to manually specify --load-via-partition-root for enum hash partitions
Eliminates data corruption risks during pg_upgrade of partitioned tables
Provides clearer dump file annotation structure
Checklist
Additional Context
Compatibility note:
For existing dumps using --inserts + --load-via-partition-root:
Parallel restore might still trigger deadlocks (legacy file limitation)
Mitigation: Use single-threaded restore or regenerate dump files
CI Skip Instructions