v4.0.0 Initial release for PG11 support. Major security requirement changes. See CHANGELOG for full details.

Author: keithf4

CHANGELOG.txt

@@ -1,3 +1,58 @@
+4.0.0
+-- Initial release to provide support PostgreSQL 11. All PG11 functionality is subject to change before its final stable release.
+
+-- IMPORTANT POSTGRESQL 11 UPGRADE STEPS
+    -- All foreign keys, any non-unique indexes, and any unique indexes that include the partition key must be created on the actual parent table and removed from the template table for those property inheritances to continue working. Foreign key inheritance from the template table no longer works at all.
+    -- Any unique indexes or primary keys that do not include the partition column must still use the template table in order to be inherited to child tables.
+    -- Indexes defined on the parent will take precidence over those defined on the template table. Duplicate indexes should not happen. If they do, please report this as a bug. 
+
+-- Note that if this version is installed on PG10 and older, the procedures will not be created. A migration script to update pg_partman to include those will be made available once PG11 stable has been released.
+
+-- Major change in privilege/security of this extension. SECURITY DEFINER is no longer in use to cause most of pg_partman's functions to run as the owner of the called function. This is mostly due to the requirements to support PROCEDUREs in PG11, but it's also generally a better idea for security in the long run.
+    -- It is recommended to create a dedicated role for pg_partman maintenance runs and give it the privileges outlined in the setup section of the top level README.
+    -- The role running maintenance must now have permissions to create tables in the target schema. If you start seeing permission errors after upgrading, this is most likely what needs to be fixed.
+    -- The role running maintenance must now always be the owner of the tables it is managing. This shouldn't be too drastic of a change since previously it was making the maintenace function owner the owner of the partitioned tables. However, if you've changed partition table ownership after initial creation, you may run into issues until you set the pg_partman role as the owner.
+    -- If you are using logical replication, the partman maintenance role must be the owner of the publication in use.
+    -- As a bonus, the ALTER DEFAULT PRIVILEGES feature should now be much easier to use since you have a dedicated role for maintaining partitions. Check the documentation for this feature so that you can set any read/write privileges you want new partition sets to have at the time they are initially created.
+    -- Two functions retain their SECURITY DEFINER status: 
+        -- check_name_length() - Used inside the trigger function and I don't want to break existing installations. It's an IMMUTABLE function so the security implications aren't that great.
+        -- apply_privileges() - Requires access to pg_authid & also potentially changes object ownership.
+    -- If you're using pg_jobmon, privileges for that extension are managed independently of pg_partman. See that extension's documentation for privileges it requires.    
+
+-- For PostgreSQL 11+, started conversion of some python scripts into PROCEDUREs that are now installed within the database. Scripts are still available for pre-11 versions, but will be phased out as soon as PG10 is no longer under support.
+    -- Note that the user that calls these procedures must have permissions to write to the partition set and create/drop partitions in the set. 
+    -- partition_data.py script is now the partition_data_proc() PROCEDURE.
+    -- undo_partition.py script is now the undo_partition_proc() PROCEDURE. 
+    -- reapply_constraints.py is now the reapply_constraints_proc() PROCEDURE.
+    -- reapply_foreign_keys.py not converted since it's not needed for PG11+. Foreign keys can be applied to the parent and will automatically be inherited.
+    -- reapply_indexes.py was not converted yet because concurrent index creation is not yet possible inside a PROCEDURE.
+    -- Unable to turn autovacuum off/on reliably in PROCEDURE as was previously done with python scripts. If autovac is an issue when (un)partitioning large tables, recommend manually disabling it.
+
+-- Consolidated undo_partition_time(), undo_partition_id(), and undo_partition_native() functions into a single undo_partition() function.
+-- The old undo_partition() function that just copied data from child tables to the parent has been removed. The only utility this really had was a possible way to support undoing partition sets not maintained by pg_partman. With native partitioning now in place, its continued usability is now questionable and it was also causing confusion since people were using it and not the previous ones specialized for pg_partman. Can incorporate back in if someone can provide a further beneficial use case.
+
+-- For PG11+, a DEFAULT partition is automatically created as part of the partition set to handle any data that does not have a matching child table. 
+    -- It's just given a _default suffix on the existing parent table name.
+    -- Support within pg_partman for moving data out of the default partition is not included in 4.0.0, but will be included in a future release.
+    -- New parameter to show_partitions() can include the DEFAULT in its output, but it's not output unless specifically requested since it's not commonly needed in internal code where it's mostly used.
+
+-- For native partitioning, added option to give a "source" table to the partition_data*() functions that is different from the parent table of the partition set. This assists with partitioning an existing table to a natively partitioned table.
+
+-- For PostgreSQL 11+, run_maintenance_proc() has been added and should cause much less contention when running against many large partition sets in a single call.
+    -- For PG11+, this is the preferred maintenance method to run over the old run_maintenance() function. The old one will continue to work for the next few major releases, but please update all pg_partman maintenence calls in your environment to use the new procedure.
+    
+-- Simplified --type argument for undo_partition.py script for PG versions 10 and below. Similar to create_parent(), values are now 'partman' or 'native'.
+
+-- Maintenance that creates new partitions on PG11+ and is using native partitioning will no longer analyze the partition sets by default. This should greatly cut down on maintenance run time and contention issues. If you're noticing any odd query plans, especially if you're using additionally configured constraints, you may need to schedule some manual analyzes or you can set the p_analyze parameter in either the function or procedure call to "true".
+
+-- Fixed bug in run_maintenance() with subpartitioning where it would throw an error if retention dropped one of the parent tables undergoing maintenance.
+
+-- Fixed template table inheritance for PG10 not working on minor version PostgreSQL 10.0 (but you really shouldn't be running this version anymore).
+
+-- Fixed compilation errors for background worker against PostgreSQL 11.
+
+
+
 3.2.1
 -- Fix bug with inheriting indexes from template tables with native partitioning. Was throwing syntax error related to format() function. Thanks to iwarford on github for reporting the issue (Github Issue #223).
 

META.json

@@ -1,7 +1,7 @@
 {
     "name": "pg_partman",
     "abstract": "Extension to manage partitioned tables by time or ID",
-    "version": "3.2.1",
+    "version": "4.0.0",
     "maintainer": [ 
         "Keith Fiske <[email protected]>"
     ],
@@ -20,9 +20,9 @@
     },
     "provides": {
         "pg_partman": {
-            "file": "sql/pg_partman--3.2.1.sql",
+            "file": "sql/pg_partman--4.0.0.sql",
             "docfile": "doc/pg_partman.md",
-            "version": "3.2.1",
+            "version": "4.0.0",
             "abstract": "Extension to manage partitioned tables by time or ID"
         }
     },

Makefile

@@ -4,24 +4,44 @@ EXTVERSION = $(shell grep default_version $(EXTENSION).control | \
 
 PG_CONFIG = pg_config
 PG94 = $(shell $(PG_CONFIG) --version | egrep " 8\.| 9\.0| 9\.1| 9\.2| 9\.3" > /dev/null && echo no || echo yes)
+PG11 = $(shell $(PG_CONFIG) --version | egrep " 8\.| 9\.| 10\." > /dev/null && echo no || echo yes)
 
 ifeq ($(PG94),yes)
+
 DOCS = $(wildcard doc/*.md)
-SCRIPTS = bin/*.py
 MODULES = src/pg_partman_bgw
+
 # If user does not want the background worker, run: make NO_BGW=1
 ifneq ($(NO_BGW),)
 	MODULES=
 endif
+
 all: sql/$(EXTENSION)--$(EXTVERSION).sql
 
+# Only include scripts common to all version of PG in PG11+. 
+# Otherwise if less than PG11, include the old python scripts along with common.
+# Only include procedures in PG11+
+ifeq ($(PG11),yes)
+SCRIPTS = bin/common/*.py
+
+sql/$(EXTENSION)--$(EXTVERSION).sql: $(sort $(wildcard sql/types/*.sql)) $(sort $(wildcard sql/tables/*.sql)) $(sort $(wildcard sql/functions/*.sql)) $(sort $(wildcard sql/procedures/*.sql))
+	cat $^ > $@
+
+else
+SCRIPTS = bin/common/*.py bin/pg10/*.py
+
 sql/$(EXTENSION)--$(EXTVERSION).sql: $(sort $(wildcard sql/types/*.sql)) $(sort $(wildcard sql/tables/*.sql)) $(sort $(wildcard sql/functions/*.sql))
 	cat $^ > $@
+	#
+# end PG11 if
+endif
 
 DATA = $(wildcard updates/*--*.sql) sql/$(EXTENSION)--$(EXTVERSION).sql
 EXTRA_CLEAN = sql/$(EXTENSION)--$(EXTVERSION).sql
 else
 $(error Minimum version of PostgreSQL required is 9.4.0)
+
+# end PG94 if
 endif
 
 PGXS := $(shell $(PG_CONFIG) --pgxs)

README.md

@@ -3,18 +3,19 @@
 PG Partition Manager
 ====================
 
-pg_partman is an extension to create and manage both time-based and serial-based table partition sets. Native partitioning in PostgreSQL 10 is supported as of pg_partman v3.0.1. Note that all the features of trigger-based partitioning are not yet supported in native, but performance in both reads & writes is significantly better.
+pg_partman is an extension to create and manage both time-based and serial-based table partition sets. Native partitioning in PostgreSQL 10 is supported as of pg_partman v3.0.1 and much more extensively as of 4.0.0 along with PostgreSQL 11. Note that all the features of trigger-based partitioning are not yet supported in native, but performance in both reads & writes is significantly better.
 
-Child table creation is all managed by the extension itself. For non-native, trigger function maintenance is also handled. For non-native partitioning, tables with existing data can have their data partitioned in easily managed smaller batches. For native partitioning, the creation of a new partitioned set is required and data will have to be migrated over separately. 
+Child table creation is all managed by the extension itself. For non-native, trigger function maintenance is also handled. For non-native partitioning, tables with existing data can have their data partitioned in easily managed smaller batches. For native partitioning, the creation of a new partitioned parent must be done first and the data migrated over after setup is complete.
 
 Optional retention policy can automatically drop partitions no longer needed for both native and non-native partitioning.
 
 A background worker (BGW) process is included to automatically run partition maintenance without the need of an external scheduler (cron, etc) in most cases.
 
 All bug reports, feature requests and general questions can be directed to the Issues section on Github. Please feel free to post here no matter how minor you may feel your issue or question may be. - https://github.com/pgpartman/pg_partman/issues
 
-If you're looking for a partitioning system that handles any range type beyond just time & serial, the new native partitioning features in PostgreSQL 10 are likely the best method for the foreseeable future. If this is something critical to your environment, start planning your upgrades now!
+If you're looking for a partitioning system that handles any range type beyond just time & serial, the new native partitioning features in PostgreSQL 10+ are likely the best method for the foreseeable future. If this is something critical to your environment, start planning your upgrades now!
 
+If you're still trying to evaluate whether partitioning is a good choice for your environment, keep an eye on the HypoPG project. Version 2 will have a hypothetical partitioning feature that will let you evaluate different partitioning schemes without requiring you to actually partition your data. I may see about integrating this feature into pg_partman once it is available. - https://hypopg.readthedocs.io
 
 INSTALLATION
 ------------
@@ -46,11 +47,19 @@ Log into PostgreSQL and run the following commands. Schema is optional (but reco
     CREATE SCHEMA partman;
     CREATE EXTENSION pg_partman SCHEMA partman;
 
-Functions must either be run as a superuser or you can set the ownership of the extension functions to a superuser role and they will also work (SECURITY DEFINER is set).
+As of version 4.0.0, the privileges required to run pg_partman have been reduced greatly. Superuser is still required to install pg_partman and a superuser must still own the extension objects for things to fully run. But the functions with SECURITY DEFINER (ie, run as a superuser) are very minimal, and in a future update, will be completely optional. It is recommended that a dedicated role is created for running pg_partman functions and to be the owner of all partition sets that pg_partman maintains. At a minimum this role will need the following privileges (assuming pg_partman is installed to the "partman" schema):
 
-I've received many requests for being able to install this extension on Amazon RDS. RDS does not support third-party extension management outside of the ones it has approved and provides itself. Therefore, I cannot provide support for running this extension in RDS if the limitations are RDS related. If you'd like to see this extension available there, please send an email to [email protected] requesting that they include it. The more people that do so, the more likely it will happen!
+    CREATE ROLE partman WITH LOGIN;
+    GRANT ALL ON ALL TABLES IN SCHEMA partman TO partman;
+    GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA partman TO partman;
+    GRANT EXECUTE ON ALL PROCEDURES IN SCHEMA partman TO partman;  -- PG11+ only
+    GRANT ALL ON SCHEMA my_partition_schema TO partman;
+
+If you need the role to also be able to create schemas, you will need to grant create on the database as well. In general this shouldn't be required as long as you give the above role CREATE privileges on any pre-existing schemas that will contain partition sets.
 
-Version 1.8.8 of pg_partman is still available on github if you're running a version of PostgreSQL older than 9.4. Note however that no further updates (bug fixes, features, etc) are being released for the 1.x series. If you encounter any issues, please plan for upgrading your database to 9.4+ so that you can use the 2.x series of pg_partman.  
+    GRANT CREATE ON DATABASE mydb TO partman;
+
+I've received many requests for being able to install this extension on Amazon RDS. RDS does not support third-party extension management outside of the ones it has approved and provides itself. Therefore, I cannot provide support for running this extension in RDS if the limitations are RDS related. If you'd like to see this extension available there, please send an email to [email protected] requesting that they include it. The more people that do so, the more likely it will happen!
 
 UPGRADE
 -------
@@ -61,7 +70,7 @@ Run "make install" same as above to put the script files and libraries in place.
 
 If you are doing a pg_dump/restore and you've upgraded pg_partman in place from previous versions, it is recommended you use the --column-inserts option when dumping and/or restoring pg_partman's configuration tables. This is due to ordering of the configuration columns possibly being different (upgrades just add the columns onto the end, whereas the default of a new install may be different).
 
-If upgrading from 1.x to 2.x or greater, please carefully read all intervening version notes in the CHANGELOG, especially those for 2.0.0 and 3.0.1. There are additional instructions for updating your trigger functions to the newer version and other important considerations for the update.
+If upgrading between any major versions of pg_partman (1.x -> 2.x, 2.x -> 3.x, etc), please carefully read all intervening version notes in the CHANGELOG, especially those notes for the major version. There are often additional instructions (Ex. updating trigger functions) and other important considerations for the updates.
 
 EXAMPLE
 -------