v4.1.0 Superuser requirement now optional. Support moving data out of default in PG11+. See CHANGELOG for more details.

Author: keithf4

CHANGELOG.txt

@@ -1,3 +1,28 @@
+4.1.0
+-- For PostgreSQL 10+ and native partitioning, privilege inheritance is now optional and turned off by default. This now allows pg_partman to run fully as a non-superuser when using native partitioning. (Github Issue #231)
+    -- This means that all interaction with these partition sets must be done through the parent table.
+    -- A new column in the part_config table, inherit_privileges, is available if you require direct access to the child tables. Note that this will then require superuser at some level (superuser ownershipt of pg_partman functions for SECURITY DEFINER or running maintenance as a superuser). Just set this option to true and run the reapply_privileges() function to set all child privileges to match the parent. From then on, all new child tables will inherit the parent's privileges.
+    -- Note that for non-native partitioning on all versions, privileges will continue to be inherited to avoid backward compatability issues and will therefore require a superuser as the above bullet notes.
+    -- All existing partition sets have had their configuration values set to true so they will continue to act as they did before this update.
+    -- Updated GRANTS that are required for a non-superuser are in the INSTALLATION section of the readme.
+
+-- The partition_data*() functions now work with the new DEFAULT table feature of PG11. Running these functions will now move data out of the default partition and into the proper child tables, creating them if necessary (Github Issue #230).
+    -- Note that normal maintenance can fail if data goes into the default table since data that exists in the default cannot have a corresponding child created without taking extra steps. Those extra steps are now handled by partition_data*() and, once that is run, maintenance should be able to proceed as normal.
+
+-- Renamed check_parent() function to check_default(). This function now checks both the parent table in trigger-based partition sets as well as the default table introduced in PostgreSQL 11.
+
+-- The reapply_indexes.py script now only works on trigger-based partition sets or native partition sets in PG10. This script will likely never work with native partitioning in PG11+ without being extremely prone to unexpected behavior, so it is currently not supported at all. Be aware that non-unique index inheritance is built in for PG11+ and that should be used if possible (Github Issue #232).
+
+-- Fixed partition_data_proc() not working for epoch time partitioned sets (Github Pull Request #239).
+
+-- Use SplitIdentifierString() in BGW for more recent versions of PostgreSQL (10.5+) (Github Pull Request #241)
+
+-- Fixed show_partition_info() to account for quarterly/3month partitioning that may not use the "q" naming convention (Github Issue #252, PR #251).
+
+-- Set the drop indexes retention feature to only be usable with trigger-based partitioning or for native, only on PG10 and lower. Natively inherited indexes cannot be dropped from children.
+
+
+
 4.0.0
 -- Initial release to provide support PostgreSQL 11. All PG11 functionality is subject to change before its final stable release.
 

README.md

@@ -21,7 +21,7 @@ INSTALLATION
 ------------
 Requirement: PostgreSQL >= 9.4
 
-Recommended: pg_jobmon (>=v1.3.2). PG Job Monitor will automatically be used if it is installed and setup properly.
+Recommended: pg_jobmon (>=v1.4.0). PG Job Monitor will automatically be used if it is installed and setup properly.
 https://github.com/omniti-labs/pg_jobmon
 
 In the directory where you downloaded pg_partman, run
@@ -47,9 +47,10 @@ Log into PostgreSQL and run the following commands. Schema is optional (but reco
     CREATE SCHEMA partman;
     CREATE EXTENSION pg_partman SCHEMA partman;
 
-As of version 4.0.0, the privileges required to run pg_partman have been reduced greatly. Superuser is still required to install pg_partman and a superuser must still own the extension objects for things to fully run. But the functions with SECURITY DEFINER (ie, run as a superuser) are very minimal, and in a future update, will be completely optional. It is recommended that a dedicated role is created for running pg_partman functions and to be the owner of all partition sets that pg_partman maintains. At a minimum this role will need the following privileges (assuming pg_partman is installed to the "partman" schema):
+As of version 4.1.0, pg_partman no longer requires a superuser to run for native partitioning. Trigger-based partitioning still requires it, so if you want to not require superuser, look to migrating to native partitioning. Superuser is still required to install pg_partman. It is recommended that a dedicated role is created for running pg_partman functions and to be the owner of all partition sets that pg_partman maintains. At a minimum this role will need the following privileges (assuming pg_partman is installed to the "partman" schema and that dedicated role is called "partman"):
 
     CREATE ROLE partman WITH LOGIN;
+    GRANT ALL ON SCHEMA partman TO partman;
     GRANT ALL ON ALL TABLES IN SCHEMA partman TO partman;
     GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA partman TO partman;
     GRANT EXECUTE ON ALL PROCEDURES IN SCHEMA partman TO partman;  -- PG11+ only
@@ -70,7 +71,7 @@ Run "make install" same as above to put the script files and libraries in place.
 
 If you are doing a pg_dump/restore and you've upgraded pg_partman in place from previous versions, it is recommended you use the --column-inserts option when dumping and/or restoring pg_partman's configuration tables. This is due to ordering of the configuration columns possibly being different (upgrades just add the columns onto the end, whereas the default of a new install may be different).
 
-If upgrading between any major versions of pg_partman (1.x -> 2.x, 2.x -> 3.x, etc), please carefully read all intervening version notes in the CHANGELOG, especially those notes for the major version. There are often additional instructions (Ex. updating trigger functions) and other important considerations for the updates.
+If upgrading between any major versions of pg_partman (2.x -> 3.x, etc), please carefully read all intervening version notes in the CHANGELOG, especially those notes for the major version. There are often additional instructions (Ex. updating trigger functions) and other important considerations for the updates.
 
 EXAMPLE
 -------
@@ -84,7 +85,7 @@ Then just run the create_parent() function with the appropriate parameters
 
     SELECT partman.create_parent('test.part_test', 'col3', 'partman', 'daily');
     or
-    SELECT partman.create_parent('test.part_test', 'col1', 'partman', '100000');
+    SELECT partman.create_parent('test.part_test', 'col1', 'native', '100000');
 
 This will turn your table into a parent table and premake 4 future partitions and also make 4 past partitions. To make new partitions, schedule the run_maintenance() function to run periodically or use the background worker settings in postgresql.conf (the latter is recommended). 
 

bin/common/reapply_indexes.py

@@ -3,7 +3,7 @@
 import argparse, psycopg2, re, sys, time
 from multiprocessing import Process
 
-partman_version = "3.1.0"
+partman_version = "4.1.0"
 
 parser = argparse.ArgumentParser(description="Script for reapplying indexes on child tables in a partition set to match the parent table. Any indexes that currently exist on the children and match the definition on the parent will be left as is. There is an option to recreate matching as well indexes if desired, as well as the primary key. Indexes that do not exist on the parent will be dropped. Commits are done after each index is dropped/created to help prevent long running transactions & locks.", epilog="NOTE: New index names are made based off the child table name & columns used, so their naming may differ from the name given on the parent. This is done to allow the tool to account for long or duplicate index names. If an index name would be duplicated, an incremental counter is added on to the end of the index name to allow it to be created. Use the --dryrun option first to see what it will do and which names may cause dupes to be handled like this.")
 parser.add_argument('-p', '--parent', help="Parent table of an already created partition set. (Required)")
@@ -20,6 +20,30 @@
 parser.add_argument('--version', action="store_true", help="Print out the minimum version of pg_partman this script is meant to work with. The version of pg_partman installed may be greater than this.")
 args = parser.parse_args()
 
+def check_compatibility(conn, partman_schema):
+    cur = conn.cursor()
+    
+    sql = """SELECT current_setting('server_version_num')::int"""
+    cur.execute(sql)
+    pg_version = int(cur.fetchone()[0])
+
+    if pg_version < 90400:
+        print("ERROR: This script requires PostgreSQL minimum version of 9.4.0")
+        sys.exit(2)
+
+    sql = "SELECT partition_type FROM " + partman_schema + ".part_config WHERE parent_table = %s"
+    cur.execute(sql, [args.parent])
+    partition_type = cur.fetchone()[0]
+
+    if pg_version >= 110000 and partition_type == "native":
+        print("This script cannot currently work with native partition sets in PG11+. Please use native index inheritance methods if possible.")
+        cur.close()
+        conn_close(conn)
+        sys.exit(2)
+
+    cur.close()
+
+
 def create_conn():
     conn = psycopg2.connect(args.connection)
     return conn
@@ -71,7 +95,7 @@ def create_index(conn, partman_schema, child_schemaname, child_tablename, child_
             while True:
                 sql = "SELECT count(*) FROM pg_class c JOIN pg_namespace n ON c.relnamespace = n.oid WHERE n.nspname = %s AND c.relname = %s"
                 cur = conn.cursor()
-                cur.execute(sql, [parent_schemaname, index_name])
+                cur.execute(sql, [child_schemaname, index_name])
                 index_exists = cur.fetchone()[0]
                 if index_exists != None and index_exists > 0:
                     index_name = child_tablename
@@ -280,14 +304,8 @@ def reindex_proc(child_schemaname, child_tablename, parent, parent_index_list, p
 
     conn = create_conn()
 
-    cur = conn.cursor()
-    cur.execute("SELECT current_setting('server_version_num')::int > 90400")
-    if cur.fetchone()[0] == False:
-        print("ERROR: This script requires PostgreSQL minimum version of 9.4.0")
-        sys.exit(2)
-    cur.close()
-
     partman_schema = get_partman_schema(conn)
+    check_compatibility(conn,partman_schema)
     parent = get_parent(conn, partman_schema)
     parent_index_list = get_parent_index_list(conn, parent)
     child_list = get_children(conn, partman_schema)