[CI only] Bump PG minors to 15.19 / 16.15 / 17.11 (#8778)

Validation PR for the-process
[#241](citusdata/the-process#241). Repoints CI
at the freshly built dev images carrying the new PostgreSQL minors
published 2026-08-14/15.

| | before | after |
|---|---|---|
| PG15 | 15.18 | 15.19 |
| PG16 | 16.14 | 16.15 |
| PG17 | 17.10 | 17.11 |

`image_suffix`: `-dev-34d3b6f`

---

## Fallout from the new minors

Validation surfaced two independent PostgreSQL security changes in this
minor set that break Citus.

### 1. psql `COPY FROM STDIN` (fixed here)

When a `COPY` fails early, the new psql silently swallows every line up
to and including the next `\.`. Tests that relied on the old behaviour
were adjusted: a terminating `\.` was added where one was missing, stray
statements after a swallowed block were removed, and expected output was
trimmed for the lines that no longer execute.

### 2. `output_plugin_libraries` (fixed here)

PostgreSQL 14.24, 15.19, 16.15, 17.11 and 18.6 add an
`output_plugin_libraries` GUC. Only the libraries it lists may be used
as logical decoding output plugins. It defaults to `pgoutput,
test_decoding` and is `superuser`-settable, so it takes a reload rather
than a restart.

**Upgrade note for operators.** Citus uses an output plugin named
`citus` for logical replication during non-blocking shard splits. On
these minors the following fail until the plugin is allowed:

* `citus_split_shard_by_split_points(..., 'force_logical')` and `'auto'`
* `create_distributed_table_concurrently()`
* `citus_isolate_tenant_to_new_shard(..., 'force_logical'` / `'auto')`

Shard **moves** and the rebalancer are unaffected -- they use
`pgoutput`. CDC through the `pgoutput` shim is unaffected; CDC through
`wal2json` needs the same treatment, exactly as it does on vanilla
PostgreSQL.

Remediation, on **every** node:

```sql
ALTER SYSTEM SET output_plugin_libraries = pgoutput, test_decoding, citus;
SELECT pg_reload_conf();
```

The GUC is `GUC_LIST_QUOTE`, so quoting the whole list (`'pgoutput,
test_decoding, citus'`) stores it as a single name and does **not**
work. Use the bare list above, or quote each element individually.

This PR does not work around the restriction -- allowing a decoder is
deliberately an operator action. Instead Citus now **fails fast and
clearly**: a preflight check runs against the source node before any
shards, publications or replication slots are created, and raises an
error naming the plugin, the node, the current allowlist value, and a
ready-to-paste `ALTER SYSTEM` hint.

### Testing

The regression and CDC harnesses start their own clusters, so in CI *we*
are the operator: `pg_regress_multi.pl` and `cdctestlib.pm` now write
the allowlist themselves, guarded by a probe of `postgresql.conf.sample`
so they stay compatible with older minors where the GUC does not exist.

To keep the un-remediated path covered, a new
`check-split-output-plugin-denied` job deliberately skips that override
(via `CITUS_TEST_SKIP_OUTPUT_PLUGIN_ALLOWLIST=1`) and asserts that a
non-blocking split fails with the new error and leaks no shards,
publications or replication slots.

---

**Draft:** these are `-dev-` images. Do not merge until the-process
[#241](citusdata/the-process#241) is merged and
this PR is repointed at the resulting release image tag.

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7c6370b2-06fd-4491-bf92-ecb811d34518

Author: ihalatci

.github/workflows/build_and_test.yml

@@ -31,12 +31,12 @@ jobs:
       pgupgrade_image_name: "ghcr.io/citusdata/pgupgradetester"
       style_checker_image_name: "ghcr.io/citusdata/stylechecker"
       style_checker_tools_version: "0.8.18"
-      sql_snapshot_pg_version: "17.10"
-      image_suffix: "-v88596db"
-      pg15_version: '{ "major": "15", "full": "15.18" }'
-      pg16_version: '{ "major": "16", "full": "16.14" }'
-      pg17_version: '{ "major": "17", "full": "17.10" }'
-      upgrade_pg_versions: "15.18-16.14-17.10"
+      sql_snapshot_pg_version: "17.11"
+      image_suffix: "-v7a09c75"
+      pg15_version: '{ "major": "15", "full": "15.19" }'
+      pg16_version: '{ "major": "16", "full": "16.15" }'
+      pg17_version: '{ "major": "17", "full": "17.11" }'
+      upgrade_pg_versions: "15.19-16.15-17.11"
     steps:
       # Since GHA jobs need at least one step we use a noop step here.
       - name: Set up parameters
@@ -144,7 +144,7 @@ jobs:
           ${{ needs.params.outputs.pg16_version }},
           ${{ needs.params.outputs.pg17_version }}
         ]
-      make_targets: '["check-split", "check-multi", "check-multi-1", "check-multi-1-create-citus", "check-multi-mx", "check-vanilla", "check-isolation", "check-operations", "check-follower-cluster", "check-add-backup-node", "check-columnar", "check-columnar-isolation", "check-enterprise", "check-enterprise-isolation", "check-enterprise-isolation-logicalrep-1", "check-enterprise-isolation-logicalrep-2", "check-enterprise-isolation-logicalrep-3"]'
+      make_targets: '["check-split", "check-split-output-plugin-denied", "check-multi", "check-multi-1", "check-multi-1-create-citus", "check-multi-mx", "check-vanilla", "check-isolation", "check-operations", "check-follower-cluster", "check-add-backup-node", "check-columnar", "check-columnar-isolation", "check-enterprise", "check-enterprise-isolation", "check-enterprise-isolation-logicalrep-1", "check-enterprise-isolation-logicalrep-2", "check-enterprise-isolation-logicalrep-3"]'
       image_suffix: ${{ needs.params.outputs.image_suffix }}
       image_name: ${{ needs.params.outputs.test_image_name }}
     secrets:

src/backend/distributed/operations/shard_split.c

@@ -69,6 +69,9 @@ typedef struct GroupedDummyShards
 	List *shardIntervals;
 } GroupedDummyShards;
 
+/* name of the logical decoding output plugin used by non-blocking splits */
+#define CITUS_SPLIT_DECODER_PLUGIN "citus"
+
 /* Function declarations */
 static void ErrorIfCannotSplitShard(SplitOperation splitOperation,
 									ShardInterval *sourceShard);
@@ -92,6 +95,8 @@ static void CreateAuxiliaryStructuresForShardGroup(List *shardGroupSplitInterval
 												   List *workersForPlacementList,
 												   bool includeReplicaIdentity);
 static void CreateReplicaIdentitiesForDummyShards(HTAB *mapOfPlacementToDummyShardList);
+static void ErrorIfOutputPluginNotAllowed(MultiConnection *connection,
+										  char *outputPlugin);
 static void CreateObjectOnPlacement(List *objectCreationCommandList,
 									WorkerNode *workerNode);
 static List *    CreateSplitIntervalsForShardGroup(List *sourceColocatedShardList,
@@ -1365,6 +1370,73 @@ AcquireNonblockingSplitLock(Oid relationId)
 }
 
 
+/*
+ * ErrorIfOutputPluginNotAllowed errors out if the given logical decoding output
+ * plugin is not listed in the "output_plugin_libraries" setting on the node
+ * behind the given connection, which is where the replication slot is created.
+ *
+ * "output_plugin_libraries" was introduced in PostgreSQL 14.24, 15.19, 16.15,
+ * 17.11 and 18.6. On older minor versions current_setting() returns NULL and we
+ * skip the check, because there is nothing to enforce.
+ */
+static void
+ErrorIfOutputPluginNotAllowed(MultiConnection *connection, char *outputPlugin)
+{
+	StringInfo command = makeStringInfo();
+	appendStringInfo(command,
+					 "SELECT current_setting('output_plugin_libraries', true), "
+					 "coalesce(bool_or(btrim(name, ' \"') = %s), false) "
+					 "FROM unnest(string_to_array(coalesce("
+					 "current_setting('output_plugin_libraries', true), %s), ',')) name",
+					 quote_literal_cstr(outputPlugin),
+					 quote_literal_cstr(outputPlugin));
+
+	PGresult *result = NULL;
+	int queryResult = ExecuteOptionalRemoteCommand(connection, command->data, &result);
+
+	if (queryResult != RESPONSE_OKAY || !IsResponseOK(result) || PQntuples(result) != 1)
+	{
+		ReportResultError(connection, result, ERROR);
+	}
+
+	char *allowedPlugins = "";
+	if (!PQgetisnull(result, 0, 0))
+	{
+		allowedPlugins = pstrdup(PQgetvalue(result, 0, 0));
+	}
+
+	bool isAllowed = (strcmp(PQgetvalue(result, 0, 1), "t") == 0);
+
+	PQclear(result);
+	ForgetResults(connection);
+
+	if (isAllowed)
+	{
+		return;
+	}
+
+	StringInfo newValue = makeStringInfo();
+	if (allowedPlugins[0] != '\0')
+	{
+		appendStringInfo(newValue, "%s, ", allowedPlugins);
+	}
+	appendStringInfoString(newValue, outputPlugin);
+
+	ereport(ERROR, (errmsg("output plugin \"%s\" is not allowed on the source node",
+						   outputPlugin),
+					errdetail("Non-blocking shard splits replicate data with the \"%s\" "
+							  "logical decoding output plugin, but PostgreSQL on node "
+							  "%s:%d only allows the plugins listed in "
+							  "\"output_plugin_libraries\", which is set to \"%s\".",
+							  outputPlugin, connection->hostname, connection->port,
+							  allowedPlugins),
+					errhint("Allow the plugin on every node and reload the "
+							"configuration, for example: ALTER SYSTEM SET "
+							"output_plugin_libraries = %s; SELECT pg_reload_conf();",
+							newValue->data)));
+}
+
+
 /*
  * SplitShard API to split a given shard (or shard group) in non-blocking fashion
  * based on specified split points to a set of destination nodes.
@@ -1423,6 +1495,12 @@ NonBlockingShardSplit(SplitOperation splitOperation,
 		databaseName);
 	ClaimConnectionExclusively(sourceConnection);
 
+	/*
+	 * Fail before creating any shards, publications or replication slots if
+	 * the source node does not allow our output plugin.
+	 */
+	ErrorIfOutputPluginNotAllowed(sourceConnection, CITUS_SPLIT_DECODER_PLUGIN);
+
 	MultiConnection *sourceReplicationConnection =
 		GetReplicationConnection(sourceShardToCopyNode->workerName,
 								 sourceShardToCopyNode->workerPort);
@@ -1493,7 +1571,7 @@ NonBlockingShardSplit(SplitOperation splitOperation,
 		groupedLogicalRepTargetsHash,
 		superUser, databaseName);
 
-	char *logicalRepDecoderPlugin = "citus";
+	char *logicalRepDecoderPlugin = CITUS_SPLIT_DECODER_PLUGIN;
 
 	/*
 	 * 6) Create replication slots and keep track of their snapshot.

src/test/cdc/t/cdctestlib.pm

@@ -142,6 +142,16 @@ max_wal_senders = 100
 max_replication_slots = 100
 ";
     $node->init(allows_streaming => 'logical');
+
+    # PostgreSQL 14.24, 15.19, 16.15, 17.11 and 18.6 restrict logical decoding
+    # to the output plugins listed in output_plugin_libraries. Setting an
+    # unknown GUC is fatal, so only set it when initdb wrote it out.
+    if (open(my $conf_fh, '<', $node->data_dir . "/postgresql.conf")) {
+        $citus_config_options = $citus_config_options .
+            "\noutput_plugin_libraries = 'pgoutput, test_decoding, wal2json, citus'"
+            if grep { /output_plugin_libraries/ } <$conf_fh>;
+        close($conf_fh);
+    }
     if ($node_type == $NODE_TYPE_COORDINATOR || $node_type == $NODE_TYPE_WORKER) {
         $node->append_conf("postgresql.conf",$citus_config_options);
     } else {

src/test/regress/Makefile

@@ -281,6 +281,12 @@ check-split: all
 	$(pg_regress_multi_check) --load-extension=citus \
 	-- $(MULTI_REGRESS_OPTS) --schedule=$(citus_abs_srcdir)/split_schedule $(EXTRA_TESTS)
 
+# Runs the split negative test against a cluster that deliberately does not
+# allow the "citus" output plugin in "output_plugin_libraries".
+check-split-output-plugin-denied: all
+	CITUS_TEST_SKIP_OUTPUT_PLUGIN_ALLOWLIST=1 $(pg_regress_multi_check) --load-extension=citus \
+	-- $(MULTI_REGRESS_OPTS) --schedule=$(citus_abs_srcdir)/split_output_plugin_denied_schedule $(EXTRA_TESTS)
+
 check-failure: all
 	$(pg_regress_multi_check) --load-extension=citus --mitmproxy \
 	-- $(MULTI_REGRESS_OPTS) --schedule=$(citus_abs_srcdir)/failure_schedule $(EXTRA_TESTS)

src/test/regress/expected/multi_follower_dml.out

@@ -227,26 +227,18 @@ COPY the_table (a, b, z) FROM STDIN WITH CSV;
 ERROR:  COPY command to Citus tables is not allowed in read-only mode
 DETAIL:  the database is read-only
 HINT:  All COPY commands to citus tables happen via 2PC, and 2PC requires the database to be in a writable state.
-\.
-invalid command \.
 COPY the_replicated_table (a, b, z) FROM STDIN WITH CSV;
 ERROR:  writing to worker nodes is not currently allowed for replicated tables such as reference tables or hash distributed tables with replication factor greater than 1.
 DETAIL:  the database is read-only
 HINT:  All modifications to replicated tables happen via 2PC, and 2PC requires the database to be in a writable state.
-\.
-invalid command \.
 COPY reference_table (a, b, z) FROM STDIN WITH CSV;
 ERROR:  writing to worker nodes is not currently allowed for replicated tables such as reference tables or hash distributed tables with replication factor greater than 1.
 DETAIL:  the database is read-only
 HINT:  All modifications to replicated tables happen via 2PC, and 2PC requires the database to be in a writable state.
-\.
-invalid command \.
 COPY citus_local_table (a, b, z) FROM STDIN WITH CSV;
 ERROR:  COPY command to Citus tables is not allowed in read-only mode
 DETAIL:  the database is read-only
 HINT:  All COPY commands to citus tables happen via 2PC, and 2PC requires the database to be in a writable state.
-\.
-invalid command \.
 -- all multi-shard modifications require 2PC hence not supported
 INSERT INTO the_table (a, b, z) VALUES (2, 3, 4), (5, 6, 7);
 ERROR:  cannot assign TransactionIds during recovery
@@ -299,20 +291,14 @@ COPY the_table (a, b, z) FROM STDIN WITH CSV;
 ERROR:  COPY command to Citus tables is not allowed in read-only mode
 DETAIL:  the database is read-only
 HINT:  All COPY commands to citus tables happen via 2PC, and 2PC requires the database to be in a writable state.
-\.
-invalid command \.
 COPY reference_table (a, b, z) FROM STDIN WITH CSV;
 ERROR:  writing to worker nodes is not currently allowed for replicated tables such as reference tables or hash distributed tables with replication factor greater than 1.
 DETAIL:  the database is read-only
 HINT:  All modifications to replicated tables happen via 2PC, and 2PC requires the database to be in a writable state.
-\.
-invalid command \.
 COPY citus_local_table (a, b, z) FROM STDIN WITH CSV;
 ERROR:  COPY command to Citus tables is not allowed in read-only mode
 DETAIL:  the database is read-only
 HINT:  All COPY commands to citus tables happen via 2PC, and 2PC requires the database to be in a writable state.
-\.
-invalid command \.
 SELECT * FROM the_table ORDER BY a;
  a | b | z
 ---------------------------------------------------------------------

src/test/regress/expected/multi_multiuser_copy.out

@@ -25,27 +25,16 @@ COPY customer_copy_hash (c_custkey,c_name) FROM STDIN;
 -- COPY FROM as user with ALL access
 SET ROLE full_access;
 COPY customer_copy_hash (c_custkey,c_name) FROM STDIN;
-;
 RESET ROLE;
 -- COPY FROM as user with SELECT access, should fail
 SET ROLE read_access;
 COPY customer_copy_hash (c_custkey,c_name) FROM STDIN;
 ERROR:  permission denied for table customer_copy_hash
-3	customer3
-\.
-invalid command \.
-;
-ERROR:  syntax error at or near "3"
 RESET ROLE;
 -- COPY FROM as user with no access, should fail
 SET ROLE no_access;
 COPY customer_copy_hash (c_custkey,c_name) FROM STDIN;
 ERROR:  permission denied for table customer_copy_hash
-4	customer4
-\.
-invalid command \.
-;
-ERROR:  syntax error at or near "4"
 RESET ROLE;
 -- COPY TO as superuser
 COPY (SELECT * FROM customer_copy_hash ORDER BY 1) TO STDOUT;

src/test/regress/expected/pg12.out

@@ -88,15 +88,12 @@ select create_distributed_table('cptest', 'id');
 
 copy cptest from STDIN with csv where val < 4;
 ERROR:  Citus does not support COPY FROM with WHERE
-1,6
-2,3
-3,2
-4,9
-5,4
-\.
-invalid command \.
 select sum(id), sum(val) from cptest;
-ERROR:  syntax error at or near "1"
+ sum | sum
+---------------------------------------------------------------------
+     |
+(1 row)
+
 -- CTE materialized/not materialized
 CREATE TABLE single_hash_repartition_first (id int, sum int, avg float);
 CREATE TABLE single_hash_repartition_second (id int primary key, sum int, avg float);

src/test/regress/expected/replicated_partitioned_table.out

@@ -126,8 +126,6 @@ HINT:  Run the query on the parent table "collections" instead.
 COPY collections_1 FROM STDIN;
 ERROR:  modifications on partitions when replication factor is greater than 1 is not supported
 HINT:  Run the query on the parent table "collections" instead.
-\.
-invalid command \.
 -- DDLs are not allowed
 CREATE INDEX index_on_partition ON collections_1(key);
 ERROR:  modifications on partitions when replication factor is greater than 1 is not supported

src/test/regress/expected/split_output_plugin_denied.out

@@ -0,0 +1,51 @@
+-- Negative coverage for the "output_plugin_libraries" allowlist that PostgreSQL
+-- 14.24, 15.19, 16.15, 17.11 and 18.6 introduced. This test is only run by the
+-- check-split-output-plugin-denied target, which deliberately starts the cluster
+-- without adding "citus" to the allowlist.
+CREATE SCHEMA split_output_plugin_denied;
+SET search_path TO split_output_plugin_denied;
+SET citus.shard_count TO 2;
+SET citus.shard_replication_factor TO 1;
+SET citus.next_shard_id TO 8990000;
+CREATE TABLE table_to_split (id bigint PRIMARY KEY, value char);
+SELECT create_distributed_table('table_to_split', 'id');
+ create_distributed_table
+---------------------------------------------------------------------
+
+(1 row)
+
+SELECT nodeid AS worker_1_node FROM pg_dist_node WHERE nodeport=:worker_1_port \gset
+SELECT nodeid AS worker_2_node FROM pg_dist_node WHERE nodeport=:worker_2_port \gset
+-- The split must be rejected up front. Terse verbosity keeps the node address
+-- and the current allowlist value out of the expected output.
+\set VERBOSITY terse
+SELECT citus_split_shard_by_split_points(
+	8990000,
+	ARRAY['-1073741826'],
+	ARRAY[:worker_1_node, :worker_2_node],
+	'force_logical');
+ERROR:  output plugin "citus" is not allowed on the source node
+\set VERBOSITY default
+-- Nothing must have been created before the split was rejected.
+SELECT count(*) FROM pg_dist_shard WHERE logicalrelid = 'table_to_split'::regclass;
+ count
+---------------------------------------------------------------------
+     2
+(1 row)
+
+SELECT run_command_on_workers($$SELECT count(*) FROM pg_replication_slots$$);
+ run_command_on_workers
+---------------------------------------------------------------------
+ (localhost,57637,t,0)
+ (localhost,57638,t,0)
+(2 rows)
+
+SELECT run_command_on_workers($$SELECT count(*) FROM pg_publication$$);
+ run_command_on_workers
+---------------------------------------------------------------------
+ (localhost,57637,t,0)
+ (localhost,57638,t,0)
+(2 rows)
+
+DROP SCHEMA split_output_plugin_denied CASCADE;
+NOTICE:  drop cascades to table table_to_split

src/test/regress/pg_regress_multi.pl

@@ -584,6 +584,19 @@ sub should_defer_libdir_swap
 # Allow CREATE SUBSCRIPTION to work
 push(@pgOptions, "wal_level='logical'");
 
+# PostgreSQL 14.24, 15.19, 16.15, 17.11 and 18.6 restrict logical decoding to
+# the output plugins listed in output_plugin_libraries. Citus uses "citus" for
+# non-blocking shard splits and "wal2json" in some CDC tests. Setting an unknown
+# GUC is fatal, so only set it when the installed PostgreSQL knows about it.
+if (!$ENV{CITUS_TEST_SKIP_OUTPUT_PLUGIN_ALLOWLIST} &&
+	open(my $confSample, '<', catfile($sharedir, "postgresql.conf.sample")))
+{
+	push(@pgOptions,
+		 "output_plugin_libraries='pgoutput, test_decoding, wal2json, citus'")
+		if grep { /output_plugin_libraries/ } <$confSample>;
+	close($confSample);
+}
+
 # Faster logical replication status update so tests with logical replication
 # run faster
 push(@pgOptions, "wal_receiver_status_interval=0");