Merge tag 'v4.1.0' into multisite

Author: ants

.github/workflows/tests.yaml

@@ -13,7 +13,7 @@ env:
 
 jobs:
   unit:
-    runs-on: ${{ fromJson('{"ubuntu":"ubuntu-22.04","windows":"windows-latest","macos":"macos-latest"}')[matrix.os] }}
+    runs-on: ${{ fromJson('{"ubuntu":"ubuntu-22.04","windows":"windows-latest","macos":"macos-15"}')[matrix.os] }}
     strategy:
       fail-fast: false
       matrix:
@@ -105,7 +105,7 @@ jobs:
       run: python -m coveralls --service=github
 
   behave:
-    runs-on: ${{ fromJson('{"ubuntu":"ubuntu-22.04","windows":"windows-latest","macos":"macos-latest"}')[matrix.os] }}
+    runs-on: ${{ fromJson('{"ubuntu":"ubuntu-22.04","windows":"windows-latest","macos":"macos-14"}')[matrix.os] }}
     env:
       DCS: ${{ matrix.dcs }}
       ETCDVERSION: 3.4.23
@@ -198,7 +198,7 @@ jobs:
 
     - uses: jakebailey/pyright-action@v2
       with:
-        version: 1.1.401
+        version: 1.1.405
 
   ydiff:
     name: Test compatibility with the latest version of ydiff

README.rst

@@ -12,7 +12,7 @@ Patroni is a template for high availability (HA) PostgreSQL solutions using Pyth
 
 We call Patroni a "template" because it is far from being a one-size-fits-all or plug-and-play replication system. It will have its own caveats. Use wisely.
 
-Currently supported PostgreSQL versions: 9.3 to 17.
+Currently supported PostgreSQL versions: 9.3 to 18.
 
 **Note to Citus users**: Starting from 3.0 Patroni nicely integrates with the `Citus <https://github.com/citusdata/citus>`__ database extension to Postgres. Please check the `Citus support page <https://github.com/patroni/patroni/blob/master/docs/citus.rst>`__ in the Patroni documentation for more info about how to use Patroni high availability together with a Citus distributed cluster.
 

docs/index.rst

@@ -10,7 +10,7 @@ Patroni is a template for high availability (HA) PostgreSQL solutions using Pyth
 
 We call Patroni a "template" because it is far from being a one-size-fits-all or plug-and-play replication system. It will have its own caveats. Use wisely. There are many ways to run high availability with PostgreSQL; for a list, see the `PostgreSQL Documentation <https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling>`__.
 
-Currently supported PostgreSQL versions: 9.3 to 17.
+Currently supported PostgreSQL versions: 9.3 to 18.
 
 **Note to Citus users**: Starting from 3.0 Patroni nicely integrates with the `Citus <https://github.com/citusdata/citus>`__ database extension to Postgres. Please check the :ref:`Citus support page <citus>` in the Patroni documentation for more info about how to use Patroni high availability together with a Citus distributed cluster.
 

docs/patronictl.rst

@@ -1136,6 +1136,7 @@ Synopsis
       [ --group CITUS_GROUP ]
       [ --wait ]
       [ --force ]
+      [ --from-leader ]
 
 .. _patronictl_reinit_description:
 
@@ -1168,6 +1169,9 @@ Parameters
 ``--force``
     Flag to skip confirmation prompts when rebuilding Postgres standby instances.
 
+``--from-leader``
+    Flag to get basebackup from leader directly.
+
     Useful for scripts.
 
 .. _patronictl_reinit_examples:
@@ -1206,6 +1210,20 @@ Request a rebuild of ``postgresql2`` and wait for it to complete:
     Waiting for reinitialize to complete on: postgresql2
     Reinitialize is completed on: postgresql2
 
+Request a rebuild of ``postgresql2`` and get basebackup from leader directly:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml reinit batman postgresql2 --from-leader
+    + Cluster: batman (7277694203142172922) -+-----------+----+-------------+-----+------------+-----+
+    | Member      | Host           | Role    | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+    +-------------+----------------+---------+-----------+----+-------------+-----+------------+-----+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  5 |             |     |            |     |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  5 |   0/40004E8 |   0 |  0/40004E8 |   0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  5 |   0/40004E8 |   0 |  0/40004E8 |   0 |
+    +-------------+----------------+---------+-----------+----+-------------+-----+------------+-----+
+    Success: reinitialize for member postgresql2
+
 .. _patronictl_reload:
 
 patronictl reload

docs/releases.rst

@@ -3,6 +3,127 @@
 Release notes
 =============
 
+Version 4.1.0
+-------------
+
+Released 2025-09-23
+
+**New features**
+
+- Add support for systemd "notify" unit type (Ronan Dunklau)
+
+  Without a notify unit type, it is possible to start Patroni and immediately send it a SIGHUP signal using systemd, effectively killing it before it had time to set up its signal handlers.
+
+- Provide receive and replay LSN/lag information in API and ctl (Polina Bungina)
+
+  Patroni REST API ``/cluster`` endpoint and ``patronictl list`` command now provide receive LSN, replay LSN, receive lag, and replay lag information for each replica member.
+
+- Ensure clean demotion to standby cluster (Polina Bungina)
+
+  Make sure the introduction of the ``standby_cluster`` section in the dynamic configuration leads to a clean cluster demotion.
+
+- Implement ``patronictl demote-cluster`` and ``promote-cluster`` commands (Polina Bungina)
+
+  New commands for cluster demotion and promotion handle both the dynamic configuration editing and checking the result status.
+
+- Implement ``sync_priority`` tag (Polina Bungina)
+
+  This parameter controls the priority a member should have during synchronous replica selection when ``synchronous_mode`` is set to ``on``.
+
+- Implement ``--print`` option for ``--validate-config`` (Polina Bungina)
+
+  Print out local configuration (including environment configuration overrides) after it has been successfully validated.
+
+- Implement ``kubernetes.bootstrap_labels`` (Polina Bungina)
+
+  This feature allows you to define labels that will be assigned to a member pod when in ``initializing new cluster``, ``running custom bootstrap script``, ``starting after custom bootstrap``, or ``creating replica`` state.
+
+- Add configuration option to suppress duplicate heartbeat logs (Michael Morris)
+
+  If set to ``true``, successive heartbeat logs that are identical shall not be output.
+
+- Add optional ``cluster_type`` attribute to permanent replication slots (Michael Banck)
+
+  This allows you to set whether a particular permanent replication slot should always be created, or just on a primary or standby cluster.
+
+- Make HTTP Server header configurable (David Grierson)
+
+  Introduce the ``restapi.server_tokens`` configuration parameter that allows you to restrict information disclosed in the HTTP Server header.
+
+- Implement readiness API checks for replication on replica members (Ants Aasma)
+
+  The previous implementation considered replicas ready as soon as PostgreSQL was started. With this change, a replica pod is only considered ready when PostgreSQL is replicating and is not too far behind the leader.
+
+
+**Improvements**
+
+- Reduce log level of watchdog configuration failure (Ants Aasma)
+
+  Show the `Could not activate Linux watchdog device` log line on debug logging level, when the watchdog is configured with ``required`` mode. It was previously shown on info level.
+
+- Take advantage of ``written_lsn`` and ``latest_end_lsn`` from ``pg_stat_wal_receiver`` (Alexander Kukushkin)
+
+  ``written_lsn``, the actual write LSN, is now preferred over the one returned by ``pg_last_wal_receive_lsn()``, which is in fact the flush LSN. ``latest_end_lsn`` points to WAL flush on the source host. In case of a primary, it allows better calculation of the replay lag, because values stored in DCS are updated only every ``loop_wait`` seconds.
+
+- Avoid interactions with slots created with the ``failover=true`` option (Alexander Kukushkin)
+
+  This change is required to make the logical failover slots feature fully functional.
+
+- Add PostgreSQL state to ``/metrics`` REST API endpoint (Ivan Filianin)
+
+  PostgreSQL instance state information is now available in the Prometheus format output of the ``/metrics`` REST API endpoint.
+
+
+Version 4.0.7
+-------------
+
+Released 2025-09-22
+
+**New features**
+
+- Add support for PostgreSQL 18 RC1 (Alexander Kukushkin)
+
+  GUC's validator rules were extended. Patroni now properly handles the new background I/O worker.
+
+
+**Bugfixes**
+
+- Fix potential issue around resolving localhost to IPv6 on Windows (András Váczi)
+
+  When configuring ``listen_addresses`` in PostgreSQL, using ``0.0.0.0`` or ``127.0.0.1`` will restrict listening to IPv4 only, excluding IPv6. On typical Windows systems, however, ``localhost`` often resolves to the IPv6 address ``::1`` by default. To ensure compatibility, Patroni now configures PostgreSQL to listen on ``127.0.0.1``, instead of ``localhost``, on Windows systems.
+
+- Return global config only when ``/config`` key exists in DCS (Alexander Kukushkin)
+
+  Patroni REST API was returning an empty configuration instead of raising an error if the ``/config`` key was missing in DCS.
+
+- Fix the issue of failsafe mode not being triggered in case of Etcd unavailability (Alexander Kukushkin)
+
+  Patroni was not always properly handling ``etcd3`` exceptions, which resulted in failsafe mode not being triggered.
+
+- Fix signal handler reentrancy deadlock (Waynerv)
+
+  Patroni running in a Docker container with ``PID=1`` in some special cases was experiencing deadlock after receiving ``SIGCHLD``.
+
+- Recreate (permanent) physical slot when it doesn't reserve WAL (Israel Barth Rubio)
+
+  Permanent physical replication slots created outside of Patroni scope without reserving WALs were causing a ``replication slot cannot be advanced`` error. To avoid this, Patroni now recreates such slots.
+
+- Handle watch cancelation messages in ``etcd3`` properly (Alexander Kukushkin)
+
+  When ``etcd3`` sends a cancelation message to the watch channel, it doesn't close the connection. This results in Patroni using stale data. Patroni now solves it by breaking a loop of reading chunked response and closing the connection on the Patroni side.
+
+- Handle case when ``HTTPConnection`` socket is wrapped with ``pyopenssl`` (Alexander Kukushkin)
+
+  Patroni was not correctly using ``pyopenssl`` interfaces, enforced in ``python-etcd``.
+
+
+**Documentation improvements**
+
+- Improve 2-node cluster guidance (Nikolay Samokhvalov)
+
+  Clarify behaviour during failover and DCS requirements.
+
+
 Version 4.0.6
 -------------
 

docs/rest_api.rst

@@ -286,7 +286,7 @@ Retrieve the Patroni metrics in Prometheus format through the ``GET /metrics`` e
 .. code-block:: bash
 
 	$ curl http://localhost:8008/metrics
-	
+
 	# HELP patroni_version Patroni semver without periods. \
 	# TYPE patroni_version gauge
 	patroni_version{scope="batman",name="patroni1"} 040000
@@ -353,6 +353,71 @@ Retrieve the Patroni metrics in Prometheus format through the ``GET /metrics`` e
 	# HELP patroni_is_paused Value is 1 if auto failover is disabled, 0 otherwise.
 	# TYPE patroni_is_paused gauge
 	patroni_is_paused{scope="batman",name="patroni1"} 1
+	# HELP patroni_postgres_state Numeric representation of Postgres state.
+	# Values: 0=initdb, 1=initdb_failed, 2=custom_bootstrap, 3=custom_bootstrap_failed, 4=creating_replica, 5=running, 6=starting, 7=bootstrap_starting, 8=start_failed, 9=restarting, 10=restart_failed, 11=stopping, 12=stopped, 13=stop_failed, 14=crashed
+	# TYPE patroni_postgres_state gauge
+	patroni_postgres_state{scope="batman",name="patroni1"} 5
+
+PostgreSQL State Values
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``patroni_postgres_state`` metric provides a numeric representation of the current PostgreSQL instance state. This is useful for monitoring and alerting systems that need to track state changes over time. The numeric values are generated using the ``PostgresqlState.get_metrics_description()`` static method.
+
+.. list-table:: PostgreSQL State Values
+   :widths: 10 20 50
+   :header-rows: 1
+
+   * - Value
+     - State Name
+     - Description
+   * - 0
+     - initdb
+     - Initializing new cluster
+   * - 1
+     - initdb_failed
+     - Initialization of new cluster failed
+   * - 2
+     - custom_bootstrap
+     - Running custom bootstrap script
+   * - 3
+     - custom_bootstrap_failed
+     - Custom bootstrap script failed
+   * - 4
+     - creating_replica
+     - Creating replica from primary
+   * - 5
+     - running
+     - PostgreSQL is running normally
+   * - 6
+     - starting
+     - PostgreSQL is starting up
+   * - 7
+     - bootstrap_starting
+     - Starting after custom bootstrap
+   * - 8
+     - start_failed
+     - PostgreSQL start failed
+   * - 9
+     - restarting
+     - PostgreSQL is restarting
+   * - 10
+     - restart_failed
+     - PostgreSQL restart failed
+   * - 11
+     - stopping
+     - PostgreSQL is stopping
+   * - 12
+     - stopped
+     - PostgreSQL is stopped
+   * - 13
+     - stop_failed
+     - PostgreSQL stop failed
+   * - 14
+     - crashed
+     - PostgreSQL has crashed
+
+.. note::
+   These numeric values are fixed and will never change to maintain backward compatibility with existing monitoring systems. If new states are added in the future, they will be assigned new numeric values without changing existing ones.
 
 
 Cluster status endpoints
@@ -637,7 +702,7 @@ In the JSON body of the ``POST`` request you must specify the ``candidate`` fiel
 	Successfully failed over to "postgresql1"
 
 .. warning::
-	:ref:`Be very careful <failover_healthcheck>` when using this endpoint, as this can cause data loss in certain situations. In most cases, :ref:`the switchover endpoint <switchover_api>` satisfies the administrator's needs. 
+	:ref:`Be very careful <failover_healthcheck>` when using this endpoint, as this can cause data loss in certain situations. In most cases, :ref:`the switchover endpoint <switchover_api>` satisfies the administrator's needs.
 
 
 ``POST /switchover`` and ``POST /failover`` endpoints are used by :ref:`patronictl_switchover` and :ref:`patronictl_failover`, respectively.
@@ -721,4 +786,6 @@ Reinitialize endpoint
 
 The call might fail if Patroni is in a loop trying to recover (restart) a failed Postgres. In order to overcome this problem one can specify ``{"force":true}`` in the request body.
 
+You can specify {"from-leader":true} in the request body to directly get basebackup from leader node. This is useful when executing reinit during all replica nodes fail.
+
 The reinitialize endpoint is used by :ref:`patronictl_reinit`.

features/patroni_api.feature

@@ -77,7 +77,7 @@ Scenario: check API requests for the primary-replica pair in the pause mode
 	Then I receive a response code 200
 	And I receive a response state running
 	And I receive a response role replica
-	When I run patronictl.py reinit batman postgres-1 --force --wait
+	When I run patronictl.py reinit batman postgres-1 --force --from-leader --wait
 	Then I receive a response returncode 0
 	And I receive a response output "Success: reinitialize for member postgres-1"
 	And postgres-1 role is the secondary after 30 seconds

patroni/api.py

@@ -733,6 +733,15 @@ def do_GET_metrics(self) -> None:
         metrics.append("# TYPE patroni_is_paused gauge")
         metrics.append("patroni_is_paused{0} {1}".format(labels, int(postgres.get('pause', 0))))
 
+        metrics.append("# HELP patroni_postgres_state Numeric representation of Postgres state.")
+        # Generate description of all state values for metrics documentation
+        state_descriptions = [f"{state.index}={state.name.lower()}" for state in PostgresqlState]
+        metrics.append(f"# Values: {', '.join(state_descriptions)}")
+        metrics.append("# TYPE patroni_postgres_state gauge")
+        current_state = postgres['state']
+        state_value = current_state.index if isinstance(current_state, PostgresqlState) else -1
+        metrics.append(f"patroni_postgres_state{labels} {state_value}")
+
         if patroni.multisite.is_active:
             metrics.append("# HELP patroni_multisite_switches Number of times multisite leader has been switched")
             metrics.append("# TYPE patroni_multisite_switches counter")
@@ -1070,6 +1079,7 @@ def do_POST_reinitialize(self) -> None:
         The request body may contain a JSON dictionary with the following key:
 
             * ``force``: ``True`` if we want to cancel an already running task in order to reinit a replica.
+            * ``from_leader``: ``True`` if we want to reinit a replica and get basebackup from the leader node.
 
         Response HTTP status codes:
 
@@ -1082,8 +1092,9 @@ def do_POST_reinitialize(self) -> None:
             logger.debug('received reinitialize request: %s', request)
 
         force = isinstance(request, dict) and parse_bool(request.get('force')) or False
+        from_leader = isinstance(request, dict) and parse_bool(request.get('from_leader')) or False
 
-        data = self.server.patroni.ha.reinitialize(force)
+        data = self.server.patroni.ha.reinitialize(force, from_leader)
         if data is None:
             status_code = 200
             data = 'reinitialize started'