Merge remote-tracking branch 'zalando/master' into multisite

Author: ants

docs/ENVIRONMENT.rst

@@ -208,6 +208,7 @@ REST API
 -  **PATRONI\_RESTAPI\_HTTP\_EXTRA\_HEADERS**: (optional) HTTP headers let the REST API server pass additional information with an HTTP response.
 -  **PATRONI\_RESTAPI\_HTTPS\_EXTRA\_HEADERS**: (optional) HTTPS headers let the REST API server pass additional information with an HTTP response when TLS is enabled. This will also pass additional information set in ``http_extra_headers``.
 -  **PATRONI\_RESTAPI\_REQUEST\_QUEUE\_SIZE**: (optional): Sets request queue size for TCP socket used by Patroni REST API.  Once the queue is full, further requests get a "Connection denied" error. The default value is 5.
+-  **PATRONI\_RESTAPI\_SERVER\_TOKENS**: (optional) Configures the value of the ``Server`` HTTP header.  ``Original`` (default) will expose the original behaviour and display the BaseHTTP and Python versions, e.g. ``BaseHTTP/0.6 Python/3.12.3``. ``Minimal``: The header will contain only the Patroni version, e.g. ``Patroni/4.0.0``. ``ProductOnly``: The header will contain only the product name, e.g. ``Patroni``.
 
 .. warning::
 

docs/README.rst

@@ -34,6 +34,10 @@ Patroni/PostgreSQL nodes are decoupled from DCS nodes (except when Patroni imple
 there is no requirement on the minimal number of nodes. Running a cluster consisting of one primary and one standby is
 perfectly fine. You can add more standby nodes later.
 
+**2-node clusters** (primary + standby) are common and provide automatic failover with high availability. Note that during failover, you'll temporarily have no redundancy until the failed node rejoins.
+
+**DCS requirements**: Your DCS (etcd, ZooKeeper, Consul) has to run with **3 or 5 nodes** for proper consensus and fault tolerance. A single DCS cluster can store information for hundreds or thousands of Patroni clusters using different namespace/scope combinations.
+
 Running and Configuring
 -----------------------
 

docs/yaml_configuration.rst

@@ -349,6 +349,10 @@ REST API
    -  **http\_extra\_headers**: (optional): HTTP headers let the REST API server pass additional information with an HTTP response.
    -  **https\_extra\_headers**: (optional): HTTPS headers let the REST API server pass additional information with an HTTP response when TLS is enabled. This will also pass additional information set in ``http_extra_headers``.
    -  **request_queue_size**: (optional): Sets request queue size for TCP socket used by Patroni REST API.  Once the queue is full, further requests get a "Connection denied" error. The default value is 5.
+   -  **server_tokens**: (optional): Configures the value of the ``Server`` HTTP header.
+      - ``Minimal``: The header will contain only the Patroni version, e.g. ``Patroni/4.0.0``.
+      - ``ProductOnly``: The header will contain only the product name, e.g. ``Patroni``.
+      - ``Original`` (default): The header will expose the original behaviour and display the BaseHTTP and Python versions, e.g. ``BaseHTTP/0.6 Python/3.12.3``.
 
 Here is an example of both **http_extra_headers** and **https_extra_headers**:
 

extras/startup-scripts/patroni.service

@@ -23,8 +23,8 @@ EnvironmentFile=-/etc/patroni_env.conf
 
 # Pre-commands to start watchdog device
 # Uncomment if watchdog is part of your patroni setup
-#ExecStartPre=-/usr/bin/sudo /sbin/modprobe softdog
-#ExecStartPre=-/usr/bin/sudo /bin/chown postgres /dev/watchdog
+#ExecStartPre=-+/sbin/modprobe softdog
+#ExecStartPre=-+/bin/chown postgres /dev/watchdog
 
 # Start the patroni process
 ExecStart=/bin/patroni /etc/patroni.yml

features/environment.py

@@ -123,6 +123,12 @@ def read_label(self, label):
         except IOError:
             return None
 
+    def read_config(self):
+        config = dict()
+        with open(self._config) as r:
+            config = yaml.safe_load(r)
+        return config
+
     @staticmethod
     def recursive_update(dst, src):
         for k, v in src.items():
@@ -132,11 +138,10 @@ def recursive_update(dst, src):
                 dst[k] = v
 
     def update_config(self, custom_config):
-        with open(self._config) as r:
-            config = yaml.safe_load(r)
-            self.recursive_update(config, custom_config)
-            with open(self._config, 'w') as w:
-                yaml.safe_dump(config, w, default_flow_style=False)
+        config = self.read_config()
+        self.recursive_update(config, custom_config)
+        with open(self._config, 'w') as w:
+            yaml.safe_dump(config, w, default_flow_style=False)
         self._scope = config.get('scope', 'batman')
 
     def add_tag_to_config(self, tag, value):
@@ -857,7 +862,7 @@ def start(self, name, max_wait_limit=40, custom_config=None):
         self._processes[name].start(max_wait_limit)
 
     def __getattr__(self, func):
-        if func not in ['stop', 'query', 'write_label', 'read_label', 'check_role_has_changed_to',
+        if func not in ['stop', 'query', 'write_label', 'read_label', 'read_config', 'check_role_has_changed_to',
                         'add_tag_to_config', 'get_watchdog', 'patroni_hang', 'backup', 'read_patroni_log']:
             raise AttributeError("PatroniPoolController instance has no attribute '{0}'".format(func))
 

features/standby_cluster.feature

@@ -64,3 +64,17 @@ Feature: standby cluster
     And I receive a response role standby_leader
     And replication works from postgres-0 to postgres-1 after 15 seconds
     And there is a postgres-1_cb.log with "on_role_change replica batman1\non_role_change standby_leader batman1" in postgres-1 data directory
+
+  Scenario: demote cluster
+    When I switch standby cluster batman1 to archive recovery
+    Then Response on GET http://127.0.0.1:8009/patroni contains replication_state=in archive recovery after 30 seconds
+    When I demote cluster batman
+    And "members/postgres-0" key in DCS has role=standby_leader after 20 seconds
+    And "members/postgres-0" key in DCS has state=running after 10 seconds
+
+  Scenario: promote cluster
+    When I issue a PATCH request to http://127.0.0.1:8009/config with {"standby_cluster": null}
+    Then I receive a response code 200
+    And postgres-1 role is the primary after 10 seconds
+    When I add the table foo2 to postgres-1
+    Then table foo2 is present on postgres-0 after 20 seconds

features/steps/standby_cluster.py

@@ -1,7 +1,9 @@
+import json
 import os
 import time
 
 from behave import step
+from patroni_api import check_response, do_request
 
 
 def callbacks(context, name):
@@ -66,3 +68,29 @@ def check_replication_status(context, pg_name1, pg_name2, timeout):
         time.sleep(1)
     else:
         assert False, "{0} is not replicating from {1} after {2} seconds".format(pg_name1, pg_name2, timeout)
+
+
+@step('I switch standby cluster {scope:name} to archive recovery')
+def standby_cluster_archive(context, scope, demote=False):
+    for name, proc in context.pctl._processes.items():
+        if proc._scope != scope or not proc._is_running:
+            continue
+
+        config = context.pctl.read_config(name)
+        url = f'http://{config["restapi"]["connect_address"]}/config'
+        data = {
+            "standby_cluster": {
+                "restore_command": config['bootstrap']['dcs']['postgresql']['parameters']['restore_command']
+            }
+        }
+        if not demote:
+            data['standby_cluster']['primary_slot_name'] = data['standby_cluster']['host'] =\
+                data['standby_cluster']['port'] = None
+        do_request(context, 'PATCH', url, json.dumps(data))
+        check_response(context, 'code', 200)
+        break
+
+
+@step('I demote cluster {scope:name}')
+def demote_cluster(context, scope):
+    standby_cluster_archive(context, scope, True)

patroni/__main__.py

@@ -417,6 +417,12 @@ def passtochild(signo: int, stack_frame: Optional[FrameType]) -> None:
         if pid:
             os.kill(pid, signo)
 
+    import multiprocessing
+    patroni = multiprocessing.Process(target=patroni_main, args=(args.configfile,))
+    patroni.start()
+    pid = patroni.pid
+
+    # Set up signal handlers after fork to prevent child from inheriting them
     if os.name != 'nt':
         signal.signal(signal.SIGCHLD, sigchld_handler)
         signal.signal(signal.SIGHUP, passtochild)
@@ -426,11 +432,6 @@ def passtochild(signo: int, stack_frame: Optional[FrameType]) -> None:
     signal.signal(signal.SIGINT, passtochild)
     signal.signal(signal.SIGABRT, passtochild)
     signal.signal(signal.SIGTERM, passtochild)
-
-    import multiprocessing
-    patroni = multiprocessing.Process(target=patroni_main, args=(args.configfile,))
-    patroni.start()
-    pid = patroni.pid
     patroni.join()
 
 

patroni/api.py

@@ -119,6 +119,16 @@ def __init__(self, request: Any,
         self.__start_time: float = 0.0
         self.path_query: Dict[str, List[str]] = {}
 
+    def version_string(self) -> str:
+        """Override the default version string to return the server header as specified in the configuration.
+
+        If the server header is not set, then it returns the default version string of the HTTP server.
+
+        :return: ``Server`` version string, which is either the server header or the default version string
+            from the :class:`BaseHTTPRequestHandler`.
+        """
+        return self.server.server_header or super().version_string()
+
     def _write_status_code_only(self, status_code: int) -> None:
         """Write a response that is composed only of the HTTP status.
 
@@ -567,7 +577,7 @@ def do_GET_config(self) -> None:
         ``502`` instead.
         """
         cluster = self.server.patroni.dcs.cluster or self.server.patroni.dcs.get_cluster()
-        if cluster.config:
+        if cluster.config and cluster.config.modify_version:
             self._write_json_response(200, cluster.config.data)
         else:
             self.send_error(502)
@@ -1517,6 +1527,33 @@ def __init__(self, patroni: Patroni, config: Dict[str, Any]) -> None:
         self.reload_config(config)
         self.daemon = True
 
+    def construct_server_tokens(self, token_config: str) -> str:
+        """Construct the value for the ``Server`` HTTP header based on *server_tokens*.
+
+        :param server_tokens: the value of ``restapi.server_tokens`` configuration option.
+
+        :returns: a string to be used as the value of ``Server`` HTTP header.
+        """
+        token = token_config.lower()
+        logger.debug('restapi.server_tokens is set to "%s".', token_config)
+
+        # If 'original' is set, we do not modify the Server header.
+        # This is useful for compatibility with existing setups that expect the original header.
+        if token == 'original':
+            return ""
+
+        # If 'productonly', or 'minimal' is set, we construct the header accordingly.
+        if token == 'productonly':  # Show only the product name, without versions.
+            return 'Patroni'
+        elif token == 'minimal':    # Show only the product name and version, without PostgreSQL version.
+            return f'Patroni/{self.patroni.version}'
+        else:
+            # Token is not valid (one of 'original', 'productonly', 'minimal') so report a warning and
+            # return an empty string.
+            logger.warning('restapi.server_tokens is set to "%s". Patroni will not modify the Server header. '
+                           'Valid values are: "Minimal", "ProductOnly".', token_config)
+            return ""
+
     def query(self, sql: str, *params: Any) -> List[Tuple[Any, ...]]:
         """Execute *sql* query with *params* and optionally return results.
 
@@ -1895,6 +1932,9 @@ def reload_config(self, config: Dict[str, Any]) -> None:
             assert isinstance(self.__listen, str)
         self.connection_string = uri(self.__protocol, config.get('connect_address') or self.__listen, 'patroni')
 
+        # Define the Server header response using the server_tokens option.
+        self.server_header = self.construct_server_tokens(config.get('server_tokens', 'original'))
+
     def handle_error(self, request: Union[socket.socket, Tuple[bytes, socket.socket]],
                      client_address: Tuple[str, int]) -> None:
         """Handle any exception that is thrown while handling a request to the REST API.

patroni/config.py

@@ -536,7 +536,7 @@ def _set_section_values(section: str, params: List[str]) -> None:
         _set_section_values('restapi', ['listen', 'connect_address', 'certfile', 'keyfile', 'keyfile_password',
                                         'cafile', 'ciphers', 'verify_client', 'http_extra_headers',
                                         'https_extra_headers', 'allowlist', 'allowlist_include_members',
-                                        'request_queue_size'])
+                                        'request_queue_size', 'server_tokens'])
         _set_section_values('ctl', ['insecure', 'cacert', 'certfile', 'keyfile', 'keyfile_password'])
         _set_section_values('postgresql', ['listen', 'connect_address', 'proxy_address',
                                            'config_dir', 'data_dir', 'pgpass', 'bin_dir'])