Merge branch 'master' into dle-4-0

Author: agneum

.gitlab/agents/k8s-cluster-1/config.yaml

@@ -0,0 +1,3 @@
+ci_access:
+  projects:
+    - id: postgres-ai/database-lab

README.md

@@ -64,10 +64,10 @@ Try it yourself right now:
     - [Pricing](https://postgres.ai/pricing) (starting at $62/month)
     - [Doc: How to install DBLab SE](https://postgres.ai/docs/how-to-guides/administration/install-dle-from-postgres-ai)
 - Demo: https://demo.aws.postgres.ai (use the token `demo-token` to access)
-- if you are looking for DBLab 4.0, with branching and snapshotting support in API/CLI/UI, check out this demo instance: https://branching.aws.postgres.ai:446/instance, use the token `demo-token` to enter
+- Looking for a free version? Install DBLab Community Edition by [following this tutorial](https://postgres.ai/docs/tutorials/database-lab-tutorial)
 
 ## How it works
-Thin cloning is fast because it uses [Copy-on-Write (CoW)](https://en.wikipedia.org/wiki/Copy-on-write#In_computer_storage). DBLab supports two technologies to enable CoW and thin cloning: [ZFS](https://en.wikipedia.org/wiki/ZFS) (default) and [LVM](https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)).
+Thin cloning is fast because it is based on [Copy-on-Write (CoW)](https://en.wikipedia.org/wiki/Copy-on-write#In_computer_storage). DBLab employs two technologies for enabling thin cloning: [ZFS](https://en.wikipedia.org/wiki/ZFS) (default) and [LVM](https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)).
 
 Using ZFS, DBLab routinely takes new snapshots of the data directory, managing a collection of them and removing old or unused ones. When requesting a fresh clone, users have the option to select their preferred snapshot.
 
@@ -87,33 +87,39 @@ Read more:
 - GitLab: [How GitLab iterates on SQL performance optimization workflow to reduce downtime risks](https://postgres.ai/resources/case-studies/gitlab)
 
 ## Features
-- Blazing-fast cloning of Postgres databases – a few seconds to create a new clone ready to accept connections and queries, regardless of database size.
-- The theoretical maximum number of snapshots and clones is 2<sup>64</sup> ([ZFS](https://en.wikipedia.org/wiki/ZFS), default).
-- The theoretical maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2<sup>128</sup> bytes ([ZFS](https://en.wikipedia.org/wiki/ZFS), default).
-- PostgreSQL major versions supported: 9.6–14.
-- Two technologies are supported to enable thin cloning ([CoW](https://en.wikipedia.org/wiki/Copy-on-write)): [ZFS](https://en.wikipedia.org/wiki/ZFS) and [LVM](https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)).
-- All components are packaged in Docker containers.
-- UI to make manual work more convenient.
-- API and CLI to automate the work with DBLab snapshots, branches, and clones (Postgres endpoints).
-- By default, PostgreSQL containers include many popular extensions ([docs](https://postgres.ai/docs/database-lab/supported-databases#extensions-included-by-default)).
-- PostgreSQL containers can be customized ([docs](https://postgres.ai/docs/database-lab/supported-databases#how-to-add-more-extensions)).
-- Source database can be located anywhere (self-managed Postgres, AWS RDS, GCP CloudSQL, Azure, Timescale Cloud, and so on) and does NOT require any adjustments. There are NO requirements to install ZFS or Docker to the source (production) databases.
-- Initial data provisioning can be done at either the physical (pg_basebackup, backup / archiving tools such as WAL-G or pgBackRest) or logical (dump/restore directly from the source or from files stored at AWS S3) level.
-- For logical mode, partial data retrieval is supported (specific databases, specific tables).
-- For physical mode, a continuously updated state is supported ("sync container"), making DBLab a specialized version of standby Postgres.
-- For logical mode, periodic full refresh is supported, automated, and controlled by DBLab. It is possible to use multiple disks containing different versions of the database, so full refresh won't require downtime.
-- Fast Point in Time Recovery (PITR) to the points available in DBLab snapshots.
-- Unused clones are automatically deleted.
-- "Deletion protection" flag can be used to block automatic or manual deletion of clones.
-- Snapshot retention policies supported in DBLab configuration.
-- Persistent clones: clones survive DBLab restarts (including full VM reboots).
-- The "reset" command can be used to switch to a different version of data.
-- DB Migration Checker component collects various artifacts useful for DB testing in CI ([docs](https://postgres.ai/docs/db-migration-checker)).
-- SSH port forwarding for API and Postgres connections.
-- Docker container config parameters can be specified in the DBLab config.
-- Resource usage quotas for clones: CPU, RAM (container quotas, supported by Docker)
-- Postgres config parameters can be specified in the DBLab config (separately for clones, the "sync" container, and the "promote" container).
-- Monitoring: auth-free `/healthz` API endpoint, extended `/status` (requires auth), [Netdata module](https://gitlab.com/postgres-ai/netdata_for_dle).
+- Speed & scale
+    - Blazing-fast cloning of Postgres databases – clone in seconds, irrespective of database size
+    - Theoretical max of snapshots/clones: 2<sup>64</sup> ([ZFS](https://en.wikipedia.org/wiki/ZFS), default)
+    - Maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2<sup>128</sup> bytes ([ZFS](https://en.wikipedia.org/wiki/ZFS), default)
+- Support & technologies
+    - Supported PostgreSQL versions: 9.6–17
+    - Thin cloning ([CoW](https://en.wikipedia.org/wiki/Copy-on-write)) technologies: [ZFS](https://en.wikipedia.org/wiki/ZFS) and [LVM](https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux))
+    - UI for manual tasks and API & CLI for automation
+    - Packaged in Docker containers for all components
+- Postgres containers
+    - Popular extensions including contrib modules, pgvector, HypoPG and many others ([docs](https://postgres.ai/docs/database-lab/supported-databases#extensions-included-by-default))
+    - Customization capabilities for containers ([docs](https://postgres.ai/docs/database-lab/supported-databases#how-to-add-more-extensions))
+    - Docker container and Postgres config parameters in DBLab config
+- Source database requirements
+    - Location flexibility: self-managed Postgres, AWS RDS, GCP CloudSQL, Azure, etc. No source adjustments needed
+    - No ZFS or Docker requirements for source databases
+- Data provisioning & retrieval
+    - Physical (pg_basebackup, WAL-G, pgBackRest) and logical (dump/restore) provisioning
+    - Partial data retrieval in logical mode (specific databases/tables)
+    - Continuous update in physical mode
+    - Periodic full refresh in logical mode without downtime
+- Recovery & management
+    - Fast Point in Time Recovery (PITR) for physical mode
+    - Auto-deletion of unused clones
+    - Snapshot retention policies in DBLab configuration
+- Clones
+    - "Deletion protection" for preventing clone deletion
+    - Persistent clones withstand DBLab restarts
+    - "Reset" command for data version switching
+    - Resource quotas: CPU, RAM
+- Monitoring & security
+    - `/healthz` API endpoint (no auth), extended `/status` endpoint ([API docs](https://api.dblab.dev))
+    - Netdata module for insights
 
 ## How to contribute
 ### Support us on GitHub/GitLab
@@ -182,7 +188,7 @@ More you can find in [the "How-to guides" section](https://postgres.ai/docs/how-
 ## License
 DBLab source code is licensed under the OSI-approved open source license [Apache 2.0](https://opensource.org/license/apache-2-0/).
 
-Reach out to the Postgres.ai team if you use or want to start using DBLab Standard Edition (DBLab SE) or Enterprise Edition (DBLab EE): [Contact page](https://postgres.ai/contact).
+Reach out to the Postgres.ai team if you want a trial or commercial license that does not contain the GPL clauses: [Contact page](https://postgres.ai/contact).
 
 ## Community & Support
 - ["Database Lab Engine Community Covenant Code of Conduct"](./CODE_OF_CONDUCT.md)

engine/.gitlab-ci.yml

@@ -467,7 +467,12 @@ bash-test-15:
 bash-test-16:
   <<: *bash_test
   variables:
-    POSTGRES_VERSION: 16rc1
+    POSTGRES_VERSION: 16
+
+bash-test-17:
+  <<: *bash_test
+  variables:
+    POSTGRES_VERSION: 17rc1
 
 integration-test:
   services:

engine/api/swagger-spec/dblab_server_swagger.yaml

@@ -21,7 +21,7 @@ externalDocs:
   url: https://gitlab.com/postgres-ai/docs/tree/master/docs/database-lab
 
 servers:
-  - url: "https://demo.aws.postgres.ai:446"
+  - url: "https://demo.aws.postgres.ai/api"
     description: "DBLab 3.x demo server; token: 'demo-token'"
     x-examples:
       Verification-Token: "demo-token"

engine/configs/config.example.logical_generic.yml

@@ -98,10 +98,13 @@ databaseContainer: &db_container
   # We need to specify which Postgres Docker image is to be used for that.
   # The default is the extended Postgres image built on top of the official Postgres image
   # (See https://postgres.ai/docs/database-lab/supported_databases).
-  # It is possible to choose any custom or official Docker image that runs Postgres. Our Dockerfile
-  # (See https://gitlab.com/postgres-ai/custom-images/-/tree/master/extended)
-  # is recommended in case if customization is needed.
-  dockerImage: "postgresai/extended-postgres:15-0.4.1"
+  #
+  # For paid customers, Postgres.AI maintains a set of images compatible with
+  # RDS and RDS Aurora, GCP CloudSQL, Heroku, Timescale Cloud, Supabase, PostGIS.
+  #
+  # IMPORTANT: Postgres version of this image should match the source's Postgres version.
+  # For logical mode, it is a recommendation. For physical mode, it is a *requirement*.
+  dockerImage: "postgresai/extended-postgres:16-0.5.0"
 
   # Container parameters, see
   # https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources

engine/configs/config.example.logical_rds_iam.yml

@@ -97,10 +97,13 @@ databaseContainer: &db_container
   # We need to specify which Postgres Docker image is to be used for that.
   # The default is the extended Postgres image built on top of the official Postgres image
   # (See https://postgres.ai/docs/database-lab/supported_databases).
-  # It is possible to choose any custom or official Docker image that runs Postgres. Our Dockerfile
-  # (See https://gitlab.com/postgres-ai/custom-images/-/tree/master/extended)
-  # is recommended in case if customization is needed.
-  dockerImage: "postgresai/extended-postgres:15-0.4.1"
+  #
+  # For paid customers, Postgres.AI maintains a set of images compatible with
+  # RDS and RDS Aurora, GCP CloudSQL, Heroku, Timescale Cloud, Supabase, PostGIS.
+  #
+  # IMPORTANT: Postgres version of this image should match the source's Postgres version.
+  # For logical mode, it is a recommendation. For physical mode, it is a *requirement*.
+  dockerImage: "postgresai/extended-postgres:16-0.5.0"
 
   # Custom parameters for containers with PostgreSQL, see
   # https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources

engine/configs/config.example.physical_generic.yml

@@ -97,10 +97,13 @@ databaseContainer: &db_container
   # We need to specify which Postgres Docker image is to be used for that.
   # The default is the extended Postgres image built on top of the official Postgres image
   # (See https://postgres.ai/docs/database-lab/supported_databases).
-  # Any custom or official Docker image that runs Postgres. Our Dockerfile
-  # (See https://gitlab.com/postgres-ai/custom-images/-/tree/master/extended)
-  # is recommended in case if customization is needed.
-  dockerImage: "postgresai/extended-postgres:15-0.4.1"
+  #
+  # For paid customers, Postgres.AI maintains a set of images compatible with
+  # RDS and RDS Aurora, GCP CloudSQL, Heroku, Timescale Cloud, Supabase, PostGIS.
+  #
+  # IMPORTANT: Postgres version of this image should match the source's Postgres version.
+  # For logical mode, it is a recommendation. For physical mode, it is a *requirement*.
+  dockerImage: "postgresai/extended-postgres:16-0.5.0"
 
   # Custom parameters for containers with PostgreSQL, see
   # https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources

engine/configs/config.example.physical_pgbackrest.yml

@@ -97,10 +97,13 @@ databaseContainer: &db_container
   # We need to specify which Postgres Docker image is to be used for that.
   # The default is the extended Postgres image built on top of the official Postgres image
   # (See https://postgres.ai/docs/database-lab/supported_databases).
-  # Any custom or official Docker image that runs Postgres. Our Dockerfile
-  # (See https://gitlab.com/postgres-ai/custom-images/-/tree/master/extended)
-  # is recommended in case if customization is needed.
-  dockerImage: "postgresai/extended-postgres:15-0.4.1"
+  #
+  # For paid customers, Postgres.AI additionally maintains a set of images compatible with
+  # RDS and RDS Aurora, GCP CloudSQL, Heroku, Timescale Cloud, Supabase, PostGIS.
+  #
+  # IMPORTANT: Postgres version of this image should match the source's Postgres version.
+  # For logical mode, it is a recommendation. For physical mode, it is a *requirement*.
+  dockerImage: "postgresai/extended-postgres:16-0.5.0"
 
   # Custom parameters for containers with PostgreSQL, see
   # https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources

engine/configs/config.example.physical_walg.yml

@@ -97,10 +97,13 @@ databaseContainer: &db_container
   # We need to specify which Postgres Docker image is to be used for that.
   # The default is the extended Postgres image built on top of the official Postgres image
   # (See https://postgres.ai/docs/database-lab/supported_databases).
-  # Any custom or official Docker image that runs Postgres. Our Dockerfile
-  # (See https://gitlab.com/postgres-ai/custom-images/-/tree/master/extended)
-  # is recommended in case if customization is needed.
-  dockerImage: "postgresai/extended-postgres:15-0.4.1"
+  #
+  # For paid customers, Postgres.AI additionally maintains a set of images compatible with
+  # RDS and RDS Aurora, GCP CloudSQL, Heroku, Timescale Cloud, Supabase, PostGIS.
+  #
+  # IMPORTANT: Postgres version of this image should match the source's Postgres version.
+  # For logical mode, it is a recommendation. For physical mode, it is a *requirement*.
+  dockerImage: "postgresai/extended-postgres:16-0.5.0"
 
   # Custom parameters for containers with PostgreSQL, see
   # https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources

engine/configs/standard/postgres/default/17/pg_hba.conf

@@ -0,0 +1,128 @@
+# PostgreSQL Client Authentication Configuration File
+# ===================================================
+#
+# Refer to the "Client Authentication" section in the PostgreSQL
+# documentation for a complete description of this file.  A short
+# synopsis follows.
+#
+# ----------------------
+# Authentication Records
+# ----------------------
+#
+# This file controls: which hosts are allowed to connect, how clients
+# are authenticated, which PostgreSQL user names they can use, which
+# databases they can access.  Records take one of these forms:
+#
+# local         DATABASE  USER  METHOD  [OPTIONS]
+# host          DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
+# hostssl       DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
+# hostnossl     DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
+# hostgssenc    DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
+# hostnogssenc  DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
+#
+# (The uppercase items must be replaced by actual values.)
+#
+# The first field is the connection type:
+# - "local" is a Unix-domain socket
+# - "host" is a TCP/IP socket (encrypted or not)
+# - "hostssl" is a TCP/IP socket that is SSL-encrypted
+# - "hostnossl" is a TCP/IP socket that is not SSL-encrypted
+# - "hostgssenc" is a TCP/IP socket that is GSSAPI-encrypted
+# - "hostnogssenc" is a TCP/IP socket that is not GSSAPI-encrypted
+#
+# DATABASE can be "all", "sameuser", "samerole", "replication", a
+# database name, a regular expression (if it starts with a slash (/))
+# or a comma-separated list thereof.  The "all" keyword does not match
+# "replication".  Access to replication must be enabled in a separate
+# record (see example below).
+#
+# USER can be "all", a user name, a group name prefixed with "+", a
+# regular expression (if it starts with a slash (/)) or a comma-separated
+# list thereof.  In both the DATABASE and USER fields you can also write
+# a file name prefixed with "@" to include names from a separate file.
+#
+# ADDRESS specifies the set of hosts the record matches.  It can be a
+# host name, or it is made up of an IP address and a CIDR mask that is
+# an integer (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that
+# specifies the number of significant bits in the mask.  A host name
+# that starts with a dot (.) matches a suffix of the actual host name.
+# Alternatively, you can write an IP address and netmask in separate
+# columns to specify the set of hosts.  Instead of a CIDR-address, you
+# can write "samehost" to match any of the server's own IP addresses,
+# or "samenet" to match any address in any subnet that the server is
+# directly connected to.
+#
+# METHOD can be "trust", "reject", "md5", "password", "scram-sha-256",
+# "gss", "sspi", "ident", "peer", "pam", "ldap", "radius" or "cert".
+# Note that "password" sends passwords in clear text; "md5" or
+# "scram-sha-256" are preferred since they send encrypted passwords.
+#
+# OPTIONS are a set of options for the authentication in the format
+# NAME=VALUE.  The available options depend on the different
+# authentication methods -- refer to the "Client Authentication"
+# section in the documentation for a list of which options are
+# available for which authentication methods.
+#
+# Database and user names containing spaces, commas, quotes and other
+# special characters must be quoted.  Quoting one of the keywords
+# "all", "sameuser", "samerole" or "replication" makes the name lose
+# its special character, and just match a database or username with
+# that name.
+#
+# ---------------
+# Include Records
+# ---------------
+#
+# This file allows the inclusion of external files or directories holding
+# more records, using the following keywords:
+#
+# include           FILE
+# include_if_exists FILE
+# include_dir       DIRECTORY
+#
+# FILE is the file name to include, and DIR is the directory name containing
+# the file(s) to include.  Any file in a directory will be loaded if suffixed
+# with ".conf".  The files of a directory are ordered by name.
+# include_if_exists ignores missing files.  FILE and DIRECTORY can be
+# specified as a relative or an absolute path, and can be double-quoted if
+# they contain spaces.
+#
+# -------------
+# Miscellaneous
+# -------------
+#
+# This file is read on server startup and when the server receives a
+# SIGHUP signal.  If you edit the file on a running system, you have to
+# SIGHUP the server for the changes to take effect, run "pg_ctl reload",
+# or execute "SELECT pg_reload_conf()".
+#
+# ----------------------------------
+# Put your actual configuration here
+# ----------------------------------
+#
+# If you want to allow non-local connections, you need to add more
+# "host" records.  In that case you will also need to make PostgreSQL
+# listen on a non-local interface via the listen_addresses
+# configuration parameter, or via the -i or -h command line switches.
+
+# CAUTION: Configuring the system for local "trust" authentication
+# allows any local user to connect as any PostgreSQL user, including
+# the database superuser.  If you do not trust all your local users,
+# use another authentication method.
+
+
+# TYPE  DATABASE        USER            ADDRESS                 METHOD
+
+# "local" is for Unix domain socket connections only
+local   all             all                                     trust
+# IPv4 local connections:
+host    all             all             127.0.0.1/32            trust
+# IPv6 local connections:
+host    all             all             ::1/128                 trust
+# Allow replication connections from localhost, by a user with the
+# replication privilege.
+local   replication     all                                     trust
+host    replication     all             127.0.0.1/32            trust
+host    replication     all             ::1/128                 trust
+
+host all all all scram-sha-256