refactor read/write connection pool (#240)

* refactor read/write connection pool

* rename settings from POSTGRES_* to PG*

* catch warnings

* typo

* Update stac_fastapi/pgstac/config.py

Co-authored-by: Henry Rodman <[email protected]>

* update from review

* update docstring

* add docs

---------

Co-authored-by: Henry Rodman <[email protected]>

Author: vincentsarago

.github/workflows/cicd.yaml

@@ -90,14 +90,10 @@ jobs:
       - name: Load data and validate
         run: python -m stac_fastapi.pgstac.app & ./scripts/wait-for-it.sh localhost:8080 && python ./scripts/ingest_joplin.py http://localhost:8080 && ./scripts/validate http://localhost:8080
         env:
-          POSTGRES_USER: username
-          POSTGRES_PASS: password
-          POSTGRES_DBNAME: postgis
-          POSTGRES_HOST_READER: localhost
-          POSTGRES_HOST_WRITER: localhost
-          POSTGRES_PORT: 5432
           PGUSER: username
           PGPASSWORD: password
+          PGHOST: localhost
+          PGPORT: 5432
           PGDATABASE: postgis
           APP_HOST: 0.0.0.0
           APP_PORT: 8080

CHANGES.md

@@ -4,12 +4,53 @@
 
 ### Changed
 
+- rename `POSTGRES_HOST_READER` to `PGHOST` in config **breaking change**
+- rename `POSTGRES_USER` to `PGUSER` in config **breaking change**
+- rename `POSTGRES_PASS` to `PGPASSWORD` in config **breaking change**
+- rename `POSTGRES_PORT` to `PGPORT` in config **breaking change**
+- rename `POSTGRES_DBNAME` to `PGDATABASE` in config **breaking change**
+  ```python
+  from stac_fastapi.pgstac.config import PostgresSettings
+
+  # before
+  settings = PostgresSettings(
+      postgres_user="user",
+      postgres_pass="password",
+      postgres_host_reader="0.0.0.0",
+      postgres_host_writer="0.0.0.0",
+      postgres_port=1111,
+      postgres_dbname="pgstac",
+  )
+
+  # now
+  settings = PostgresSettings(
+      pguser="user",
+      pgpassword="password",
+      pghost="0.0.0.0",
+      pgport=1111,
+      pgdatabase="pgstac",
+  )
+  ```
+
+- rename `reader_connection_string` to `connection_string` in `PostgresSettings` class **breaking change**
 - add `ENABLE_TRANSACTIONS_EXTENSIONS` env variable to enable `transaction` extensions
 - disable transaction and bulk_transactions extensions by default **breaking change**
 - update `stac-fastapi-*` version requirements to `>=5.2,<6.0`
 - add pgstac health-check in `/_mgmt/health`
 - switch from using pygeofilter to cql2
 
+### Added
+
+- add `write_connection_pool` option in `stac_fastapi.pgstac.db.connect_to_db` function
+- add `write_postgres_settings` option in `stac_fastapi.pgstac.db.connect_to_db` function to set specific settings for the `writer` DB connection pool
+
+### removed
+
+- `stac_fastapi.pgstac.db.DB` class
+- `POSTGRES_HOST_WRITER` in config
+- `writer_connection_string` in `PostgresSettings` class
+- `testing_connection_string` in `PostgresSettings` class
+
 ## [5.0.2] - 2025-04-07
 
 ### Fixed
@@ -183,7 +224,7 @@ As a part of this release, this repository was extracted from the main
 ### Added
 
 - Nginx service as second docker-compose stack to demonstrate proxy ([#503](https://github.com/stac-utils/stac-fastapi/pull/503))
-- Validation checks in CI using [stac-api-validator](github.com/stac-utils/stac-api-validator) ([#508](https://github.com/stac-utils/stac-fastapi/pull/508))
+- Validation checks in CI using [stac-api-validator](https://github.com/stac-utils/stac-api-validator) ([#508](https://github.com/stac-utils/stac-fastapi/pull/508))
 - Required links to the sqlalchemy ItemCollection endpoint ([#508](https://github.com/stac-utils/stac-fastapi/pull/508))
 - Publication of docker images to GHCR ([#525](https://github.com/stac-utils/stac-fastapi/pull/525))
 

docker-compose.yml

@@ -8,12 +8,11 @@ services:
       - APP_PORT=8082
       - RELOAD=true
       - ENVIRONMENT=local
-      - POSTGRES_USER=username
-      - POSTGRES_PASS=password
-      - POSTGRES_DBNAME=postgis
-      - POSTGRES_HOST_READER=database
-      - POSTGRES_HOST_WRITER=database
-      - POSTGRES_PORT=5432
+      - PGUSER=username
+      - PGPASS=password
+      - PGDATABASE=postgis
+      - PGHOST=database
+      - PGPORT=5432
       - WEB_CONCURRENCY=10
       - VSI_CACHE=TRUE
       - GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES

docs/api/stac_fastapi/pgstac/version.md

@@ -0,0 +1,64 @@
+
+
+### Application Extension
+
+The default `stac-fastapi-pgstac` application comes will **all** extensions enabled (except transaction). Users can use `ENABLED_EXTENSIONS` environment variable to limit the supported extensions.
+
+Available values for `ENABLED_EXTENSIONS`:
+
+- `query`
+- `sort`
+- `fields`
+- `filter`
+- `free_text` (only for collection-search)
+- `pagination`
+- `collection_search`
+
+Example: `ENABLED_EXTENSIONS="pagination,sort"`
+
+
+Since `6.0.0`, the transaction extension is not enabled by default. To add the transaction endpoints, users can set `ENABLE_TRANSACTIONS_EXTENSIONS=TRUE/YES/1`.
+
+### Database config
+
+- `PGUSER`: postgres username
+- `PGPASSWORD`: postgres password
+- `PGHOST`: hostname for the connection
+- `PGPORT`: database port
+- `PGDATABASE`: database name
+- `DB_MIN_CONN_SIZE`: Number of connection the pool will be initialized with. Defaults to `1`
+- `DB_MAX_CONN_SIZE` Max number of connections in the pool. Defaults to `10`
+- `DB_MAX_QUERIES`: Number of queries after a connection is closed and replaced with a new connection. Defaults to `50000`
+- `DB_MAX_INACTIVE_CONN_LIFETIME`: Number of seconds after which inactive connections in the pool will be closed. Defaults to `300`
+- `SEARCH_PATH`: Postgres search path. Defaults to `"pgstac,public"`
+- `APPLICATION_NAME`: PgSTAC Application name. Defaults to `"pgstac"`
+
+##### Deprecated
+
+In version `6.0.0` we've renamed the PG configuration variable to match the official naming convention:
+
+- `POSTGRES_USER` -> `PGUSER`
+- `POSTGRES_PASS` -> `PGPASSWORD`
+- `POSTGRES_HOST_READER` -> `PGHOST`
+- `POSTGRES_HOST_WRITER` -> `PGHOST`*
+- `POSTGRES_PORT` -> `PGPORT`
+- `POSTGRES_DBNAME` -> `PGDATABASE`
+
+\* Since version `6.0`, users cannot set a different host for `writer` and `reader` database but will need to customize the application and pass a specific `stac_fastapi.pgstac.config.PostgresSettings` instance to the `connect_to_db` function.
+
+### Validation/Serialization
+
+- `ENABLE_RESPONSE_MODELS`: use pydantic models to validate endpoint responses. Defaults to `False`
+- `ENABLE_DIRECT_RESPONSE`: by-pass the default FastAPI serialization by wrapping the endpoint responses into `starlette.Response` classes. Defaults to `False`
+
+### Misc
+
+- `STAC_FASTAPI_VERSION` (string) is the version number of your API instance (this is not the STAC version)
+- `STAC FASTAPI_TITLE` (string) should be a self-explanatory title for your API
+- `STAC FASTAPI_DESCRIPTION` (string) should be a good description for your API. It can contain CommonMark
+- `STAC_FASTAPI_LANDING_ID` (string) is a unique identifier for your Landing page
+- `ROOT_PATH`: set application root-path (when using proxy)
+- `CORS_ORIGINS`: A list of origins that should be permitted to make cross-origin requests. Defaults to `*`
+- `CORS_METHODS`: A list of HTTP methods that should be allowed for cross-origin requests. Defaults to `"GET,POST,OPTIONS"`
+- `USE_API_HYDRATE`: perform hydration of stac items within stac-fastapi
+- `INVALID_ID_CHARS`: list of characters that are not allowed in item or collection ids (used in Transaction endpoints)

docs/settings.md

@@ -15,6 +15,7 @@ extra:
 # Layout
 nav:
   - Home: "index.md"
+  - Configuration: "settings.md"
   - API:
       - stac_fastapi.pgstac:
           - module: api/stac_fastapi/pgstac/index.md

mkdocs.yml

@@ -17,6 +17,7 @@
     "brotli_asgi",
     "cql2>=0.3.6",
     "pypgstac>=0.8,<0.10",
+    "typing_extensions>=4.9.0",
 ]
 
 extra_reqs = {

setup.py

@@ -95,7 +95,12 @@
 
 application_extensions = []
 
-if os.environ.get("ENABLE_TRANSACTIONS_EXTENSIONS", "").lower() in ["yes", "true", "1"]:
+with_transactions = os.environ.get("ENABLE_TRANSACTIONS_EXTENSIONS", "").lower() in [
+    "yes",
+    "true",
+    "1",
+]
+if with_transactions:
     application_extensions.append(
         TransactionExtension(
             client=TransactionsClient(),
@@ -150,7 +155,7 @@
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     """FastAPI Lifespan."""
-    await connect_to_db(app)
+    await connect_to_db(app, add_write_connection_pool=with_transactions)
     yield
     await close_db_connection(app)
 

stac_fastapi/pgstac/app.py

@@ -1,9 +1,10 @@
 """Postgres API configuration."""
 
-from typing import List, Type
+import warnings
+from typing import Annotated, Any, List, Optional, Type
 from urllib.parse import quote_plus as quote
 
-from pydantic import BaseModel, field_validator
+from pydantic import BaseModel, Field, field_validator, model_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 from stac_fastapi.types.config import ApiSettings
 
@@ -52,22 +53,60 @@ class PostgresSettings(BaseSettings):
     """Postgres-specific API settings.
 
     Attributes:
-        postgres_user: postgres username.
-        postgres_pass: postgres password.
-        postgres_host_reader: hostname for the reader connection.
-        postgres_host_writer: hostname for the writer connection.
-        postgres_port: database port.
-        postgres_dbname: database name.
-        use_api_hydrate: perform hydration of stac items within stac-fastapi.
-        invalid_id_chars: list of characters that are not allowed in item or collection ids.
+        pguser: postgres username.
+        pgpassword: postgres password.
+        pghost: hostname for the connection.
+        pgport: database port.
+        pgdatabase: database name.
+
     """
 
-    postgres_user: str
-    postgres_pass: str
-    postgres_host_reader: str
-    postgres_host_writer: str
-    postgres_port: int
-    postgres_dbname: str
+    postgres_user: Annotated[
+        Optional[str],
+        Field(
+            deprecated="`postgres_user` is deprecated, please use `pguser`", default=None
+        ),
+    ]
+    postgres_pass: Annotated[
+        Optional[str],
+        Field(
+            deprecated="`postgres_pass` is deprecated, please use `pgpassword`",
+            default=None,
+        ),
+    ]
+    postgres_host_reader: Annotated[
+        Optional[str],
+        Field(
+            deprecated="`postgres_host_reader` is deprecated, please use `pghost`",
+            default=None,
+        ),
+    ]
+    postgres_host_writer: Annotated[
+        Optional[str],
+        Field(
+            deprecated="`postgres_host_writer` is deprecated, please use `pghost`",
+            default=None,
+        ),
+    ]
+    postgres_port: Annotated[
+        Optional[int],
+        Field(
+            deprecated="`postgres_port` is deprecated, please use `pgport`", default=None
+        ),
+    ]
+    postgres_dbname: Annotated[
+        Optional[str],
+        Field(
+            deprecated="`postgres_dbname` is deprecated, please use `pgdatabase`",
+            default=None,
+        ),
+    ]
+
+    pguser: str
+    pgpassword: str
+    pghost: str
+    pgport: int
+    pgdatabase: str
 
     db_min_conn_size: int = 1
     db_max_conn_size: int = 10
@@ -78,24 +117,52 @@ class PostgresSettings(BaseSettings):
 
     model_config = {"env_file": ".env", "extra": "ignore"}
 
+    @model_validator(mode="before")
+    @classmethod
+    def _pg_settings_compat(cls, data: Any) -> Any:
+        if isinstance(data, dict):
+            compat = {
+                "postgres_user": "pguser",
+                "postgres_pass": "pgpassword",
+                "postgres_host_reader": "pghost",
+                "postgres_host_writer": "pghost",
+                "postgres_port": "pgport",
+                "postgres_dbname": "pgdatabase",
+            }
+            for old_key, new_key in compat.items():
+                if val := data.get(old_key, None):
+                    warnings.warn(
+                        f"`{old_key}` is deprecated, please use `{new_key}`",
+                        DeprecationWarning,
+                        stacklevel=1,
+                    )
+                    data[new_key] = val
+
+            if (pgh_reader := data.get("postgres_host_reader")) and (
+                pgh_writer := data.get("postgres_host_writer")
+            ):
+                if pgh_reader != pgh_writer:
+                    raise ValueError(
+                        "In order to use different host values for reading and writing "
+                        "you must explicitly provide write_postgres_settings to the connect_to_db function"
+                    )
+
+        return data
+
     @property
-    def reader_connection_string(self):
+    def connection_string(self):
         """Create reader psql connection string."""
-        return f"postgresql://{self.postgres_user}:{quote(self.postgres_pass)}@{self.postgres_host_reader}:{self.postgres_port}/{self.postgres_dbname}"
+        return f"postgresql://{self.pguser}:{quote(self.pgpassword)}@{self.pghost}:{self.pgport}/{self.pgdatabase}"
 
-    @property
-    def writer_connection_string(self):
-        """Create writer psql connection string."""
-        return f"postgresql://{self.postgres_user}:{quote(self.postgres_pass)}@{self.postgres_host_writer}:{self.postgres_port}/{self.postgres_dbname}"
 
-    @property
-    def testing_connection_string(self):
-        """Create testing psql connection string."""
-        return f"postgresql://{self.postgres_user}:{quote(self.postgres_pass)}@{self.postgres_host_writer}:{self.postgres_port}/pgstactestdb"
+class Settings(ApiSettings):
+    """API settings.
 
+    Attributes:
+        use_api_hydrate: perform hydration of stac items within stac-fastapi.
+        invalid_id_chars: list of characters that are not allowed in item or collection ids.
 
-class Settings(ApiSettings):
-    """Api Settings."""
+    """
 
     use_api_hydrate: bool = False
     invalid_id_chars: List[str] = DEFAULT_INVALID_ID_CHARS