Merge branch 'main' into examples/oidc-docker-compose

Author: alukach

README.md

@@ -1,45 +1,208 @@
-# STAC Auth Proxy
+<div align="center">
+  <h1 style="font-family: monospace">stac auth proxy</h1>
+  <p align="center">Reverse proxy to apply auth*n to STAC APIs.</p>
+</div>
+
+---
 
 > [!WARNING]
 > This project is currently in active development and may change drastically in the near future while we work towards solidifying a first release.
 
-STAC Auth Proxy is a proxy API that mediates between the client and and some internally accessible STAC API in order to provide a flexible authentication mechanism.
+STAC Auth Proxy is a proxy API that mediates between the client and an internally accessible STAC API to provide a flexible authentication, authorization, and content-filtering mechanism.
 
 ## Features
 
-- 🔐 Selectively apply OIDC auth to some or all endpoints & methods
-- 📖 Augments [OpenAPI](https://swagger.io/specification/) with auth information, keeping auto-generated docs (e.g. [Swagger UI](https://swagger.io/tools/swagger-ui/)) accurate
+- 🔐 Authentication: Selectively apply OIDC auth to some or all endpoints & methods
+- 🎟️ Content Filtering: Apply CQL2 filters to client requests, filtering API content based on user context
+- 📖 OpenAPI Augmentation: Update [OpenAPI](https://swagger.io/specification/) with security requirements, keeping auto-generated docs/UIs accurate (e.g. [Swagger UI](https://swagger.io/tools/swagger-ui/))
 
-### CQL2 Filters
+## Usage
 
-| Method   | Endpoint                                       | Action | Filter | Strategy                                                                                                   |
-| -------- | ---------------------------------------------- | ------ | ------ | ---------------------------------------------------------------------------------------------------------- |
-| `POST`   | `/search`                                      | Read   | Item   | Append body with generated CQL2 query.                                                                     |
-| `GET`    | `/search`                                      | Read   | Item   | Append query params with generated CQL2 query.                                                             |
-| `GET`    | `/collections/{collection_id}/items`           | Read   | Item   | Append query params with generated CQL2 query.                                                             |
-| `POST`   | `/collections/{collection_id}/items`           | Create | Item   | Validate body with generated CQL2 query.                                                                   |
-| `PUT`    | `/collections/{collection_id}/items/{item_id}` | Update | Item   | Fetch STAC Item and validate CQL2 query; merge STAC Item with body and validate with generated CQL2 query. |
-| `DELETE` | `/collections/{collection_id}/items/{item_id}` | Delete | Item   | Fetch STAC Item and validate with CQL2 query.                                                              |
+> [!NOTE]
+> Currently, the project can only be installed by downloading the repository. It will eventually be available on Docker ([#5](https://github.com/developmentseed/stac-auth-proxy/issues/5)) and PyPi ([#30](https://github.com/developmentseed/stac-auth-proxy/issues/30)).
 
-#### Recipes
+### Installation
 
-Only return collections that are mentioned in a `collections` array encoded within the auth token.
+For local development, we use [`uv`](https://docs.astral.sh/uv/) to manage project dependencies and environment.
 
+```sh
+uv sync
 ```
-"A_CONTAINEDBY(id, ('{{ token.collections | join(\"', '\") }}' ))"
+
+Otherwise, the application can be installed as a standard Python module:
+
+```sh
+pip install -e .
 ```
 
-## Installation
+### Running
 
-Set up connection to upstream STAC API and the OpenID Connect provider by setting the following environment variables:
+The simplest way to run the project is by calling the module directly:
 
-```bash
-export STAC_AUTH_PROXY_UPSTREAM_URL="https://some.url"
-export STAC_AUTH_PROXY_OIDC_DISCOVERY_URL="https://your-openid-connect-provider.com/.well-known/openid-configuration"
+```sh
+python -m stac_auth_proxy
 ```
 
-Install software:
+Alternatively, the application's factory can be passed to Uvicorn:
 
-```bash
-uv run python -m stac_auth_proxy
+```sh
+uvicorn --factory stac_auth_proxy:create_app
 ```
+
+### Configuration
+
+The application is configurable via environment variables.
+
+- `UPSTREAM_URL`
+  - The STAC API to proxy requests to
+  - **Type:** HTTP(S) URL
+  - **Required:** Yes
+  - **Example:** `https://your-stac-api.com/stac`
+- `OIDC_DISCOVERY_URL`
+  - OpenID Connect discovery document URL
+  - **Type:** HTTP(S) URL
+  - **Required:** Yes
+  - **Example:** `https://auth.example.com/.well-known/openid-configuration`
+- `OIDC_DISCOVERY_INTERNAL_URL`
+  - The internal network OpenID Connect discovery document URL
+  - **Type:** HTTP(S) URL
+  - **Required:** No, defaults to the value of `OIDC_DISCOVERY_URL`
+  - **Example:** `http://auth/.well-known/openid-configuration`
+- `DEFAULT_PUBLIC`
+  - **Description:** Default access policy for endpoints
+  - **Type:** boolean
+  - **Default:** `false`
+  - **Example:** `false`, `1`, `True`
+- `PRIVATE_ENDPOINTS`
+  - **Description:** Endpoints explicitly marked as requiring authentication, for use when `DEFAULT_PUBLIC == True`
+  - **Type:** JSON object mapping regex patterns to HTTP methods OR tuples of HTTP methods and an array of strings representing required scopes
+  - **Default:**
+    ```json
+    {
+      "^/collections$": ["POST"],
+      "^/collections/([^/]+)$": ["PUT", "PATCH", "DELETE"],
+      "^/collections/([^/]+)/items$": ["POST"],
+      "^/collections/([^/]+)/items/([^/]+)$": ["PUT", "PATCH", "DELETE"],
+      "^/collections/([^/]+)/bulk_items$": ["POST"]
+    }
+    ```
+- `PUBLIC_ENDPOINTS`
+  - **Description:** Endpoints explicitly marked as not requiring authentication, for use when `DEFAULT_PUBLIC == False`
+  - **Type:** JSON object mapping regex patterns to HTTP methods
+  - **Default:**
+    ```json
+    {
+      "^/api.html$": ["GET"],
+      "^/api$": ["GET"]
+    }
+    ```
+- `OPENAPI_SPEC_ENDPOINT`
+  - Path to serve OpenAPI specification
+  - **Type:** string or null
+  - **Default:** `null` (disabled)
+  - **Example:** `/api`
+- `ITEMS_FILTER`
+  - Configuration for item-level filtering
+  - **Type:** JSON object with class configuration
+  - **Default:** `null`
+  - Components:
+    - `cls`: Python import path
+    - `args`: List of positional arguments
+    - `kwargs`: Dictionary of keyword arguments
+  - **Example:**
+    ```json
+    {
+      "cls": "my_package.filters.OrganizationFilter",
+      "args": ["org1"],
+      "kwargs": {
+        "field_name": "properties.organization"
+      }
+    }
+    ```
+- `ITEMS_FILTER_ENDPOINTS`
+  - Where to apply item filtering
+  - **Type:** JSON object mapping regex patterns to HTTP methods
+  - **Default:**
+    ```json
+    {
+      "^/search$": ["GET", "POST"],
+      "^/collections/([^/]+)/items$": ["GET", "POST"]
+    }
+    ```
+
+### Customization
+
+While this project aims to provide utility out-of-the-box as a runnable application, it's likely won't address every project's needs. In these situations, this codebase can instead be treated as a library of components that can be used to augment any webserver that makes use of the [ASGI protocol](https://asgi.readthedocs.io/en/latest/) (e.g. [Django](https://docs.djangoproject.com/en/3.0/topics/async/), [Falcon](https://falconframework.org/), [FastAPI](https://github.com/tiangolo/fastapi), [Litestar](https://litestar.dev/), [Responder](https://responder.readthedocs.io/en/latest/), [Sanic](https://sanic.dev/), [Starlette](https://www.starlette.io/)). Review [`app.py`](https://github.com/developmentseed/stac-auth-proxy/blob/main/src/stac_auth_proxy/app.py) to get a sense of how we make use of the various components to construct a FastAPI application.
+
+## Architecture
+
+### Middleware Stack
+
+Requests pass through a chain of middleware, each performing individual tasks:
+
+1. **EnforceAuthMiddleware**
+
+   - Handles authentication and authorization
+   - Configurable public/private endpoints
+   - OIDC integration
+   - Places auth token payload in request state
+
+2. **BuildCql2FilterMiddleware**
+
+   - Builds CQL2 filters based on request context/state
+   - Places [CQL2 expression](http://developmentseed.org/cql2-rs/latest/python/#cql2.Expr) in request state
+
+3. **ApplyCql2FilterMiddleware**
+
+   - Retrieves [CQL2 expression](http://developmentseed.org/cql2-rs/latest/python/#cql2.Expr) from request state
+   - Augments request with CQL2 filter:
+     - Modifies query strings for `GET` requests
+     - Modifies JSON bodies for `POST`/`PUT`/`PATCH` requests
+
+4. **OpenApiMiddleware**
+
+   - Modifies OpenAPI specification based on endpoint configuration, adding security requirements
+   - Only active if `openapi_spec_endpoint` is configured
+
+5. **AddProcessTimeHeaderMiddleware**
+   - Adds processing time headers
+   - Useful for monitoring/debugging
+
+### Data filtering via CQL2
+
+The system supports generating CQL2 filters based on request context to provide row-level content filtering. These CQL2 filters are then set on outgoing requests prior to the upstream API.
+
+> [!IMPORTANT]
+> The upstream STAC API must support the [STAC API Filter Extension](https://github.com/stac-api-extensions/filter/blob/main/README.md), including the [Features Filter](http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/features-filter) conformance class on to the Features resource (`/collections/{cid}/items`) [#37](https://github.com/developmentseed/stac-auth-proxy/issues/37).
+
+> [!TIP]
+> Integration with external authorization systems (e.g. [Open Policy Agent](https://www.openpolicyagent.org/)) can be achieved by specifying an `ITEMS_FILTER` that points to a class/function that, once initialized, returns a [`cql2.Expr` object](https://developmentseed.org/cql2-rs/latest/python/#cql2.Expr) when called with the request context.
+
+#### Example GET Request Flow
+
+```mermaid
+sequenceDiagram
+    Client->>Proxy: GET /collections
+    Note over Proxy: EnforceAuth checks credentials
+    Note over Proxy: BuildCql2Filter creates filter immediately
+    Note over Proxy: ApplyCql2Filter modifies query string
+    Proxy->>STAC API: GET /collection?filter=(collection=landsat)
+    STAC API->>Client: Response
+```
+
+#### Filters
+
+| Supported                                                | Method   | Endpoint                                       | Action | Filter     | Strategy                                                                                               |
+| -------------------------------------------------------- | -------- | ---------------------------------------------- | ------ | ---------- | ------------------------------------------------------------------------------------------------------ |
+| ✅                                                       | `POST`   | `/search`                                      | Read   | Item       | Append body with generated CQL2 query.                                                                 |
+| ✅                                                       | `GET`    | `/search`                                      | Read   | Item       | Append query params with generated CQL2 query.                                                         |
+| ❌ ([#22](https://github.com/developmentseed/stac-auth-proxy/issues/22)) | `POST`   | `/collections/`                                | Create | Collection | Validate body with generated CQL2 query.                                                               |
+| ❌ ([#23](https://github.com/developmentseed/stac-auth-proxy/issues/23)) | `GET`    | `/collections/{collection_id}`                 | Read   | Collection | Append query params with generated CQL2 query.                                                         |
+| ❌ ([#22](https://github.com/developmentseed/stac-auth-proxy/issues/22)) | `PUT`    | `/collections/{collection_id}}`                | Update | Collection | Fetch Collection and validate CQL2 query; merge Item with body and validate with generated CQL2 query. |
+| ❌ ([#22](https://github.com/developmentseed/stac-auth-proxy/issues/22)) | `DELETE` | `/collections/{collection_id}`                 | Delete | Collection | Fetch Collectiion and validate with CQL2 query.                                                        |
+| ✅                                                       | `GET`    | `/collections/{collection_id}/items`           | Read   | Item       | Append query params with generated CQL2 query.                                                         |
+| ❌ ([#25](https://github.com/developmentseed/stac-auth-proxy/issues/25)) | `GET`    | `/collections/{collection_id}/items/{item_id}` | Read   | Item       | Validate response against CQL2 query.                                                                  |
+| ❌ ([#21](https://github.com/developmentseed/stac-auth-proxy/issues/21)) | `POST`   | `/collections/{collection_id}/items`           | Create | Item       | Validate body with generated CQL2 query.                                                               |
+| ❌ ([#21](https://github.com/developmentseed/stac-auth-proxy/issues/21)) | `PUT`    | `/collections/{collection_id}/items/{item_id}` | Update | Item       | Fetch Item and validate CQL2 query; merge Item with body and validate with generated CQL2 query.       |
+| ❌ ([#21](https://github.com/developmentseed/stac-auth-proxy/issues/21)) | `DELETE` | `/collections/{collection_id}/items/{item_id}` | Delete | Item       | Fetch Item and validate with CQL2 query.                                                               |
+| ❌ ([#21](https://github.com/developmentseed/stac-auth-proxy/issues/21)) | `POST`   | `/collections/{collection_id}/bulk_items`      | Create | Item       | Validate items in body with generated CQL2 query.                                                      |

docker-compose.yaml

@@ -1,6 +1,6 @@
 services:
   stac:
-    image: ghcr.io/stac-utils/stac-fastapi-pgstac
+    image: ghcr.io/stac-utils/stac-fastapi-pgstac:5.0.0
     environment:
       APP_HOST: 0.0.0.0
       APP_PORT: 8001

pyproject.toml

@@ -8,7 +8,7 @@ classifiers = [
 dependencies = [
   "authlib>=1.3.2",
   "brotli>=1.1.0",
-  "cql2>=0.3.4",
+  "cql2>=0.3.5",
   "fastapi>=0.115.5",
   "httpx>=0.28.0",
   "jinja2>=3.1.4",

src/stac_auth_proxy/app.py

@@ -90,8 +90,7 @@ def create_app(settings: Optional[Settings] = None) -> FastAPI:
         public_endpoints=settings.public_endpoints,
         private_endpoints=settings.private_endpoints,
         default_public=settings.default_public,
-        oidc_config_url=settings.oidc_discovery_url,
-        oidc_config_internal_url=settings.oidc_discovery_internal_url,
+        oidc_config_url=settings.oidc_discovery_internal_url,
     )
 
     app.add_middleware(

src/stac_auth_proxy/config.py

@@ -1,9 +1,9 @@
 """Configuration for the STAC Auth Proxy."""
 
 import importlib
-from typing import Literal, Optional, Sequence, TypeAlias, Union
+from typing import Any, Literal, Optional, Sequence, TypeAlias, Union
 
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, Field, model_validator
 from pydantic.networks import HttpUrl
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
@@ -37,7 +37,7 @@ class Settings(BaseSettings):
     # External URLs
     upstream_url: HttpUrl
     oidc_discovery_url: HttpUrl
-    oidc_discovery_internal_url: Optional[HttpUrl] = None
+    oidc_discovery_internal_url: HttpUrl
 
     wait_for_upstream: bool = True
 
@@ -66,8 +66,16 @@ class Settings(BaseSettings):
     # Filters
     items_filter: Optional[ClassInput] = None
     items_filter_endpoints: Optional[EndpointMethods] = {
-        r"^/search$": ["POST"],
+        r"^/search$": ["GET", "POST"],
         r"^/collections/([^/]+)/items$": ["GET", "POST"],
     }
 
     model_config = SettingsConfigDict()
+
+    @model_validator(mode="before")
+    @classmethod
+    def default_oidc_discovery_internal_url(cls, data: Any) -> Any:
+        """Set the internal OIDC discovery URL to the public URL if not set."""
+        if not data.get("oidc_discovery_internal_url"):
+            data["oidc_discovery_internal_url"] = data.get("oidc_discovery_url")
+        return data

src/stac_auth_proxy/handlers/reverse_proxy.py

@@ -44,6 +44,12 @@ async def proxy_request(self, request: Request) -> httpx.Response:
             headers=headers,
             content=request.stream(),
         )
+
+        # NOTE: HTTPX adds headers, so we need to trim them before sending request
+        for h in rp_req.headers:
+            if h not in headers:
+                del rp_req.headers[h]
+
         logger.debug(f"Proxying request to {rp_req.url}")
 
         start_time = time.perf_counter()

src/stac_auth_proxy/lifespan/ServerHealthCheck.py

@@ -15,9 +15,9 @@ class ServerHealthCheck:
     """Health check for upstream API."""
 
     url: str | HttpUrl
-    max_retries: int = 5
-    retry_delay: float = 0.25
-    retry_delay_max: float = 10.0
+    max_retries: int = 10
+    retry_delay: float = 0.5
+    retry_delay_max: float = 5.0
     timeout: float = 5.0
 
     def __post_init__(self):

src/stac_auth_proxy/middleware/ApplyCql2FilterMiddleware.py

@@ -0,0 +1,61 @@
+"""Middleware to apply CQL2 filters."""
+
+import json
+from dataclasses import dataclass
+from logging import getLogger
+
+from starlette.requests import Request
+from starlette.types import ASGIApp, Message, Receive, Scope, Send
+
+from ..utils import filters
+
+logger = getLogger(__name__)
+
+
+@dataclass(frozen=True)
+class ApplyCql2FilterMiddleware:
+    """Middleware to apply the Cql2Filter to the request."""
+
+    app: ASGIApp
+
+    state_key: str = "cql2_filter"
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        """Add the Cql2Filter to the request."""
+        if scope["type"] != "http":
+            return await self.app(scope, receive, send)
+
+        request = Request(scope)
+
+        if request.method == "GET":
+            cql2_filter = getattr(request.state, self.state_key, None)
+            if cql2_filter:
+                # TODO: Differentiate between list/search and lookup
+                scope["query_string"] = filters.append_qs_filter(
+                    request.url.query, cql2_filter
+                )
+            return await self.app(scope, receive, send)
+
+        elif request.method in ["POST", "PUT", "PATCH"]:
+
+            async def receive_and_apply_filter() -> Message:
+                message = await receive()
+                if message["type"] != "http.request":
+                    return message
+
+                cql2_filter = getattr(request.state, self.state_key, None)
+                if cql2_filter:
+                    try:
+                        body = message.get("body", b"{}")
+                    except json.JSONDecodeError as e:
+                        logger.warning("Failed to parse request body as JSON")
+                        # TODO: Return a 400 error
+                        raise e
+
+                    new_body = filters.append_body_filter(json.loads(body), cql2_filter)
+                    message["body"] = json.dumps(new_body).encode("utf-8")
+                return message
+
+            return await self.app(scope, receive_and_apply_filter, send)
+
+        return await self.app(scope, receive, send)