Merge branch 'main' into feature/s3-url-signing

Author: alukach

README.md

@@ -19,7 +19,7 @@ STAC Auth Proxy is a proxy API that mediates between the client and an internall
 ## Usage
 
 > [!NOTE]
-> Currently, the project can only be installed by downloading the repository. It will eventually be available on Docker ([#5](https://github.com/developmentseed/issues/5)) and PyPi ([#30](https://github.com/developmentseed/issues/30)).
+> Currently, the project can only be installed by downloading the repository. It will eventually be available on Docker ([#5](https://github.com/developmentseed/stac-auth-proxy/issues/5)) and PyPi ([#30](https://github.com/developmentseed/stac-auth-proxy/issues/30)).
 
 ### Installation
 
@@ -125,7 +125,7 @@ The application is configurable via environment variables.
   - **Default:**
     ```json
     {
-      "^/search$": ["POST"],
+      "^/search$": ["GET", "POST"],
       "^/collections/([^/]+)/items$": ["GET", "POST"]
     }
     ```
@@ -138,7 +138,7 @@ While this project aims to provide utility out-of-the-box as a runnable applicat
 
 ### Middleware Stack
 
-The middleware stack is processed in reverse order (bottom to top):
+Requests pass through a chain of middleware, each performing individual tasks:
 
 1. **EnforceAuthMiddleware**
 
@@ -156,8 +156,8 @@ The middleware stack is processed in reverse order (bottom to top):
 
    - Retrieves [CQL2 expression](http://developmentseed.org/cql2-rs/latest/python/#cql2.Expr) from request state
    - Augments request with CQL2 filter:
-     - Modifies query strings for GET requests
-     - Modifies JSON bodies for POST/PUT/PATCH requests
+     - Modifies query strings for `GET` requests
+     - Modifies JSON bodies for `POST`/`PUT`/`PATCH` requests
 
 4. **OpenApiMiddleware**
 
@@ -173,10 +173,10 @@ The middleware stack is processed in reverse order (bottom to top):
 The system supports generating CQL2 filters based on request context to provide row-level content filtering. These CQL2 filters are then set on outgoing requests prior to the upstream API.
 
 > [!IMPORTANT]
-> The upstream STAC API must support the [STAC API Filter Extension](https://github.com/stac-api-extensions/filter/blob/main/README.md).
+> The upstream STAC API must support the [STAC API Filter Extension](https://github.com/stac-api-extensions/filter/blob/main/README.md), including the [Features Filter](http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/features-filter) conformance class on to the Features resource (`/collections/{cid}/items`) [#37](https://github.com/developmentseed/stac-auth-proxy/issues/37).
 
 > [!TIP]
-> Integration with external authorization systems (e.g. [Open Policy Agent](https://www.openpolicyagent.org/)) can be achieved by replacing the default `BuildCql2FilterMiddleware` with a custom async middleware that is capable of generating [`cql2.Expr` objects](https://developmentseed.org/cql2-rs/latest/python/#cql2.Expr).
+> Integration with external authorization systems (e.g. [Open Policy Agent](https://www.openpolicyagent.org/)) can be achieved by specifying an `ITEMS_FILTER` that points to a class/function that, once initialized, returns a [`cql2.Expr` object](https://developmentseed.org/cql2-rs/latest/python/#cql2.Expr) when called with the request context.
 
 #### Example GET Request Flow
 
@@ -196,13 +196,13 @@ sequenceDiagram
 | -------------------------------------------------------- | -------- | ---------------------------------------------- | ------ | ---------- | ------------------------------------------------------------------------------------------------------ |
 | ✅                                                       | `POST`   | `/search`                                      | Read   | Item       | Append body with generated CQL2 query.                                                                 |
 | ✅                                                       | `GET`    | `/search`                                      | Read   | Item       | Append query params with generated CQL2 query.                                                         |
-| ❌ ([#22](https://github.com/developmentseed/issues/22)) | `POST`   | `/collections/`                                | Create | Collection | Validate body with generated CQL2 query.                                                               |
-| ❌ ([#23](https://github.com/developmentseed/issues/23)) | `GET`    | `/collections/{collection_id}`                 | Read   | Collection | Append query params with generated CQL2 query.                                                         |
-| ❌ ([#22](https://github.com/developmentseed/issues/22)) | `PUT`    | `/collections/{collection_id}}`                | Update | Collection | Fetch Collection and validate CQL2 query; merge Item with body and validate with generated CQL2 query. |
-| ❌ ([#22](https://github.com/developmentseed/issues/22)) | `DELETE` | `/collections/{collection_id}`                 | Delete | Collection | Fetch Collectiion and validate with CQL2 query.                                                        |
+| ❌ ([#22](https://github.com/developmentseed/stac-auth-proxy/issues/22)) | `POST`   | `/collections/`                                | Create | Collection | Validate body with generated CQL2 query.                                                               |
+| ❌ ([#23](https://github.com/developmentseed/stac-auth-proxy/issues/23)) | `GET`    | `/collections/{collection_id}`                 | Read   | Collection | Append query params with generated CQL2 query.                                                         |
+| ❌ ([#22](https://github.com/developmentseed/stac-auth-proxy/issues/22)) | `PUT`    | `/collections/{collection_id}}`                | Update | Collection | Fetch Collection and validate CQL2 query; merge Item with body and validate with generated CQL2 query. |
+| ❌ ([#22](https://github.com/developmentseed/stac-auth-proxy/issues/22)) | `DELETE` | `/collections/{collection_id}`                 | Delete | Collection | Fetch Collectiion and validate with CQL2 query.                                                        |
 | ✅                                                       | `GET`    | `/collections/{collection_id}/items`           | Read   | Item       | Append query params with generated CQL2 query.                                                         |
-| ❌ ([#25](https://github.com/developmentseed/issues/25)) | `GET`    | `/collections/{collection_id}/items/{item_id}` | Read   | Item       | Validate response against CQL2 query.                                                                  |
-| ❌ ([#21](https://github.com/developmentseed/issues/21)) | `POST`   | `/collections/{collection_id}/items`           | Create | Item       | Validate body with generated CQL2 query.                                                               |
-| ❌ ([#21](https://github.com/developmentseed/issues/21)) | `PUT`    | `/collections/{collection_id}/items/{item_id}` | Update | Item       | Fetch Item and validate CQL2 query; merge Item with body and validate with generated CQL2 query.       |
-| ❌ ([#21](https://github.com/developmentseed/issues/21)) | `DELETE` | `/collections/{collection_id}/items/{item_id}` | Delete | Item       | Fetch Item and validate with CQL2 query.                                                               |
-| ❌ ([#21](https://github.com/developmentseed/issues/21)) | `POST`   | `/collections/{collection_id}/bulk_items`      | Create | Item       | Validate items in body with generated CQL2 query.                                                      |
+| ❌ ([#25](https://github.com/developmentseed/stac-auth-proxy/issues/25)) | `GET`    | `/collections/{collection_id}/items/{item_id}` | Read   | Item       | Validate response against CQL2 query.                                                                  |
+| ❌ ([#21](https://github.com/developmentseed/stac-auth-proxy/issues/21)) | `POST`   | `/collections/{collection_id}/items`           | Create | Item       | Validate body with generated CQL2 query.                                                               |
+| ❌ ([#21](https://github.com/developmentseed/stac-auth-proxy/issues/21)) | `PUT`    | `/collections/{collection_id}/items/{item_id}` | Update | Item       | Fetch Item and validate CQL2 query; merge Item with body and validate with generated CQL2 query.       |
+| ❌ ([#21](https://github.com/developmentseed/stac-auth-proxy/issues/21)) | `DELETE` | `/collections/{collection_id}/items/{item_id}` | Delete | Item       | Fetch Item and validate with CQL2 query.                                                               |
+| ❌ ([#21](https://github.com/developmentseed/stac-auth-proxy/issues/21)) | `POST`   | `/collections/{collection_id}/bulk_items`      | Create | Item       | Validate items in body with generated CQL2 query.                                                      |

docker-compose.yaml

@@ -1,6 +1,6 @@
 services:
   stac:
-    image: ghcr.io/stac-utils/stac-fastapi-pgstac
+    image: ghcr.io/stac-utils/stac-fastapi-pgstac:5.0.0
     environment:
       APP_HOST: 0.0.0.0
       APP_PORT: 8001

pyproject.toml

@@ -8,7 +8,7 @@ classifiers = [
 dependencies = [
   "authlib>=1.3.2",
   "brotli>=1.1.0",
-  "cql2>=0.3.4",
+  "cql2>=0.3.5",
   "fastapi>=0.115.5",
   "httpx>=0.28.0",
   "jinja2>=3.1.4",

src/stac_auth_proxy/config.py

@@ -69,7 +69,7 @@ class Settings(BaseSettings):
     # Filters
     items_filter: Optional[ClassInput] = None
     items_filter_endpoints: Optional[EndpointMethods] = {
-        r"^/search$": ["POST"],
+        r"^/search$": ["GET", "POST"],
         r"^/collections/([^/]+)/items$": ["GET", "POST"],
     }
 

src/stac_auth_proxy/handlers/reverse_proxy.py

@@ -44,6 +44,12 @@ async def proxy_request(self, request: Request) -> httpx.Response:
             headers=headers,
             content=request.stream(),
         )
+
+        # NOTE: HTTPX adds headers, so we need to trim them before sending request
+        for h in rp_req.headers:
+            if h not in headers:
+                del rp_req.headers[h]
+
         logger.debug(f"Proxying request to {rp_req.url}")
 
         start_time = time.perf_counter()

src/stac_auth_proxy/lifespan/ServerHealthCheck.py

@@ -15,9 +15,9 @@ class ServerHealthCheck:
     """Health check for upstream API."""
 
     url: str | HttpUrl
-    max_retries: int = 5
-    retry_delay: float = 0.25
-    retry_delay_max: float = 10.0
+    max_retries: int = 10
+    retry_delay: float = 0.5
+    retry_delay_max: float = 5.0
     timeout: float = 5.0
 
     def __post_init__(self):

tests/test_authn.py

@@ -112,79 +112,3 @@ def test_scopes(
     )
     expected_status_code = 200 if expected_permitted else 401
     assert response.status_code == expected_status_code
-
-
-# @pytest.mark.parametrize(
-#     "is_valid, path, method",
-#     [
-#         *[
-#             [True, *endpoint_method]
-#             for endpoint_method in [
-#                 ["/collections", "POST"],
-#                 ["/collections/foo", "PUT"],
-#                 ["/collections/foo", "PATCH"],
-#                 ["/collections/foo/items", "POST"],
-#                 ["/collections/foo/items/bar", "PUT"],
-#                 ["/collections/foo/items/bar", "PATCH"],
-#             ]
-#         ],
-#         *[
-#             [False, *endpoint_method]
-#             for endpoint_method in [
-#                 ["/collections/foo", "DELETE"],
-#                 ["/collections/foo/items/bar", "DELETE"],
-#             ]
-#         ],
-#     ],
-# )
-# def test_scopes(source_api_server, token_builder, is_valid, path, method):
-#     """Private endpoints permit access with a valid token."""
-#     test_app = app_factory(
-#         upstream_url=source_api_server,
-#         default_public=True,
-#         private_endpoints={
-#             r"^/collections$": [
-#                 ("POST", ["collections:create"]),
-#             ],
-#             r"^/collections/([^/]+)$": [
-#                 # ("PUT", ["collections:update"]),
-#                 # ("PATCH", ["collections:update"]),
-#                 ("DELETE", ["collections:delete"]),
-#             ],
-#             r"^/collections/([^/]+)/items$": [
-#                 ("POST", ["items:create"]),
-#             ],
-#             r"^/collections/([^/]+)/items/([^/]+)$": [
-#                 # ("PUT", ["items:update"]),
-#                 # ("PATCH", ["items:update"]),
-#                 ("DELETE", ["items:delete"]),
-#             ],
-#             r"^/collections/([^/]+)/bulk_items$": [
-#                 ("POST", ["items:create"]),
-#             ],
-#         },
-#     )
-#     valid_auth_token = token_builder(
-#         {
-#             "scopes": " ".join(
-#                 [
-#                     "collection:create",
-#                     "items:create",
-#                     "collections:update",
-#                     "items:update",
-#                 ]
-#             )
-#         }
-#     )
-#     client = TestClient(test_app)
-
-#     response = client.request(
-#         method=method,
-#         url=path,
-#         headers={"Authorization": f"Bearer {valid_auth_token}"},
-#         json={} if method != "DELETE" else None,
-#     )
-#     if is_valid:
-#         assert response.status_code == 200
-#     else:
-#         assert response.status_code == 403

tests/test_filters_jinja2.py

@@ -1,14 +1,11 @@
 """Tests for Jinja2 CQL2 filter (simplified for readability)."""
 
 import json
-from typing import cast
-from unittest.mock import MagicMock
 
 import cql2
 import pytest
 from fastapi.testclient import TestClient
-from httpx import Request
-from utils import AppFactory, parse_query_string
+from utils import AppFactory, get_upstream_request
 
 FILTER_EXPR_CASES = [
     pytest.param(
@@ -148,14 +145,6 @@ def _build_client(
     return TestClient(app, headers=headers)
 
 
-async def _get_upstream_request(mock_upstream: MagicMock):
-    """Fetch the raw body and query params from the single upstream request."""
-    assert mock_upstream.call_count == 1
-    [request] = cast(list[Request], mock_upstream.call_args[0])
-    req_body = request._streamed_body
-    return req_body.decode(), parse_query_string(request.url.query.decode("utf-8"))
-
-
 @pytest.mark.parametrize(
     "filter_template_expr, expected_auth_filter, expected_anon_filter",
     FILTER_EXPR_CASES,
@@ -182,8 +171,8 @@ async def test_search_post(
     response.raise_for_status()
 
     # Retrieve the JSON body that was actually sent upstream
-    proxied_body_str = (await _get_upstream_request(mock_upstream))[0]
-    proxied_body = json.loads(proxied_body_str)
+    proxied_request = await get_upstream_request(mock_upstream)
+    proxied_body = json.loads(proxied_request.body)
 
     # Determine the expected combined filter
     proxy_filter = cql2.Expr(
@@ -231,8 +220,8 @@ async def test_search_get(
     response.raise_for_status()
 
     # For GET, we expect the upstream body to be empty, but URL params to be appended
-    proxied_body, upstream_query = await _get_upstream_request(mock_upstream)
-    assert proxied_body == ""
+    proxied_request = await get_upstream_request(mock_upstream)
+    assert proxied_request.body == ""
 
     # Determine the expected combined filter
     proxy_filter = cql2.Expr(
@@ -253,7 +242,7 @@ async def test_search_get(
         "filter-lang": filter_lang,
     }
     assert (
-        upstream_query == expected_output
+        proxied_request.query_params == expected_output
     ), "GET query should combine filter expressions."
 
 
@@ -284,15 +273,15 @@ async def test_items_list(
     response.raise_for_status()
 
     # For GET items, we also expect an empty body and appended querystring
-    proxied_body, proxied_query = await _get_upstream_request(mock_upstream)
-    assert proxied_body == ""
+    proxied_request = await get_upstream_request(mock_upstream)
+    assert proxied_request.body == ""
 
     # Only the appended filter (no input_filter merges in these particular tests),
     # but you could do similar merging logic if needed.
     proxy_filter = cql2.Expr(
         expected_auth_filter if is_authenticated else expected_anon_filter
     )
-    assert proxied_query == {
+    assert proxied_request.query_params == {
         "filter-lang": "cql2-text",
         "filter": (
             proxy_filter + cql2.Expr(qs_filter)

tests/test_proxy.py

@@ -0,0 +1,40 @@
+"""Test authentication cases for the proxy app."""
+
+from fastapi.testclient import TestClient
+from utils import AppFactory, get_upstream_request
+
+app_factory = AppFactory(
+    oidc_discovery_url="https://example-stac-api.com/.well-known/openid-configuration",
+    default_public=True,
+    public_endpoints={},
+    private_endpoints={},
+)
+
+
+async def test_proxied_headers_no_encoding(source_api_server, mock_upstream):
+    """Clients that don't accept encoding should not receive it."""
+    test_app = app_factory(upstream_url=source_api_server)
+
+    client = TestClient(test_app)
+    req = client.build_request(method="GET", url="/", headers={})
+    for h in req.headers:
+        if h in ["accept-encoding"]:
+            del req.headers[h]
+    client.send(req)
+
+    proxied_request = await get_upstream_request(mock_upstream)
+    assert "accept-encoding" not in proxied_request.headers
+
+
+async def test_proxied_headers_with_encoding(source_api_server, mock_upstream):
+    """Clients that do accept encoding should receive it."""
+    test_app = app_factory(upstream_url=source_api_server)
+
+    client = TestClient(test_app)
+    req = client.build_request(
+        method="GET", url="/", headers={"accept-encoding": "gzip"}
+    )
+    client.send(req)
+
+    proxied_request = await get_upstream_request(mock_upstream)
+    assert proxied_request.headers.get("accept-encoding") == "gzip"

tests/utils.py

@@ -2,10 +2,12 @@
 
 import json
 from dataclasses import dataclass
-from typing import Callable
+from typing import Callable, cast
+from unittest.mock import MagicMock
 from urllib.parse import parse_qs, unquote
 
 import httpx
+from httpx import Headers, Request
 
 from stac_auth_proxy import Settings, create_app
 
@@ -68,3 +70,24 @@ def parse_query_string(qs: str) -> dict:
             result[key] = unquote(value)
 
     return result
+
+
+async def get_upstream_request(mock_upstream: MagicMock) -> "UpstreamRequest":
+    """Fetch the raw body and query params from the single upstream request."""
+    assert mock_upstream.call_count == 1
+    [request] = cast(list[Request], mock_upstream.call_args[0])
+    req_body = request._streamed_body
+    return UpstreamRequest(
+        body=req_body.decode(),
+        query_params=parse_query_string(request.url.query.decode("utf-8")),
+        headers=request.headers,
+    )
+
+
+@dataclass
+class UpstreamRequest:
+    """The raw body and query params from the single upstream request."""
+
+    body: str
+    query_params: dict
+    headers: Headers