Merge branch 'main' into release/v6.5.0

Author: jonhealy1

CHANGELOG.md

@@ -17,14 +17,16 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### Added
 
+- Environment variable `ENABLE_COLLECTIONS_SEARCH_ROUTE` to turn on/off the `/collections-search` endpoint. [#478](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/478)
+- POST and GET `/collections-search` endpoint for collections search queries, needed because POST /collections search will not work when the Transactions Extension is enabled. Defaults to `False` [#478](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/478)
 - GET `/collections` collection search structured filter extension with support for both cql2-json and cql2-text formats. [#475](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/475)
 - GET `/collections` collection search query extension. [#477](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/477)
 - GET `/collections` collections search datetime filtering support. [#476](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/476)
 
 ### Changed
+- Refactored `/collections` endpoint implementation to support both GET and POST methods. [#478](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/478)
 
 ### Fixed
-
 - support of disabled nested attributes in the properties dictionary. [#474](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/474)
 
 ## [v6.4.0] - 2025-09-24

README.md

@@ -66,13 +66,13 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
 ## Table of Contents
 
 - [stac-fastapi-elasticsearch-opensearch](#stac-fastapi-elasticsearch-opensearch)
-  - [Sponsors \& Supporters](#sponsors--supporters)
+  - [Sponsors & Supporters](#sponsors--supporters)
   - [Project Introduction - What is SFEOS?](#project-introduction---what-is-sfeos)
   - [Common Deployment Patterns](#common-deployment-patterns)
   - [Technologies](#technologies)
   - [Table of Contents](#table-of-contents)
   - [Collection Search Extensions](#collection-search-extensions)
-  - [Documentation \& Resources](#documentation--resources)
+  - [Documentation & Resources](#documentation--resources)
   - [Package Structure](#package-structure)
   - [Examples](#examples)
   - [Performance](#performance)
@@ -115,7 +115,11 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
 
 ## Collection Search Extensions
 
-SFEOS implements extended capabilities for the `/collections` endpoint, allowing for more powerful collection discovery:
+SFEOS provides enhanced collection search capabilities through two primary routes:
+- **GET/POST `/collections`**: The standard STAC endpoint with extended query parameters
+- **GET/POST `/collections-search`**: A custom endpoint that supports the same parameters, created to avoid conflicts with the STAC Transactions extension if enabled (which uses POST `/collections` for collection creation)
+
+These endpoints support advanced collection discovery features including:
 
 - **Sorting**: Sort collections by sortable fields using the `sortby` parameter
   - Example: `/collections?sortby=+id` (ascending sort by ID)
@@ -146,11 +150,11 @@ SFEOS implements extended capabilities for the `/collections` endpoint, allowing
   - Collections are matched if their temporal extent overlaps with the provided datetime parameter
   - This allows for efficient discovery of collections based on time periods
 
-> **Note on HTTP Methods**: All collection search extensions (sorting, field selection, free text search, structured filtering, and datetime filtering) currently only support GET requests. POST requests with these parameters in the request body are not yet supported.
-
 These extensions make it easier to build user interfaces that display and navigate through collections efficiently.
 
-> **Configuration**: Collection search extensions (sorting, field selection, free text search, structured filtering, and datetime filtering) can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
+> **Configuration**: Collection search extensions (sorting, field selection, free text search, structured filtering, and datetime filtering) for the `/collections` endpoint can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
+> 
+> **Configuration**: The custom `/collections-search` endpoint can be enabled by setting the `ENABLE_COLLECTIONS_SEARCH_ROUTE` environment variable to `true`. By default, this endpoint is **disabled**.
 
 > **Note**: Sorting is only available on fields that are indexed for sorting in Elasticsearch/OpenSearch. With the default mappings, you can sort on:
 > - `id` (keyword field)
@@ -161,6 +165,7 @@ These extensions make it easier to build user interfaces that display and naviga
 >
 > **Important**: Adding keyword fields to make text fields sortable can significantly increase the index size, especially for large text fields. Consider the storage implications when deciding which fields to make sortable.
 
+
 ## Package Structure
 
 This project is organized into several packages, each with a specific purpose:
@@ -291,8 +296,9 @@ You can customize additional settings in your `.env` file:
 | `ENABLE_DIRECT_RESPONSE`     | Enable direct response for maximum performance (disables all FastAPI dependencies, including authentication, custom status codes, and validation) | `false`                  | Optional                       |
 | `RAISE_ON_BULK_ERROR`        | Controls whether bulk insert operations raise exceptions on errors. If set to `true`, the operation will stop and raise an exception when an error occurs. If set to `false`, errors will be logged, and the operation will continue. **Note:** STAC Item and ItemCollection validation errors will always raise, regardless of this flag. | `false` | Optional |
 | `DATABASE_REFRESH`           | Controls whether database operations refresh the index immediately after changes. If set to `true`, changes will be immediately searchable. If set to `false`, changes may not be immediately visible but can improve performance for bulk operations. If set to `wait_for`, changes will wait for the next refresh cycle to become visible. | `false` | Optional |
-| `ENABLE_COLLECTIONS_SEARCH`  | Enable collection search extensions (sort, fields, free text search, structured filtering, and datetime filtering).                                 | `true`                   | Optional                                                                                    |
-| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. If set to `false`, the POST `/collections` route and related transaction endpoints (including bulk transaction operations) will be unavailable in the API. This is useful for deployments where mutating the catalog via the API should be prevented. | `true` | Optional |
+| `ENABLE_COLLECTIONS_SEARCH`  | Enable collection search extensions (sort, fields, free text search, structured filtering, and datetime filtering) on the core `/collections` endpoint. | `true`                   | Optional                                                                                    |
+| `ENABLE_COLLECTIONS_SEARCH_ROUTE` | Enable the custom `/collections-search` endpoint (both GET and POST methods). When disabled, the custom endpoint will not be available, but collection search extensions will still be available on the core `/collections` endpoint if `ENABLE_COLLECTIONS_SEARCH` is true. | `false` | Optional |
+| `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. This is useful for deployments where mutating the catalog via the API should be prevented. If set to `true`, the POST `/collections` route for search will be unavailable in the API. | `true` | Optional |
 | `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
 | `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
 | `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
@@ -442,7 +448,6 @@ The system uses a precise naming convention:
   - `ENABLE_COLLECTIONS_SEARCH`: Set to `true` (default) to enable collection search extensions (sort, fields). Set to `false` to disable.
   - `ENABLE_TRANSACTIONS_EXTENSIONS`: Set to `true` (default) to enable transaction extensions. Set to `false` to disable.
 
-
 ## Collection Pagination
 
 - **Overview**: The collections route supports pagination through optional query parameters.

stac_fastapi/core/stac_fastapi/core/core.py

@@ -136,6 +136,20 @@ def _landing_page(
                     "href": urljoin(base_url, "search"),
                     "method": "POST",
                 },
+                {
+                    "rel": "collections-search",
+                    "type": "application/json",
+                    "title": "Collections Search",
+                    "href": urljoin(base_url, "collections-search"),
+                    "method": "GET",
+                },
+                {
+                    "rel": "collections-search",
+                    "type": "application/json",
+                    "title": "Collections Search",
+                    "href": urljoin(base_url, "collections-search"),
+                    "method": "POST",
+                },
             ],
             stac_extensions=extension_schemas,
         )
@@ -227,8 +241,9 @@ async def landing_page(self, **kwargs) -> stac_types.LandingPage:
     async def all_collections(
         self,
         datetime: Optional[str] = None,
+        limit: Optional[int] = None,
         fields: Optional[List[str]] = None,
-        sortby: Optional[str] = None,
+        sortby: Optional[Union[str, List[str]]] = None,
         filter_expr: Optional[str] = None,
         filter_lang: Optional[str] = None,
         q: Optional[Union[str, List[str]]] = None,
@@ -239,6 +254,7 @@ async def all_collections(
 
         Args:
             datetime (Optional[str]): Filter collections by datetime range.
+            limit (Optional[int]): Maximum number of collections to return.
             fields (Optional[List[str]]): Fields to include or exclude from the results.
             sortby (Optional[str]): Sorting options for the results.
             filter_expr (Optional[str]): Structured filter expression in CQL2 JSON or CQL2-text format.
@@ -252,7 +268,36 @@ async def all_collections(
         """
         request = kwargs["request"]
         base_url = str(request.base_url)
-        limit = int(request.query_params.get("limit", os.getenv("STAC_ITEM_LIMIT", 10)))
+
+        # Get the global limit from environment variable
+        global_limit = None
+        env_limit = os.getenv("STAC_ITEM_LIMIT")
+        if env_limit:
+            try:
+                global_limit = int(env_limit)
+            except ValueError:
+                # Handle invalid integer in environment variable
+                pass
+
+        # Apply global limit if it exists
+        if global_limit is not None:
+            # If a limit was provided, use the smaller of the two
+            if limit is not None:
+                limit = min(limit, global_limit)
+            else:
+                limit = global_limit
+        else:
+            # No global limit, use provided limit or default
+            if limit is None:
+                query_limit = request.query_params.get("limit")
+                if query_limit:
+                    try:
+                        limit = int(query_limit)
+                    except ValueError:
+                        limit = 10
+                else:
+                    limit = 10
+
         token = request.query_params.get("token")
 
         # Process fields parameter for filtering collection properties
@@ -262,7 +307,8 @@ async def all_collections(
                 if field[0] == "-":
                     excludes.add(field[1:])
                 else:
-                    includes.add(field[1:] if field[0] in "+ " else field)
+                    include_field = field[1:] if field[0] in "+ " else field
+                    includes.add(include_field)
 
         sort = None
         if sortby:
@@ -337,6 +383,7 @@ async def all_collections(
                     raise HTTPException(
                         status_code=400, detail=f"Error parsing filter: {e}"
                     )
+
             except Exception as e:
                 raise HTTPException(
                     status_code=400, detail=f"Invalid filter parameter: {e}"
@@ -346,7 +393,7 @@ async def all_collections(
         if datetime:
             parsed_datetime = format_datetime_range(date_str=datetime)
 
-        collections, next_token = await self.database.get_all_collections(
+        collections, next_token, maybe_count = await self.database.get_all_collections(
             token=token,
             limit=limit,
             request=request,
@@ -380,7 +427,91 @@ async def all_collections(
             next_link = PagingLinks(next=next_token, request=request).link_next()
             links.append(next_link)
 
-        return stac_types.Collections(collections=filtered_collections, links=links)
+        return stac_types.Collections(
+            collections=filtered_collections,
+            links=links,
+            numberMatched=maybe_count,
+            numberReturned=len(filtered_collections),
+        )
+
+    async def post_all_collections(
+        self, search_request: BaseSearchPostRequest, request: Request, **kwargs
+    ) -> stac_types.Collections:
+        """Search collections with POST request.
+
+        Args:
+            search_request (BaseSearchPostRequest): The search request.
+            request (Request): The request.
+
+        Returns:
+            A Collections object containing all the collections in the database and links to various resources.
+        """
+        request.postbody = search_request.model_dump(exclude_unset=True)
+
+        fields = None
+
+        # Check for field attribute (ExtendedSearch format)
+        if hasattr(search_request, "field") and search_request.field:
+            fields = []
+
+            # Handle include fields
+            if (
+                hasattr(search_request.field, "includes")
+                and search_request.field.includes
+            ):
+                for field in search_request.field.includes:
+                    fields.append(f"+{field}")
+
+            # Handle exclude fields
+            if (
+                hasattr(search_request.field, "excludes")
+                and search_request.field.excludes
+            ):
+                for field in search_request.field.excludes:
+                    fields.append(f"-{field}")
+
+        # Convert sortby parameter from POST format to all_collections format
+        sortby = None
+        # Check for sortby attribute
+        if hasattr(search_request, "sortby") and search_request.sortby:
+            # Create a list of sort strings in the format expected by all_collections
+            sortby = []
+            for sort_item in search_request.sortby:
+                # Handle different types of sort items
+                if hasattr(sort_item, "field") and hasattr(sort_item, "direction"):
+                    # This is a Pydantic model with field and direction attributes
+                    field = sort_item.field
+                    direction = sort_item.direction
+                elif isinstance(sort_item, dict):
+                    # This is a dictionary with field and direction keys
+                    field = sort_item.get("field")
+                    direction = sort_item.get("direction", "asc")
+                else:
+                    # Skip this item if we can't extract field and direction
+                    continue
+
+                if field:
+                    # Create a sort string in the format expected by all_collections
+                    # e.g., "-id" for descending sort on id field
+                    prefix = "-" if direction.lower() == "desc" else ""
+                    sortby.append(f"{prefix}{field}")
+
+        # Pass all parameters from search_request to all_collections
+        return await self.all_collections(
+            limit=search_request.limit if hasattr(search_request, "limit") else None,
+            fields=fields,
+            sortby=sortby,
+            filter_expr=search_request.filter
+            if hasattr(search_request, "filter")
+            else None,
+            filter_lang=search_request.filter_lang
+            if hasattr(search_request, "filter_lang")
+            else None,
+            query=search_request.query if hasattr(search_request, "query") else None,
+            q=search_request.q if hasattr(search_request, "q") else None,
+            request=request,
+            **kwargs,
+        )
 
     async def get_collection(
         self, collection_id: str, **kwargs

stac_fastapi/core/stac_fastapi/core/extensions/__init__.py

@@ -1,5 +1,11 @@
 """elasticsearch extensions modifications."""
 
+from .collections_search import CollectionsSearchEndpointExtension
 from .query import Operator, QueryableTypes, QueryExtension
 
-__all__ = ["Operator", "QueryableTypes", "QueryExtension"]
+__all__ = [
+    "Operator",
+    "QueryableTypes",
+    "QueryExtension",
+    "CollectionsSearchEndpointExtension",
+]