Merge branch 'main' into CAT-1382-2

Author: YuriZmytrakov

CHANGELOG.md

@@ -9,23 +9,29 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### Added
 
+- Spatial search support for collections via `bbox` parameter on `/collections` endpoint. Collections are now indexed with a `bbox_shape` field (GeoJSON polygon) derived from their spatial extent for efficient geospatial queries when created or updated. [#481](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/481)
+- Introduced SFEOS Tools (`sfeos_tools/`) - An installable Click-based CLI package for managing SFEOS deployments. Initial command `add-bbox-shape` adds the `bbox_shape` field to existing collections for spatial search compatibility. Install with `pip install sfeos-tools[elasticsearch]` or `pip install sfeos-tools[opensearch]`. [#481](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/481)
+- Moved SFEOS Tools to its own repository at [Healy-Hyperspatial/sfeos-tools](https://github.com/Healy-Hyperspatial/sfeos-tools). The CLI package is now maintained separately. [#PR_NUMBER]
 - CloudFerro logo to sponsors and supporters list [#485](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/485)
 - Latest news section to README [#485](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/485)
 - Added Redis caching configuration for navigation pagination support, enabling proper `prev` and `next` links in paginated responses. [#488](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/488)
 
 ### Changed
 
+- Removed ENV_MAX_LIMIT environment variable; maximum limits are now handled by the default global limit environment variable. [#482](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/482)
+- Changed the default and maximum pagination limits for collections/items endpoints. [#482](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/482)
+
 ### Fixed
 
-[v6.5.1] - 2025-09-30
+## [v6.5.1] - 2025-09-30
 
 ### Fixed
 
 - Issue where token, query param was not being passed to POST collections search logic [#483](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/483)
 - Issue where datetime param was not being passed from POST collections search logic to Elasticsearch [#483](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/483)
 - Collections search tests to ensure both GET /collections and GET/POST /collections-search endpoints are tested [#483](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/483)
 
-[v6.5.0] - 2025-09-29
+## [v6.5.0] - 2025-09-29
 
 ### Added
 

README.md

@@ -30,11 +30,20 @@ The following organizations have contributed time and/or funding to support the
 
 ## Latest News
 
-<div style="max-height: 200px; overflow-y: auto; padding: 10px; border: 1px solid #eaecef; border-radius: 6px; margin-bottom: 16px;">
+- **10/15/2025:** 🚀 SFEOS Tools v0.1.0 Released! - The new `sfeos-tools` CLI is now available on [PyPI](https://pypi.org/project/sfeos-tools/)
+- **10/15/2025:** Added `reindex` command to **[SFEOS-tools](https://github.com/Healy-Hyperspatial/sfeos-tools)** for zero-downtime index updates when changing mappings or settings. The new `reindex` command makes it easy to apply mapping changes, update index settings, or migrate to new index structures without any service interruption, ensuring high availability of your STAC API during maintenance operations.
+- **10/12/2025:** Collections search **bbox** functionality added! The collections search extension now supports bbox queries. Collections will need to be updated via the API or with the new **[SFEOS-tools](https://github.com/Healy-Hyperspatial/sfeos-tools)** CLI package to support geospatial discoverability. 🙏 Thanks again to **CloudFerro** for their sponsorship of this work!
+- **10/04/2025:** The **[CloudFerro](https://cloudferro.com/)** logo has been added to the sponsors and supporters list above. Their sponsorship of the ongoing collections search extension work has been invaluable. This is in addition to the many other important changes and updates their developers have added to the project.
 
-- 10/04/2025: The [CloudFerro](https://cloudferro.com/) logo has been added to the sponsors and supporters list above. Their sponsorship of the ongoing collections search extension work has been invaluable. This is in addition to the many other important changes and updates their developers have added to the project.
+<details style="border: 1px solid #eaecef; border-radius: 6px; padding: 10px; margin-bottom: 16px; background-color: #f9f9f9;">
+<summary style="cursor: pointer; font-weight: bold; margin: -10px -10px 0; padding: 10px; background-color: #f0f0f0; border-bottom: 1px solid #eaecef; border-top-left-radius: 6px; border-top-right-radius: 6px;">View Older News (Click to Expand)</summary>
+
+-------------
+- **09/25/2025:** v6.5.0 adds a new GET/POST /collections-search endpoint (disabled by default via ENABLE_COLLECTIONS_SEARCH_ROUTE) to avoid conflicts with the Transactions Extension, and enhances collections search with structured filtering (CQL2 JSON/text), query, and datetime filtering. These changes make collection discovery more powerful and configurable while preserving compatibility with transaction-enabled deployments.
+<!-- Add more older news items here in Markdown format; GitHub will parse them thanks to the blank line implicit in this structure -->
+
+</details>
 
-</div>
 
 ## Project Introduction - What is SFEOS?
 
@@ -105,6 +114,7 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
   - [Interacting with the API](#interacting-with-the-api)
   - [Configure the API](#configure-the-api)
   - [Collection Pagination](#collection-pagination)
+  - [SFEOS Tools CLI](#sfeos-tools-cli)
   - [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
   - [Elasticsearch Mappings](#elasticsearch-mappings)
   - [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
@@ -160,9 +170,21 @@ These endpoints support advanced collection discovery features including:
   - Collections are matched if their temporal extent overlaps with the provided datetime parameter
   - This allows for efficient discovery of collections based on time periods
 
+- **Spatial Filtering**: Filter collections by their spatial extent using the `bbox` parameter
+  - Example: `/collections?bbox=-10,35,40,70` (finds collections whose spatial extent intersects with this bounding box)
+  - Example: `/collections?bbox=-180,-90,180,90` (finds all collections with global coverage)
+  - Supports both 2D bounding boxes `[minx, miny, maxx, maxy]` and 3D bounding boxes `[minx, miny, minz, maxx, maxy, maxz]` (altitude values are ignored for spatial queries)
+  - Collections are matched if their spatial extent (stored in the `extent.spatial.bbox` field) intersects with the provided bbox parameter
+  - **Implementation Note**: When collections are created or updated, a `bbox_shape` field is automatically generated from the collection's spatial extent and indexed as a GeoJSON polygon for efficient geospatial queries
+  - **Migrating Legacy Collections**: Collections created before this feature was added will not be discoverable via bbox search until they have the `bbox_shape` field added. You can either:
+    - Update each collection via the API (PUT `/collections/{collection_id}` with the existing collection data)
+    - Use the [SFEOS Tools CLI](https://github.com/Healy-Hyperspatial/sfeos-tools) (install with `pip install sfeos-tools[elasticsearch]` or `pip install sfeos-tools[opensearch]`):
+      - `sfeos-tools add-bbox-shape --backend elasticsearch --no-ssl`
+      - `sfeos-tools add-bbox-shape --backend opensearch --host db.example.com --no-ssl`
+
 These extensions make it easier to build user interfaces that display and navigate through collections efficiently.
 
-> **Configuration**: Collection search extensions (sorting, field selection, free text search, structured filtering, and datetime filtering) for the `/collections` endpoint can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
+> **Configuration**: Collection search extensions (sorting, field selection, free text search, structured filtering, datetime filtering, and spatial filtering) for the `/collections` endpoint can be disabled by setting the `ENABLE_COLLECTIONS_SEARCH` environment variable to `false`. By default, these extensions are enabled.
 > 
 > **Configuration**: The custom `/collections-search` endpoint can be enabled by setting the `ENABLE_COLLECTIONS_SEARCH_ROUTE` environment variable to `true`. By default, this endpoint is **disabled**.
 
@@ -309,16 +331,18 @@ You can customize additional settings in your `.env` file:
 | `ENABLE_COLLECTIONS_SEARCH`  | Enable collection search extensions (sort, fields, free text search, structured filtering, and datetime filtering) on the core `/collections` endpoint. | `true`                   | Optional                                                                                    |
 | `ENABLE_COLLECTIONS_SEARCH_ROUTE` | Enable the custom `/collections-search` endpoint (both GET and POST methods). When disabled, the custom endpoint will not be available, but collection search extensions will still be available on the core `/collections` endpoint if `ENABLE_COLLECTIONS_SEARCH` is true. | `false` | Optional |
 | `ENABLE_TRANSACTIONS_EXTENSIONS` | Enables or disables the Transactions and Bulk Transactions API extensions. This is useful for deployments where mutating the catalog via the API should be prevented. If set to `true`, the POST `/collections` route for search will be unavailable in the API. | `true` | Optional |
-| `STAC_ITEM_LIMIT` | Sets the environment variable for result limiting to SFEOS for the number of returned items and STAC collections. | `10` | Optional |
+| `STAC_GLOBAL_COLLECTION_MAX_LIMIT` | Configures the maximum number of STAC collections that can be returned in a single search request. | N/A | Optional |
+| `STAC_DEFAULT_COLLECTION_LIMIT` | Configures the default number of STAC collections returned when no limit parameter is specified in the request. | `300` | Optional |
+| `STAC_GLOBAL_ITEM_MAX_LIMIT` | Configures the maximum number of STAC items that can be returned in a single search request. | N/A | Optional |
+| `STAC_DEFAULT_ITEM_LIMIT` | Configures the default number of STAC items returned when no limit parameter is specified in the request. | `10` | Optional |
 | `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
-| `ENV_MAX_LIMIT` | Configures the environment variable in SFEOS to override the default `MAX_LIMIT`, which controls the limit parameter for returned items and STAC collections. | `10,000` | Optional |
 | `USE_DATETIME` | Configures the datetime search behavior in SFEOS. When enabled, searches both datetime field and falls back to start_datetime/end_datetime range for items with null datetime. When disabled, searches only by start_datetime/end_datetime range. | `true` | Optional |
 
 > [!NOTE]
 > The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
 
-**Redis for Navigation:**  
-These Redis configuration variables enable proper navigation functionality in STAC FastAPI. The Redis cache stores navigation state for paginated results, allowing the system to maintain previous page links using tokens. The configuration supports either Redis Sentinel or standalone Redis setups.
+## Redis for Navigation:
+These Redis configuration variables to enable proper navigation functionality in STAC FastAPI.
 
 | Variable                      | Description                                                                                  | Default                  | Required                                                                                     |
 |-------------------------------|----------------------------------------------------------------------------------------------|--------------------------|---------------------------------------------------------------------------------------------|
@@ -497,6 +521,67 @@ The system uses a precise naming convention:
   curl -X "GET" "http://localhost:8080/collections?limit=1&token=example_token"
   ```
 
+## SFEOS Tools CLI
+
+- **Overview**: [SFEOS Tools](https://github.com/Healy-Hyperspatial/sfeos-tools) is an installable CLI package for managing and maintaining SFEOS deployments. This CLI package provides utilities for managing and maintaining SFEOS deployments.
+
+- **Installation**:
+  ```shell
+  # For Elasticsearch (from PyPI)
+  pip install sfeos-tools[elasticsearch]
+  
+  # For OpenSearch (from PyPI)
+  pip install sfeos-tools[opensearch]
+  
+  ```
+
+- **Available Commands**:
+  - `add-bbox-shape`: Add bbox_shape field to existing collections for spatial search support
+  - `reindex`: Reindex all STAC indices (collections and per-collection items) to new versioned indices and update aliases; supports both Elasticsearch and OpenSearch backends. Use this when you need to apply mapping changes, update index settings, or migrate to a new index structure. The command handles the entire process including creating new indices, reindexing data, and atomically updating aliases with zero downtime.
+
+- **Basic Usage**:
+  ```shell
+  sfeos-tools add-bbox-shape --backend elasticsearch
+  sfeos-tools add-bbox-shape --backend opensearch
+  ```
+
+- **Connection Options**: Configure database connection via CLI flags or environment variables:
+  - `--host`: Database host (default: `localhost` or `ES_HOST` env var)
+  - `--port`: Database port (default: `9200` or `ES_PORT` env var)
+  - `--use-ssl` / `--no-ssl`: Use SSL connection (default: `true` or `ES_USE_SSL` env var)
+  - `--user`: Database username (default: `ES_USER` env var)
+  - `--password`: Database password (default: `ES_PASS` env var)
+
+- **Examples**:
+  ```shell
+  # Local Docker Compose (no SSL)
+  sfeos-tools add-bbox-shape --backend elasticsearch --no-ssl
+  
+  # Remote server with SSL
+  sfeos-tools add-bbox-shape \
+    --backend elasticsearch \
+    --host db.example.com \
+    --port 9200 \
+    --user admin \
+    --password secret
+  
+  # Cloud deployment with environment variables
+  ES_HOST=my-es-cluster.cloud.com ES_PORT=9243 ES_USER=elastic ES_PASS=changeme \
+    sfeos-tools add-bbox-shape --backend elasticsearch
+  
+  # Using --help for more information
+  sfeos-tools --help
+  sfeos-tools add-bbox-shape --help
+  sfeos-tools reindex --help
+
+  ```
+
+- **Documentation**:
+  For complete documentation, examples, and advanced usage, please visit the [SFEOS Tools GitHub repository](https://github.com/Healy-Hyperspatial/sfeos-tools).
+
+- **Contributing**:
+  Contributions, bug reports, and feature requests are welcome! Please file them on the [SFEOS Tools issue tracker](https://github.com/Healy-Hyperspatial/sfeos-tools/issues).
+
 ## Ingesting Sample Data CLI Tool
 
 - **Overview**: The `data_loader.py` script provides a convenient way to load STAC items into the database.
@@ -530,6 +615,15 @@ The system uses a precise naming convention:
     python3 data_loader.py --base-url http://localhost:8080 --use-bulk
     ```
 
+## Redis for Navigation
+
+The Redis cache stores navigation state for paginated results, allowing the system to maintain previous page links using tokens. The configuration supports both Redis Sentinel and standalone Redis setups.
+
+Steps to configure:
+1. Ensure that a Redis instance is available, either a standalone server or a Sentinel-managed cluster.
+2. Establish a connection between STAC FastAPI and Redis instance by setting the appropriate [**environment variables**](#redis-for-navigation)These define the Redis host, port, authentication, and optional Sentinel settings.
+3. Control whether Redis caching is activated using the `REDIS_ENABLE` environment variable to `True` or `False`.
+
 ## Elasticsearch Mappings
 
 - **Overview**: Mappings apply to search index, not source data. They define how documents and their fields are stored and indexed.

stac_fastapi/core/stac_fastapi/core/base_database_logic.py

@@ -3,6 +3,8 @@
 import abc
 from typing import Any, Dict, Iterable, List, Optional, Tuple
 
+from stac_pydantic.shared import BBox
+
 
 class BaseDatabaseLogic(abc.ABC):
     """
@@ -19,17 +21,27 @@ async def get_all_collections(
         limit: int,
         request: Any = None,
         sort: Optional[List[Dict[str, Any]]] = None,
-    ) -> Tuple[List[Dict[str, Any]], Optional[str]]:
+        bbox: Optional[BBox] = None,
+        q: Optional[List[str]] = None,
+        filter: Optional[Dict[str, Any]] = None,
+        query: Optional[Dict[str, Dict[str, Any]]] = None,
+        datetime: Optional[str] = None,
+    ) -> Tuple[List[Dict[str, Any]], Optional[str], Optional[int]]:
         """Retrieve a list of collections from the database, supporting pagination.
 
         Args:
             token (Optional[str]): The pagination token.
             limit (int): The number of results to return.
             request (Any, optional): The FastAPI request object. Defaults to None.
             sort (Optional[List[Dict[str, Any]]], optional): Optional sort parameter. Defaults to None.
+            bbox (Optional[BBox], optional): Bounding box to filter collections by spatial extent. Defaults to None.
+            q (Optional[List[str]], optional): Free text search terms. Defaults to None.
+            filter (Optional[Dict[str, Any]], optional): Structured query in CQL2 format. Defaults to None.
+            query (Optional[Dict[str, Dict[str, Any]]], optional): Query extension parameters. Defaults to None.
+            datetime (Optional[str], optional): Temporal filter. Defaults to None.
 
         Returns:
-            A tuple of (collections, next pagination token if any).
+            A tuple of (collections, next pagination token if any, optional count).
         """
         pass