feat: Add Redis caching for navigation pagination (stac-utils#488)

YuriZmytrakov · Yuri Zmytrakov · jonhealy1 · web-flow · commit 3dbdf2366be9 · 2025-10-23T17:28:49.000+08:00
**Related Issue(s):** - stac-utils#439 **Description:** Add Redis caching support for navigation pagination to enable proper `prev`/`next` links in STAC API responses. **PR Checklist:** - [x] Code is formatted and linted (run `pre-commit run --all-files`) - [x] Tests pass (run `make test`) - [x] Documentation has been updated to reflect changes, if applicable - [x] Changes are added to the changelog --------- Co-authored-by: Yuri Zmytrakov <yzmytrakovNB@yzmytrakovNB.local> Co-authored-by: Jonathan Healy <jonathan.d.healy@gmail.com>
diff --git a/.github/workflows/cicd.yml b/.github/workflows/cicd.yml
@@ -56,6 +56,16 @@ jobs:
           --ulimit nofile=65536:65536
           --ulimit memlock=-1:-1
 
+      redis:
+        image: redis:7-alpine
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          - 6379:6379
+
     strategy:
       matrix:
         python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
@@ -115,4 +125,7 @@ jobs:
           ES_USE_SSL: false
           DATABASE_REFRESH: true
           ES_VERIFY_CERTS: false
-          BACKEND: ${{ matrix.backend == 'elasticsearch8' && 'elasticsearch' || 'opensearch' }}
+          REDIS_ENABLE: true
+          REDIS_HOST: localhost
+          REDIS_PORT: 6379
+          BACKEND: ${{ matrix.backend == 'elasticsearch8' && 'elasticsearch' || 'opensearch' }}
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -31,7 +31,8 @@ repos:
         ]
         additional_dependencies: [
           "types-attrs",
-          "types-requests"
+          "types-requests",
+          "types-redis"
         ]
   - repo: https://github.com/PyCQA/pydocstyle
     rev: 6.1.1
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ### Added
 
 - Environment variable `EXCLUDED_FROM_QUERYABLES` to exclude specific fields from queryables endpoint and filtering. Supports comma-separated list of fully qualified field names (e.g., `properties.auth:schemes,properties.storage:schemes`) [#489](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/489)
+- Added Redis caching configuration for navigation pagination support, enabling proper `prev` and `next` links in paginated responses. [#488](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/488)
 
 ### Changed
 
diff --git a/Makefile b/Makefile
@@ -117,4 +117,4 @@ docs-image:
 .PHONY: docs
 docs: docs-image
 	docker compose -f compose.docs.yml \
-		run docs
+		run docs
diff --git a/README.md b/README.md
@@ -118,6 +118,7 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
   - [Collection Pagination](#collection-pagination)
   - [SFEOS Tools CLI](#sfeos-tools-cli)
   - [Ingesting Sample Data CLI Tool](#ingesting-sample-data-cli-tool)
+  - [Redis for navigation](#redis-for-navigation)
   - [Elasticsearch Mappings](#elasticsearch-mappings)
   - [Managing Elasticsearch Indices](#managing-elasticsearch-indices)
     - [Snapshots](#snapshots)
@@ -344,6 +345,32 @@ You can customize additional settings in your `.env` file:
 > [!NOTE]
 > The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.
 
+## Redis for Navigation environment variables:
+These Redis configuration variables to enable proper navigation functionality in STAC FastAPI.
+
+| Variable                      | Description                                                                                  | Default                  | Required                                                                                     |
+|-------------------------------|----------------------------------------------------------------------------------------------|--------------------------|---------------------------------------------------------------------------------------------|
+| `REDIS_ENABLE`                | Enables or disables Redis caching for navigation. Set to `true` to use Redis, or `false` to disable. | `false`                  | **Required** (determines whether Redis is used at all)                                      |
+| **Redis Sentinel**            |                                                                                              |                          |                                                                                             |
+| `REDIS_SENTINEL_HOSTS`        | Comma-separated list of Redis Sentinel hostnames/IP addresses.                               | `""`                     | Conditional (required if using Sentinel)                                                    |
+| `REDIS_SENTINEL_PORTS`        | Comma-separated list of Redis Sentinel ports (must match order).                             | `"26379"`                | Conditional (required if using Sentinel)                                                    |
+| `REDIS_SENTINEL_MASTER_NAME`  | Name of the Redis master node in Sentinel configuration.                                     | `"master"`               | Conditional (required if using Sentinel)                                                    |
+| **Redis**                     |                                                                                              |                          |                                                                                             |
+| `REDIS_HOST`                  | Redis server hostname or IP address for Redis configuration.                                 | `""`                     | Conditional (required for standalone Redis)                                                 |
+| `REDIS_PORT`                  | Redis server port for Redis configuration.                                                   | `6379`                   | Conditional (required for standalone Redis)                                                 |
+| **Both**                      |                                                                                              |                          |                                                                                             |
+| `REDIS_DB`                    | Redis database number to use for caching.                                                    | `0` (Sentinel) / `15` (Standalone) | Optional                                                                                    |
+| `REDIS_MAX_CONNECTIONS`       | Maximum number of connections in the Redis connection pool.                                  | `10`                     | Optional                                                                                    |
+| `REDIS_RETRY_TIMEOUT`         | Enable retry on timeout for Redis operations.                                                | `true`                   | Optional                                                                                    |
+| `REDIS_DECODE_RESPONSES`      | Automatically decode Redis responses to strings.                                             | `true`                   | Optional                                                                                    |
+| `REDIS_CLIENT_NAME`           | Client name identifier for Redis connections.                                                | `"stac-fastapi-app"`     | Optional                                                                                    |
+| `REDIS_HEALTH_CHECK_INTERVAL` | Interval in seconds for Redis health checks.                                                 | `30`                     | Optional                                                                                    |
+| `REDIS_SELF_LINK_TTL` | Time-to-live (TTL) in seconds for storing self-links in Redis, used for pagination caching. | 1800 | Optional |
+
+
+> [!NOTE]
+> Use either the Sentinel configuration (`REDIS_SENTINEL_HOSTS`, `REDIS_SENTINEL_PORTS`, `REDIS_SENTINEL_MASTER_NAME`) OR the Redis configuration (`REDIS_HOST`, `REDIS_PORT`), but not both.
+
 ## Excluding Fields from Queryables
 
 You can exclude specific fields from being exposed in the queryables endpoint and from filtering by setting the `EXCLUDED_FROM_QUERYABLES` environment variable. This is useful for hiding sensitive or internal fields that should not be queryable by API users.
@@ -615,6 +642,19 @@ The system uses a precise naming convention:
     python3 data_loader.py --base-url http://localhost:8080 --use-bulk
     ```
 
+## Redis for Navigation
+
+The Redis cache stores navigation state for paginated results, allowing the system to maintain previous page links using tokens. The configuration supports both Redis Sentinel and standalone Redis setups.
+
+Steps to configure:
+1. Ensure that a Redis instance is available, either a standalone server or a Sentinel-managed cluster.
+2. Establish a connection between STAC FastAPI and Redis instance by setting the appropriate [**environment variables**](#redis-for-navigation-environment-variables). These define the Redis host, port, authentication, and optional Sentinel settings.
+3. Control whether Redis caching is activated using the `REDIS_ENABLE` environment variable to `True` or `False`.
+4. Ensure the appropriate version of `Redis` is installed:
+```
+pip install stac-fastapi-elasticsearch[redis]
+```
+
 ## Elasticsearch Mappings
 
 - **Overview**: Mappings apply to search index, not source data. They define how documents and their fields are stored and indexed.
diff --git a/compose.yml b/compose.yml
@@ -23,6 +23,9 @@ services:
       - BACKEND=elasticsearch
       - DATABASE_REFRESH=true
       - ENABLE_COLLECTIONS_SEARCH_ROUTE=true
+      - REDIS_ENABLE=true
+      - REDIS_HOST=redis
+      - REDIS_PORT=6379
     ports:
       - "8080:8080"
     volumes:
@@ -31,6 +34,7 @@ services:
       - ./esdata:/usr/share/elasticsearch/data
     depends_on:
       - elasticsearch
+      - redis
     command:
       bash -c "./scripts/wait-for-it-es.sh es-container:9200 && python -m stac_fastapi.elasticsearch.app"
 
@@ -58,6 +62,9 @@ services:
       - BACKEND=opensearch
       - STAC_FASTAPI_RATE_LIMIT=200/minute
       - ENABLE_COLLECTIONS_SEARCH_ROUTE=true
+      - REDIS_ENABLE=true
+      - REDIS_HOST=redis
+      - REDIS_PORT=6379
     ports:
       - "8082:8082"
     volumes:
@@ -66,6 +73,7 @@ services:
       - ./osdata:/usr/share/opensearch/data
     depends_on:
       - opensearch
+      - redis
     command:
       bash -c "./scripts/wait-for-it-es.sh os-container:9202 && python -m stac_fastapi.opensearch.app"
 
@@ -96,3 +104,14 @@ services:
       - ./opensearch/snapshots:/usr/share/opensearch/snapshots
     ports:
       - "9202:9202"
+
+  redis:
+    image: redis:7-alpine
+    hostname: redis
+    ports:
+      - "6379:6379"
+    volumes:
+      - redis_test_data:/data
+    command: redis-server
+volumes:
+  redis_test_data:
diff --git a/stac_fastapi/core/pyproject.toml b/stac_fastapi/core/pyproject.toml
@@ -44,6 +44,7 @@ dependencies = [
     "pygeofilter~=0.3.1",
     "jsonschema~=4.0.0",
     "slowapi~=0.1.9",
+    "redis==6.4.0",
 ]
 
 [project.urls]
diff --git a/stac_fastapi/core/stac_fastapi/core/core.py b/stac_fastapi/core/stac_fastapi/core/core.py
@@ -24,9 +24,10 @@
 from stac_fastapi.core.base_settings import ApiBaseSettings
 from stac_fastapi.core.datetime_utils import format_datetime_range
 from stac_fastapi.core.models.links import PagingLinks
+from stac_fastapi.core.redis_utils import redis_pagination_links
 from stac_fastapi.core.serializers import CollectionSerializer, ItemSerializer
 from stac_fastapi.core.session import Session
-from stac_fastapi.core.utilities import filter_fields
+from stac_fastapi.core.utilities import filter_fields, get_bool_env
 from stac_fastapi.extensions.core.transaction import AsyncBaseTransactionsClient
 from stac_fastapi.extensions.core.transaction.request import (
     PartialCollection,
@@ -262,6 +263,7 @@ async def all_collections(
             A Collections object containing all the collections in the database and links to various resources.
         """
         base_url = str(request.base_url)
+        redis_enable = get_bool_env("REDIS_ENABLE", default=False)
 
         global_max_limit = (
             int(os.getenv("STAC_GLOBAL_COLLECTION_MAX_LIMIT"))
@@ -417,6 +419,14 @@ async def all_collections(
             },
         ]
 
+        if redis_enable:
+            await redis_pagination_links(
+                current_url=str(request.url),
+                token=token,
+                next_token=next_token,
+                links=links,
+            )
+
         if next_token:
             next_link = PagingLinks(next=next_token, request=request).link_next()
             links.append(next_link)
@@ -761,8 +771,8 @@ async def post_search(
         search_request.limit = limit
 
         base_url = str(request.base_url)
-
         search = self.database.make_search()
+        redis_enable = get_bool_env("REDIS_ENABLE", default=False)
 
         if search_request.ids:
             search = self.database.apply_ids_filter(
@@ -866,6 +876,34 @@ async def post_search(
         ]
         links = await PagingLinks(request=request, next=next_token).get_links()
 
+        collection_links = []
+        # Add "collection" and "parent" rels only for /collections/{collection_id}/items
+        if search_request.collections and "/items" in str(request.url):
+            for collection_id in search_request.collections:
+                collection_links.extend(
+                    [
+                        {
+                            "rel": "collection",
+                            "type": "application/json",
+                            "href": urljoin(base_url, f"collections/{collection_id}"),
+                        },
+                        {
+                            "rel": "parent",
+                            "type": "application/json",
+                            "href": urljoin(base_url, f"collections/{collection_id}"),
+                        },
+                    ]
+                )
+        links.extend(collection_links)
+
+        if redis_enable:
+            await redis_pagination_links(
+                current_url=str(request.url),
+                token=token_param,
+                next_token=next_token,
+                links=links,
+            )
+
         return stac_types.ItemCollection(
             type="FeatureCollection",
             features=items,
diff --git a/stac_fastapi/core/stac_fastapi/core/redis_utils.py b/stac_fastapi/core/stac_fastapi/core/redis_utils.py
diff --git a/stac_fastapi/tests/redis/__init__.py b/stac_fastapi/tests/redis/__init__.py
diff --git a/stac_fastapi/tests/redis/test_redis_pagination.py b/stac_fastapi/tests/redis/test_redis_pagination.py
diff --git a/stac_fastapi/tests/redis/test_redis_utils.py b/stac_fastapi/tests/redis/test_redis_utils.py

Original file line number	Diff line number	Diff line change
`@@ -31,7 +31,8 @@ repos:`
`31`	`31`	`]`
`32`	`32`	`additional_dependencies: [`
`33`	`33`	`"types-attrs",`
`34`		`- "types-requests"`
	`34`	`+ "types-requests",`
	`35`	`+ "types-redis"`
`35`	`36`	`]`
`36`	`37`	`- repo: https://github.com/PyCQA/pydocstyle`
`37`	`38`	`rev: 6.1.1`
Original file line number	Diff line number	Diff line change
`@@ -44,6 +44,7 @@ dependencies = [`
`44`	`44`	`"pygeofilter~=0.3.1",`
`45`	`45`	`"jsonschema~=4.0.0",`
`46`	`46`	`"slowapi~=0.1.9",`
	`47`	`+ "redis==6.4.0",`
`47`	`48`	`]`
`48`	`49`
`49`	`50`	`[project.urls]`