Create shared sfeos helpers module (#376)

**Related Issue(s):**

- #369 

**Description:**

- This pr moves shared elasticsearch/ opensearch logic out of `core` and
into the `sfeos_helpers` module.
- Other code duplicated in es/ os `database_logic` files has been moved
into the `sfeos_helpers` module as well.
- The readme has been updated/ reorganized and a table of contents has
been added.

**PR Checklist:**

- [x] Code is formatted and linted (run `pre-commit run --all-files`)
- [x] Tests pass (run `make test`)
- [x] Documentation has been updated to reflect changes, if applicable
- [x] Changes are added to the changelog

Author: jonhealy1

.github/workflows/cicd.yml

@@ -96,6 +96,10 @@ jobs:
         run: |
           pip install ./stac_fastapi/core
 
+      - name: Install helpers library stac-fastapi
+        run: |
+          pip install ./stac_fastapi/sfeos_helpers
+          
       - name: Install elasticsearch stac-fastapi
         run: |
           pip install ./stac_fastapi/elasticsearch[dev,server]

.github/workflows/publish.yml

@@ -35,6 +35,18 @@ jobs:
           # Publish to PyPI
           twine upload dist/*
 
+      - name: Build and publish sfeos_helpers
+        working-directory: stac_fastapi/sfeos_helpers
+        env:
+          TWINE_USERNAME: "__token__"
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+        run: |
+          # Build package
+          python setup.py sdist bdist_wheel
+  
+          # Publish to PyPI
+          twine upload dist/*
+
       - name: Build and publish stac-fastapi-elasticsearch
         working-directory: stac_fastapi/elasticsearch
         env:

CHANGELOG.md

@@ -5,24 +5,33 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+
 ## [Unreleased]
 
+## [v5.0.0a0] - 2025-05-29
+
 ### Added
 
+- Created new `sfeos_helpers` package to improve code organization and maintainability [#376](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/376)
 - Added introduction section - What is stac-fastapi-elasticsearch-opensearch? - to README [#384](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/384)
 
 ### Changed
 
+- Refactored utility functions into dedicated modules within `sfeos_helpers` [#376](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/376):
+  - Created `database` package with specialized modules for index, document, and utility operations
+  - Created `aggregation` package for Elasticsearch/OpenSearch-specific aggregation functionality
+  - Moved shared logic from core module to helper functions for better code reuse
+  - Separated utility functions from constant mappings for clearer code organization
+- Updated documentation to reflect recent code refactoring  [#376](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/376)
 - Improved README documentation with consistent formatting and enhanced sections [#381](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/381):
   - Added sfeos logo and banner
   - Added a comprehensive Quick Start guide
   - Reorganized sections for better navigation
   - Reformatted content with bullet points for improved readability
   - Added more detailed examples for API interaction
-
+  
 ### Fixed
 
-
 ## [v4.2.0] - 2025-05-15
 
 ### Added
@@ -397,7 +406,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Use genexp in execute_search and get_all_collections to return results.
 - Added db_to_stac serializer to item_collection method in core.py.
 
-[Unreleased]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v4.2.0...main
+[Unreleased]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v5.0.0a0...main
+[v5.0.0a0]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v4.2.0...v5.0.0a0
 [v4.2.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v4.1.0...v4.2.0
 [v4.1.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v4.0.0...v4.1.0
 [v4.0.0]: https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/compare/v3.2.5...v4.0.0

Makefile

@@ -95,7 +95,8 @@ pybase-install:
 	pip install -e ./stac_fastapi/api[dev] && \
 	pip install -e ./stac_fastapi/types[dev] && \
 	pip install -e ./stac_fastapi/extensions[dev] && \
-	pip install -e ./stac_fastapi/core
+	pip install -e ./stac_fastapi/core && \
+	pip install -e ./stac_fastapi/sfeos_helpers
 
 .PHONY: install-es
 install-es: pybase-install

README.md

@@ -3,7 +3,6 @@
 <!-- markdownlint-disable MD033 MD041 -->
 
 
-
 <p align="left">
   <img src="https://raw.githubusercontent.com/stac-utils/stac-fastapi-elasticsearch-opensearch/refs/heads/main/assets/sfeos.png" width=1000>
 </p>
@@ -68,6 +67,7 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
 ## Table of Contents
 
 - [Documentation & Resources](#documentation--resources)
+- [Package Structure](#package-structure)
 - [Examples](#examples)
 - [Performance](#performance)
 - [Quick Start](#quick-start)
@@ -95,6 +95,21 @@ This project is built on the following technologies: STAC, stac-fastapi, FastAPI
   - [Gitter Chat](https://app.gitter.im/#/room/#stac-fastapi-elasticsearch_community:gitter.im) - For real-time discussions
   - [GitHub Discussions](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/discussions) - For longer-form questions and answers
 
+## Package Structure
+
+This project is organized into several packages, each with a specific purpose:
+
+- **stac_fastapi_core**: Core functionality that's database-agnostic, including API models, extensions, and shared utilities. This package provides the foundation for building STAC API implementations with any database backend. See [stac-fastapi-mongo](https://github.com/Healy-Hyperspatial/stac-fastapi-mongo) for a working example.
+
+- **sfeos_helpers**: Shared helper functions and utilities used by both the Elasticsearch and OpenSearch backends. This package includes:
+  - `database`: Specialized modules for index, document, and database utility operations
+  - `aggregation`: Elasticsearch/OpenSearch-specific aggregation functionality
+  - Shared logic and utilities that improve code reuse between backends
+
+- **stac_fastapi_elasticsearch**: Complete implementation of the STAC API using Elasticsearch as the backend database. This package depends on both `stac_fastapi_core` and `sfeos_helpers`.
+- 
+- **stac_fastapi_opensearch**: Complete implementation of the STAC API using OpenSearch as the backend database. This package depends on both `stac_fastapi_core` and `sfeos_helpers`.
+
 ## Examples
 
 The `/examples` directory contains several useful examples and reference implementations:
@@ -181,6 +196,7 @@ There are two main ways to run the API locally:
 - **Compatibility**: The most recent Elasticsearch 7.x versions should also work. See the [opensearch-py docs](https://github.com/opensearch-project/opensearch-py/blob/main/COMPATIBILITY.md) for compatibility information.
 
 
+
 ## Configuration Reference
 
 You can customize additional settings in your `.env` file:
@@ -518,5 +534,6 @@ You can customize additional settings in your `.env` file:
   - Limits each client to a specified number of requests per time period (e.g., 500 requests per minute)
   - Helps prevent API abuse and maintains system stability
   - Ensures fair resource allocation among all clients
-
+  
 - **Examples**: Implementation examples are available in the [examples/rate_limit](examples/rate_limit) directory.
+

compose.yml

@@ -9,7 +9,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-elasticsearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Elasticsearch backend
-      - STAC_FASTAPI_VERSION=4.2.0
+      - STAC_FASTAPI_VERSION=5.0.0a0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-elasticsearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8080
@@ -42,7 +42,7 @@ services:
     environment:
       - STAC_FASTAPI_TITLE=stac-fastapi-opensearch
       - STAC_FASTAPI_DESCRIPTION=A STAC FastAPI with an Opensearch backend
-      - STAC_FASTAPI_VERSION=4.2.0
+      - STAC_FASTAPI_VERSION=5.0.0a0
       - STAC_FASTAPI_LANDING_PAGE_ID=stac-fastapi-opensearch
       - APP_HOST=0.0.0.0
       - APP_PORT=8082

dockerfiles/Dockerfile.ci.es

@@ -12,6 +12,7 @@ RUN apt-get update && \
 COPY . /app/
 
 RUN pip3 install --no-cache-dir -e ./stac_fastapi/core && \
+    pip3 install --no-cache-dir -e ./stac_fastapi/sfeos_helpers && \
     pip3 install --no-cache-dir ./stac_fastapi/elasticsearch[server]
 
 USER root

dockerfiles/Dockerfile.ci.os

@@ -12,6 +12,7 @@ RUN apt-get update && \
 COPY . /app/
 
 RUN pip3 install --no-cache-dir -e ./stac_fastapi/core && \
+    pip3 install --no-cache-dir -e ./stac_fastapi/sfeos_helpers && \
     pip3 install --no-cache-dir ./stac_fastapi/opensearch[server]
 
 USER root

dockerfiles/Dockerfile.deploy.es

@@ -13,6 +13,7 @@ WORKDIR /app
 COPY . /app
 
 RUN pip install --no-cache-dir -e ./stac_fastapi/core
+RUN pip install --no-cache-dir -e ./stac_fastapi/sfeos_helpers
 RUN pip install --no-cache-dir ./stac_fastapi/elasticsearch[server]
 
 EXPOSE 8080

dockerfiles/Dockerfile.deploy.os

@@ -13,6 +13,7 @@ WORKDIR /app
 COPY . /app
 
 RUN pip install --no-cache-dir -e ./stac_fastapi/core
+RUN pip install --no-cache-dir -e ./stac_fastapi/sfeos_helpers
 RUN pip install --no-cache-dir ./stac_fastapi/opensearch[server]
 
 EXPOSE 8080