Skip to content

Added support for searching large amount of indices #412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 18, 2025
Merged

Added support for searching large amount of indices #412

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
506a186
Added support for searching large amount of indices by moving the ind…
simonvb00 Jul 16, 2025
bd84128
not x in -> x not in
simonvb00 Jul 16, 2025
ab4f56e
update CHANGELOG.md
StijnCaerts Jul 17, 2025
06999c9
Create add_collections_to_body in sfeos_helpers and added search supp…
simonvb00 Jul 18, 2025
be48bd8
Fixed import.
simonvb00 Jul 18, 2025
51a7a76
Final import fix. Tests are passing again.
simonvb00 Jul 18, 2025
552b483
Precommit.
simonvb00 Jul 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

- Added the ability to set timeout for Opensearch and Elasticsearch clients by setting the environmental variable `ES_TIMEOUT` [#408](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/408)

### Changed

- Updated collection to index logic to support searching a large amount of indices [#412](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/412)

## [v6.0.0] - 2025-06-22

### Added
Expand Down
12 changes: 12 additions & 0 deletions stac_fastapi/opensearch/stac_fastapi/opensearch/database_logic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@
from stac_fastapi.types.stac import Collection, Item

logger = logging.getLogger(__name__)
ES_MAX_URL_LENGTH = 4096


async def create_index_templates() -> None:
Expand Down Expand Up @@ -546,6 +547,17 @@ async def execute_search(

index_param = indices(collection_ids)

if len(index_param) > ES_MAX_URL_LENGTH - 300:
index_param = ITEM_INDICES
index_filter = {"terms": {"collection": collection_ids}}
if "bool" not in search_body["query"]:
search_body["query"]["bool"] = {}
if "filter" not in search_body["query"]["bool"]:
search_body["query"]["bool"]["filter"] = [index_filter]
filters = search_body["query"]["bool"]["filter"]
if index_filter not in filters:
filters.append(index_filter)

max_result_window = MAX_LIMIT

size_limit = min(limit + 1, max_result_window)
Expand Down