Added support for searching large amount of indices #412
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ices from the request url to the body of the request when size is larger than 4096 bytes.
simonvb00
changed the title
StijnCaerts
previously approved these changes
Jul 16, 2025
Collaborator
StijnCaerts
left a comment
StijnCaerts
left a comment
There was a problem hiding this comment.
Nice work, I think this will be a useful fix for large catalogs, especially now that people are looking for ways to limit searches to certain collections based on authorizations (#409).
StijnCaerts
previously approved these changes
Jul 17, 2025
jonhealy1
requested changes
Jul 17, 2025
stac_fastapi/opensearch/stac_fastapi/opensearch/database_logic.py
Outdated
Show resolved
Hide resolved
…ort for large amount of queries to ElasticSearch database logic.
Contributor
Author
|
Hi @jonhealy1! Thank you for merging and closing the PR! Do you have an estimated timeline for releasing this change? We'd benefit from having it in our production system soon, as the number of indices is growing quite large. |
Collaborator
|
Hi @simonvb00 Done https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/releases/tag/v6.1.0 Thanks. Reach out anytime |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
When searching the catalog with the
/search-endpoint, aGET /<indices>/_searchrequest is done with all indices listed in the URL path. However when doing such a search on a large amount of indices, it is possible that the size of the endpoint exceeds Elasticsearch’s maximum allowed HTTP line length (4096 bytes), resulting in the following error:{"code":"RequestError","description":"RequestError(400, 'too_long_http_line_exception', 'An HTTP line is larger than 4096 bytes.')"}The solution in this commit moves the indices from the endpoint to the body of the request once the amount of indices passes a certain threshold. The indices of the endpoint will be replaced by
ITEM_INDICES. Since the query still filters on the correct indices, this change preserves the behavior while avoiding the URL length limitation.PR Checklist:
pre-commit run --all-files)make test)