Skip to content

Zoekt indexer trigram and file size limits #1436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 19, 2025
Merged

Zoekt indexer trigram and file size limits #1436

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6e85fac
Update search indexing exclusion criteria
natehessler Nov 19, 2025
310917e
Update search indexing exclusion criteria
natehessler Nov 19, 2025
64d24ad
Fix link to search.largeFiles setting in documentation
natehessler Nov 19, 2025
7737198
Remove markdown link formatting in search.mdx
natehessler Nov 19, 2025
b43ab58
Add link to search.largeFiles setting in documentation
natehessler Nov 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions docs/admin/search.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,18 @@ will not return any result.

## Indexed search

Sourcegraph indexes the code on the default branch of each repository. This speeds up searches that hit many repositories at once. Not all files in a repository branch are indexed, we skip files that are [larger than 1 MB](#maximum-file-size) and binary files. To view which files are skipped during indexing, visit the repository settings page and click on indexing.
Sourcegraph indexes the code on the default branch of each repository. This speeds up searches that hit many repositories at once. Not all files in a repository branch are indexed. We skip:

For large deployments we recommend horizontally scaling indexed search. You can do this by [adjusting the number of replicas](https://github.com/sourcegraph/deploy-sourcegraph/blob/master/docs/configure#configure-indexed-search-replica-count). Sourcegraph shards repository indexes across replicas. When the replica count changes Sourcegraph will slowly rebalance indexes to ensure availability of existing indexes.
- Files that are [larger than 1 MB](#maximum-file-size).
- Binary files.
- Files exceeding 20,000 unique trigrams (sequences of three characters).
- Files that are not valid UTF-8.

To view which files are skipped during indexing, visit the repository settings page and click on **Indexing**.

To force the indexer to include specific files (like `yarn.lock` or other large text files) that are otherwise skipped, add their file path or a glob pattern to the [search.largeFiles](https://sourcegraph.com/docs/admin/search#maximum-file-size) setting in your site configuration and reindex the repository. Note that files must still be valid UTF-8 to be indexed, even if added to `search.largeFiles`.

For large deployments we recommend horizontally scaling indexed search. You can do this by adjusting the [number of replicas](https://sourcegraph.com/docs/admin/deploy/kubernetes/configure). Sourcegraph shards repository indexes across replicas. When the replica count changes Sourcegraph will slowly rebalance indexes to ensure availability of existing indexes.

The resource requirements for indexed search vary considerably based on the text contents of your repositories, but a good estimate is that the node should have enough memory to hold the entire text contents of the default branch of each repository.

Expand Down