Skip to content
Closed
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
3a513a9
Copied over text from existing rewrite, changing to markdown
Mar 10, 2025
8859bb8
Update link
thekofimensah Mar 11, 2025
784816c
Add deploy-manage security landing page (#703)
leemthompo Mar 10, 2025
4e63214
updating regions (#707)
georgewallace Mar 10, 2025
430618e
apm-server: Remove server information endpoint `POST /` support (#647)
carsonip Mar 10, 2025
03428ae
typo: samling->sampling (#717)
carsonip Mar 10, 2025
6565df7
[D&M] Security basic structure (#706)
florent-leborgne Mar 10, 2025
03d78c5
[Docs migration] Cleans up detection and alerts files (#566)
nastasha-solomon Mar 10, 2025
f459c09
Routing optimization for logsdb is gated by licensing (#714)
kkrik-es Mar 10, 2025
3ef351c
Updates links pointing to cloud-on-k8s (#719)
szabosteve Mar 10, 2025
96a7eb2
Update steps to configure dashboard for Security: Host module (#720)
szabosteve Mar 10, 2025
0efe399
change the supported stack versions for ECK (#709)
kvalliyurnatt Mar 10, 2025
8940500
[obs] Fixes and versioning (#712)
karenzone Mar 10, 2025
c8e4bdb
[Upgrade 9.0] Adds Kibana and Elasticsearch upgrade instructions (#691)
jmikell821 Mar 11, 2025
d13c54c
[D&M] Improve deploy-manage/security landing page (#721)
leemthompo Mar 11, 2025
40905ce
Some cleanup for Manage data -> Ingest from apps (#722)
kilfoyle Mar 11, 2025
39d60d3
[D&M] Secure Elastic Cloud organization (#723)
florent-leborgne Mar 11, 2025
34d8711
Updates Upgrade ECK page (#702)
kosabogi Mar 11, 2025
40e0bbe
[Observability metrics reference] Remove duplicated file (#669)
alaudazzi Mar 11, 2025
82822e0
[D&M] Orchestrator security (#727)
florent-leborgne Mar 11, 2025
1a67fe1
Add docs for enrollment handling for containerized agents (#729)
kilfoyle Mar 11, 2025
2cec964
[D&M] Move MFA mention to cloud page only (#730)
florent-leborgne Mar 11, 2025
d21af7b
Increase maximum Osquery timeout (#726)
natasha-moore-elastic Mar 11, 2025
5930f41
[integration-docs] Update integration docs links (#731)
colleenmcginnis Mar 11, 2025
f49b894
Fixes incorrect integration reference (#708)
benironside Mar 11, 2025
06df300
Cleans up ai for security section (#737)
benironside Mar 11, 2025
25b8875
ECE: map Universal Profiling domains to proxy certificates (#659)
inge4pres Mar 11, 2025
560b5f3
[D&M] Move SELinux ECE page to security section (#732)
florent-leborgne Mar 11, 2025
8de643a
Removes outdated refs to an Assistant tab in Timeline (#739)
benironside Mar 11, 2025
2952d31
Adds ECE upgrade guide (#715)
kosabogi Mar 12, 2025
7cda000
Audit logs moved to security (#743)
eedugon Mar 12, 2025
a1fc466
[D&M] Explains pricing on Serverless for adaptive resources (#745)
szabosteve Mar 12, 2025
d97000d
[9.1] Update templates.md with fleet-synced-integrations (#555)
juliaElastic Mar 12, 2025
1625f1f
Changing links for Plugin management (#734)
charlotte-hoblik Mar 12, 2025
ecc237d
[Inference] Alibaba Cloud Inference Service adds support for deepseek…
szabosteve Mar 12, 2025
7bc1ad6
Update Plugin Management link (#752)
charlotte-hoblik Mar 12, 2025
54d8ec4
Remove unused links from docset.yml and fix links (#753)
bmorelli25 Mar 12, 2025
d4cd15d
[E&A][bugbash] Fixes an admonition in HuggingFace inference integrati…
szabosteve Mar 12, 2025
ec73797
Updates links pointing to apm-agent-rum-js (#746)
szabosteve Mar 12, 2025
25b536c
Remove OpenAI inference API (#740)
lcawl Mar 12, 2025
005bfde
Fixes automatic import and AI Assistant pages (#756)
benironside Mar 12, 2025
70a3ead
Resize icons in Security docs (#754)
natasha-moore-elastic Mar 12, 2025
849adc3
Made intro more concise, and the rest more scannable
thekofimensah Mar 19, 2025
737ed66
Fixed asciidoc styling to md
Mar 19, 2025
c189e08
Update deploy-manage/production-guidance/optimize-performance/size-sh…
thekofimensah Mar 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,25 @@

# Size your shards [size-your-shards]

Each index in {{es}} is divided into one or more shards, each of which may be replicated across multiple nodes to protect against hardware failures. If you are using [Data streams](../../../manage-data/data-store/data-streams.md) then each data stream is backed by a sequence of indices. There is a limit to the amount of data you can store on a single node so you can increase the capacity of your cluster by adding nodes and increasing the number of indices and shards to match. However, each index and shard has some overhead and if you divide your data across too many shards then the overhead can become overwhelming. A cluster with too many indices or shards is said to suffer from *oversharding*. An oversharded cluster will be less efficient at responding to searches and in extreme cases it may even become unstable.
## What is a shard? [what-is-a-shard]

A shard is a basic unit of storage in {{es}}. Every index is divided into one or more shards to help distribute data and workload across nodes in a cluster. This division allows {{es}} to handle large datasets and perform operations like searches and indexing efficiently but not without cost. Each index and shard has some overhead and if you divide your data across too many shards then the overhead will degrade performance. Shards play several key roles in {{es}}:

* **Data Distribution:** Each shard contains a portion of the data from the index. When you add more nodes to your cluster, {{es}} will spread the shards across the nodes, balancing the workload between them.
* **Replication:** Shards can have replicas which are copies of the original shard. Replicas ensure data availability and improve search performance by allowing multiple nodes to handle requests for that shard.
* **Parallel Processing:** Shards enable {{es}} to distribute indexing of documents, and process queries in parallel across shards, making ingestion and searches faster and more efficient.

By effectively using shards, {{es}} can scale horizontally and provide fault tolerance, ensuring your data is distributed and indexing and searches are processed efficiently.

## Sizing Shard Guidelines [sizing-shard-guidelines]

Proper shard sizing is crucial for maintaining the performance and stability of an {{es}} cluster. _Oversharding_ occurs when data is distributed across an excessive number of shards (primary or replica), which can degrade search performance and make the cluster unstable. Conversely, very large shards may slow down search operations and prolong recovery times after failures.

To strike the right balance, the [general guidelines](#shard-size-recommendation) are to aim for shard sizes between 10GB and 50GB, keeping the per-shard document count below 200 million. To ensure that each node is working optimally, it's important to distribute shards evenly across nodes. Uneven distribution can cause some nodes to work harder than others, leading to performance degradation and instability. While Elasticsearch automatically balances shards, it’s important to configure your indices with an appropriate number of shards and replicas to facilitate even distribution across nodes.

If you are using [data streams](../../../manage-data/data-store/data-streams.md#data-streams), each data stream is backed by a sequence of indices, each index potentially having multiple shards.

Check failure on line 24 in deploy-manage/production-guidance/optimize-performance/size-shards.md

View workflow job for this annotation

GitHub Actions / preview / build

`data-streams` does not exist in data-streams.md.

Despite these general guidelines, it is good to develop a tailored [sharding strategy](#create-a-sharding-strategy) that considers your specific infrastructure, use case, and performance expectations.


## Create a sharding strategy [create-a-sharding-strategy]
Expand Down