shard request cache

Author: shainaraskas

docs/reference/commands/node-tool.asciidoc

@@ -309,7 +309,7 @@ Unsafe cluster bootstrapping is only possible if there is at least one
 surviving master-eligible node. If there are no remaining master-eligible nodes
 then the cluster metadata is completely lost. However, the individual data
 nodes also contain a copy of the index metadata corresponding with their shards. This sometimes allows a new cluster to import these shards as
-<<dangling-indices-api,dangling indicesi>>. You can sometimes
+<<dangling-indices,dangling indices>>. You can sometimes
 recover some indices after the loss of all main-eligible nodes in a cluster
 by creating a new cluster and then using the `elasticsearch-node
 detach-cluster` command to move any surviving nodes into this new cluster. Once the new cluster is fully formed,

docs/reference/data-store-architecture.asciidoc

@@ -9,10 +9,14 @@ from any node.
 The topics in this section provides information about the architecture of {es} and how it stores and retrieves data: 
 
 * <<nodes-shards,Nodes and shards>>: Learn about the basic building blocks of an {es} cluster, including nodes, shards, primaries, and replicas.
+* <<node-roles-overview,Node roles>>: Learn about the different roles that nodes can have in an {es} cluster.
 * <<docs-replication,Reading and writing documents>>: Learn how {es} replicates read and write operations across shards and shard copies.
 * <<shard-allocation-relocation-recovery,Shard allocation, relocation, and recovery>>: Learn how {es} allocates and balances shards across nodes.
+* <<shard-request-cache,Shard request cache>>: Learn how {es} caches search requests to improve performance.
 --
 
 include::nodes-shards.asciidoc[]
+include::node-roles.asciidoc[]
 include::docs/data-replication.asciidoc[leveloffset=-1]
-include::modules/shard-ops.asciidoc[]
+include::modules/shard-ops.asciidoc[]
+include::shard-request-cache.asciidoc[leveloffset=-1]

docs/reference/modules/discovery/publishing.asciidoc

@@ -1,5 +1,23 @@
+[[cluster-state-overview]]
+=== Cluster state
+
+The _cluster state_ is an internal data structure which keeps track of a
+variety of information needed by every node, including:
+
+* The identity and attributes of the other nodes in the cluster
+
+* Cluster-wide settings
+
+* Index metadata, including the mapping and settings for each index
+
+* The location and status of every shard copy in the cluster
+
+The elected master node ensures that every node in the cluster has a copy of
+the same cluster state. The <<cluster-state,cluster state API>> lets you retrieve a
+representation of this internal state for debugging or diagnostic purposes.
+
 [[cluster-state-publishing]]
-=== Publishing the cluster state
+==== Publishing the cluster state
 
 The elected master node is the only node in a cluster that can make changes to
 the cluster state. The elected master node processes one batch of cluster state
@@ -58,3 +76,16 @@ speed of the storage on each master-eligible node, as well as the reliability
 and latency of the network interconnections between all nodes in the cluster.
 You must therefore ensure that the storage and networking available to the
 nodes in your cluster are good enough to meet your performance goals.
+
+[[dangling-indices]]
+==== Dangling indices
+
+When a node joins the cluster, if it finds any shards stored in its local
+data directory that do not already exist in the cluster state, it will consider
+those shards to belong to a "dangling" index. You can list, import or
+delete dangling indices using the <<dangling-indices-api,Dangling indices
+API>>.
+
+NOTE: The API cannot offer any guarantees as to whether the imported data
+truly represents the latest state of the data when the index was still part
+of the cluster.

docs/reference/modules/indices/request_cache.asciidoc

@@ -1,4 +1,4 @@
-[[shard-request-cache]]
+[[shard-request-cache-settings]]
 === Shard request cache settings
 
 When a search request is run against an index or against many indices, each
@@ -10,139 +10,16 @@ The shard-level request cache module caches the local results on each shard.
 This allows frequently used (and potentially heavy) search requests to return
 results almost instantly. The requests cache is a very good fit for the logging
 use case, where only the most recent index is being actively updated --
-results from older indices will be served directly from the cache.
+results from older indices will be served directly from the cache. You can use shard request cache settings to control the size and expiration of the cache.
 
-[IMPORTANT]
-===================================
-
-By default, the requests cache will only cache the results of search requests
-where `size=0`, so it will not cache `hits`,
-but it will cache `hits.total`,  <<search-aggregations,aggregations>>, and
-<<search-suggesters,suggestions>>.
-
-Most queries that use `now` (see <<date-math>>) cannot be cached.
-
-Scripted queries that use the API calls which are non-deterministic, such as
-`Math.random()` or `new Date()` are not cached.
-===================================
-
-[discrete]
-==== Cache invalidation
-
-The cache is smart -- it keeps the same _near real-time_ promise as uncached
-search.
-
-Cached results are invalidated automatically whenever the shard refreshes to
-pick up changes to the documents or when you update the mapping. In other
-words you will always get the same results from the cache as you would for an
-uncached search request.
-
-The longer the refresh interval, the longer that cached entries will remain
-valid even if there are changes to the documents. If the cache is full, the
-least recently used cache keys will be evicted.
-
-The cache can be expired manually with the <<indices-clearcache,`clear-cache` API>>:
-
-[source,console]
-------------------------
-POST /my-index-000001,my-index-000002/_cache/clear?request=true
-------------------------
-// TEST[s/^/PUT my-index-000001\nPUT my-index-000002\n/]
-
-[discrete]
-==== Enabling and disabling caching
-
-The cache is enabled by default, but can be disabled when creating a new
-index as follows:
-
-[source,console]
------------------------------
-PUT /my-index-000001
-{
-  "settings": {
-    "index.requests.cache.enable": false
-  }
-}
------------------------------
-
-It can also be enabled or disabled dynamically on an existing index with the
-<<indices-update-settings,`update-settings`>> API:
-
-[source,console]
------------------------------
-PUT /my-index-000001/_settings
-{ "index.requests.cache.enable": true }
------------------------------
-// TEST[continued]
-
-
-[discrete]
-==== Enabling and disabling caching per request
-
-The `request_cache` query-string parameter can be used to enable or disable
-caching on a *per-request* basis. If set, it overrides the index-level setting:
-
-[source,console]
------------------------------
-GET /my-index-000001/_search?request_cache=true
-{
-  "size": 0,
-  "aggs": {
-    "popular_colors": {
-      "terms": {
-        "field": "colors"
-      }
-    }
-  }
-}
------------------------------
-// TEST[continued]
-
-Requests where `size` is greater than 0 will not be cached even if the request cache is
-enabled in the index settings. To cache these requests you will need to use the
-query-string parameter detailed here.
-
-[discrete]
-==== Cache key
-
-A hash of the whole JSON body is used as the cache key. This means that if the JSON
-changes -- for instance if keys are output in a different order -- then the
-cache key will not be recognised.
-
-TIP: Most JSON libraries support a _canonical_ mode which ensures that JSON
-keys are always emitted in the same order. This canonical mode can be used in
-the application to ensure that a request is always serialized in the same way.
+To learn more about the shard request cache, see <<shard-request-cache>>.
 
 [discrete]
 ==== Cache settings
 
-The cache is managed at the node level, and has a default maximum size of `1%`
-of the heap. This can be changed in the `config/elasticsearch.yml` file with:
-
-[source,yaml]
---------------------------------
-indices.requests.cache.size: 2%
---------------------------------
-
-Also, you can use the +indices.requests.cache.expire+ setting to specify a TTL
-for cached results, but there should be no reason to do so. Remember that
-stale results are automatically invalidated when the index is refreshed. This
-setting is provided for completeness' sake only.
-
-[discrete]
-==== Monitoring cache usage
-
-The size of the cache (in bytes) and the number of evictions can be viewed
-by index, with the <<indices-stats,`indices-stats`>> API:
-
-[source,console]
-------------------------
-GET /_stats/request_cache?human
-------------------------
+indices.requests.cache.size::
+(<<static-cluster-setting,Static>>) The maximum size of the cache, as a percentage of the heap. Default: `1%`.
 
-or by node with the <<cluster-nodes-stats,`nodes-stats`>> API:
+indices.requests.cache.expire::
+(<<static-cluster-setting,Static>>) The TTL for cached results. Stale results are automatically invalidated when the index is refreshed, so you shouldn't need to use this setting.
 
-[source,console]
-------------------------
-GET /_nodes/stats/indices/request_cache?human
-------------------------