Change Elasticsearch timeouts to 429 response instead of 5xx (#116026)

When a user passes a timeout to an Elasticsearch request, the user is
telling the system to limit how long the request should take. If that
timeout is breached, it is because the user selected a timeout less than
the time the request needed. Therefore, it is up to the user to select a
long enough time. If they don't, it's on the user to adjust their
request.

Given the above, a breached timeout is an issue with the user request,
and the http response code should refelect that. This commit changes
timeout exceptions to give a 429 response code instead of the current
5xx.

Author: rjernst

docs/changelog/116026.yaml

@@ -0,0 +1,13 @@
+pr: 116026
+summary: Change Elasticsearch timeouts to 429 response instead of 5xx
+area: Infra/Core
+type: breaking
+issues: []
+breaking:
+  title: Change most Elasticsearch timeouts to 429 response instead of 5xx
+  area: REST API
+  details: When a timeout occurs in most REST requests, whether via a per-request timeout, or a system default, the
+    request would return a 5xx response code. The response code from those APIs when a timeout occurs is now 429.
+  impact: Adjust any code relying on retrying on 5xx responses for timeouts to look for a 429 response code and
+    inspect the response to determine whether a timeout occured.
+  notable: false

server/src/main/java/org/elasticsearch/ElasticsearchTimeoutException.java

@@ -10,6 +10,7 @@
 package org.elasticsearch;
 
 import org.elasticsearch.common.io.stream.StreamInput;
+import org.elasticsearch.rest.RestStatus;
 
 import java.io.IOException;
 
@@ -34,4 +35,10 @@ public ElasticsearchTimeoutException(String message, Object... args) {
     public ElasticsearchTimeoutException(String message, Throwable cause, Object... args) {
         super(message, cause, args);
     }
+
+    @Override
+    public RestStatus status() {
+        // closest thing to "your request took longer than you asked for"
+        return RestStatus.TOO_MANY_REQUESTS;
+    }
 }

server/src/main/java/org/elasticsearch/cluster/metadata/ProcessClusterEventTimeoutException.java

@@ -28,6 +28,6 @@ public ProcessClusterEventTimeoutException(StreamInput in) throws IOException {
 
     @Override
     public RestStatus status() {
-        return RestStatus.SERVICE_UNAVAILABLE;
+        return RestStatus.TOO_MANY_REQUESTS;
     }
 }

server/src/main/java/org/elasticsearch/search/query/SearchTimeoutException.java

@@ -31,7 +31,7 @@ public SearchTimeoutException(StreamInput in) throws IOException {
 
     @Override
     public RestStatus status() {
-        return RestStatus.GATEWAY_TIMEOUT;
+        return RestStatus.TOO_MANY_REQUESTS;
     }
 
     /**