Added Basic Client side Metrics

CHANGELOG.md

@@ -3,10 +3,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ## [Unreleased 3.x]
 ### Added
-
+- Metrics support includes micrometer integration and Prometheus support with a custom client-side metrics. [Metrics](./guides/metrics.md)
 ### Dependencies
 - Bump `org.apache.httpcomponents.client5:httpclient5` from 5.4.4 to 5.5 ([#1578](https://github.com/opensearch-project/opensearch-java/pull/1578))
 - Bump `org.junit:junit-bom` from 5.12.2 to 5.13.0 ([#1587](https://github.com/opensearch-project/opensearch-java/pull/1587))
+- Added micrometer dependency `io.micrometer:micrometer-core` version 1.13.13
 
 ### Changed
 

USER_GUIDE.md

@@ -200,6 +200,7 @@ You can find a working sample of the above code in [IndexingBasics.java](./sampl
 - [Search](./guides/search.md)
 - [Generic Client](./guides/generic.md)
 - [Json](./guides/json.md)
+- [Metrics](./guides/metrics.md)
 
 ## Plugins
 

guides/metrics.md

@@ -0,0 +1,75 @@
+- [SDK Metrics](#SDK-Metrics)
+    - [How to enable metrics](#get-client)
+    - [Metrics Collected](#metrics-collected)
+
+# SDK Metrics
+
+We've integrated a robust metrics solution into the OpenSearch Java client to provide comprehensive insights into its API usage and performance. This includes the collection of vital operational metrics, such as overall throughput, request latency, and error rate. Furthermore, we're capturing more granular details like request and response payload sizes, distinct success and failure rates for operations, and real-time OpenSearch node status to provide a holistic view of client behavior and cluster health.
+
+## How to enable metrics
+
+We should create a MetricOptions instance and set it in the ApacheHttpClient5TransportBuilder when creating the client. Take a look at below code snippet for an example of how to create a client with metrics enabled:
+
+```java
+public class CreateClient {
+  public static OpenSearchClient createClientWithMetrics() throws NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
+    var env = System.getenv();
+    var https = Boolean.parseBoolean(env.getOrDefault("HTTPS", "true"));
+    var hostname = env.getOrDefault("HOST", "localhost");
+    var port = Integer.parseInt(env.getOrDefault("PORT", "9200"));
+    var user = env.getOrDefault("USERNAME", "admin");
+    var pass = env.getOrDefault("PASSWORD", "admin");
+    var metricEnabled = true;
+    double PERCENTILE_99 = 0.99;
+    double PERCENTILE_95 = 0.95;
+    var meterRegistry =  new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
+
+    final var hosts = new HttpHost[]{new HttpHost(https ? "https" : "http", hostname, port)};
+
+    final var sslContext = SSLContextBuilder.create().loadTrustMaterial(null, (chains, authType) -> true).build();
+
+    MetricOptions metricOptions = MetricOptions.builder()
+            .setMetricsEnabled(metricEnabled) // required to turn metrics on/off
+            .setMeterRegistry(meterRegistry) 
+            .setPercentiles(PERCENTILE_99, PERCENTILE_95)
+            .setAdditionalMetricGroups(MetricGroup.NETWORK_DETAILS)
+            .setExcludedTags(MetricTag.HOST_CONTACTED)
+            .build();
+
+    final var transport = ApacheHttpClient5TransportBuilder.builder(hosts)
+            .setMapper(new JacksonJsonpMapper())
+            .setMetricOptions(metricOptions)
+            .setHttpClientConfigCallback(httpClientBuilder -> {
+              final var credentialsProvider = new BasicCredentialsProvider();
+              for (final var host : hosts) {
+                credentialsProvider.setCredentials(new AuthScope(host), new UsernamePasswordCredentials(user, pass.toCharArray()));
+              }
+
+// Disable SSL/TLS verification as our local testing clusters use self-signed certificates
+              final var tlsStrategy = ClientTlsStrategyBuilder.create()
+                      .setSslContext(sslContext)
+                      .setHostnameVerifier(NoopHostnameVerifier.INSTANCE)
+                      .build();
+
+              final var connectionManager = PoolingAsyncClientConnectionManagerBuilder.create().setTlsStrategy(tlsStrategy).build();
+
+              return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider).setConnectionManager(connectionManager);
+            })
+            .build();
+    return new OpenSearchClient(transport);
+  }
+}
+```
+
+## Metrics Collected
+
+The OpenSearch Java client collects a variety of metrics to provide insights into its operations. Below is a summary of the key metrics collected, along with their descriptions and important dimensions, such as status code and request type, that can be used for filtering and analysis.
+
+| Metric                                | Description                                                     | Important Dimensions(tags)     |
+|---------------------------------------|-----------------------------------------------------------------|--------------------------------|
+| os.client.request.seconds             | End-to-end request execution latency                            | StatusCodeOrException, Request |
+| os.client.request.seconds.count       | request throughput and error rate based on status tags          | StatusCodeOrException, Request |
+| os.client.request.payload.size.bytes  | Request payload size in bytes                                   | Request                        |
+| os.client.response.payload.size.bytes | Response payload size in bytes                                  | Request                        |
+| os.client.active.nodes                | Number of OpenSearch active nodes from a client's perspective   |                                |
+| os.client.inactive.nodes              | Number of OpenSearch inactive nodes from a client's perspective |                                |

java-client/build.gradle.kts

@@ -246,6 +246,12 @@ dependencies {
     testImplementation("junit", "junit" , "4.13.2") {
         exclude(group = "org.hamcrest")
     }
+
+    // Micrometer
+    implementation("io.micrometer:micrometer-core:1.13.13")
+
+    // Awaitility
+    testImplementation("org.awaitility:awaitility:4.2.0")
 }
 
 licenseReport {

java-client/src/main/java/org/opensearch/client/transport/client_metrics/ExecutionMetricContext.java

@@ -0,0 +1,54 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.client.transport.client_metrics;
+
+import java.time.Duration;
+
+/**
+ * Contains necessary information about a request execution to be used for metric recordings.
+ */
+public abstract class ExecutionMetricContext {
+
+    public static final int DEFAULT_EMPTY_STATUS_CODE = -1;
+    private Throwable throwable = null;
+    private int statusCode = DEFAULT_EMPTY_STATUS_CODE;
+    private Duration executionTime = null;
+
+    protected ExecutionMetricContext() {}
+
+    protected ExecutionMetricContext(Throwable throwable, int statusCode, Duration executionTime) {
+        this.throwable = throwable;
+        this.statusCode = statusCode;
+        this.executionTime = executionTime;
+    }
+
+    public Throwable getThrowable() {
+        return throwable;
+    }
+
+    public int getStatusCode() {
+        return statusCode;
+    }
+
+    public Duration getRequestExecutionTime() {
+        return executionTime;
+    }
+
+    public void setThrowable(Throwable throwable) {
+        this.throwable = throwable;
+    }
+
+    public void setStatusCode(int statusCode) {
+        this.statusCode = statusCode;
+    }
+
+    public void setRequestExecutionTime(Duration executionTime) {
+        this.executionTime = executionTime;
+    }
+}

java-client/src/main/java/org/opensearch/client/transport/client_metrics/MeterOptions.java

@@ -0,0 +1,55 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.client.transport.client_metrics;
+
+import static org.opensearch.client.transport.client_metrics.MetricConstants.DEFAULT_EXCLUDED_TAGS;
+import static org.opensearch.client.transport.client_metrics.MetricConstants.DEFAULT_PERCENTILES;
+
+import io.micrometer.core.instrument.Tags;
+import java.util.Set;
+
+/**
+ * Contains settings for configuring a meter
+ */
+public class MeterOptions {
+    private final double[] percentiles;
+    private final Tags commonTags;
+    private final Set<MetricTag> excludedTagNames;
+
+    public MeterOptions(double[] percentiles, Tags commonTags, Set<MetricTag> excludedTagNames) {
+        this.percentiles = percentiles == null ? DEFAULT_PERCENTILES.clone() : percentiles.clone();
+        this.commonTags = commonTags == null ? Tags.empty() : commonTags;
+        this.excludedTagNames = excludedTagNames == null ? DEFAULT_EXCLUDED_TAGS : excludedTagNames;
+    }
+
+    /**
+     * Get percentiles to publish for Timer/Distribution meters
+     * @return a double array
+     */
+    public double[] getPercentiles() {
+        return percentiles;
+    }
+
+    /**
+     * Get common {@link io.micrometer.core.instrument.Tags} that this meter need to have
+     * @return a {@link io.micrometer.core.instrument.Tags}
+     */
+    public Tags getCommonTags() {
+        return commonTags;
+    }
+
+    /**
+     * Get tag names that a meter are excluded.
+     * @return a set of {@link MetricTag}
+     */
+    public Set<MetricTag> getExcludedTagNames() {
+        return excludedTagNames;
+    }
+
+}

java-client/src/main/java/org/opensearch/client/transport/client_metrics/MetricConstants.java

@@ -0,0 +1,21 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.client.transport.client_metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import io.micrometer.core.instrument.Metrics;
+import java.util.EnumSet;
+import java.util.Set;
+
+public class MetricConstants {
+    public static final double[] DEFAULT_PERCENTILES = new double[] { 0.99, 0.95, 0.9, 0.75, 0.5 };
+    public static final MeterRegistry DEFAULT_REGISTRY = Metrics.globalRegistry;
+    public static final Set<MetricTag> DEFAULT_EXCLUDED_TAGS = EnumSet.noneOf(MetricTag.class);
+    public static final Set<MetricGroup> DEFAULT_ADDITIONAL_METRIC_GROUPS = EnumSet.noneOf(MetricGroup.class);
+}

java-client/src/main/java/org/opensearch/client/transport/client_metrics/MetricGroup.java

@@ -0,0 +1,20 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.client.transport.client_metrics;
+
+import java.util.EnumSet;
+import java.util.Set;
+
+public enum MetricGroup {
+    GENERAL,
+    NETWORK_DETAILS;
+
+    public static final Set<MetricGroup> REQUIRED_GROUPS = EnumSet.of(GENERAL);
+    public static final Set<MetricGroup> ALL = EnumSet.allOf(MetricGroup.class);
+}

java-client/src/main/java/org/opensearch/client/transport/client_metrics/MetricName.java

@@ -0,0 +1,30 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.client.transport.client_metrics;
+
+public enum MetricName {
+    REQUEST("request"),
+    NETWORK_REQUEST("network.request"),
+    ACTIVE_NODES("active.nodes"),
+    INACTIVE_NODES("inactive.nodes"),
+    REQUEST_PAYLOAD_SIZE("request.payload.size"),
+    RESPONSE_PAYLOAD_SIZE("response.payload.size");
+
+    private static final String PREFIX = "os.client";
+    private final String metricName;
+
+    MetricName(String metricName) {
+        this.metricName = metricName;
+    }
+
+    @Override
+    public String toString() {
+        return PREFIX + "." + metricName;
+    }
+}