Update Kafka JMX Script (#216)

* Enhance kafka

* Appease spotless

* Remove another test from integration for now

* Remove another metric not available in integration test

* Fix documentation

* Fix expected type of election metrics

* Fix issues with integration tests

* Update decription of metric in expected

* Resolve PR feedback

* Fix accidental commit of local files

* Address PR feedback

* Update the way kafka producer container is loaded for multiple tests

* Update docs to reflect removal of topic attributes

Author: Sam DeHaan

jmx-metrics/docs/target-systems/kafka.md

@@ -7,116 +7,104 @@ These metrics are sourced from Kafka's exposed Yammer metrics for each instance:
 
 ### Broker Metrics
 
-* Name: `kafka.messages.in`
-* Description: Number of messages in per second
-* Unit: `1`
-* Instrument Type: LongValueObserver
+* Name: `kafka.message.count`
+* Description: The number of messages received by the broker
+* Unit: `{messages}`
+* Instrument Type: LongCounterObserver
+
+* Name: `kafka.request.count`
+* Description: The number of requests received by the broker
+* Unit: `{requests}`
+* Attributes: `type`
+* Instrument Type: LongCounterObserver
+
+* Name: `kafka.request.failed`
+* Description: The number of requests to the broker resulting in a failure
+* Unit: `{requests}`
+* Attributes: `type`
+* Instrument Type: LongCounterObserver
+
+* Name: `kafka.request.time.total`
+* Description: The total time the broker has taken to service requests
+* Unit: `ms`
+* Attributes: `type`
+* Instrument Type: LongCounterObserver
+
+* Name: `kafka.request.time.50p`
+* Description: The 50th percentile time the broker has taken to service requests
+* Unit: `ms`
+* Attributes: `type`
+* Instrument Type: DoubleValueObserver
 
-* Name: `kafka.bytes.in`
-* Description: Bytes in per second from clients
+* Name: `kafka.request.time.99p`
+* Description: The 99th percentile time the broker has taken to service requests
+* Unit: `ms`
+* Attributes: `type`
+* Instrument Type: DoubleValueObserver
+
+* Name: `kafka.network.io`
+* Description: The bytes received or sent by the broker
 * Unit: `by`
+* Attributes: `state`
+* Instrument Type: LongCounterObserver
+
+* Name: `kafka.purgatory.size`
+* Description: The number of requests waiting in purgatory
+* Unit: `{requests}`
+* Attributes: `type`
 * Instrument Type: LongValueObserver
 
-* Name: `kafka.bytes.out`
-* Description: Bytes out per second to clients
-* Unit: `by`
+* Name: `kafka.partition.count`
+* Description: The number of partitions on the broker
+* Unit: `{partitions}`
 * Instrument Type: LongValueObserver
 
-* Name: `kafka.isr.shrinks`
-* Description: In-sync replica shrinks per second
-* Unit: `1`
+* Name: `kafka.partition.offline`
+* Description: The number of partitions offline
+* Unit: `{partitions}`
 * Instrument Type: LongValueObserver
 
-* Name: `kafka.isr.expands`
-* Description: In-sync replica expands per second
-* Unit: `1`
+* Name: `kafka.partition.under_replicated`
+* Description: The number of under replicated partitions
+* Unit: `{partitions}`
 * Instrument Type: LongValueObserver
 
+* Name: `kafka.isr.operation.count`
+* Description: The number of in-sync replica shrink and expand operations
+* Unit: `{operations}`
+* Attributes: `operation`
+* Instrument Type: LongCounterObserver
+
 * Name: `kafka.max.lag`
 * Description: Max lag in messages between follower and leader replicas
-* Unit: `1`
+* Unit: `{messages}`
 * Instrument Type: LongValueObserver
 
 * Name: `kafka.controller.active.count`
 * Description: Controller is active on broker
-* Unit: `1`
-* Instrument Type: LongValueObserver
-
-* Name: `kafka.partitions.offline.count`
-* Description: Number of partitions without an active leader
-* Unit: `1`
-* Instrument Type: LongValueObserver
-
-* Name: `kafka.partitions.underreplicated.count`
-* Description: Number of under replicated partitions
-* Unit: `1`
+* Unit: `{controllers}`
 * Instrument Type: LongValueObserver
 
 * Name: `kafka.leader.election.rate`
-* Description: Leader election rate - non-zero indicates broker failures
-* Unit: `1`
-* Instrument Type: LongValueObserver
+* Description: Leader election rate - increasing indicates broker failures
+* Unit: `{elections}`
+* Instrument Type: LongCounterObserver
 
 * Name: `kafka.unclean.election.rate`
-* Description: Unclean leader election rate - non-zero indicates broker failures
-* Unit: `1`
-* Instrument Type: LongValueObserver
+* Description: Unclean leader election rate - increasing indicates broker failures
+* Unit: `{elections}`
+* Instrument Type: LongCounterObserver
 
 * Name: `kafka.request.queue`
 * Description: Size of the request queue
-* Unit: `1`
+* Unit: `{requests}`
 * Instrument Type: LongValueObserver
 
-* Name: `kafka.fetch.consumer.total.time.count`
-* Description: Fetch consumer request count
-* Unit: `1`
-* Instrument Type: LongSumObserver
-
-* Name: `kafka.fetch.consumer.total.time.median`
-* Description: Fetch consumer request time - 50th percentile
-* Unit: `ms`
-* Instrument Type: DoubleValueObserver
-
-* Name: `kafka.fetch.consumer.total.time.99p`
-* Description: Fetch consumer request time - 99th percentile
-* Unit: `ms`
-* Instrument Type: DoubleValueObserver
-
-* Name: `kafka.fetch.follower.total.time.count`
-* Description: Fetch follower request count
-* Unit: `1`
-* Instrument Type: LongSumObserver
-
-* Name: `kafka.fetch.follower.total.time.median`
-* Description: Fetch follower request time - 50th percentile
-* Unit: `ms`
-* Instrument Type: DoubleValueObserver
-
-* Name: `kafka.fetch.follower.total.time.99p`
-* Description: Fetch follower request time - 99th percentile
-* Unit: `ms`
-* Instrument Type: DoubleValueObserver
-
-* Name: `kafka.produce.total.time.count`
-* Description: Produce request count
-* Unit: `1`
-* Instrument Type: LongSumObserver
-
-* Name: `kafka.produce.total.time.median`
-* Description: Produce request time - 50th percentile
-* Unit: `ms`
-* Instrument Type: DoubleValueObserver
-
-* Name: `kafka.produce.total.time.99p`
-* Description: Produce request time - 99th percentile
-* Unit: `ms`
-* Instrument Type: DoubleValueObserver
-
 ### Log metrics
 
 * Name: `kafka.logs.flush.time.count`
 * Description: Log flush count
-* Unit: `1`
+* Unit: `ms`
 * Instrument Type: LongSumObserver
 
 * Name: `kafka.logs.flush.time.median`

jmx-metrics/src/integrationTest/java/io/opentelemetry/contrib/jmxmetrics/AbstractIntegrationTest.java

@@ -125,8 +125,7 @@ protected static GenericContainer<?> cassandraContainer() {
         .waitingFor(Wait.forLogMessage(".*Startup complete.*", 1));
   }
 
-  @SafeVarargs
-  protected final void waitAndAssertMetrics(Consumer<Metric>... assertions) {
+  protected final void waitAndAssertMetrics(Iterable<Consumer<Metric>> assertions) {
     await()
         .atMost(Duration.ofSeconds(30))
         .untilAsserted(
@@ -158,6 +157,12 @@ protected final void waitAndAssertMetrics(Consumer<Metric>... assertions) {
             });
   }
 
+  @SafeVarargs
+  @SuppressWarnings("varargs")
+  protected final void waitAndAssertMetrics(Consumer<Metric>... assertions) {
+    waitAndAssertMetrics(Arrays.asList(assertions));
+  }
+
   protected void assertGauge(Metric metric, String name, String description, String unit) {
     assertThat(metric.getName()).isEqualTo(name);
     assertThat(metric.getDescription()).isEqualTo(description);