Migrate to OpenTelemetry Java agent for automatic instrumentation (#590)

* Migrate to OpenTelemetry Java agent for automatic instrumentation

Replaces manual instrumentation with the OTel Java agent (v2.26.0).
Removes manual instrumentation libraries for AWS SDK, OkHttp, and Logback.
The agent automatically instruments these libraries without code changes.

Changes:
- Add OTel agent download and distribution to all deployment paths
- Upgrade OTel SDK from 1.45.0 to 1.60.1
- Remove manual instrumentation dependencies and code
- Simplify AWS client configuration (agent handles tracing)
- Remove Logback OTel appender (agent handles log export)

All deployment paths now use the agent: development, distribution, and containers.

* Remove manual OpenTelemetry instrumentation

The OpenTelemetry Java agent handles all instrumentation automatically.
Manual instrumentation is redundant and creates version conflicts.

Changes:
- Remove opentelemetry-sdk-extension-autoconfigure dependency (conflicts with agent)
- Remove all OpenTelemetry SDK dependencies from libs.versions.toml
- Delete entire observability/ package (TelemetryProvider, OtelTelemetryProvider, etc.)
- Remove all manual instrumentation calls (withSpan, recordDuration, incrementCounter)
- Remove telemetryProvider from DefaultRemoteOperationsService, CommandExecutor, K8s services
- Remove unused telemetryProvider parameter from AWSModule and AWSClientFactory
- Update all tests to remove TelemetryProvider mocks

The Java agent now handles all instrumentation: AWS SDK, HTTP clients,
JDBC/Cassandra driver, JVM metrics, and logging.

* Use version catalog for opentelemetry-javaagent dependency

Instead of hardcoding the version in build.gradle.kts, reference the
opentelemetry-agent version from libs.versions.toml to avoid drift.

* Update OpenTelemetry documentation for agent-only approach

Rewrites docs/reference/opentelemetry.md to reflect the removal of
manual instrumentation. Key changes:

- Clarifies that the agent is always loaded but only exports telemetry
  when OTEL_EXPORTER_OTLP_ENDPOINT is set (minimal overhead when unset)
- Removes "Instrumented Operations" section describing deleted custom spans
- Removes "Metrics" table listing deleted custom metrics
- Removes "Shutdown Warnings" troubleshooting entry for deleted shutdown hook
- Emphasizes "automatic instrumentation only - there is no custom manual
  instrumentation in the CLI tool code"

Updates CLAUDE.md to remove outdated reference to deleted observability/
package and replace with accurate description of Java agent-based
automatic instrumentation.

* Fix OTLP endpoint configuration in end-to-end tests

Configures the OpenTelemetry Java agent to use gRPC protocol when
connecting to the OTLP collector in docker compose. The agent was
defaulting to HTTP protocol and attempting to append paths to the
endpoint, causing "unexpected end of stream" errors.

Changes:
- Discover OTLP collector port from docker compose using `port` command
- Export OTEL_EXPORTER_OTLP_ENDPOINT with the discovered port mapping
- Export OTEL_EXPORTER_OTLP_PROTOCOL=grpc to force gRPC protocol

This ensures tests can successfully export telemetry to the local
collector without connection errors.

* Suppress logging output to prevent pollution of machine-readable stdout

Commands like `show-iam-policies` (sip) output JSON that's consumed by
shell scripts. kotlin-logging initialization messages and OpenTelemetry
agent logs were polluting stdout, causing AWS IAM to reject the output
as malformed JSON.

Changes:
- Make Main.kt logger lazy and suppress kotlin-logging startup message
- Route OTel agent logs through logback instead of stdout
- Use consistent agent path in bin/easy-db-lab (installDist location)

This ensures commands can safely redirect stdout for machine parsing.

* Suppress JVM Class Data Sharing warning

Add -Xshare:off to suppress "Sharing is only supported for boot loader
classes because bootstrap classpath has been appended" warning that
occurs when using the OpenTelemetry javaagent.

Author: rustyrazorblade

CLAUDE.md

@@ -285,7 +285,7 @@ All observability K8s resources are built programmatically using Fabric8 manifes
 
 **Grafana** (port 3000): Dashboards built via `GrafanaManifestBuilder`. Dashboard JSON in `configuration/grafana/dashboards/` with `__KEY__` template substitution. `GrafanaDashboard` enum is the single source of truth for dashboard metadata.
 
-**OTel Instrumentation in Kotlin**: The `observability/` package provides `TelemetryProvider` with `withSpan()`, `recordDuration()`, `incrementCounter()`. See `docs/reference/opentelemetry.md`.
+**CLI Tool Instrumentation**: The CLI tool uses the OpenTelemetry Java Agent for automatic instrumentation of AWS SDK calls, HTTP clients, JDBC operations, and JVM metrics. No manual instrumentation exists in the Kotlin code. See `docs/reference/opentelemetry.md`.
 
 **CLI commands**: `grafana update-config`, `logs query/backup/import/ls`, `metrics backup/import/ls`
 

bin/easy-db-lab

@@ -12,6 +12,7 @@ EASY_DB_LAB_USER_DATA=~/.easy-db-lab/
 VERSION=$(grep '^version' "${APP_HOME}/gradle.properties" | cut -d '=' -f2)
 
 JAR="${APP_HOME}/build/libs/easy-db-lab-${VERSION}-all.jar"
+OTEL_AGENT="${APP_HOME}/build/install/easy-db-lab/agents/opentelemetry-javaagent.jar"
 
 # Start OTel collector
 if [ "${EASY_DB_LAB_SKIP_DOCKER}" != "1" ]; then
@@ -26,4 +27,8 @@ OTEL_PORT=$(docker compose -f "${APP_HOME}/docker-compose.yml" port otel-collect
 export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:${OTEL_PORT}"
 export EASY_DB_LAB_REDIS_URL="redis://localhost:6399/easydblab-events"
 
-java -Deasydblab.version=$VERSION -Deasydblab.apphome=$APP_HOME -jar $JAR "$@"
+# Route OTel agent logs through application logger (logback) instead of stdout
+# This prevents agent messages from polluting machine-readable output (like JSON policies)
+export OTEL_JAVAAGENT_LOGGING=application
+
+java -Xshare:off -Deasydblab.version=$VERSION -Deasydblab.apphome=$APP_HOME -javaagent:"$OTEL_AGENT" -jar $JAR "$@"

bin/end-to-end-test

@@ -1457,6 +1457,13 @@ export EASY_DB_LAB_SKIP_BUILD=1
 
 # Start Docker services once upfront (Redis, OTel collector), then skip per-invocation restarts
 docker compose -f "$PROJECT_ROOT/docker-compose.yml" up -d
+
+# Discover the dynamically assigned OTLP port and export for test execution
+OTEL_PORT=$(docker compose -f "$PROJECT_ROOT/docker-compose.yml" port otel-collector 4317 2>/dev/null | cut -d: -f2)
+export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:${OTEL_PORT}"
+export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
+echo "OpenTelemetry OTLP endpoint: $OTEL_EXPORTER_OTLP_ENDPOINT (protocol: grpc)"
+
 export EASY_DB_LAB_SKIP_DOCKER=1
 
 # Change to work directory

build.gradle.kts

@@ -21,6 +21,23 @@ plugins {
     alias(libs.plugins.ktor)
 }
 
+// Configuration for OpenTelemetry Java agent
+val otelAgentConfig: Configuration by configurations.creating {
+    isCanBeConsumed = false
+    isCanBeResolved = true
+}
+
+val copyOtelAgent =
+    tasks.register<Copy>("copyOtelAgent") {
+        from(otelAgentConfig)
+        into("${project.layout.buildDirectory.get()}/otel-agent")
+        rename { "opentelemetry-javaagent.jar" }
+    }
+
+dependencies {
+    otelAgentConfig("io.opentelemetry.javaagent:opentelemetry-javaagent:${libs.versions.opentelemetry.agent.get()}")
+}
+
 group = "com.rustyrazorblade"
 
 tasks.withType<ShadowJar> {
@@ -36,6 +53,7 @@ application {
     mainClass.set("com.rustyrazorblade.easydblab.MainKt")
     applicationDefaultJvmArgs =
         listOf(
+            "-javaagent:\$APP_HOME/agents/opentelemetry-javaagent.jar",
             "-Deasydblab.ami.name=rustyrazorblade/images/easy-db-lab-cassandra-amd64-$version",
             "-Deasydblab.version=$version",
         )
@@ -64,14 +82,6 @@ tasks.named<CreateStartScripts>("startScripts") {
     }
 }
 
-// Force OpenTelemetry incubator versions before Gradle upgrades them
-configurations.all {
-    resolutionStrategy {
-        force("io.opentelemetry:opentelemetry-api-incubator:1.45.0-alpha")
-        force("io.opentelemetry:opentelemetry-sdk-extension-incubator:1.45.0-alpha")
-    }
-}
-
 // In this section you declare the dependencies for your production and test code
 dependencies {
     // Logging
@@ -137,9 +147,6 @@ dependencies {
     // Cassandra Driver
     implementation(libs.cassandra.driver.core)
 
-    // OpenTelemetry
-    implementation(libs.bundles.opentelemetry)
-
     // Testing
     testImplementation(libs.bundles.testing)
     testImplementation(libs.bundles.koin.test)
@@ -241,6 +248,10 @@ distributions {
             from("packer") {
                 into("packer")
             }
+            // Include the OTel agent in the distribution
+            from(copyOtelAgent) {
+                into("agents")
+            }
         }
     }
 }
@@ -251,7 +262,14 @@ tasks.distTar {
 }
 
 tasks.named("installDist") {
-    dependsOn(tasks.named("shadowJar"))
+    dependsOn(tasks.named("shadowJar"), copyOtelAgent)
+    doLast {
+        // Copy agent to installDist location
+        copy {
+            from("${project.layout.buildDirectory.get()}/otel-agent")
+            into(layout.buildDirectory.dir("install/easy-db-lab/agents"))
+        }
+    }
 }
 
 tasks.assemble {
@@ -330,6 +348,7 @@ jib {
         appRoot = "/app"
         jvmFlags =
             listOf(
+                "-javaagent:/agents/opentelemetry-javaagent.jar",
                 "-Deasydblab.ami.name=rustyrazorblade/images/easy-db-lab-cassandra-amd64-$version",
                 "-Deasydblab.version=$version",
                 "-Deasydblab.apphome=/app",
@@ -345,6 +364,10 @@ jib {
                     setFrom(file("packer"))
                     into = "/app/packer"
                 }
+                path {
+                    setFrom("${project.layout.buildDirectory.get()}/otel-agent")
+                    into = "/agents"
+                }
             }
         }
         creationTime = "USE_CURRENT_TIMESTAMP"
@@ -364,9 +387,11 @@ jib {
 
 // Jib doesn't fully support configuration cache yet
 tasks.named("jib") {
+    dependsOn(copyOtelAgent)
     notCompatibleWithConfigurationCache("Jib plugin doesn't fully support configuration cache yet")
 }
 
 tasks.named("jibDockerBuild") {
+    dependsOn(copyOtelAgent)
     notCompatibleWithConfigurationCache("Jib plugin doesn't fully support configuration cache yet")
 }

docs/reference/opentelemetry.md

@@ -2,7 +2,16 @@
 
 easy-db-lab includes optional OpenTelemetry (OTel) instrumentation for distributed tracing and metrics. When enabled, traces and metrics are exported to an OTLP-compatible collector.
 
-## Enabling OpenTelemetry
+## CLI Tool Instrumentation
+
+The easy-db-lab CLI tool runs with the OpenTelemetry Java Agent, which automatically instruments:
+
+- **AWS SDK calls** - EC2, S3, IAM, EMR, STS, OpenSearch operations
+- **HTTP clients** - OkHttp and other HTTP libraries
+- **JDBC/Cassandra driver** - Database operations
+- **JVM metrics** - Memory, threads, garbage collection
+
+### Enabling Instrumentation
 
 Set the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable to your OTLP collector endpoint:
 
@@ -13,52 +22,15 @@ easy-db-lab up
 
 When this environment variable is:
 - **Set**: Traces and metrics are exported via gRPC to the specified endpoint
-- **Not set**: OpenTelemetry is completely disabled with zero overhead
-
-## Instrumented Operations
-
-### Commands
-
-All CLI commands are automatically instrumented with:
-- A span for each command execution
-- Duration metrics
-- Command count metrics
-- Success/failure attributes
+- **Not set**: The agent is still loaded but no telemetry is exported (minimal overhead)
 
-### SSH Operations
+The agent uses automatic instrumentation only - there is no custom manual instrumentation in the CLI tool code.
 
-Remote SSH operations are instrumented including:
-- Command execution (`ssh.execute`)
-- File uploads (`ssh.upload`, `ssh.upload_directory`)
-- File downloads (`ssh.download`, `ssh.download_directory`)
+## Cluster Node Instrumentation
 
-Span attributes include host alias, target IP, and (for non-secret commands) the command text.
+The following instrumentation applies to cluster nodes (Cassandra, stress, Spark) and is separate from the CLI tool:
 
-### Kubernetes Operations
-
-K8s operations via the fabric8 client are instrumented:
-- Manifest application (`k8s.apply_manifests`, `k8s.apply_yaml`)
-- Namespace deletion (`k8s.delete_namespace`)
-- StatefulSet scaling (`k8s.scale_statefulset`)
-
-### AWS SDK Calls
-
-When OTel is enabled, AWS SDK calls are automatically instrumented using the OpenTelemetry AWS SDK instrumentation library. This includes:
-- EC2 operations
-- S3 operations
-- IAM operations
-- EMR operations
-- STS operations
-- OpenSearch operations
-
-## Resource Attributes
-
-Traces include the following resource attributes:
-- `service.name`: `easy-db-lab`
-- `service.version`: Application version
-- `host.name`: Local hostname
-
-## Node Role Labeling
+### Node Role Labeling
 
 The OTel Collector on cluster nodes uses the `k8sattributes` processor to read the K8s node label `type` and set it as the `node_role` resource attribute. This label is used by Grafana dashboards (e.g., System Overview) for hostname and service filtering.
 
@@ -73,20 +45,7 @@ The `k8sattributes` processor runs in the `metrics/local` and `logs/local` pipel
 
 The processor requires RBAC access to the K8s API. The OTel Collector DaemonSet runs with a dedicated ServiceAccount (`otel-collector`) that has read-only access to pods and nodes.
 
-## Metrics
-
-The following metrics are exported:
-
-| Metric | Type | Description |
-|--------|------|-------------|
-| `easydblab.command.duration` | Histogram | Command execution duration (ms) |
-| `easydblab.command.count` | Counter | Number of commands executed |
-| `easydblab.ssh.operation.duration` | Histogram | SSH operation duration (ms) |
-| `easydblab.ssh.operation.count` | Counter | Number of SSH operations |
-| `easydblab.k8s.operation.duration` | Histogram | K8s operation duration (ms) |
-| `easydblab.k8s.operation.count` | Counter | Number of K8s operations |
-
-## Stress Job Metrics
+### Stress Job Metrics
 
 When running cassandra-easy-stress as K8s Jobs, metrics are automatically collected via an OTel collector sidecar container. The sidecar scrapes the stress process's Prometheus endpoint (`localhost:9500`) and forwards metrics via OTLP to the node's OTel DaemonSet, which then exports them to VictoriaMetrics.
 
@@ -100,7 +59,7 @@ The Prometheus scrape job is named `cassandra-easy-stress`. The following labels
 
 Short-lived stress commands (`list`, `info`, `fields`) do not include the sidecar since they complete quickly and don't produce meaningful metrics.
 
-## Spark JVM Instrumentation
+### Spark JVM Instrumentation
 
 EMR Spark jobs are auto-instrumented with the OpenTelemetry Java Agent (v2.25.0) and Pyroscope Java Agent (v2.3.0), both installed via an EMR bootstrap action. The OTel agent is activated through `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions`.
 
@@ -115,7 +74,7 @@ Key configuration:
 - **Service name**: `spark-<job-name>` (set per job)
 - **Profiling**: CPU, allocation (512k threshold), lock (10ms threshold) profiles in JFR format sent to Pyroscope server
 
-## Cassandra Sidecar Instrumentation
+### Cassandra Sidecar Instrumentation
 
 The Cassandra Sidecar process is instrumented with the OpenTelemetry Java Agent and Pyroscope Java Agent, matching the pattern used for Cassandra itself. Both agents are loaded via `-javaagent` flags set in `/etc/default/cassandra-sidecar`, which is written by the `setup-instances` command.
 
@@ -127,7 +86,7 @@ Key configuration:
 - **Profiling**: CPU, allocation (512k threshold), lock (10ms threshold) profiles sent to Pyroscope server
 - **Activation**: Gated on `/etc/default/cassandra-sidecar` — the systemd `EnvironmentFile=-` directive makes it optional, so the sidecar starts normally without instrumentation if the file doesn't exist
 
-## Tool Runner Log Collection
+### Tool Runner Log Collection
 
 Commands run via `exec run` are executed through `systemd-run`, which captures stdout and stderr to log files under `/var/log/easydblab/tools/`. The OTel Collector's `filelog/tools` receiver watches this directory and ships log entries to VictoriaLogs with the attribute `source: tool-runner`.
 
@@ -139,7 +98,7 @@ Key details:
 - **Foreground commands**: Output displayed after completion, also logged to file
 - **Background commands** (`--bg`): Output logged to file only, tool runs as a systemd transient unit
 
-## YACE CloudWatch Scrape
+### YACE CloudWatch Scrape
 
 YACE (Yet Another CloudWatch Exporter) runs on the control node and scrapes AWS CloudWatch metrics for services used by the cluster. It uses tag-based auto-discovery with the `easy_cass_lab=1` tag to find relevant resources.
 
@@ -153,15 +112,24 @@ EMR metrics are collected directly via OTel Collectors on Spark nodes (see Spark
 
 YACE exposes scraped metrics as Prometheus-compatible metrics on port 5001, which are then scraped by the OTel Collector and forwarded to VictoriaMetrics. This replaces the previous CloudWatch datasource in Grafana with a Prometheus-based approach, giving dashboards access to CloudWatch metrics through VictoriaMetrics queries.
 
+## Resource Attributes
+
+Traces from the CLI tool and cluster nodes include the following resource attributes:
+- `service.name`: Service identifier (e.g., `easy-db-lab`, `cassandra-sidecar`, `spark-<job-name>`)
+- `service.version`: Application version (CLI tool only)
+- `host.name`: Hostname
+
 ## Configuration
 
 The following environment variables are supported:
 
 | Variable | Description | Default |
 |----------|-------------|---------|
-| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP gRPC endpoint | None (OTel disabled) |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP gRPC endpoint | None (no export) |
+| `OTEL_SERVICE_NAME` | Override service name | `easy-db-lab` |
+| `OTEL_RESOURCE_ATTRIBUTES` | Additional resource attributes | None |
 
-Additional standard OTel environment variables may work depending on the SDK defaults.
+Additional standard OTel environment variables are supported by the agent. See the [OpenTelemetry Java Agent documentation](https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/) for details.
 
 ## Example: Using with Jaeger
 
@@ -198,12 +166,8 @@ easy-db-lab up
 
 1. Verify the endpoint is correct and reachable
 2. Check that the collector accepts gRPC OTLP (port 4317 is standard)
-3. Look for OpenTelemetry initialization logs on startup
+3. Look for OpenTelemetry agent logs on startup (use `-Dotel.javaagent.debug=true` to enable debug logging)
 
 ### High Latency
 
 Traces are batched before export (default 1 second delay). This is normal and reduces overhead.
-
-### Shutdown Warnings
-
-A shutdown hook flushes remaining telemetry on exit. Brief delays during shutdown are expected.

gradle/libs.versions.toml

@@ -66,9 +66,7 @@ resilience4j = "2.4.0"
 fabric8-kubernetes = "7.5.0"
 
 # OpenTelemetry
-opentelemetry = "1.45.0"
-opentelemetry-instrumentation = "2.26.0-alpha"
-opentelemetry-semconv = "1.29.0-alpha"
+opentelemetry-agent = "2.26.0"
 
 # Gradle Plugins
 shadow = "9.3.2"
@@ -172,18 +170,6 @@ fabric8-kubernetes-client = { module = "io.fabric8:kubernetes-client", version.r
 # Cassandra Driver
 cassandra-driver-core = { module = "org.apache.cassandra:java-driver-core", version.ref = "cassandra-driver" }
 
-# OpenTelemetry
-opentelemetry-api = { module = "io.opentelemetry:opentelemetry-api", version.ref = "opentelemetry" }
-opentelemetry-sdk = { module = "io.opentelemetry:opentelemetry-sdk", version.ref = "opentelemetry" }
-opentelemetry-sdk-extension-autoconfigure = { module = "io.opentelemetry:opentelemetry-sdk-extension-autoconfigure", version.ref = "opentelemetry" }
-opentelemetry-exporter-otlp = { module = "io.opentelemetry:opentelemetry-exporter-otlp", version.ref = "opentelemetry" }
-opentelemetry-api-incubator = { module = "io.opentelemetry:opentelemetry-api-incubator", version = "1.45.0-alpha" }
-opentelemetry-sdk-extension-incubator = { module = "io.opentelemetry:opentelemetry-sdk-extension-incubator", version = "1.45.0-alpha" }
-opentelemetry-semconv = { module = "io.opentelemetry.semconv:opentelemetry-semconv", version.ref = "opentelemetry-semconv" }
-opentelemetry-semconv-incubating = { module = "io.opentelemetry.semconv:opentelemetry-semconv-incubating", version.ref = "opentelemetry-semconv" }
-opentelemetry-instrumentation-logback = { module = "io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0", version.ref = "opentelemetry-instrumentation" }
-opentelemetry-aws-sdk = { module = "io.opentelemetry.instrumentation:opentelemetry-aws-sdk-2.2", version.ref = "opentelemetry-instrumentation" }
-opentelemetry-okhttp = { module = "io.opentelemetry.instrumentation:opentelemetry-okhttp-3.0", version.ref = "opentelemetry-instrumentation" }
 
 [bundles]
 # Logging bundle
@@ -217,8 +203,6 @@ sshd = ["sshd-core", "sshd-scp"]
 # Resilience4j bundle
 resilience4j = ["resilience4j-retry", "resilience4j-kotlin"]
 
-# OpenTelemetry bundle
-opentelemetry = ["opentelemetry-api", "opentelemetry-api-incubator", "opentelemetry-sdk", "opentelemetry-sdk-extension-autoconfigure", "opentelemetry-sdk-extension-incubator", "opentelemetry-exporter-otlp", "opentelemetry-aws-sdk", "opentelemetry-okhttp", "opentelemetry-instrumentation-logback"]
 
 [plugins]
 kotlin-jvm = { id = "org.jetbrains.kotlin.jvm", version.ref = "kotlin" }