Otel collector not installed on all spark nodes

[rustyrazorblade]

Looks like it's only on the master.

Should set a service name of spark-master or spark-worker and let me filter on the emr dashboard.

Add pyroscope to the emr dashboard. One for master one for workers. Create a section for master metrics and another for worker. Do the os stats as well as spark metrics.

Make a proposal.

[rustyrazorblade]

@claude start on this

[claude]

Claude finished @rustyrazorblade's task in 6m 43s —— View job • claude/issue-564-20260313-0245 • Create PR ➔

---

Proposal: EMR OTel Master/Worker Role Differentiation

• Gather context on current EMR/Spark OTel collector setup
• Analyze current bootstrap script and OTel collector config
• Review EMR dashboard JSON and Pyroscope setup
• Create proposal, design, and tasks artifacts

---

Root Cause

The OTel Collector runs on all EMR nodes via the bootstrap action — that part is already correct. The actual gaps are:

1. otel-collector-config.yaml hardcodes node_role: spark for every node — no master/worker distinction
2. OTEL_SERVICE_NAME=spark and -Dpyroscope.application.name=spark are set cluster-wide in spark-defaults — same for all nodes
3. The EMR dashboard has a single "Host Metrics" section with no role split and no Pyroscope panels

---

Proposed Changes

1. otel-collector-config.yaml — Add __NODE_ROLE__ placeholder

Change the hardcoded value: spark in the resource/role processor to value: __NODE_ROLE__. This placeholder is left unresolved at S3 upload time (TemplateService won't touch it) and is patched per-node at bootstrap time.

2. bootstrap-otel.sh — Detect IS_MASTER and resolve role

EMR sets IS_MASTER=true on the master node at bootstrap execution time. The script:

• Sets NODE_ROLE=spark-master if IS_MASTER=true, else NODE_ROLE=spark-worker
• Runs sed -i "s/__NODE_ROLE__/$NODE_ROLE/g" "$COLLECTOR_CONFIG" after writing the config
• Appends export OTEL_SERVICE_NAME=$NODE_ROLE and export PYROSCOPE_APPLICATION_NAME=$NODE_ROLE to /etc/spark/conf/spark-env.sh

3. EMRProvisioningService — Remove static cluster-wide service name

Remove OTEL_SERVICE_NAME=spark from otelEnvVars and -Dpyroscope.application.name=spark from pyroscopeFlags — these are now set per-node by the bootstrap script. The per-job OTEL_SERVICE_NAME=spark-{jobName} override in EMRSparkService is unaffected.

4. emr.json — Master/worker sections + Pyroscope panels

Restructure dashboard into:

• Master — OS Stats row (CPU, memory, disk, network filtered by node_role="spark-master")
• Master — Spark JVM row (heap, GC, threads filtered by node_role="spark-master")
• Worker — OS Stats row (same metrics filtered by node_role="spark-worker")
• Worker — Spark JVM row (same JVM metrics filtered by node_role="spark-worker")
• Profiling row with Pyroscope flamegraph panels for spark-master and spark-worker

5. openspec/specs/spark-emr/spec.md — Spec update

Add requirement for per-node role labeling and remove static service name requirement.

---

Artifacts

All artifacts committed to branch claude/issue-564-20260313-0245:

• openspec/changes/emr-otel-master-worker-roles/proposal.md
• openspec/changes/emr-otel-master-worker-roles/design.md
• openspec/changes/emr-otel-master-worker-roles/tasks.md

Ready for implementation — run /opsx:apply emr-otel-master-worker-roles when you're ready to proceed.

|

[rustyrazorblade]

Put the workers on top. They are more useful.

Apply after that

[rustyrazorblade]

@claude

[claude]

Claude finished @rustyrazorblade's task in 8m 23s —— View job • claude/issue-564-20260313-0307 • Create PR ➔

---

Implementing EMR OTel Master/Worker Role Differentiation

• Gather context from previous proposal
• Update plan: workers section before masters
• Implement changes
  • Update otel-collector-config.yaml with __NODE_ROLE__ placeholder
  • Update bootstrap-otel.sh to detect IS_MASTER and set role
  • Update EMRProvisioningService.kt to remove static service name
  • Update EMR dashboard with workers-first layout + Pyroscope panels
  • Update OpenSpec spec
• Commit and push changes

Summary

All changes implemented and pushed to claude/issue-564-20260313-0307:

otel-collector-config.yaml — changed node_role: spark to node_role: __NODE_ROLE__ so the placeholder is patched per-node at runtime.

bootstrap-otel.sh — detects EMR's IS_MASTER env var at bootstrap time, sets NODE_ROLE=spark-master or spark-worker, patches __NODE_ROLE__ in the collector config via sed, then appends export OTEL_SERVICE_NAME=$NODE_ROLE and export PYROSCOPE_APPLICATION_NAME=$NODE_ROLE to /etc/spark/conf/spark-env.sh.

EMRProvisioningService.kt — removed the static OTEL_SERVICE_NAME=spark env var and -Dpyroscope.application.name=spark JVM flag from the cluster-wide spark-defaults classification. Both are now set per-node by the bootstrap script.

emr.json — restructured dashboard layout (workers first as requested):

1. Workers — OS Stats (CPU, Memory, Disk I/O, Network I/O, Load Average, Filesystem)
2. Workers — Spark JVM (Heap, GC Duration, Threads, Classes Loaded)
3. Workers — Profiling (CPU + Memory allocation flamegraphs filtered by service_name="spark-worker")
4. Master — OS Stats
5. Master — Spark JVM
6. Master — Profiling (flamegraphs filtered by service_name="spark-master")

The hostname template variable now matches node_role=~"spark-.*" to capture both roles.

openspec/specs/spark-emr/spec.md — added requirement for per-node role labeling.

• claude/issue-564-20260313-0307