Use the latest sidecar image

[rustyrazorblade]

Now that we have builds passing GH CI we can start using the sidecar image instead of the custom build during packer. Should save 5-10 minutes during the build process.

ghcr.io/apache/cassandra-sidecar:latest

Need to ensure the config gets mounted in the right spot for the container.

[rustyrazorblade]

@claude

[claude]

Claude finished @rustyrazorblade's task in 5m 42s —— View job • claude/issue-458-20260312-1634 • Create PR ➔

---

Working on Issue #458: Use the latest sidecar image

• Gather context - explore packer scripts and sidecar configuration
• Understand current sidecar build/install process
• Update install_sidecar.sh to install Docker CE and pre-pull ghcr.io/apache/cassandra-sidecar:latest
• Update cassandra-sidecar.service to run via Docker with volume mounts
• Commit and push changes

Changes Made

packer/cassandra/install/install_sidecar.sh
Replaced the ~5-10 minute git clone + Gradle build with:

1. Install docker.io via apt
2. Enable Docker on boot
3. Pre-pull ghcr.io/apache/cassandra-sidecar:latest to cache the image in the AMI

packer/cassandra/services/cassandra-sidecar.service
Updated the systemd service to run the sidecar as a Docker container:

• After=docker.service / Requires=docker.service for proper dependency ordering
• ExecStartPre cleans up any stale container
• docker run --rm --network host with volume mounts:
  • /etc/cassandra-sidecar → config directory (read-only)
  • /mnt/db1/cassandra → data, logs, and staging directories
  • /usr/local/otel → OTel Java agent (read-only)
  • /usr/local/pyroscope → Pyroscope Java agent (read-only)
• Environment variables JAVA_OPTS, OTEL_SERVICE_NAME, and OTEL_EXPORTER_OTLP_ENDPOINT are passed through from the systemd env (populated by setup-instances)
• ExecStop gracefully stops the container

The JAVA_OPTS environment variable (set by SetupInstance.kt) already includes -Dsidecar.config=file:///etc/cassandra-sidecar/cassandra-sidecar.yaml and -Dsidecar.logdir=/mnt/db1/cassandra/logs/sidecar, so config is mounted at exactly the path the sidecar expects.

[rustyrazorblade]

@claude Sidecar should be run by k3s not docker. Do not install docker.

[rustyrazorblade]

@claude Sidecar should be run by k3s not docker. Do not install docker.

[claude]

Claude finished @rustyrazorblade's task in 17m 59s —— View job

---

Implementation Complete

Replaced the 5-10 minute build-from-source with a K8s DaemonSet running ghcr.io/apache/cassandra-sidecar:latest via k3s.

Changes

Packer (packer/cassandra/)

• install/install_sidecar.sh — Replaces git clone + ./gradlew shadowjar with skopeo copy to download the pre-built image into the k3s airgap images directory (/var/lib/rancher/k3s/agent/images/cassandra-sidecar.tar). k3s loads this at startup so no network pull is needed when the DaemonSet is deployed.
• services/cassandra-sidecar.service — Deleted. The sidecar is now managed by k3s, not systemd.

New: SidecarManifestBuilder (configuration/sidecar/SidecarManifestBuilder.kt)

• Fabric8-based DaemonSet targeting type=db nodes with hostNetwork: true
• Mounts config from /etc/cassandra-sidecar (hostPath), data from /mnt/db1/cassandra
• OTel and Pyroscope agents mounted from host (/usr/local/otel, /usr/local/pyroscope)
• JAVA_OPTS uses K8s $(VAR_NAME) env var interpolation for per-node Pyroscope labels (hostname, cluster, control IP)

SidecarService (rewritten from SystemD → K8s)

• Was: interface SidecarService : SystemDServiceManager with per-host SSH commands
• Now: cluster-level operations via K8sService:
  • deploy(controlHost) — applies DaemonSet (idempotent)
  • rolloutRestart(controlHost) — rolling restart of the DaemonSet
  • remove(controlHost) — deletes DaemonSet by label

Commands updated (Start.kt, Stop.kt, Restart.kt)

• Now call sidecarService.deploy/remove/rolloutRestart(controlHost) instead of per-host systemd operations

GrafanaUpdateConfig.kt

• Deploys sidecar DaemonSet as part of update-config (alongside other observability stack components)
• Added cassandra-sidecar to the rollout-restart list

Tests

• SidecarServiceTest — rewrites tests for K8s DaemonSet behavior
• K8sServiceIntegrationTest — added @Order(19) test that applies the DaemonSet to a real K3s TestContainers cluster and verifies it exists

[rustyrazorblade]

Get rid of the packer part. The image is available on the apache cassandra-sidecar repo. Use latest.

Make an openspec proposal

[rustyrazorblade]

@claude

[claude]

Claude finished @rustyrazorblade's task in 4m 33s —— View job • claude/issue-458-20260313-0206 • Create PR ➔

---

OpenSpec Proposal: Sidecar K8s DaemonSet Migration

• Gather context - read current branch state
• Create OpenSpec proposal for sidecar K8s migration

---

Created the change at openspec/changes/2026-03-13-sidecar-k8s-daemonset/ with three artifacts:

proposal.md — Why & what changes:

• Remove packer source build (install_sidecar.sh git clone + gradlew) — saves 5–10 min per AMI build
• Remove cassandra-sidecar.service systemd unit
• Run ghcr.io/apache/cassandra-sidecar:latest as a K8s DaemonSet on type=db nodes via k3s
• New SidecarManifestBuilder (Fabric8), rewritten SidecarService (K8s instead of SystemD), updated Start/Stop/Restart commands

design.md — Architecture decisions:

• Use latest tag, pull at deploy time (no airgap pre-pull needed — nodes have internet access)
• hostNetwork: true, nodeSelector: {type: db}
• Volume mounts: /etc/cassandra-sidecar (config, ro), /mnt/db1/cassandra (data), /usr/local/otel + /usr/local/pyroscope (agent JARs, ro)
• OTel/Pyroscope env vars injected via DaemonSet spec using K8s fieldRef + ConfigMap refs — no more systemd EnvironmentFile
• New SidecarService interface: deploy(), rolloutRestart(), remove() (cluster-level, not per-host)

tasks.md — 8 task groups covering: packer cleanup, SidecarManifestBuilder, SidecarService rewrite, command updates, SetupInstance cleanup, GrafanaUpdateConfig, tests, and docs.

Ready to implement with /opsx:apply.

• claude/issue-458-20260313-0206

[rustyrazorblade]

@claude apply

[claude]

Claude finished @rustyrazorblade's task in 16m 30s —— View job • claude/issue-458-20260313-0247 • Create PR ➔

---

Implementation Complete: Sidecar K8s DaemonSet Migration

Using change: 2026-03-13-sidecar-k8s-daemonset
Progress: 8/8 task groups complete ✓

• Gather context - read proposal and current codebase state
• Understand the request and tasks
• Remove packer source build (install_sidecar.sh git clone + gradlew removed)
• Remove cassandra-sidecar.service systemd unit
• Create SidecarManifestBuilder (Fabric8 DaemonSet in configuration/sidecar/)
• Rewrite SidecarService (K8s cluster-level: deploy, remove, rolloutRestart)
• Update commands (Start.kt, Stop.kt, Restart.kt)
• Update SetupInstance.kt (removed setupSidecarSystemdEnv())
• Update GrafanaUpdateConfig.kt to deploy sidecar DaemonSet
• Register SidecarManifestBuilder in ServicesModule.kt
• Update SidecarServiceTest.kt (K8s-based tests)
• Add sidecar to K8sServiceIntegrationTest (@Order(20))
• Commit and push changes

Summary

Packer cleanup:

• install_sidecar.sh — replaced 10+ line git clone + Gradle build with a comment; no-op
• cassandra-sidecar.service — deleted (sidecar now managed by k3s, not systemd)

New: SidecarManifestBuilder (configuration/sidecar/SidecarManifestBuilder.kt)

• Fabric8 DaemonSet running ghcr.io/apache/cassandra-sidecar:latest
• nodeSelector: {type: db} — runs only on db nodes
• hostNetwork: true — sidecar reaches Cassandra on localhost
• Volume mounts: /etc/cassandra-sidecar (config, ro), /mnt/db1/cassandra (data), /usr/local/otel and /usr/local/pyroscope (Java agents, ro)
• JAVA_OPTS uses K8s $(VAR_NAME) interpolation for per-node Pyroscope labels (hostname, cluster, control IP from cluster-config ConfigMap)

Rewritten: SidecarService

• Interface: deploy(controlHost), rolloutRestart(controlHost), remove(controlHost)
• Implementation delegates to K8sService — no more per-host SSH

Commands updated:

• Start.kt — calls sidecarService.deploy(controlHost) after Cassandra starts
• Stop.kt — calls sidecarService.remove(controlHost) after Cassandra stops
• Restart.kt — calls sidecarService.rolloutRestart(controlHost) after Cassandra restarts

SetupInstance.kt — setupSidecarSystemdEnv() removed; env vars now injected via K8s

GrafanaUpdateConfig.kt — deploys sidecar DaemonSet as part of update-config; added cassandra-sidecar to rollout-restart list

All tasks complete! You can archive this change with /opsx:archive.

• claude/issue-458-20260313-0247