Spark refactor (#554)

* Add live metrics streaming via Redis pub/sub and archive completed changes

* Fix all 68 detekt violations and split DefaultK8sService into focused classes

- Rename metrics events: SystemSnapshot→System, CassandraSnapshot→Cassandra, NodeMetrics→Node
- Split DefaultK8sService (1138 lines) into 4 delegation classes following SRP:
  K8sClientProvider, DefaultK8sManifestOperations, DefaultK8sJobOperations,
  DefaultK8sNamespaceOperations, DefaultK8sStorageOperations
- Extract extension functions: AWSIamExtensions, AWSS3Extensions,
  ClusterStateExtensions, ClusterS3PathExtensions, ClusterS3PathConfigExtensions
- Introduce data classes to reduce parameter counts: EmrClusterProvisioningConfig,
  ProvisioningCallbacks, InfrastructureContext, SparkJobRequest, StressJobConfig,
  RequirementCheckDeps, BackupTargetResult
- Extract helpers to reduce method length/complexity across McpServer, Up,
  K3sClusterService, EMRSparkService, OpenSearchStart, StatusCache, and others
- Tune detekt config: ignorePrivate/ignoreOverridden for TooManyFunctions,
  ignoreDefaultParameters for LongParameterList
- Fix EmptyFunctionBlock, MagicNumber, ComplexCondition, InstanceOfCheckForException,
  NestedBlockDepth, LoopWithTooManyJumpStatements violations

* Simplify: consolidate HTTP_OK constant, fix redundant null-check, optimize metrics collection

- Move duplicated HTTP_OK=200 from 3 Victoria services into Constants.HttpStatus.OK
- Fix redundant ?. operator after guaranteed non-null assignment in McpServer
- Convert per-call buildSectionExtractors() into reusable sectionSerializers property in StatusCache
- Replace O(n×m) findValueForHost() linear search with O(1) indexByHost() map lookups in MetricsCollector

* Structural improvements: shared K8sClientProvider, extract K8sPodUtils, consolidate delete pattern

- Share single K8sClientProvider across all 4 K8s operation classes via Koin singleton
- Extract checkForPodFailure into K8sPodUtils, removing circular dependency from DefaultK8sService
- Replace 120 lines of copy-paste K8s resource deletion with generic deleteMatchingResources helper
- Collect system metrics independently of db node presence in MetricsCollector
- Replace KoinComponent service-locator with constructor injection in StressJobService

* Rename "MCP Server" to "Server" across docs, specs, and code comments

The server command provides more than just MCP — it includes REST status
endpoints, background StatusCache, and optional MetricsCollector. Renamed
all user-facing references from "MCP Server" to "Server" to reflect the
full scope. Kotlin class names (McpServer, McpToolRegistry) and the mcp/
package stay unchanged since they specifically implement the MCP protocol.

* Reorganize Spark modules under unified spark/ parent with shared config

Move 4 scattered Spark submodules (spark-shared, bulk-writer, connector-writer,
spark-connector-test1) into spark/ with nested Gradle structure. Split bulk-writer
into bulk-writer-sidecar and bulk-writer-s3. Extract shared SparkJobConfig into
spark/common for unified spark.easydblab.* configuration across all modules.

Delete AbstractBulkWriter in favor of standalone main classes. Update all bin/
scripts, e2e tests, CI workflow, and documentation for new module paths.

* Deduplicate Spark module config and fix compiler warnings

Gradle: Extract shared deps and shadow config into parent spark/build.gradle.kts
using scoped configure() blocks for bulk-writer-* and connector-* modules. Each
subproject build file now only declares mainClass and archiveBaseName. Fail fast
at compileJava if cassandra-analytics is not built.

Java: Add helper methods to SparkJobConfig (generateTestData, buildBulkWriteOptions,
configureCassandraConnector) and constants for write option keys, connector format,
and connector config strings. All writers now use shared helpers instead of
duplicated inline code.

Fix 3 compiler warnings: redundant elvis operator in AWSResourceSetupService,
redundant toString() calls on ClusterS3Path.toUri() in EMRSparkService.

* Archive spark-refactor change and sync specs

Archive completed spark-refactor change (37/37 tasks).
Add new spark-modules spec (unified config, module organization, deployable JARs).
Merge reorganized JAR path scenario into existing spark-emr spec.

* Add Spark integration test harness, skip S3 bulk writer, patch storage_compatibility_mode

- Add TestContainers-based SparkWriterIntegrationTest for local verification
- Skip bulk-writer-s3 e2e step until DATA_TRANSPORT_EXTENSION_CLASS is implemented
- Patch storage_compatibility_mode: NONE before starting Cassandra 5

* Archive rename-mcp-server-to-server change

* Address code review: replace System.exit with exceptions, add unit tests, fail-fast on skipDdl

- SparkJobConfig throws IllegalArgumentException instead of System.exit(1)
- S3BulkWriter does the same for missing s3.bucket
- Add SparkJobConfigTest covering defaults, parsing, and missing property errors
- setupSchema validates table exists when skipDdl=true to fail fast
- Add storage_compatibility_mode comment explaining why NONE is needed
- Expand comment on unpublished cassandra-analytics JARs
- Remove unused LocalStackContainer import

* Address code review: remove reflection, add isRunningCassandra(), fix e2e test gaps

- Remove step_bulk_writer_s3 from e2e step list (stub gives false confidence)
- Replace reflection in MetricsCollectorTest with internal collect() method
- Add ClusterState.isRunningCassandra() to replace fragile clickHouseConfig null check
- Document detekt.yml threshold relaxations with triggering classes
- Add JAR existence check to submit_spark_writer before submission
- Deduplicate hostNames with .toSet() in collectSystemMetrics()
- Update skipDdl docs to mention validation behavior

* Verify spark writer row counts at LOCAL_QUORUM in e2e test

- Parse COUNT(*) output and assert it matches expected 10000 rows
- Use CONSISTENCY LOCAL_QUORUM for verification queries
- Bump replicationFactor from 1 to 3 to support LOCAL_QUORUM reads

* Address code review: strengthen test assertions, thread-safe MetricsCollector, portable grep

- SparkSubmitTest: replace bare any() with argThat to verify clusterId, jarPath,
  and mainClass are correctly forwarded through execute()
- MetricsCollector: add @synchronized to start()/stop() to prevent double-start
  from concurrent calls
- bin/end-to-end-test: replace grep -oP (Linux PCRE) with portable grep -Eo
- bin/end-to-end-test: add design doc reference to step_bulk_writer_s3 TODO

* Widen CI artifact paths to capture all modules, add jacoco for spark

- Test results: use ** globs to capture root + spark submodule JUnit XML
- Quote dorny/test-reporter path to fix "No test report files found" error
- Upload jacoco reports from spark modules alongside Kover coverage
- Add jacoco plugin to spark subprojects for Java coverage reporting

* Clean up tests: remove mock-echo test, fix container lifecycle, use project.root

- Delete SparkSubmitTest `command validates required parameters` (only checked defaults)
- SparkWriterIntegrationTest: make spark container @Container-managed so
  TestContainers handles full lifecycle, preventing leaks on setup failure
- Replace fragile getParent().getParent() path resolution with Gradle
  project.root system property (with fallback for IDE runs)
- Add dependsOn(:spark:connector-writer:shadowJar) so integration test
  JAR is always built before tests run

* Use LOCAL_QUORUM via Java driver instead of cqlsh CONSISTENCY syntax

- DefaultCqlSessionService: execute all queries at LOCAL_QUORUM via
  SimpleStatement.setConsistencyLevel() instead of default CL
- Remove invalid cqlsh CONSISTENCY syntax from e2e test shell script
  (the cql command uses the Java driver, not cqlsh)

* Formatting fix

Author: rustyrazorblade

.claude/commands/test-workflow.md

@@ -1,12 +1,12 @@
 ---
 command: "/test-workflow"
 category: "Testing & Demonstration"
-purpose: "End-to-end Cassandra cluster workflow demonstration using MCP servers"
+purpose: "End-to-end Cassandra cluster workflow demonstration using server tools"
 ---
 
 # Cassandra Cluster Test Workflow
 
-Execute a complete end-to-end workflow demonstrating easy-db-lab capabilities using only MCP server tools.
+Execute a complete end-to-end workflow demonstrating easy-db-lab capabilities using only server tools.
 
 ## Workflow Steps
 
@@ -38,7 +38,7 @@ Use `mcp__easy-db-lab__use` to set Cassandra version:
 Use `mcp__easy-db-lab__start` to start all services:
 - Starts Cassandra on database nodes
 - Starts cassandra-easy-stress on stress nodes
-- Starts monitoring and MCP servers on control nodes
+- Starts monitoring and servers on control nodes
 - Report when services are ready
 
 ### 5. Get Cassandra Host IP
@@ -49,8 +49,8 @@ Use `mcp__easy-db-lab__hosts` to retrieve host information:
 ### 6. Wait for Cluster Readiness
 Wait ~30 seconds for Cassandra to fully initialize and be ready to accept connections.
 
-### 7. Check MCP Server Status
-Use `mcp__easy-db-lab__get_server_status` to check if MCP servers are running:
+### 7. Check Server Status
+Use `mcp__easy-db-lab__get_server_status` to check if servers are running:
 - Verify easy-cass-mcp is accessible
 - Verify cassandra-easy-stress is accessible
 - If any are disconnected, report but continue (manual reconnection may be needed)

.github/workflows/pr-checks.yml

@@ -27,25 +27,25 @@ jobs:
           cache-read-only: ${{ github.ref != 'refs/heads/main' }}
 
       - name: Run tests
-        # Exclude bulk-writer module - requires cassandra-analytics SNAPSHOTs from local Maven
-        run: ./gradlew test koverXmlReport --no-daemon -x :bulk-writer:compileJava -x :bulk-writer:test
+        # Exclude bulk-writer modules - require cassandra-analytics SNAPSHOTs from local Maven
+        run: ./gradlew test koverXmlReport --no-daemon -x :spark:bulk-writer-sidecar:compileJava -x :spark:bulk-writer-sidecar:test -x :spark:bulk-writer-s3:compileJava -x :spark:bulk-writer-s3:test
 
       - name: Upload test results
         if: always()
         uses: actions/upload-artifact@v6
         with:
           name: test-results
           path: |
-            build/reports/tests/test/
-            build/test-results/test/
+            **/build/reports/tests/test/
+            **/build/test-results/test/
           retention-days: 30
 
       - name: Publish Test Results
         uses: dorny/test-reporter@v2
         if: always()
         with:
           name: Test Results
-          path: build/test-results/test/*.xml
+          path: '**/build/test-results/test/*.xml'
           reporter: java-junit
           fail-on-error: false
 
@@ -66,6 +66,7 @@ jobs:
           name: coverage-reports
           path: |
             build/reports/kover/
+            **/build/reports/jacoco/
           retention-days: 30
 
   # Run linting and static analysis in parallel with tests

.gitignore

@@ -155,7 +155,8 @@ site/
 
 # Cassandra Analytics build artifacts (see bin/build-cassandra-analytics)
 .cassandra-analytics/
-bulk-writer/libs/*.jar
+spark/bulk-writer-sidecar/libs/*.jar
+spark/bulk-writer-s3/libs/*.jar
 clusters/
 
 # Claude Code config (mounted in devcontainer)

CLAUDE.md

@@ -13,14 +13,17 @@ The project follows a layered architecture:
 
 - **Commands (PicoCLI)** delegate to **Services**, which interact with **External Systems** (K8s, AWS, Filesystem)
 - Commands and Services emit events via the **EventBus**
-- **Listeners** consume events: `ConsoleEventListener` (stdout/stderr), `McpEventListener` (MCP server status), `RedisEventListener` (pub/sub, optional)
+- **Listeners** consume events: `ConsoleEventListener` (stdout/stderr), `McpEventListener` (server MCP status), `RedisEventListener` (pub/sub, optional)
 
 ### Project Modules
 
 The Gradle project has multiple modules:
 - **Root module** (`:`) — the main CLI application
-- **`bulk-writer`** — Cassandra bulk writer (requires cassandra-analytics built with JDK 11)
-- **`spark-shared`** — shared Spark utilities
+- **`spark/common`** — shared Spark config (`SparkJobConfig`), data generation, CQL setup
+- **`spark/bulk-writer-sidecar`** — Cassandra Analytics bulk writer, direct sidecar transport (requires cassandra-analytics built with JDK 11)
+- **`spark/bulk-writer-s3`** — Cassandra Analytics bulk writer, S3 staging transport (requires cassandra-analytics built with JDK 11)
+- **`spark/connector-writer`** — Standard Spark Cassandra Connector writer
+- **`spark/connector-read-write`** — Read→transform→write example using Spark Cassandra Connector
 
 ### Layer Responsibilities
 
@@ -40,10 +43,10 @@ See [`commands/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/command
 
 See [`providers/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/providers/CLAUDE.md) for AWS/SSH/Docker patterns and retry logic.
 
-### MCP Server & REPL
+### Server & REPL
 
 Two commands run as long-lived processes instead of the typical run-and-exit pattern:
-- **`Server`** — starts an MCP server (Ktor + SSE) for AI agent integration. See [`mcp/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/mcp/CLAUDE.md).
+- **`Server`** — starts a hybrid HTTP server with MCP protocol support (Ktor + SSE), REST status endpoints, and background services. See [`mcp/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/mcp/CLAUDE.md).
 - **`Repl`** — starts an interactive REPL to reduce typing for repeated commands.
 
 ### Dependency Injection
@@ -297,6 +300,6 @@ Detailed patterns live in package-level CLAUDE.md files:
 - [`services/aws/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/services/aws/CLAUDE.md) — AWS service classes (AMI, EC2, EMR, OpenSearch, S3)
 - [`providers/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/providers/CLAUDE.md) — AWS SDK wrappers, SSH/Docker patterns, retry logic
 - [`configuration/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/configuration/CLAUDE.md) — cluster state, templates, K8s manifest builders, observability stack details
-- [`mcp/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/mcp/CLAUDE.md) — MCP server architecture
+- [`mcp/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/mcp/CLAUDE.md) — Server architecture (MCP, REST, background services)
 - [`kubernetes/CLAUDE.md`](src/main/kotlin/com/rustyrazorblade/easydblab/kubernetes/CLAUDE.md) — K8s client patterns
 - [`src/test/.../CLAUDE.md`](src/test/kotlin/com/rustyrazorblade/easydblab/CLAUDE.md) — test infrastructure

README.md

@@ -28,7 +28,7 @@ A tool for creating database lab environments in AWS. Designed for testing, benc
 
 ### Developer Experience
 
-- **MCP Server** - AI assistant integration for cluster management
+- **Server** - AI assistant integration, REST status endpoints, live metrics
 - **Interactive CLI** - REPL mode for reduced typing
 - **Homebrew and Docker** - Multiple installation options
 
@@ -470,33 +470,33 @@ bcc-tools is a useful package of tools
 
 https://rustyrazorblade.com/post/2023/2023-11-14-bcc-tools/
 
-## MCP Server Integration
+## Server Integration
 
-easy-db-lab includes a Model Context Protocol (MCP) server that enables AI assistants like Claude Code to interact directly with your database clusters.
+easy-db-lab includes a server mode that enables AI assistants like Claude Code to interact directly with your database clusters via MCP (Model Context Protocol), and provides REST status endpoints for programmatic access.
 
-### Starting the MCP Server
+### Starting the Server
 
-To start the MCP server:
+To start the server:
 
 ```shell
 easy-db-lab server --port 8888
 ```
 
-This starts the MCP server on port 8888 (you can use any available port).
+This starts the server on port 8888 (you can use any available port).
 
 ### Integrating with Claude Code
 
-Once the MCP server is running, add it to Claude Code:
+Once the server is running, add it to Claude Code:
 
 ```shell
 claude mcp add --transport sse easy-db-lab http://127.0.0.1:8888/sse
 ```
 
-This establishes a Server-Sent Events (SSE) connection between Claude Code and your easy-db-lab MCP server.
+This establishes a Server-Sent Events (SSE) connection between Claude Code and the server.
 
 ### What You Can Do
 
-With MCP integration, Claude Code can:
+With the server integration, Claude Code can:
 
 * Manage and provision clusters directly
 * Configure and deploy Cassandra, ClickHouse, and OpenSearch
@@ -505,7 +505,7 @@ With MCP integration, Claude Code can:
 * Troubleshoot issues by analyzing logs and metrics
 * Automate complex multi-step cluster operations
 
-For detailed documentation, see the [MCP Integration section in the user manual](https://rustyrazorblade.github.io/easy-db-lab/integrations/mcp-server/).
+For detailed documentation, see the [Server section in the user manual](https://rustyrazorblade.github.io/easy-db-lab/integrations/server/).
 
 ## Sanity Check Test
 

bin/dev

@@ -60,22 +60,15 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
 WORKSPACE_FOLDER="$PROJECT_ROOT"
 
-# Build mount args for Claude credentials if ENABLE_CLAUDE=1
-CLAUDE_MOUNT_ARGS=()
-if [[ "${ENABLE_CLAUDE:-}" == "1" ]]; then
-    if [[ -f "$HOME/.claude/.credentials.json" ]]; then
-        CLAUDE_MOUNT_ARGS=(--mount "type=bind,source=$HOME/.claude/.credentials.json,target=/home/node/.claude/.credentials.json")
-    else
-        echo -e "${YELLOW}Warning: ~/.claude/.credentials.json not found. Claude auth will not persist.${NC}"
-    fi
-fi
-
 # Colors for output
 RED='\033[0;31m'
 GREEN='\033[0;32m'
 YELLOW='\033[0;33m'
 NC='\033[0m' # No Color
 
+# Ensure Claude config directory exists (mounted by devcontainer.json)
+mkdir -p "$PROJECT_ROOT/.devcontainer/claude"
+
 usage() {
     cat <<EOF
 Usage: $(basename "$0") <command> [args...]
@@ -101,12 +94,8 @@ Commands:
     clean           Full cleanup (containers, volumes)
     help            Show this help message
 
-Environment Variables:
-    ENABLE_CLAUDE=1     Mount ~/.claude config directory (for Claude Code users)
-
 Examples:
     $(basename "$0") start
-    ENABLE_CLAUDE=1 $(basename "$0") start
     $(basename "$0") claude
     $(basename "$0") shell
     $(basename "$0") test
@@ -156,7 +145,7 @@ get_compose_project() {
 cmd_start() {
     check_devcontainer
     info "Starting dev container..."
-    devcontainer up --workspace-folder "$WORKSPACE_FOLDER" ${CLAUDE_MOUNT_ARGS[@]+"${CLAUDE_MOUNT_ARGS[@]}"}
+    devcontainer up --workspace-folder "$WORKSPACE_FOLDER"
 }
 
 cmd_stop() {
@@ -174,7 +163,7 @@ cmd_stop() {
 cmd_rebuild() {
     check_devcontainer
     info "Rebuilding dev container..."
-    devcontainer up --workspace-folder "$WORKSPACE_FOLDER" --remove-existing-container ${CLAUDE_MOUNT_ARGS[@]+"${CLAUDE_MOUNT_ARGS[@]}"}
+    devcontainer up --workspace-folder "$WORKSPACE_FOLDER" --remove-existing-container
 }
 
 cmd_build() {