✨ Add memory profiling plugin for e2e tests

Add comprehensive memory profiling and analysis tooling to help identify
and optimize memory usage during e2e test execution.

Features:
- Automatic heap profile collection during test runs
- Support for all e2e test types (test-e2e, test-experimental-e2e,
  test-extension-developer-e2e, test-upgrade-e2e, test-upgrade-experimental-e2e)
- Detailed analysis of memory allocators and growth patterns
- Side-by-side comparison of test runs
- Integration with Claude Code via /memory-profile command

Usage:
  ./hack/tools/memory-profiling/memory-profile.sh run <test-name> [test-target]
  ./hack/tools/memory-profiling/memory-profile.sh analyze <test-name>
  ./hack/tools/memory-profiling/memory-profile.sh compare <test1> <test2>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

✨ Add Prometheus alerts tracking to memory profiling

Enhance memory profiling plugin to capture and analyze Prometheus alerts
during e2e test execution:

1. Set E2E_SUMMARY_OUTPUT environment variable during test runs
   - Captures prometheus alerts, test failures, and other metrics
   - Saved to e2e-summary.json in the profile directory

2. Updated analyze-profiles.sh to extract and display alerts
   - Parses e2e-summary.json using jq (if available)
   - Shows alert names, severities, and descriptions
   - Includes test failures in the analysis report

3. Updated compare-profiles.sh to compare alerts between runs
   - Shows alert counts for both tests
   - Lists alerts detected in each test
   - Helps identify if optimizations introduced new alerts

This allows correlating memory usage with system health metrics,
making it easier to identify if memory optimizations have any
negative side effects on system stability.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

🐛 Fix: Use absolute path for E2E_SUMMARY_OUTPUT

The e2e tests change directories during execution, causing the relative
path to e2e-summary.json to fail. Convert OUTPUT_DIR to an absolute path
before passing it to E2E_SUMMARY_OUTPUT.

This fixes the error:
  'failed to write e2e test summary output: no such file or directory'

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

♻️ Rename e2e-summary.json to e2e-summary.md

The E2E test summary file contains Markdown content (with mermaid charts),
not JSON. This updates the file extension and parsing logic throughout the
memory profiling scripts to correctly handle the Markdown format.

Changes:
- run-profiled-test.sh: Output to e2e-summary.md instead of .json
- analyze-profiles.sh: Parse Markdown Alerts section instead of JSON
- compare-profiles.sh: Compare Markdown alert sections between tests

✨ Add multi-component memory profiling support

Refactored memory profiling tools to simultaneously analyze both
operator-controller and catalogd components:

**collect-profiles.sh:**
- Hardcoded dual-component collection (operator-controller + catalogd)
- Separate port forwards: localhost:6060 → operator-controller:6060,
  localhost:6061 → catalogd:6060
- Creates component subdirectories for organized profile storage
- Collects profiles from both components simultaneously

**analyze-profiles.sh:**
- Removed backward compatibility for single-component analysis
- Now requires both component directories to exist
- Added analyze_component() function with stderr redirects to prevent
  log pollution in captured output
- Generates combined analysis.md with sections for each component
- Executive Summary shows peak memory for both components
- Prometheus Alerts section remains test-wide (not per-component)

**run-profiled-test.sh:**
- Updated progress display to show both components:
  "operator-controller: X, catalogd: Y"
- Enhanced summary output for dual-component results

Directory structure:
memory-profiles/
└── test-name/
    ├── operator-controller/
    │   └── heap*.pprof
    ├── catalogd/
    │   └── heap*.pprof
    ├── analysis.md
    └── e2e-summary.md

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

🐛 Clean output directory before profiling runs

Previously, run-profiled-test.sh would create the output directory
with mkdir -p but not clean up existing files. This caused old profile
files to remain and potentially confuse analysis when mixing
single-component and multi-component data.

Now the script:
- Warns if the output directory already exists
- Removes the entire directory to start fresh
- Creates a clean directory for the new profiling run

This ensures each test run has clean, isolated data.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

🐛 Fix Prometheus alert parsing to include level-3 headers

The sed pattern '/^##/' was incorrectly matching both level-2 (##) and
level-3 (###) markdown headers, causing it to stop at "### Firing Alerts"
instead of continuing to "## Performance". This resulted in empty alert
sections even when alerts were present.

Changed pattern to '/^## /' (with space) to match only level-2 headers.

**Impact:**
- analyze-profiles.sh now correctly extracts and displays all alerts
- compare-profiles.sh now correctly detects alerts in both test runs

**Example:**
Before: "No Prometheus alerts detected" (incorrect)
After: Shows both pending alerts:
  - operator-controller-memory-growth: 132.4kB/sec
  - operator-controller-memory-usage: 107.9MB

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

♻️ Delete empty heap profiles after collection

Empty heap profiles (0 bytes) are created when the cluster is torn down
and the pod is killed during the final profile collection attempt. These
empty files skew analysis results by showing memory as 0K at the end,
incorrectly suggesting all memory was freed.

Now collect-profiles.sh deletes any empty heap*.pprof files at the end
of collection, ensuring analysis only processes valid profiles captured
during actual test execution.

**Impact:**
- Profile counts now reflect actual valid profiles (e.g., 25 instead of 26)
- Memory growth tables no longer show misleading "0K" final entry
- Peak memory correctly identified from real execution data

**Example:**
Before: heap24 (160K) → heap25 (0K) - misleading
After: heap24 (160K) as actual peak

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

Author: tmshort

.claude/commands/memory-profile.md

@@ -0,0 +1,216 @@
+---
+description: Profile memory usage during e2e tests and analyze results
+---
+
+# Memory Profiling Plugin
+
+Analyze memory usage during e2e tests by collecting pprof heap profiles and generating comprehensive analysis reports.
+
+## Commands
+
+### /memory-profile run [test-name] [test-target]
+
+Run an e2e test with continuous memory profiling:
+
+1. Start the specified e2e test (defaults to `make test-experimental-e2e`)
+2. Wait for the operator-controller pod to be ready
+3. Collect heap profiles every 15 seconds to `./memory-profiles/[test-name]/`
+4. Continue until the test completes or is interrupted
+5. Generate a summary report
+
+**Test Targets:**
+- `test-e2e` - Standard e2e tests
+- `test-experimental-e2e` - Experimental e2e tests (default)
+- `test-extension-developer-e2e` - Extension developer e2e tests
+- `test-upgrade-e2e` - Upgrade e2e tests
+- `test-upgrade-experimental-e2e` - Upgrade experimental e2e tests
+
+**Examples:**
+```
+/memory-profile run baseline
+/memory-profile run baseline test-e2e
+/memory-profile run with-caching test-experimental-e2e
+/memory-profile run upgrade-test test-upgrade-e2e
+```
+
+### /memory-profile analyze [test-name]
+
+Analyze collected heap profiles for a specific test run:
+
+1. Load all heap profiles from `./memory-profiles/[test-name]/`
+2. Analyze memory growth patterns
+3. Identify top allocators
+4. Find OpenAPI, JSON, and other hotspots
+5. Generate detailed markdown report
+
+**Example:**
+```
+/memory-profile analyze baseline
+```
+
+### /memory-profile compare [test1] [test2]
+
+Compare two test runs to measure the impact of changes:
+
+1. Load profiles from both test runs
+2. Compare peak memory usage
+3. Compare memory growth rates
+4. Identify differences in allocation patterns
+5. Generate side-by-side comparison report with charts
+
+**Example:**
+```
+/memory-profile compare baseline with-caching
+```
+
+### /memory-profile collect
+
+Manually collect a single heap profile from the running operator-controller pod:
+
+1. Find the operator-controller pod
+2. Set up port forwarding to pprof endpoint
+3. Download heap profile
+4. Save to `./memory-profiles/manual/heap-[timestamp].pprof`
+
+**Example:**
+```
+/memory-profile collect
+```
+
+## Task Breakdown
+
+When you invoke this command, I will:
+
+1. **Setup Phase**
+   - Create `./memory-profiles/[test-name]` directory
+   - Verify `make test-experimental-e2e` is available
+   - Check kubectl access to the cluster
+
+2. **Collection Phase**
+   - Start the e2e test in background
+   - Monitor for pod readiness
+   - Set up port forwarding to pprof endpoint (port 6060)
+   - Collect heap profiles every 15 seconds
+   - Save profiles with sequential naming (heap0.pprof, heap1.pprof, ...)
+
+3. **Monitoring Phase**
+   - Track test progress
+   - Monitor profile file sizes for growth patterns
+   - Detect if test crashes or completes
+
+4. **Analysis Phase**
+   - Use `go tool pprof` to analyze profiles
+   - Extract key metrics:
+     - Peak memory usage
+     - Memory growth over time
+     - Top allocators
+     - OpenAPI-related allocations
+     - JSON deserialization overhead
+     - Informer/cache allocations
+
+5. **Reporting Phase**
+   - Generate markdown report with:
+     - Executive summary
+     - Memory timeline chart
+     - Top allocators table
+     - Allocation breakdown
+     - Recommendations for optimization
+
+## Configuration
+
+The plugin uses these defaults (customizable via environment variables):
+
+```bash
+# Namespace where operator-controller runs
+MEMORY_PROFILE_NAMESPACE=olmv1-system
+
+# Deployment name to monitor
+MEMORY_PROFILE_DEPLOYMENT=operator-controller-controller-manager
+
+# Label selector for pod
+MEMORY_PROFILE_POD_LABEL="app.kubernetes.io/name=operator-controller"
+
+# Pprof endpoint port
+MEMORY_PROFILE_PPROF_PORT=6060
+
+# Collection interval in seconds
+MEMORY_PROFILE_INTERVAL=15
+
+# Output directory base
+MEMORY_PROFILE_DIR=./memory-profiles
+```
+
+## Output Structure
+
+```
+memory-profiles/
+├── baseline/
+│   ├── heap0.pprof
+│   ├── heap1.pprof
+│   ├── ...
+│   ├── heap23.pprof
+│   ├── test.log
+│   └── analysis.md
+├── with-caching/
+│   ├── heap0.pprof
+│   ├── ...
+│   └── analysis.md
+└── comparisons/
+    └── baseline-vs-with-caching.md
+```
+
+## Tool Location
+
+The memory profiling scripts are located at:
+```
+hack/tools/memory-profiling/
+├── memory-profile.sh        # Main entry point
+├── run-profiled-test.sh     # Run test with profiling
+├── collect-profiles.sh      # Collect heap profiles
+├── analyze-profiles.sh      # Generate analysis
+├── compare-profiles.sh      # Compare two runs
+├── README.md               # Full documentation
+└── USAGE_EXAMPLES.md       # Real-world examples
+```
+
+You can run them directly:
+```bash
+./hack/tools/memory-profiling/memory-profile.sh run baseline
+./hack/tools/memory-profiling/memory-profile.sh analyze baseline
+./hack/tools/memory-profiling/memory-profile.sh compare baseline optimized
+```
+
+## Requirements
+
+- kubectl with access to the cluster
+- go tool pprof
+- make (for running tests)
+- curl (for fetching profiles)
+- Port 6060 available for forwarding
+
+## Example Workflow
+
+```bash
+# 1. Run baseline test with profiling
+/memory-profile run baseline
+
+# 2. Make code changes (e.g., add caching)
+# ... edit code ...
+
+# 3. Run new test with profiling
+/memory-profile run with-caching
+
+# 4. Compare results
+/memory-profile compare baseline with-caching
+
+# 5. Review the comparison report
+# Opens: memory-profiles/comparisons/baseline-vs-with-caching.md
+```
+
+## Notes
+
+- The test will run until completion or manual interruption (Ctrl+C)
+- Each heap profile is ~11-150KB depending on memory usage
+- Analysis requires all heap files to be present
+- Port forwarding runs in background and auto-cleans on exit
+- Reports are generated in markdown format for easy viewing

.claude/settings.local.json

@@ -0,0 +1,18 @@
+{
+  "permissions": {
+    "allow": [
+      "Read(//home/tshort/experimental-e2e-testing/**)",
+      "Bash(go tool pprof:*)",
+      "Bash(for i in 0 5 10 15 20 23)",
+      "Bash(do echo \"=== heap$i.pprof ===\")",
+      "Bash(done)",
+      "Bash(awk:*)",
+      "Bash(go doc:*)",
+      "Bash(go list:*)",
+      "Read(//home/tshort/go/pkg/mod/k8s.io/**)",
+      "Bash(go build:*)"
+    ],
+    "deny": [],
+    "ask": []
+  }
+}

.gitignore

@@ -38,9 +38,6 @@ vendor/
 \#*\#
 .\#*
 
-# AI temp files files
-.claude/
-
 # documentation website asset folder
 site
 
@@ -50,3 +47,6 @@ site
 
 # Temporary files and directories
 /test/regression/convert/testdata/tmp/*
+
+# Memory profiling artifacts
+memory-profiles/

hack/tools/memory-profiling/.gitignore

@@ -0,0 +1,3 @@
+# Ignore generated analysis files in example directories
+memory-profiles/*/analysis.md
+memory-profiles/comparisons/