chore: Update and organize README (#22)

Author: teeohhem

README.md

@@ -2,78 +2,147 @@
 
 Welcome to the official HyperDX Helm charts repository. This guide provides instructions on how to install, configure, and manage your HyperDX V2 deployment using Helm.
 
-## Prerequisites
+## Table of Contents
+
+- [Quick Start](#quick-start)
+- [Deployment Options](#deployment-options)
+  - [Full Stack (Default)](#full-stack-default)
+  - [External ClickHouse](#external-clickhouse)
+  - [External OTEL Collector](#external-otel-collector)
+  - [Minimal Deployment](#minimal-deployment)
+- [Cloud Deployment](#cloud-deployment)
+  - [Google Kubernetes Engine (GKE)](#google-kubernetes-engine-gke)
+  - [Amazon EKS](#amazon-eks)
+  - [Azure AKS](#azure-aks)
+- [Configuration](#configuration)
+  - [API Key Setup](#api-key-setup)
+  - [Task Configuration](#task-configuration)
+  - [Using Secrets](#using-secrets)
+- [Operations](#operations)
+  - [Upgrading](#upgrading-the-chart)
+  - [Uninstalling](#uninstalling-hyperdx)
+- [Troubleshooting](#troubleshooting)
+- [Contributing](#contributing)
+
+## Quick Start
+
+### Prerequisites
 
 - [Helm](https://helm.sh/) v3+
 - Kubernetes cluster (v1.20+ recommended)
 - `kubectl` configured to interact with your cluster
 
-## Adding the HyperDX Helm Repository
-
-First, add the HyperDX Helm repository:
+### Install HyperDX (Full Stack)
 
 ```sh
+# Add the HyperDX Helm repository
 helm repo add hyperdx https://hyperdxio.github.io/helm-charts
 helm repo update
+
+# Install with default values (includes ClickHouse, OTEL collector, MongoDB)
+helm install my-hyperdx hyperdx/hdx-oss-v2
+
+# Get the external IP (for cloud deployments)
+kubectl get services
+
+# Access the UI at http://<EXTERNAL-IP>:3000
 ```
 
-## Installing HyperDX
+**That's it!** HyperDX is now running with all components included.
 
-To install the HyperDX chart with default values:
+## Deployment Options
+
+### Full Stack (Default)
+
+By default, this Helm chart deploys the complete HyperDX stack including:
+- **HyperDX Application** (API, UI, and OpAMP server)
+- **ClickHouse** (for storing logs, traces, and metrics)
+- **OTEL Collector** (for receiving and processing telemetry data)
+- **MongoDB** (for application metadata)
+
+To install the full stack with default values:
 
 ```sh
 helm install my-hyperdx hyperdx/hdx-oss-v2
 ```
 
-You can customize settings by editing `values.yaml` or using `--set` flags.
+### External ClickHouse
 
-```sh
-helm install my-hyperdx hyperdx/hdx-oss-v2 -f values.yaml
-```
+If you have an existing ClickHouse cluster:
+
+```yaml
+# values-external-clickhouse.yaml
+clickhouse:
+  enabled: false  # Disable the built-in ClickHouse
 
-or
+otel:
+  clickhouseEndpoint: "tcp://your-clickhouse-server:9000"
+  clickhousePrometheusEndpoint: "http://your-clickhouse-server:9363"  # Optional
 
-```sh
-helm install my-hyperdx hyperdx/hdx-oss-v2 --set key=value
+hyperdx:
+  defaultConnections: |
+    [
+      {
+        "name": "External ClickHouse",
+        "host": "http://your-clickhouse-server:8123",
+        "port": 8123,
+        "username": "your-username",
+        "password": "your-password"
+      }
+    ]
 ```
 
-To retrieve the default values:
+### External OTEL Collector
 
-```sh
-helm show values hyperdx/hdx-oss-v2 > values.yaml
+If you have an existing OTEL collector setup:
+
+```yaml
+# values-external-otel.yaml
+otel:
+  enabled: false  # Disable the built-in OTEL collector
+
+hyperdx:
+  # Point to your external OTEL collector endpoint
+  otelExporterEndpoint: "http://your-otel-collector:4318"
 ```
 
-### Example Configuration
+### Minimal Deployment
+
+For organizations with existing infrastructure:
 
 ```yaml
-replicaCount: 2
-resources:
-  limits:
-    cpu: 500m
-    memory: 512Mi
-  requests:
-    cpu: 250m
-    memory: 256Mi
-ingress:
-  enabled: true
-  annotations:
-    kubernetes.io/ingress.class: nginx
-  hosts:
-    - host: hyperdx.example.com
-      paths:
-        - path: /
-          pathType: ImplementationSpecific
+# values-minimal.yaml
+clickhouse:
+  enabled: false
+
+otel:
+  enabled: false
+
+hyperdx:
+  otelExporterEndpoint: "http://your-otel-collector:4318"
+  defaultConnections: |
+    [
+      {
+        "name": "External ClickHouse",
+        "host": "http://your-clickhouse-server:8123",
+        "port": 8123,
+        "username": "your-username",
+        "password": "your-password"
+      }
+    ]
 ```
 
-## Post-Installation: Configuring Telemetry API Key
+## Configuration
+
+### API Key Setup
 
 After successfully deploying HyperDX, you'll need to configure the API key to enable the app's telemetry data collection:
 
 1. **Access your HyperDX instance** via the configured ingress or service endpoint
 2. **Log into the HyperDX dashboard** and navigate to Team settings to generate or retrieve your API key
 3. **Update your deployment** with the API key using one of the following methods:
 
-### Method 1: Update via Helm upgrade with values file
+#### Method 1: Update via Helm upgrade with values file
 
 Add the API key to your `values.yaml`:
 
@@ -88,7 +157,7 @@ Then upgrade your deployment:
 helm upgrade my-hyperdx hyperdx/hdx-oss-v2 -f values.yaml
 ```
 
-### Method 2: Update via Helm upgrade with --set flag
+#### Method 2: Update via Helm upgrade with --set flag
 
 ```sh
 helm upgrade my-hyperdx hyperdx/hdx-oss-v2 --set hyperdx.apiKey="your-api-key-here"
@@ -161,7 +230,9 @@ By default, there is one task in the chart setup as a cronjob, responsible for c
 | `tasks.checkAlerts.schedule` | Cron schedule for the check-alerts task | `*/1 * * * *` |
 | `tasks.checkAlerts.resources` | Resource requests and limits for the check-alerts task | See `values.yaml` |
 
-## Upgrading the Chart
+## Operations
+
+### Upgrading the Chart
 
 To upgrade to a newer version:
 
@@ -175,7 +246,7 @@ To check available chart versions:
 helm search repo hyperdx
 ```
 
-## Uninstalling HyperDX
+### Uninstalling HyperDX
 
 To remove the deployment:
 
@@ -185,6 +256,103 @@ helm uninstall my-hyperdx
 
 This will remove all resources associated with the release, but persistent data (if any) may remain.
 
+## Cloud Deployment
+
+### Google Kubernetes Engine (GKE)
+
+When deploying to GKE, you may need to override certain values due to cloud-specific networking behavior:
+
+#### LoadBalancer DNS Resolution Issue
+
+GKE's LoadBalancer service can cause internal DNS resolution issues where pod-to-pod communication resolves to external IPs instead of staying within the cluster network. This specifically affects the OTEL collector's connection to the OpAMP server.
+
+**Symptoms:**
+- OTEL collector logs showing "connection refused" errors with cluster IP addresses
+- OpAMP connection failures like: `dial tcp 34.118.227.30:4320: connect: connection refused`
+
+**Solution:**
+Use the fully qualified domain name (FQDN) for the OpAMP server URL:
+
+```bash
+helm install my-hyperdx hyperdx/hdx-oss-v2 \
+  --set hyperdx.appUrl="http://your-external-ip-or-domain.com" \
+  --set otel.opampServerUrl="http://my-hyperdx-hdx-oss-v2-app.default.svc.cluster.local:4320"
+```
+
+#### Other GKE Considerations
+
+```yaml
+# values-gke.yaml
+hyperdx:
+  appUrl: "http://34.123.61.99"  # Use your LoadBalancer external IP
+  
+otel:
+  opampServerUrl: "http://my-hyperdx-hdx-oss-v2-app.default.svc.cluster.local:4320"
+
+# Adjust for GKE pod networking if needed
+clickhouse:
+  config:
+    clusterCidrs:
+      - "10.8.0.0/16"  # GKE commonly uses this range
+      - "10.0.0.0/8"   # Fallback for other configurations
+```
+
+### Amazon EKS
+
+For EKS deployments, consider these common configurations:
+
+```yaml
+# values-eks.yaml
+hyperdx:
+  appUrl: "http://your-alb-domain.com"
+
+# EKS typically uses these pod CIDRs
+clickhouse:
+  config:
+    clusterCidrs:
+      - "192.168.0.0/16"
+      - "10.0.0.0/8"
+
+# Enable ingress for production
+hyperdx:
+  ingress:
+    enabled: true
+    host: "hyperdx.yourdomain.com"
+    tls:
+      enabled: true
+```
+
+### Azure AKS
+
+For AKS deployments:
+
+```yaml
+# values-aks.yaml
+hyperdx:
+  appUrl: "http://your-azure-lb.com"
+
+# AKS pod networking
+clickhouse:
+  config:
+    clusterCidrs:
+      - "10.244.0.0/16"  # Common AKS pod CIDR
+      - "10.0.0.0/8"
+```
+
+### Production Cloud Deployment Checklist
+
+- [ ] Configure proper `appUrl` with your external domain/IP
+- [ ] Set up ingress with TLS for HTTPS access
+- [ ] Override `otel.opampServerUrl` with FQDN if experiencing connection issues
+- [ ] Adjust `clickhouse.config.clusterCidrs` for your pod network CIDR
+- [ ] Configure persistent storage for production workloads
+- [ ] Set appropriate resource requests and limits
+- [ ] Enable monitoring and alerting
+
+### Browser Compatibility Notes
+
+For HTTP-only deployments (development/testing), some browsers may show crypto API errors due to secure context requirements. For production deployments, use HTTPS with proper TLS certificates through ingress configuration.
+
 ## Troubleshooting
 
 ### Checking Logs
@@ -193,6 +361,18 @@ This will remove all resources associated with the release, but persistent data
 kubectl logs -l app.kubernetes.io/name=hdx-oss-v2
 ```
 
+### OTEL Collector OpAMP Connection Issues
+
+If you see connection refused errors in OTEL collector logs:
+
+```sh
+# Check OTEL collector logs
+kubectl logs -l app=otel-collector
+
+# Verify service DNS resolution
+kubectl exec -it deployment/my-hyperdx-hdx-oss-v2-otel-collector -- nslookup my-hyperdx-hdx-oss-v2-app
+```
+
 ### Debugging a Failed Install
 
 ```sh