test: enhance smoke test suite reliability and add comprehensive checks

Major improvements to the ClickHouse Helm chart test suite addressing
timing issues, adding new test scenarios, and improving robustness.

## Timing and Reliability Fixes

- Increase timeout for system.clusters verification (60s → 180s)
  Allows proper cluster topology propagation after deployment

- Add retry logic for ON CLUSTER queries during upgrades
  Wait for cluster config propagation before distributed operations

- Add service endpoints verification with retry (60s timeout)
  Ensure all endpoints register after pod creation

- Fix PVC verification to only check mounted volumes
  Prevents false failures from orphaned PVCs (reclaimPolicy: Retain)
  Affects: verify_clickhouse_pvc_size, verify_log_persistence, verify_keeper_storage

## New Test Features

- Add data survival verification for upgrade scenarios
  Creates test table before upgrade, verifies data after
  Smart detection: only runs for in-place upgrades (same nameOverride)
  Skips cluster replacement scenarios with informative note

- Add Keeper high availability chaos test
  Deletes one Keeper pod from 3-node quorum
  Verifies ClickHouse remains writable with 2/3 Keepers
  Runs automatically for replicated deployments

- Add metrics endpoint verification
  Tests operator metrics endpoint accessibility
  Validates Prometheus format output

## Robustness Improvements

- Implement get_ready_clickhouse_pod() helper
  Selects first Running pod instead of blindly using pods[0]
  Eliminates race conditions during pod initialization
  Applied to: verify_clickhouse_version, verify_user_connection, verify_extra_config_values

- Refactor get_cluster_topology() to use JSON parsing
  Changed from FORMAT TabSeparated to FORMAT JSON
  Adds proper error handling with json.JSONDecodeError
  Type-safe field access with .get() methods

- Refactor verify_metrics_endpoint() to use operator pod
  Access via localhost from operator pod (more reliable)
  Implement curl/wget fallback for tool availability
  Removes dependency on ClickHouse pod having network tools

- Add get_operator_pod() helper function
  Finds operator pod (excludes CHI pods)
  Used for metrics endpoint testing

## Warning Cleanup

- Fix 'level' setting warning in extraConfig parsing
  Add 'level' to skip list (nested logger element)

- Improve secrets verification messaging
  Change warning to informational for inline credentials
  Acknowledges inline config is valid for smoke tests

- Remove non-existent upgrade scenario from test list
  Cleanup UPGRADE_SCENARIOS list

## New Helper Functions

tests/steps/clickhouse.py:
- get_ready_clickhouse_pod() - Select first Running pod
- get_operator_pod() - Find operator pod
- create_test_data() - Create test data for upgrade verification
- verify_data_survival() - Verify data survived upgrade
- test_keeper_high_availability() - Chaos test for Keeper quorum
- verify_metrics_endpoint() - Test metrics endpoint (refactored)

tests/steps/kubernetes.py:
- delete_pod() - Delete a Kubernetes pod

## Files Modified

- tests/scenarios/smoke.py
- tests/steps/clickhouse.py
- tests/steps/deployment.py
- tests/steps/kubernetes.py

Fixes timing issues, adds production-ready test scenarios, and improves
overall test suite reliability and maintainability.

Author: Elmo33

tests/scenarios/smoke.py

@@ -4,6 +4,7 @@
 import tests.steps.kubernetes as kubernetes
 import tests.steps.minikube as minikube
 import tests.steps.helm as helm
+import tests.steps.clickhouse as clickhouse
 from tests.steps.deployment import HelmState
 
 
@@ -17,7 +18,6 @@
 
 UPGRADE_SCENARIOS = [
     ("fixtures/upgrade/initial.yaml", "fixtures/upgrade/upgrade.yaml"),
-    ("fixtures/upgrade/simple-initial.yaml", "fixtures/upgrade/complex-upgraded.yaml"),
 ]
 
 
@@ -56,6 +56,19 @@ def check_deployment(self, fixture_file, skip_external_keeper=True):
 
     with Then("verify deployment state"):
         state.verify_all(namespace=namespace)
+    
+    # Add Keeper HA test for replicated deployments with 3+ keepers
+    if "replicated" in fixture_name:
+        with And("test Keeper high availability (chaos test)"):
+            admin_password = state.clickhouse_config.get("defaultUser", {}).get("password", "")
+            clickhouse.test_keeper_high_availability(
+                namespace=namespace,
+                admin_password=admin_password
+            )
+    
+    # Verify metrics endpoint is accessible
+    with And("verify metrics endpoint"):
+        clickhouse.verify_metrics_endpoint(namespace=namespace)
 
     with Finally("cleanup deployment"):
         helm.uninstall(namespace=namespace, release_name=release_name)
@@ -92,6 +105,23 @@ def check_upgrade(self, initial_fixture, upgrade_fixture):
 
     with Then("verify initial deployment state"):
         initial_state.verify_all(namespace=namespace)
+    
+    # Only test data survival if nameOverride stays the same (in-place upgrade)
+    initial_name = initial_state.values.get("nameOverride", "")
+    upgrade_name = upgrade_state.values.get("nameOverride", "")
+    is_inplace_upgrade = (initial_name == upgrade_name)
+    
+    if is_inplace_upgrade:
+        with And("create test data for upgrade survival verification"):
+            admin_password = initial_state.clickhouse_config.get("defaultUser", {}).get("password", "")
+            clickhouse.create_test_data(
+                namespace=namespace,
+                admin_password=admin_password,
+                table_name="pre_upgrade_data",
+                test_value=f"upgrade_survival_{namespace}"
+            )
+    else:
+        note(f"Skipping data survival test: nameOverride changed from '{initial_name}' to '{upgrade_name}' (cluster replacement scenario)")
 
     with When("upgrade ClickHouse to new configuration"):
         helm.upgrade(
@@ -100,6 +130,21 @@ def check_upgrade(self, initial_fixture, upgrade_fixture):
 
     with Then("verify upgraded deployment state"):
         upgrade_state.verify_all(namespace=namespace)
+    
+    if is_inplace_upgrade:
+        with And("verify data survived the upgrade"):
+            admin_password = upgrade_state.clickhouse_config.get("defaultUser", {}).get("password", "")
+            clickhouse.verify_data_survival(
+                namespace=namespace,
+                admin_password=admin_password,
+                table_name="pre_upgrade_data",
+                expected_value=f"upgrade_survival_{namespace}"
+            )
+    else:
+        note(f"Data survival verification skipped for cluster replacement scenario")
+    
+    with And("verify metrics endpoint"):
+        clickhouse.verify_metrics_endpoint(namespace=namespace)
 
     with Finally("cleanup deployment"):
         helm.uninstall(namespace=namespace, release_name=release_name)
@@ -139,5 +184,5 @@ def feature(self):
         kubernetes.use_context(context_name="minikube")
 
     Feature(run=check_all_fixtures)
-    #
-    # Feature(run=check_all_upgrades)
+
+    Feature(run=check_all_upgrades)