Splunk Operator: Support updating VolumeClaimTemplate in StatefulSets

[ductrung-nguyen]

Please select the type of request

Enhancement

Tell us more

Describe the request

Currently, when you update the VolumeClaimTemplate in a Splunk CR - like changing storage class or increasing storage size - the operator doesn't pick up the change.
We didn't handle this case, maybe because K8s doesn't allow VCT modifications on existing StatefulSets (?!)

It'd be really helpful if the operator could handle these scenarios:

• Migrating to a different storage class (e.g., cloud provider deprecates an old one)
• Expanding storage capacity as indexes grow
• Fixing a misconfigured VCT without having to delete everything

The current workaround is manual: delete the StatefulSet with orphan cascade, update PVCs via script, wait for reconcile. Works but it's error-prone and takes time.

Expected behavior

The operator should detect VCT changes and handle them automatically:

For immutable changes (storage class, access modes, VCT added/removed):

• Delete the StatefulSet with orphan cascade (keeps pods running)
• Recreate the StatefulSet on next reconcile with the new VCT config
• Preserve existing PVCs to avoid data loss

For storage size increases:

• Patch the existing PVCs directly with the new size
• Relies on storage class supporting expansion (allowVolumeExpansion: true)

For storage size decreases:

• Detect and report that this requires manual intervention (shrinking PVCs isn't supported)

General:

• Publish K8s events so users can see what's happening

Splunk setup on K8S

This affects all StatefulSet-based components:

• IndexerCluster
• SearchHeadCluster
• ClusterManager
• LicenseManager
• Standalone
• etc.

Reproduction/Testing steps

1. Deploy an IndexerCluster with a storage class, for example: "ssd-standard"
2. Update the CR to use another storage class, for example: "ssd-premium"
3. Observe that the StatefulSet VCT remains unchanged
4. Check logs - no indication of the VCT mismatch

K8s environment

• K8s 1.20+
• Any cloud provider or on-prem with CSI drivers
• Storage class needs allowVolumeExpansion: true for resize to work

Proposed changes (optional)

The operator could:

1. Compare current vs desired VCTs after fetching the existing StatefulSet
2. For immutable changes -> orphan delete + recreate StatefulSet
3. For storage expansion -> patch PVCs directly

Additional context

This comes up often during migrations and capacity planning. Automating this would reduce manual work and lower the risk of mistakes in production.