diff --git a/docs/migrate-premiumlrs-to-premiumv2lrs.md b/docs/migrate-premiumlrs-to-premiumv2lrs.md new file mode 100644 index 0000000000..9396ca9075 --- /dev/null +++ b/docs/migrate-premiumlrs-to-premiumv2lrs.md @@ -0,0 +1,837 @@ +# Premium_LRS → PremiumV2_LRS Migration Guide + +This guide explains how to use the migration scripts in `hack/` to move Azure Disk backed PVCs from Premium_LRS to PremiumV2_LRS. It now covers three supported modes (`inplace`, `dual`, and `attrclass` / VolumeAttributesClass), zone-aware preparation, prerequisites, validation steps, safety / rollback, cleanup, and troubleshooting. + +--- + +## 1. Goals & When to Use These Scripts + +These scripts automate: +- Zone-aware preparation and StorageClass creation for PremiumV2_LRS requirements +- Snapshotting existing Premium_LRS volumes +- Creating PremiumV2_LRS replacement PVCs / PVs +- (Optionally) staging intermediate CSI objects for in-tree disks +- Applying safety checks, labels, audit logging, and rollback metadata + +They are intended for controlled batches (not fire‑and‑forget across an entire large cluster without review). + +--- + +## 2. Modes Overview + +| Mode | Script | Summary | Pros | Trade‑offs / Cons | Typical Use | +|------|--------|---------|------|-------------------|-------------| +| In-place | `hack/premium-to-premiumv2-migrator-inplace.sh` | Deletes original PVC (keeping original PV), recreates same name PVC pointing to snapshot and PremiumV2 SC | Same name preserved; minimal object sprawl | Short window where PVC is absent; workload must be quiesced/detached; rollback relies on retained PV | Smaller batches, controlled maintenance windows | +| Dual (pv1→pv2) | `hack/premium-to-premiumv2-migrator-dualpvc.sh` | Creates intermediate CSI PV/PVC (if source was in-tree), snapshots, creates a *pv2* PVC (suffix), monitors migration events | Keeps original PVC around longer (reduced disruption); clearer staged artifacts | More objects (intermediate PV/PVC + target); higher cleanup burden; naming complexity | Migration where minimizing initial disruption matters or need visibility before switch | +| AttrClass (in-place attribute update) | `hack/premium-to-premiumv2-migrator-vac.sh` | (Optionally) converts in-tree PV to CSI same-name first, then applies a `VolumeAttributesClass` to mutate the disk SKU | No new pv2 PVC; minimal object churn; preserves PVC name; avoids creating SC variants | Requires cluster & driver support for VolumeAttributesClass; rollback of SKU change requires another class or snapshot-based restore | Clusters already CSI-enabled or ready to convert; desire lowest object churn | + +Recommendation: +1. **Always start with zone-aware preparation** (Step 4.1 below) +2. Pilot on a tiny subset using `inplace` (simpler) in a non-prod namespace. +3. If you need prolonged coexistence / observation, use `dual`. +4. If your cluster + Azure Disk CSI driver support `VolumeAttributesClass`, prefer `attrclass` for lowest object churn (especially when most PVs are already CSI). +5. Always label PVCs explicitly to opt them in (staged adoption). + +### 2.1 AttrClass Mode Details + +`hack/premium-to-premiumv2-migrator-vac.sh`: +- Ensures (or recreates if forced) a `VolumeAttributesClass` (default `azuredisk-premiumv2`) with `parameters.skuName=PremiumV2_LRS`. +- For CSI Premium_LRS PVCs: patches `spec.volumeAttributesClassName` only (no new PVC/PV). +- For in-tree azureDisk PVs: performs a one-time snapshot-based same-name CSI recreation (like a narrowed "inplace" convert) then patches attr class. +- Central monitoring loop watches both: + - PV `.spec.csi.volumeAttributes.skuName|skuname` flip to `PremiumV2_LRS`. + - `SKUMigration*` events (if emitted) similar to other modes. +- Rollback before SKU change: same as inplace (retained original PV + annotation / backup). After successful SKU mutation: must apply a different attr class pointing back to Premium_LRS (not auto-created) or restore from snapshot. + +Example: +```bash +kubectl label pvc data-app-a -n team-a disk.csi.azure.com/pv2migration=true +cd hack +./premium-to-premiumv2-migrator-vac.sh | tee run-attrclass-$(date +%Y%m%d-%H%M%S).log +``` + +Additional env (see section 5): +``` +ATTR_CLASS_NAME=azuredisk-premiumv2 +ATTR_CLASS_API_VERSION=storage.k8s.io/v1beta1 # or storage.k8s.io/v1 when GA +TARGET_SKU=PremiumV2_LRS +ATTR_CLASS_FORCE_RECREATE=false +``` + +--- + +## 3. Key Scripts & Shared Library + +- Zone-aware preparation: `hack/premium-to-premiumv2-zonal-aware-helper.sh` +- In-place runner: `hack/premium-to-premiumv2-migrator-inplace.sh` +- Dual runner: `hack/premium-to-premiumv2-migrator-dualpvc.sh` +- AttrClass runner: `hack/premium-to-premiumv2-migrator-vac.sh` +- Shared logic: `hack/lib-premiumv2-migration-common.sh` + +The common library provides: +- RBAC preflight +- Snapshot class creation (`ensure_snapshot_class`) +- StorageClass variant creation (`-pv1`, `-pv2`) +- Intermediate PV/PVC creation for in-tree disks +- Migration event parsing (`SKUMigration*`) +- Cleanup report & audit logging +- Base64 backup and rollback metadata (in-place mode) + +--- + +## 4. Zone-Aware Preparation (REQUIRED First Step) + +**⚠️ CRITICAL: Zone Preparation Must Be Run Before Migration** + +PremiumV2_LRS disks have strict zone requirements and must be provisioned in the same availability zone as the target workloads. The zone-aware preparation helper **must be run before any migration script** to ensure proper zone-specific StorageClass creation and PVC annotation. + +### 4.1 Zone-Aware Migration Helper Script + +**Script**: `hack/premium-to-premiumv2-zonal-aware-helper.sh` + +**Purpose**: +- Automatically detects zones for existing PVCs using multiple detection methods +- Creates zone-specific StorageClasses with proper topology constraints +- Annotates PVCs with the appropriate zone-specific StorageClass for migration +- Preserves all original StorageClass properties while adding zone constraints + +### 4.2 Zone Detection Logic (Automatic) + +The script uses a three-tier detection approach: + +1. **StorageClass allowedTopologies** - If single zone constraint exists, uses it +2. **PV nodeAffinity** - Extracts zone from PV's node affinity requirements +3. **Zone mapping file** - Falls back to user-provided disk-to-zone mappings + +### 4.3 Usage + +#### Step 1: Generate Zone Mapping Template (if needed) +```bash +cd hack +./premium-to-premiumv2-zonal-aware-helper.sh generate-template +``` + +This creates `disk-zone-mapping-template.txt` with entries for all PVCs marked for migration: +``` +# Azure Disk Zone Mapping File +# Format: = +# Example zones: uksouth-1, uksouth-2, uksouth-3 + +# PVC: default/my-app-data, PV: pv-abc123 +/subscriptions/12345678-1234-1234-1234-123456789012/resourceGroups/myRG/providers/Microsoft.Compute/disks/myDisk1=uksouth-1 +``` + +#### Step 2: Edit Zone Mapping (if automatic detection insufficient) +```bash +# Edit the generated file to specify correct zones +vim disk-zone-mapping-template.txt + +# Rename to active mapping file +mv disk-zone-mapping-template.txt disk-zone-mapping.txt +``` + +#### Step 3: Run Zone Preparation +```bash +cd hack + +# Label PVCs for migration first +kubectl label pvc data-app-a -n team-a disk.csi.azure.com/pv2migration=true +kubectl label pvc data-app-b -n team-b disk.csi.azure.com/pv2migration=true + +# Run zone preparation (processes all labeled PVCs) +./premium-to-premiumv2-zonal-aware-helper.sh process + +# Or with zone mapping file +ZONE_MAPPING_FILE=disk-zone-mapping.txt ./premium-to-premiumv2-zonal-aware-helper.sh process +``` + +### 4.4 What the Zone Helper Creates + +For each original StorageClass (e.g., `managed-premium`), the script creates zone-specific variants: + +```yaml +# Example: managed-premium-uksouth-1 +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: managed-premium-uksouth-1 + labels: + disk.csi.azure.com/created-by: azuredisk-pv1-to-pv2-migrator + # All original labels preserved + # All original annotations preserved +provisioner: disk.csi.azure.com +parameters: + skuName: PremiumV2_LRS + cachingMode: None + # All compatible original parameters preserved +reclaimPolicy: Retain # Preserved from original +allowVolumeExpansion: true # Preserved from original +volumeBindingMode: WaitForFirstConsumer # Set for zone awareness +allowedTopologies: + - matchLabelExpressions: + - key: topology.kubernetes.io/zone + values: ["uksouth-1"] # Zone-specific constraint +# Original mountOptions preserved if present +``` + +### 4.5 PVC Annotations Added + +The script annotates each processed PVC: +```yaml +metadata: + annotations: + disk.csi.azure.com/migration-sourcesc: "managed-premium-uksouth-1" +``` + +This annotation tells the migration scripts which zone-specific StorageClass to use instead of creating generic variants. + +### 4.6 Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `ZONE_MAPPING_FILE` | `disk-zone-mapping.txt` | Path to disk-zone mapping file | +| `MIGRATION_LABEL` | `disk.csi.azure.com/pv2migration=true` | PVC label selector (same as migration scripts) | +| `NAMESPACE` | (empty) | Limit to specific namespace | +| `MAX_PVCS` | `50` | Maximum PVCs to process in one run | +| `ZONE_SC_ANNOTATION_KEY` | `disk.csi.azure.com/migration-sourcesc` | Annotation key for zone-specific StorageClass | + +### 4.7 Verification + +After running zone preparation, verify the results: + +```bash +# Check created zone-specific StorageClasses +kubectl get sc | grep "disk.csi.azure.com/created-by=azuredisk-pv1-to-pv2-migrator" + +# Check PVC annotations +kubectl get pvc -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,ZONE-SC:.metadata.annotations.disk\.csi\.azure\.com/migration-sourcesc" + +# Verify zone topology in created StorageClasses +kubectl get sc managed-premium-uksouth-1 -o yaml | grep -A 5 allowedTopologies +``` + +### 4.8 Error Handling & Skipping + +The script will: +- **Skip** PVCs that already have the zone annotation (safe to re-run) +- **Error** if zone cannot be determined from any detection method +- **Warn** about PVCs without bound PVs or missing StorageClass +- **Validate** that created StorageClasses match expected configuration + +### 4.9 Integration with Migration Scripts + +**IMPORTANT**: Run the zone preparation **before** any migration script if zonal information is not present in StorageClass or PersistentVolume: + +```bash +# 1. FIRST: Label PVCs for migration +kubectl label pvc data-app-a -n team-a disk.csi.azure.com/pv2migration=true + +# 2. SECOND: Run zone preparation +./premium-to-premiumv2-zonal-aware-helper.sh process + +# 3. THIRD: Run your chosen migration script +./premium-to-premiumv2-migrator-inplace.sh +# OR +./premium-to-premiumv2-migrator-dualpvc.sh +# OR +./premium-to-premiumv2-migrator-vac.sh +``` + +The migration scripts automatically detect and use the zone-specific StorageClass annotation, ensuring PremiumV2 volumes are provisioned in the correct zones. + +--- + +## 5. Label-Driven Selection + +PVCs are selected by (default): +``` +disk.csi.azure.com/pv2migration=true +``` +Environment variable: +``` +MIGRATION_LABEL="disk.csi.azure.com/pv2migration=true" +``` +Change the label (or add additional selectors externally) to control scope. Only labeled, *Bound* PVCs under the size threshold are processed. + +--- + +## 6. Environment Variables (Key Tunables) + +| Variable | Default | Meaning / Guidance | +|----------|---------|--------------------| +| `MIG_SUFFIX` | `csi` | Suffix for intermediate & pv2 naming (`-[-pv2]`) – keep stable. | +| `MAX_PVCS` | `50` | Upper bound per run; script truncates beyond this to avoid huge batches. | +| `MAX_PVC_CAPACITY_GIB` | `2048` | Skip PVCs at or above this (safety / PremiumV2 size comfort). | +| `WAIT_FOR_WORKLOAD` | `true` | If true, tries to ensure detachment before migration (in-place more critical). | +| `WORKLOAD_DETACH_TIMEOUT_MINUTES` | `5` | >0 to enforce a max wait for volume detach. | +| `BIND_TIMEOUT_SECONDS` | `60` | Wait for new pv2 PVC binding. | +| `MONITOR_TIMEOUT_MINUTES` | `300` | Global migration monitor upper bound. | +| `MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES` | `3` | Force a migration-inprogress label on original PV if events lag (in both modes). | +| `BACKUP_ORIGINAL_PVC` | `true` | In-place only: store raw YAML (under `PVC_BACKUP_DIR`). | +| `PVC_BACKUP_DIR` | `pvc-backups` | Backup directory root. | +| `ROLLBACK_ON_TIMEOUT` | `true` (if set by you) | In-place: attempt rollback automatically on bind timeout / monitor timeout. | +| `SNAPSHOT_CLASS` | `csi-azuredisk-vsc` | Snapshot class name (created if missing). | +| `SNAPSHOT_MAX_AGE_SECONDS` | `7200` | Reuse snapshot if younger unless stale logic triggers. | +| `SNAPSHOT_RECREATE_ON_STALE` | `false` | If `true`, stale snapshot is deleted and recreated. | +| `MIGRATION_LABEL` | see above | PVC selection. | +| `AUDIT_ENABLE` | `true` | Enable audit log lines. | +| `AUDIT_LOG_FILE` | `pv1-pv2-migration-audit.log` | Rolling append log file. | +| `ATTR_CLASS_NAME` | `azuredisk-premiumv2` | (AttrClass mode) Name of VolumeAttributesClass to apply. | +| `ATTR_CLASS_API_VERSION` | `storage.k8s.io/v1beta1` | API version for VolumeAttributesClass (adjust if GA). | +| `TARGET_SKU` | `PremiumV2_LRS` | Target skuName parameter for the VolumeAttributesClass. | +| `ATTR_CLASS_FORCE_RECREATE` | `false` | Recreate the attr class each run. | +| `PV_POLL_INTERVAL_SECONDS` | `10` | (AttrClass) Poll interval for sku check. | +| `SKU_UPDATE_TIMEOUT_MINUTES` | `60` | (AttrClass optional blocking helper) Per-PVC sku update wait if used directly. | +| `ONE_SC_FOR_MULTIPLE_ZONES` | `true` | When true, creates single StorageClass for multiple zones; when false, creates zone-specific StorageClasses. | +| `ZONE_SC_ANNOTATION_KEY` | `disk.csi.azure.com/migration-sourcesc` | Annotation key for zone-specific StorageClass reference. | + +(See top of `lib-premiumv2-migration-common.sh` for the complete list.) + +--- + +## 7. Prerequisites & Validation Checklist (Updated) + +Before running migration scripts: + +1. **⚠️ MANDATORY: Run Zone-Aware Preparation** (see Section 4) + - **MUST** be completed before any migration script + - Creates zone-specific StorageClasses with proper topology constraints + - Annotates PVCs with appropriate zone-specific StorageClass names + +2. **RBAC**: Ensure your principal can `get/list/create/patch/delete` PV/PVC/Snapshot/SC as required. Script will abort if critical verbs fail. + +3. **Quota**: Check PremiumV2 disk quotas in target subscription/region (script does NOT enforce). + +4. **StorageClasses**: Confirm original SC(s) are Premium_LRS (cachingMode=none, no unsupported encryption combos). + +5. **Zone Topology Verification**: + + After running the zone-aware helper (Section 4), verify that zone-specific StorageClasses were created correctly: + + ```bash + # Check that zone-specific StorageClasses exist + kubectl get sc | grep "disk.csi.azure.com/created-by=azuredisk-pv1-to-pv2-migrator" + + # Verify PVC annotations are in place + kubectl get pvc -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,ZONE-SC:.metadata.annotations.disk\.csi\.azure\.com/migration-sourcesc" + + # Confirm zone constraints in created StorageClasses + kubectl get sc -o yaml | grep -A 10 allowedTopologies + ``` + +6. **Workload readiness**: Plan for pods referencing target PVCs to be idle / safe to pause if using in-place. + +7. **Snapshot CRDs**: Ensure `VolumeSnapshot` CRDs installed (the script creates a class if absent). + +8. **Label small test set**: + ```bash + kubectl label pvc data-app-a -n team-a disk.csi.azure.com/pv2migration=true + ``` + +9. **Dry run *logic* (syntax & preflight only)**: + ```bash + bash -n hack/premium-to-premiumv2-zonal-aware-helper.sh + bash -n hack/premium-to-premiumv2-migrator-inplace.sh + bash -n hack/premium-to-premiumv2-migrator-dualpvc.sh + bash -n hack/premium-to-premiumv2-migrator-vac.sh + ``` + +10. **Optional**: Run with a deliberately empty label selector to validate preflight (set `MIGRATION_LABEL="doesnotexist=true"` temporarily). + +**Complete Workflow Order**: +```bash +# 1. Label PVCs for migration +kubectl label pvc data-app-a -n team-a disk.csi.azure.com/pv2migration=true + +# 2. Run zone-aware preparation (MANDATORY FIRST) +cd hack +./premium-to-premiumv2-zonal-aware-helper.sh process + +# 3. Verify zone preparation results +kubectl get pvc data-app-a -n team-a -o jsonpath='{.metadata.annotations.disk\.csi\.azure\.com/migration-sourcesc}' + +# 4. Run chosen migration script +./premium-to-premiumv2-migrator-inplace.sh # or dual/vac +``` + +--- + +## 8. Running the Scripts + +Change to repository root or `hack/` directory. + +**Zone preparation (MANDATORY FIRST STEP IF ZONAL INFORMATION NOT AVAILABLE IN SC/PV)**: +```bash +cd hack +# Run zone preparation for all labeled PVCs +./premium-to-premiumv2-zonal-aware-helper.sh process 2>&1 | tee zone-prep-$(date +%Y%m%d-%H%M%S).log +``` + +**In-place example**: +```bash +cd hack +# Limit to first few PVCs, raise verbosity by tee'ing output +MAX_PVCS=5 MIG_SUFFIX=csi \ + ./premium-to-premiumv2-migrator-inplace.sh 2>&1 | tee run-inplace-$(date +%Y%m%d-%H%M%S).log +``` + +**Dual example**: +```bash +cd hack +MAX_PVCS=5 MIG_SUFFIX=csi \ + ./premium-to-premiumv2-migrator-dualpvc.sh 2>&1 | tee run-dual-$(date +%Y%m%d-%H%M%S).log +``` + +**AttrClass example**: +```bash +cd hack +MAX_PVCS=5 ATTR_CLASS_NAME=azuredisk-premiumv2 \ + ./premium-to-premiumv2-migrator-vac.sh 2>&1 | tee run-attrclass-$(date +%Y%m%d-%H%M%S).log +``` + +Important runtime phases (all migration modes): +1. Pre-req scan (size, SC parameters, binding). +2. RBAC preflight. +3. Zone-aware StorageClass detection (looks for PVC annotations from zone helper). +4. StorageClass variant creation (uses zone-specific SC if annotated, fallback to generic variants). +5. Snapshot creation or reuse. +6. PVC/PV creation (intermediate for in-tree in dual; immediate replacement in inplace). +7. Bind wait & event monitoring (`SKUMigrationStarted/Progress/Completed`). +8. Labeling original PVC (`migration-done=true`) on completion. +9. Summary + cleanup report + audit summary. + +--- + +## 9. Rollback (In-place Mode) + +Each migrated PVC stores: +- Base64 annotation with sanitized pre-migration spec (`rollback-pvc-yaml`) +- Original PV name annotation + +Automatic rollback triggers: +- Bind timeout (rc=2) if `ROLLBACK_ON_TIMEOUT=true` +- Monitor timeout per PVC (also loops through for rollback) + +### Manual rollback steps + +Primary (annotation-based) method: +```bash +# 1. Fetch encoded sanitized spec from annotation +enc=$(kubectl get pvc mypvc -n ns -o jsonpath="{.metadata.annotations['disk.csi.azure.com/rollback-pvc-yaml']}") +# 2. Decode to a file +echo "$enc" | base64 -d > original.yaml + +# 3. Delete current pv2 PVC (it references the PremiumV2 volume) +kubectl delete pvc mypvc -n ns + +# 4. Clear claimRef on original PV (name stored in annotation) +origpv=$(kubectl get pvc mypvc -n ns -o jsonpath="{.metadata.annotations['disk.csi.azure.com/rollback-orig-pv']}") +kubectl patch pv "$origpv" -p '{"spec":{"claimRef":null}}' + +# 5. Recreate original PVC from saved spec +kubectl apply -f original.yaml +``` + +#### Alternative if annotation is missing (e.g., pv2 PVC already removed or annotations pruned) + +If the pv2 PVC (with rollback annotations) was deleted before you captured the encoded spec, you can fall back to the raw backup taken when `BACKUP_ORIGINAL_PVC=true`. + +1. Locate the most recent backup file: + ``` + ls -1 pvc-backups// | grep '^-.*\.yaml$' | sort | tail -n1 + ``` + Example: + ``` + latest_backup=$(ls -1 pvc-backups/ns-example/ | grep '^mypvc-.*\.yaml$' | sort | tail -n1) + cp "pvc-backups/ns-example/$latest_backup" restore.yaml + ``` + +2. Inspect & sanitize if needed (the backup is the full original object; it may still contain fields you don’t want to apply directly in rare cluster/version mismatches): + - Remove `status:` block (if present). + - Ensure metadata does NOT include: `resourceVersion`, `uid`, `creationTimestamp`, `managedFields`, `finalizers`. + Quick one-liner to strip common runtime fields: + ```bash + yq 'del(.status, .metadata.uid, .metadata.resourceVersion, .metadata.managedFields, .metadata.creationTimestamp, .metadata.finalizers)' restore.yaml > restore.clean.yaml + mv restore.clean.yaml restore.yaml + ``` + +3. Get the original PV name from the backup spec (or from audit log lines): + ```bash + origpv=$(yq -r '.spec.volumeName // ""' restore.yaml) + [ -z "$origpv" ] && echo "Could not determine original PV name" && exit 1 + ``` + +4. Ensure the PV reclaim policy is still `Retain` (script should have patched it earlier): + ```bash + kubectl get pv "$origpv" -o jsonpath='{.spec.persistentVolumeReclaimPolicy}'; echo + ``` + +5. Clear the `claimRef` on the original PV so it can rebind: + ```bash + kubectl patch pv "$origpv" -p '{"spec":{"claimRef":null}}' + ``` + +6. (Optional) Double-check no pv2 PVC with the same name still exists: + ```bash + kubectl get pvc mypvc -n ns && echo "A PVC named mypvc still exists; delete it first" && exit 1 || true + ``` + +7. Recreate the original PVC: + ```bash + kubectl apply -f restore.yaml + ``` + +8. Wait for binding: + ```bash + kubectl wait --for=jsonpath='{.status.phase}=Bound' pvc/mypvc -n ns --timeout=5m + ``` + +9. Validate pod/workload mounts (if you restart workloads): + ```bash + kubectl describe pvc mypvc -n ns | grep -E 'Volume|Status' + ``` + +Notes & cautions: +- If multiple backups exist, always choose the latest timestamped file unless you have a reason to revert further back. +- If the original PV was manually modified post-migration (uncommon), verify it still points to the original disk resource. +- Audit log file (`pv1-pv2-migration-audit.log`) can help correlate the PV name and timing if the backup spec is ambiguous. + +Validation after restore: +```bash +# Confirm PVC bound to original PV (not a PremiumV2 one) +kubectl get pvc mypvc -n ns -o jsonpath='{.spec.volumeName}'; echo +kubectl get pv "$origpv" -o jsonpath='{.spec.csi.driver}'; echo # likely empty for in-tree +kubectl get pv "$origpv" -o jsonpath='{.spec.azureDisk.diskURI}'; echo +``` + +If you also plan to retry the migration later: +- Re-apply the migration label to the restored PVC. +- Ensure the reclaim policy is still `Retain`. +- Remove any stale snapshot (or keep it if you want faster retry and it’s recent). + +--- + +## 10. Interpreting Output + +Key log prefixes: +- `[OK]` – success milestones +- `[WARN]` – transient issues, retries, or manual review needed +- `[ERROR]` – abort conditions + +Events: +- The script inspects `kubectl get events` for `SKUMigration*` reasons to drive progress/state. + +Audit log (`pv1-pv2-migration-audit.log`): +- Pipe-delimited lines: `timestamp|action|kind|namespace|name|revertCommand|extra` +- At the end, a "best-effort revert command summary" is printed for quick manual cleanup / rollback. + +Backups (in-place mode): +- Raw PVC YAMLs stored under `pvc-backups//`. +- Keep this directory until you are fully confident in the migration; then archive or delete. + +--- + +## 10. Cleanup Report & Post-Migration Tasks + +Final section (`print_migration_cleanup_report`) highlights: +- Intermediate PV/PVC candidates (dual mode) +- Snapshots safe to remove +- Released PremiumV2 PVs that still reference the claimRef (especially leftover post rollback or naming transitions) +- Original PV references (in dual) that may no longer be needed once you fully switch workloads + +Actions to consider after verifying data integrity: +```bash +# Delete intermediate artifacts (dual mode example) +kubectl delete pvc -csi -n # if listed & unused +kubectl delete pv -csi # intermediate PV +kubectl delete volumesnapshot ss--csi-pv # snapshot if not needed + +# Delete released PVs (after ensuring data & rollback not required) +kubectl delete pv +``` + +Ensure workloads are successfully using the new PremiumV2 PVC: +```bash +kubectl describe pvc mypvc -n ns | grep -i "StorageClass" +kubectl get pv $(kubectl get pvc mypvc -n ns -o jsonpath='{.spec.volumeName}') -o jsonpath='{.spec.csi.volumeAttributes.skuName}' ; echo +``` +Expect `PremiumV2_LRS` (or `skuname` attribute). + +--- + +## 10.1 Original Premium_LRS Disk Lifecycle & Safe Deletion + +During migration the scripts intentionally set or preserve `persistentVolumeReclaimPolicy: Retain` on the *original* Premium_LRS PV. This ensures: +- Rollback remains possible (data & PV object remain). +- The underlying managed disk in Azure is **not** deleted automatically when the PVC is deleted or replaced. + +Implications: +- After you confirm migration success and no rollback need, those original Premium_LRS disks continue to incur cost until explicitly removed. +- Simply deleting the *PVC* (in dual mode, or an old intermediate PVC) does NOT delete a retained PV’s backing disk. +- Deleting a PV with reclaimPolicy=Retain only removes the Kubernetes PV object; the Azure managed disk still survives (becomes an “unattached” disk visible in your resource group). + +Recommended deletion workflow (when you are 100% certain rollback is unnecessary): + +1. Identify the original PV(s): + ```bash + # For a migrated PVC + kubectl get pvc mypvc -n ns -o jsonpath='{.metadata.annotations["disk.csi.azure.com/rollback-orig-pv"]}'; echo + # Or from audit log or cleanup report + ``` + +2. (Optional) Final verification: + - Mount / read data from the new PremiumV2 PVC. + - Confirm application-level integrity checks (db consistency, file checksums). + +3. Decide whether you want Kubernetes to clean up the disk automatically or to delete manually: + - Automatic deletion path: + a. Patch reclaim policy to Delete. + b. Delete the PV (Kubernetes will then ask the in-tree/CSI provisioner to release & remove the Azure disk—verify it actually does for your scenario; in-tree azureDisk with Retain→Delete patch + PV deletion should delete the managed disk). + - Manual deletion path: + a. Delete the PV (still Retain). + b. Locate and delete the managed disk via Azure CLI / Portal. + +4. Automatic deletion path commands: + ```bash + PV=orig-pv-name + # Patch reclaim policy + kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}' + # Double-check + kubectl get pv "$PV" -o jsonpath='{.spec.persistentVolumeReclaimPolicy}'; echo + # Delete PV (this should trigger disk deletion) + kubectl delete pv "$PV" + ``` + +5. Manual deletion path commands: + ```bash + PV=orig-pv-name + # Delete PV object but leave disk (policy still Retain) + kubectl delete pv "$PV" + # Find disk URI (from prior audit log or: + # kubectl get pv $PV -o jsonpath='{.spec.azureDisk.diskURI}' + # If you noted it earlier, delete with Azure CLI: + az disk delete --ids "" --yes + ``` + +6. Post-deletion validation: + ```bash + # PV gone + kubectl get pv "$PV" || echo "PV deleted" + # (If automatic) Ensure disk no longer appears: + az disk show --ids "" || echo "Disk deleted" + ``` + +Cautions: +- Never patch to Delete *before* you are certain rollback is unnecessary. +- If you batch-delete PVs after mass migration, maintain an inventory (audit log + cleanup report) so you can reconcile against Azure resource group disks and ensure no unexpected survivors or accidental deletions. +- For Released PVs (phase=Released, reclaimPolicy=Retain) that refer to a PremiumV2 volume you’ve decided to discard: + - Same procedure applies: patch to Delete then delete PV, or delete PV + disk manually. + +Quick identification of retained original PVs older than N days (example 7): +```bash +kubectl get pv -o json \ + | jq -r ' + [.items[] + | select(.spec.persistentVolumeReclaimPolicy=="Retain") + | select(.status.phase=="Released" or .status.phase=="Available") + | {name:.metadata.name, creation:.metadata.creationTimestamp} + ] + | map(select(((now - ( .creation | fromdateiso8601 ))/86400) > 7)) + | .[] + | "\(.name)\t\(.creation)"' +``` + +Summary: +- Migration scripts keep you safe by retaining source disks. +- You must perform an explicit, audited cleanup pass to avoid ongoing Premium_LRS cost. +- Choose Delete vs manual removal consciously, and record what was deleted (append to your audit log or change tracking system). + +--- + +## 11. Recommended Validation Before Scaling Up + +1. Single test PVC end-to-end (snapshot + bind + event completion + cleanup). +2. Run read/write workload (e.g., fill a file, checksum before & after). +3. Confirm no unexpected resizing or mode changes. +4. Validate rollback path (simulate a forced rollback on a dummy PVC). +5. Check storage account / disk inventory (ensure no orphaned or unexpectedly deleted disks). +6. Measure migration time per PVC to forecast batch durations. + +--- + +## 12. Scaling Strategy + +- Start with `MAX_PVCS=5` (pilot) +- Increase to 20–30 (during low traffic) +- Always review cleanup report before next batch +- Keep a gap (e.g., 15–30 minutes) between batches to allow asynchronous controller reconciliation and quota feedback. + +--- + +## 13. Troubleshooting + +| Symptom | Likely Cause | Action | +|---------|--------------|--------| +| Syntax error near `)` | Hidden character or earlier unclosed heredoc | Run `bash -n` and retype suspicious line; normalize line endings | +| Snapshot never ready | Snapshot class or CSI driver issues | `kubectl describe volumesnapshot` and underlying `VolumeSnapshotContent` | +| PVC stuck Pending | Missing `*-pv2` StorageClass or quota | Verify SC creation logs and `kubectl describe pvc` | +| No `SKUMigration*` events | Controller not emitting or watch delay | Force in-progress label (script auto after threshold) | +| Released PV leftovers | Rollback or partial batch | Confirm not needed → delete PV | +| Rollback failed to rebind | claimRef not cleared or PV reclaimPolicy=Delete | Ensure reclaimPolicy changed to Retain earlier | +| AttrClass PVC never flips sku | Driver / cluster lacks VolumeAttributesClass update support | Confirm driver version & feature gate; inspect PV `.spec.csi.volumeAttributes` | +| AttrClass run shows no events | Controller not emitting `SKUMigration*` | Rely on sku attribute polling; consider driver log inspection | +| AttrClass rollback after sku change | SKU already mutated on disk | Apply alternate attr class (Premium_LRS) or snapshot restore | +| Zone preparation fails | Missing zone mapping or detection failure | Check zone mapping file, verify PV node affinity, use `generate-template` command | +| Zone-specific SC not used | Missing zone annotation or incorrect annotation key | Verify PVC has `disk.csi.azure.com/migration-sourcesc` annotation | + +--- + +## 14. Safety & Data Integrity Notes + +- The script relies on *snapshots*; ensure snapshot storage is regionally redundant as per policy. +- Retain raw PVC backups and audit logs until a post-migration verification window closes. +- Consider taking an application-layer backup (database dump, etc.) before large waves. + +--- + +## 15. Example End-to-End (In-Place, Small Batch) + +```bash +# Label a target PVC +kubectl label pvc data-app-a -n team-a disk.csi.azure.com/pv2migration=true + +# Run zone preparation (MANDATORY FIRST IF ZONAL INFORMATION NOT AVAILABLE IN PV/SC) +cd hack +./premium-to-premiumv2-zonal-aware-helper.sh process + +# Verify zone preparation +kubectl get pvc data-app-a -n team-a -o jsonpath='{.metadata.annotations.disk\.csi\.azure\.com/migration-sourcesc}'; echo + +# Run migration +MAX_PVCS=1 BIND_TIMEOUT_SECONDS=120 MONITOR_TIMEOUT_MINUTES=60 \ + ./premium-to-premiumv2-migrator-inplace.sh | tee mig-inplace-a.log + +# Inspect cleanup report in output +# Verify migration: +kubectl get pvc data-app-a -n team-a -o wide +kubectl describe pv $(kubectl get pvc data-app-a -n team-a -o jsonpath='{.spec.volumeName}') | grep -i sku +``` + +### 15.1 Example AttrClass (CSI-native PVC) +```bash +kubectl label pvc data-app-b -n team-b disk.csi.azure.com/pv2migration=true + +# Run zone preparation first (IF ZONAL INFORMATION NOT AVAILABLE IN PV/SC) +cd hack +./premium-to-premiumv2-zonal-aware-helper.sh process + +# Run AttrClass migration +./premium-to-premiumv2-migrator-vac.sh | tee mig-attrclass-b.log + +# Verify: +kubectl get pvc data-app-b -n team-b -o wide +pv=$(kubectl get pvc data-app-b -n team-b -o jsonpath='{.spec.volumeName}') +kubectl get pv "$pv" -o jsonpath='{.spec.csi.volumeAttributes.skuName}'; echo +``` + +--- + +## 16. After Everything Looks Good + +- Archive `pv1-pv2-migration-audit.log` +- Archive or prune `pvc-backups/` +- Remove migration label from PVCs (optional): + ``` + kubectl label pvc data-app-a -n team-a disk.csi.azure.com/pv2migration- + ``` +- Enforce a policy (OPA / Kyverno) to prefer PremiumV2 SCs for new claims. + +--- + +## 17. Limitations / Non-Goals + +- Does not auto-resize or convert AccessModes. +- Does not verify disk-level performance metrics. +- Does not handle non-Premium_LRS source SKUs (skips them). +- Does not auto-delete original PVs; leaves operator in control. + +--- + +## 18. Contributing / Extending + +Ideas: +- Add a dry-run (`NO_MUTATE=true`) that stops before creations. +- Metrics export (JSON summary). +- Parallelization with rate limits. + +Open a PR if you extend; keep safety-first defaults. + +--- + +## 19. Quick Variable Cheat Sheet + +``` +# Common overrides: +export MAX_PVCS=10 +export MIG_SUFFIX=csi +export MONITOR_TIMEOUT_MINUTES=180 +export BIND_TIMEOUT_SECONDS=120 +export WORKLOAD_DETACH_TIMEOUT_MINUTES=15 +export BACKUP_ORIGINAL_PVC=true +export ROLLBACK_ON_TIMEOUT=true +export ATTR_CLASS_NAME=azuredisk-premiumv2 +export TARGET_SKU=PremiumV2_LRS +export ATTR_CLASS_FORCE_RECREATE=false +export PV_POLL_INTERVAL_SECONDS=10 +export MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES=3 +export ONE_SC_FOR_MULTIPLE_ZONES=true +export ZONE_SC_ANNOTATION_KEY=disk.csi.azure.com/migration-sourcesc +``` + +--- + +## 20. Final Checklist (Per Batch) + +1. Label set? (Only intended PVCs show up) +2. `bash -n` passes +3. **Zone preparation complete?** (MANDATORY) +4. Run script → watch logs until summary +5. Review cleanup report +6. Verify data & app workload on PremiumV2 (PV attributes or events) +7. (Dual/In-place) Cleanup intermediate / snapshot artifacts +8. (AttrClass) Confirm attr class applied (PVC.spec.volumeAttributesClassName) & PV sku updated +9. Archive audit + backups +10. Proceed to next batch + +--- + +## 21. Zone-Aware Migration Complete Example + +```bash +# Complete end-to-end example with zone-aware preparation + +# 1. Label target PVCs +kubectl label pvc data-app-a -n team-a disk.csi.azure.com/pv2migration=true +kubectl label pvc data-app-b -n team-b disk.csi.azure.com/pv2migration=true + +# 2. MANDATORY: Run zone-aware preparation (IF ZONAL INFORMATION NOT AVAILABLE IN PV/SC) +cd hack +./premium-to-premiumv2-zonal-aware-helper.sh process | tee zone-prep-$(date +%Y%m%d-%H%M%S).log + +# 3. Verify zone preparation results +kubectl get pvc data-app-a -n team-a -o jsonpath='{.metadata.annotations.disk\.csi\.azure\.com/migration-sourcesc}'; echo +kubectl get sc | grep "disk.csi.azure.com/created-by=azuredisk-pv1-to-pv2-migrator" + +# 4. Run migration with zone-aware StorageClasses +MAX_PVCS=2 ./premium-to-premiumv2-migrator-inplace.sh | tee mig-inplace-zoneaware.log + +# 5. Verify PremiumV2 with correct zone constraints +kubectl get pvc data-app-a -n team-a -o wide +pv=$(kubectl get pvc data-app-a -n team-a -o jsonpath='{.spec.volumeName}') +kubectl get pv "$pv" -o jsonpath='{.spec.csi.volumeAttributes.skuName}'; echo +kubectl get sc $(kubectl get pvc data-app-a -n team-a -o jsonpath='{.spec.storageClassName}') -o yaml | grep -A 5 allowedTopologies +``` + +--- + +Happy migrating! Review each summary carefully—intentional operator review is a built-in safety step, not an inefficiency. \ No newline at end of file diff --git a/hack/lib-premiumv2-migration-common.sh b/hack/lib-premiumv2-migration-common.sh new file mode 100644 index 0000000000..db257b18d0 --- /dev/null +++ b/hack/lib-premiumv2-migration-common.sh @@ -0,0 +1,1628 @@ +#!/usr/bin/env bash +# Common library for Premium_LRS -> PremiumV2_LRS migration helpers +set -euo pipefail +IFS=$'\n\t' + +# ---------- Logging / Audit ---------- +ts() { date +'%Y-%m-%dT%H:%M:%S'; } +info() { echo "$(ts) [INFO] $*"; } +warn() { echo "$(ts) [WARN] $*" >&2; } +err() { echo "$(ts) [ERROR] $*" >&2; } +ok() { echo "$(ts) [OK] $*"; } + +# ---------- Globals ---------- +declare -A SC_SET +declare -g -A SC_CACHE +declare -g -A PVC_SC_CACHE +declare -g -A PVCS_MIGRATION_INPROGRESS_MARKED_CACHE +declare -A _UNIQUE_ZONES=() +declare -g -A SNAPSHOTS_ARRAY + +# ----------- Common configurations ----------- +MIG_SUFFIX="${MIG_SUFFIX:-csi}" +AUDIT_ENABLE="${AUDIT_ENABLE:-true}" +AUDIT_LOG_FILE="${AUDIT_LOG_FILE:-pv1-pv2-migration-audit.log}" +SNAPSHOT_MAX_AGE_SECONDS="${SNAPSHOT_MAX_AGE_SECONDS:-7200}" +SNAPSHOT_RECREATE_ON_STALE="${SNAPSHOT_RECREATE_ON_STALE:-false}" +SNAPSHOT_CLASS="${SNAPSHOT_CLASS:-csi-azuredisk-vsc}" +MIGRATION_LABEL="${MIGRATION_LABEL:-disk.csi.azure.com/pv2migration=true}" +NAMESPACE="${NAMESPACE:-}" +MAX_PVCS="${MAX_PVCS:-50}" +POLL_INTERVAL="${POLL_INTERVAL:-120}" +WAIT_FOR_WORKLOAD="${WAIT_FOR_WORKLOAD:-true}" +MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES="${MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES:-3}" +ONE_SC_FOR_MULTIPLE_ZONES="${ONE_SC_FOR_MULTIPLE_ZONES:-true}" +SINGLE_ZONE_USE_GENERIC_PV2_SC="${SINGLE_ZONE_USE_GENERIC_PV2_SC:-true}" + +# In-place rollback keys +ROLLBACK_PVC_YAML_ANN="${ROLLBACK_PVC_YAML_ANN:-disk.csi.azure.com/rollback-pvc-yaml}" +ROLLBACK_ORIG_PV_ANN="${ROLLBACK_ORIG_PV_ANN:-disk.csi.azure.com/rollback-orig-pv}" + +# Maximum PVC size (in GiB) eligible for migration (default 2TiB = 2048GiB). PVCs >= this are skipped. +MAX_PVC_CAPACITY_GIB="${MAX_PVC_CAPACITY_GIB:-2048}" +declare -a AUDIT_LINES=() +MIGRATION_LABEL_KEY="${MIGRATION_LABEL%%=*}" +MIGRATION_LABEL_VALUE="${MIGRATION_LABEL#*=}" +BACKUP_ORIGINAL_PVC="${BACKUP_ORIGINAL_PVC:-true}" +PVC_BACKUP_DIR="${PVC_BACKUP_DIR:-pvc-backups}" + +# ---------- Timeouts ---------- +BIND_TIMEOUT_SECONDS="${BIND_TIMEOUT_SECONDS:-60}" +MONITOR_TIMEOUT_MINUTES="${MONITOR_TIMEOUT_MINUTES:-300}" +WORKLOAD_DETACH_TIMEOUT_MINUTES="${WORKLOAD_DETACH_TIMEOUT_MINUTES:-5}" +EXIT_ON_WORKLOAD_DETACH_TIMEOUT="${EXIT_ON_WORKLOAD_DETACH_TIMEOUT:-false}" + +# ---------- Kubectl retry configurations ---------- +KUBECTL_MAX_RETRIES="${KUBECTL_MAX_RETRIES:-5}" +KUBECTL_RETRY_BASE_DELAY="${KUBECTL_RETRY_BASE_DELAY:-2}" +KUBECTL_RETRY_MAX_DELAY="${KUBECTL_RETRY_MAX_DELAY:-30}" +KUBECTL_TRANSIENT_REGEX="${KUBECTL_TRANSIENT_REGEX:-(connection refused|i/o timeout|timeout exceeded|TLS handshake timeout|context deadline exceeded|Service Unavailable|Too Many Requests|EOF|transport is closing|Internal error|no route to host|Connection reset by peer)}" + +# ---------- Labels / Annotations ---------- +CREATED_BY_LABEL_KEY="${CREATED_BY_LABEL_KEY:-disk.csi.azure.com/created-by}" +MIGRATION_TOOL_ID="${MIGRATION_TOOL_ID:-azuredisk-pv1-to-pv2-migrator}" +MIGRATION_DONE_LABEL_KEY="${MIGRATION_DONE_LABEL_KEY:-disk.csi.azure.com/migration-done}" +MIGRATION_DONE_LABEL_VALUE="${MIGRATION_DONE_LABEL_VALUE:-true}" +MIGRATION_DONE_LABEL_VALUE_FALSE="${MIGRATION_DONE_LABEL_VALUE_FALSE:-false}" +MIGRATION_INPROGRESS_LABEL_KEY="${MIGRATION_INPROGRESS_LABEL_KEY:-disk.csi.azure.com/migration-in-progress}" +MIGRATION_INPROGRESS_LABEL_VALUE="${MIGRATION_INPROGRESS_LABEL_VALUE:-true}" +ZONE_SC_ANNOTATION_KEY="${ZONE_SC_ANNOTATION_KEY:-disk.csi.azure.com/migration-sourcesc}" + +# ------------ State ---------- +LAST_RUN_WITHOUT_ERREXIT_RC=0 +GENERIC_PV2_SC_MODE=0 + +is_direct_exec() { + # realpath + -ef handles symlinks and differing relative paths + [[ "$(realpath "${BASH_SOURCE[0]}")" -ef "$(realpath "$0")" ]] +} + +audit_add() { + [[ "$AUDIT_ENABLE" != "true" ]] && return 0 + local kind="$1" name="$2" namespace="$3" action="$4" revert="$5" extra="$6" + local line + line="$(ts)|${action}|${kind}|${namespace}|${name}|${revert}|${extra}" + AUDIT_LINES+=("$line") + if [[ -n "$AUDIT_LOG_FILE" ]]; then + printf '%s\n' "$line" >>"$AUDIT_LOG_FILE" || true + fi +} + +human_duration() { + local total=${1:-0} + local h=$(( total / 3600 )) + local m=$(( (total % 3600) / 60 )) + local s=$(( total % 60 )) + if (( h>0 )); then printf '%dh%02dm%02ds' "$h" "$m" "$s" + elif (( m>0 )); then printf '%dm%02ds' "$m" "$s" + else printf '%ds' "$s"; fi +} + +finalize_audit_summary() { + [[ "$AUDIT_ENABLE" != "true" ]] && return 0 + local start_ts="$1" start_epoch="$2" + local end_ts end_epoch dur_sec + end_ts="$(date +'%Y-%m-%dT%H:%M:%S')" + end_epoch="$(date +%s)" + dur_sec=$(( end_epoch - start_epoch )) + local dur_fmt + dur_fmt="$(human_duration "$dur_sec")" + echo + info "Run Timing:" + echo " Start: ${start_ts}" + echo " End: ${end_ts}" + echo " Elapsed: ${dur_fmt} (${dur_sec}s)" + info "Audit Trail" + if (( ${#AUDIT_LINES[@]} == 0 )); then + echo " (no mutating actions recorded)" + else + echo + info "Best-effort summary to revert if required:" + local line act revert + for line in "${AUDIT_LINES[@]}"; do + IFS='|' read -r _ act _ _ _ revert _ <<<"$line" + [[ -z "$revert" || "$revert" == "N/A" ]] && continue + printf ' %s\n' "$revert" + done + fi + if [[ -n "$AUDIT_LOG_FILE" ]]; then + { + printf 'RUN_END|%s|durationSeconds=%d|durationHuman=%s\n' "$end_ts" "$dur_sec" "$dur_fmt" + } >>"$AUDIT_LOG_FILE" 2>/dev/null || true + fi +} + +kubectl_retry() { + local attempt=1 rc output + while true; do + set +e + output=$(kubectl "$@" 2>&1) + rc=$? + set -e + if [[ $rc -eq 0 ]]; then + printf '%s' "$output" + return 0 + fi + if ! grep -qiE "$KUBECTL_TRANSIENT_REGEX" <<<"$output"; then + echo "$output" >&2 + return $rc + fi + if (( attempt >= KUBECTL_MAX_RETRIES )); then + warn "kubectl retry exhausted ($attempt): kubectl $*" + echo "$output" >&2 + return $rc + fi + local sleep_time=$(( KUBECTL_RETRY_BASE_DELAY * 2 ** (attempt-1) )) + (( sleep_time > KUBECTL_RETRY_MAX_DELAY )) && sleep_time=$KUBECTL_RETRY_MAX_DELAY + warn "kubectl transient (attempt $attempt) -> retry in ${sleep_time}s: $(head -n1 <<<"$output")" + sleep "$sleep_time" + attempt=$(( attempt + 1 )) + done +} + +kcmd() { + kubectl_retry "$@" +} + +kapply_retry() { + local tmp rc attempt=1 out + tmp="$(mktemp)" + cat >"$tmp" + while true; do + set +e + out=$(kubectl apply -f "$tmp" 2>&1) + rc=$? + set -e + if [[ $rc -eq 0 ]]; then + printf '%s\n' "$out" + rm -f "$tmp" + return 0 + fi + if ! grep -qiE "$KUBECTL_TRANSIENT_REGEX" <<<"$out"; then + echo "$out" >&2 + rm -f "$tmp" + return $rc + fi + if (( attempt >= KUBECTL_MAX_RETRIES )); then + warn "kubectl apply retry exhausted ($attempt)" + echo "$out" >&2 + rm -f "$tmp" + return $rc + fi + local sleep_time=$(( KUBECTL_RETRY_BASE_DELAY * 2 ** (attempt-1) )) + (( sleep_time > KUBECTL_RETRY_MAX_DELAY )) && sleep_time=$KUBECTL_RETRY_MAX_DELAY + warn "kubectl apply transient (attempt $attempt) -> retry in ${sleep_time}s: $(head -n1 <<<"$out")" + sleep "$sleep_time" + attempt=$(( attempt + 1 )) + done +} + +# ---------- Naming ---------- +name_csi_pvc() { local pvc="$1"; echo "${pvc}-${MIG_SUFFIX}"; } +name_csi_pv() { local pv="$1"; echo "${pv}-${MIG_SUFFIX}"; } +name_snapshot(){ local pv="$1"; echo "ss-$(name_csi_pv "$pv")"; } +name_pv2_pvc() { local pvc="$1"; echo "${pvc}-${MIG_SUFFIX}-pv2"; } +name_pv1_sc() { local sc="$1"; sc=$(get_srcsc_of_sc "$sc"); echo "${sc}-pv1"; } +name_pv2_sc() { local sc="$1"; sc=$(get_srcsc_of_sc "$sc"); echo "${sc}-pv2"; } + +# ---------- Utilities ---------- +require_bins() { + local missing=() + for b in kubectl jq base64; do + command -v "$b" >/dev/null 2>&1 || missing+=("$b") + done + if (( ${#missing[@]} > 0 )); then + err "Missing required binaries: ${missing[*]}" + exit 1 + fi +} + +# Convert a Kubernetes size string to an integer GiB (ceiling for sub-Gi units). +# Supports Ki, Mi, Gi, Ti, Pi (with optional 'i'). Returns 0 if unparseable. +size_to_gi_ceiling() { + local raw="$1" + [[ -z "$raw" ]] && { echo 0; return; } + if [[ "$raw" =~ ^([0-9]+)([KMGTP]i?)$ ]]; then + local n="${BASH_REMATCH[1]}" + local u="${BASH_REMATCH[2]}" + case "${u}" in + Ki|K) echo 0 ;; # effectively negligible; treat as 0 Gi + Mi|M) echo $(( (n + 1023) / 1024 )) ;; + Gi|G) echo $n ;; + Ti|T) echo $(( n * 1024 )) ;; + Pi|P) echo $(( n * 1024 * 1024 )) ;; + *) echo 0 ;; + esac + else + # Handles plain numbers (assume Gi) + if [[ "$raw" =~ ^[0-9]+$ ]]; then + echo "$raw" + else + echo 0 + fi + fi +} + +b64e() { base64 -w0; } +b64d() { base64 -d; } + +get_pv_of_pvc() { + local ns="$1" pvc="$2" + kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true +} + +is_in_tree_pv() { + local pv="$1" + local diskuri + diskuri=$(kcmd get pv "$pv" -o jsonpath='{.spec.azureDisk.diskURI}' 2>/dev/null || true) + [[ -n "$diskuri" ]] +} + +is_csi_pv() { + local pv="$1" + local drv + drv=$(kcmd get pv "$pv" -o jsonpath='{.spec.csi.driver}' 2>/dev/null || true) + [[ "$drv" == "disk.csi.azure.com" ]] +} + +ensure_reclaim_policy_retain() { + local pv="$1" + local current + current=$(kcmd get pv "$pv" -o jsonpath='{.spec.persistentVolumeReclaimPolicy}' 2>/dev/null || true) + [[ -z "$current" ]] && return 0 + if [[ "$current" == "Delete" ]]; then + info "Patching reclaimPolicy Delete->Retain for PV $pv" + kcmd patch pv "$pv" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' >/dev/null 2>&1 || true + audit_add "PersistentVolume" "$pv" "" "patch" \ + "kubectl patch pv $pv -p '{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Delete\"}}'" \ + "reclaimPolicy Delete->Retain" + fi +} + +mark_source_in_progress() { + local pvc_ns="$1" pvc="$2" + + if [[ -n ${PVCS_MIGRATION_INPROGRESS_MARKED_CACHE["$pvc_ns/$pvc"]:-} ]]; then + return 0 + fi + + kcmd label pvc "$pvc" -n "$pvc_ns" "${MIGRATION_INPROGRESS_LABEL_KEY}=${MIGRATION_INPROGRESS_LABEL_VALUE}" --overwrite >/dev/null + + if [[ $? -eq 0 ]]; then + PVCS_MIGRATION_INPROGRESS_MARKED_CACHE["$pvc_ns/$pvc"]=1 + fi + + audit_add "PersistentVolumeClaim" "$pvc" "$pvc_ns" "label" \ + "kubectl label pvc $pvc -n $pvc_ns ${MIGRATION_INPROGRESS_LABEL_KEY}-" \ + "in-progress=${MIGRATION_INPROGRESS_LABEL_VALUE}" + +} + +mark_source_done() { + local pvc_ns="$1" pvc="$2" + kcmd label pvc "$pvc" -n "$pvc_ns" "${MIGRATION_DONE_LABEL_KEY}=${MIGRATION_DONE_LABEL_VALUE}" --overwrite >/dev/null + audit_add "PersistentVolumeClaim" "$pvc" "$pvc_ns" "label" \ + "kubectl label pvc $pvc -n $pvc_ns ${MIGRATION_DONE_LABEL_KEY}-" \ + "done=${MIGRATION_DONE_LABEL_VALUE}" + + kcmd label pvc "$pvc" -n "$pvc_ns" "${MIGRATION_INPROGRESS_LABEL_KEY}-" --overwrite >/dev/null + audit_add "PersistentVolumeClaim" "$pvc" "$pvc_ns" "label-remove" \ + "kubectl label pvc $pvc -n $pvc_ns ${MIGRATION_INPROGRESS_LABEL_KEY}-" \ + "remove in-progress label" +} + +mark_source_notdone() { + local pvc_ns="$1" pvc="$2" + kcmd label pvc "$pvc" -n "$pvc_ns" "${MIGRATION_DONE_LABEL_KEY}=${MIGRATION_DONE_LABEL_VALUE_FALSE}" --overwrite >/dev/null + audit_add "PersistentVolumeClaim" "$pvc" "$pvc_ns" "label" \ + "kubectl label pvc $pvc -n $pvc_ns ${MIGRATION_DONE_LABEL_KEY}-" \ + "done=${MIGRATION_DONE_LABEL_VALUE_FALSE}" + + kcmd label pvc "$pvc" -n "$pvc_ns" "${MIGRATION_INPROGRESS_LABEL_KEY}-" --overwrite >/dev/null + audit_add "PersistentVolumeClaim" "$pvc" "$pvc_ns" "label-remove" \ + "kubectl label pvc $pvc -n $pvc_ns ${MIGRATION_INPROGRESS_LABEL_KEY}-" \ + "remove in-progress label" +} + +wait_pvc_bound() { + local ns="$1" name="$2" timeout="${3:-600}" poll=5 phase + local end=$(( $(date +%s) + timeout )) + while (( $(date +%s) < end )); do + phase=$(kcmd get pvc "$name" -n "$ns" -o jsonpath='{.status.phase}' 2>/dev/null || true) + [[ "$phase" == "Bound" ]] && return 0 + sleep "$poll" + done + return 1 +} + +get_pvc_encoded_json() { + local pvc="$1" ns="$2" + local orig_json + orig_json=$(kcmd get pvc "$pvc" -n "$ns" -o json | jq '{ + apiVersion, + kind, + metadata: { + name: .metadata.name, + namespace: .metadata.namespace, + labels: (.metadata.labels // {}), + annotations: ( + (.metadata.annotations // {}) + | with_entries( + select( + .key + | test("^(pv\\.kubernetes\\.io/|volume\\.kubernetes\\.io/|kubectl\\.kubernetes\\.io/|control-plane\\.|kubernetes\\.io/created-by)$") + | not + ) + ) + ) + }, + spec + }') + # Base64 encode sanitized JSON for rollback annotation + printf '%s' "$orig_json" | jq 'del(.status) | { + apiVersion, + kind, + metadata: { + name: .metadata.name, + namespace: .metadata.namespace, + labels: (.metadata.labels // {}), + annotations: ( + (.metadata.annotations // {}) + | with_entries( + select( + .key + | test("^(pv\\.kubernetes\\.io/|volume\\.kubernetes\\.io/|kubectl\\.kubernetes\\.io/|control-plane\\.|kubernetes\\.io/created-by)$") + | not + ) + ) + ) + }, + spec + }' | b64e +} + +get_pv_encoded_json() { + local pvc="$1" ns="$2" + local orig_json + local pv + + pv=$(get_pv_of_pvc "$ns" "$pvc") + + [[ -z "$pv" ]] && { err "Original PVC has no bound PV (cannot proceed)"; return 1; } + + orig_json=$(kcmd get pv "$pv" -o json | jq '{ + apiVersion, + kind, + metadata: { + name: .metadata.name, + labels: (.metadata.labels // {}), + annotations: ( + (.metadata.annotations // {}) + | with_entries( + select( + .key + | test("^(pv\\.kubernetes\\.io/|volume\\.kubernetes\\.io/|kubectl\\.kubernetes\\.io/|control-plane\\.|kubernetes\\.io/created-by)$") + | not + ) + ) + ) + }, + spec + }') + # Base64 encode sanitized JSON for rollback annotation + printf '%s' "$orig_json" | jq 'del(.status) | { + apiVersion, + kind, + metadata: { + name: .metadata.name, + labels: (.metadata.labels // {}), + annotations: ( + (.metadata.annotations // {}) + | with_entries( + select( + .key + | test("^(pv\\.kubernetes\\.io/|volume\\.kubernetes\\.io/|kubectl\\.kubernetes\\.io/|control-plane\\.|kubernetes\\.io/created-by)$") + | not + ) + ) + ) + }, + spec + }' | b64e +} + +backup_pvc() { + local pvc="$1" ns="$2" + if [[ "${BACKUP_ORIGINAL_PVC}" == "true" ]]; then + mkdir -p "${PVC_BACKUP_DIR}/${ns}" + local stamp tmp_file backup_file sc + stamp="$(date +%Y%m%d-%H%M%S)" + tmp_file="${PVC_BACKUP_DIR}/${ns}/${pvc}-${stamp}.yaml.tmp" + backup_file="${PVC_BACKUP_DIR}/${ns}/${pvc}-${stamp}.yaml" + sc=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.storageClassName}' 2>/dev/null || true) + + if ! kcmd get pvc "$pvc" -n "$ns" -o yaml >"$tmp_file" 2>/dev/null; then + err "Failed to fetch PVC $ns/$pvc for backup; skipping migration of this PVC." + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "backup-failed" "kubectl get pvc $pvc -n $ns -o yaml" "dest=$tmp_file reason=kubectlError" + return 1 + fi + if [[ ! -s "$tmp_file" ]] || ! grep -q '^kind: *PersistentVolumeClaim' "$tmp_file"; then + err "Backup validation failed for $ns/$pvc (empty or malformed); skipping migration." + rm -f "$tmp_file" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "backup-invalid" "kubectl get pvc $pvc -n $ns -o yaml" "dest=$backup_file reason=validation" + return 1 + fi + mv "$tmp_file" "$backup_file" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "backup" "rm -f $backup_file" "dest=$backup_file size=$(wc -c <"$backup_file")B sc=${sc}" + fi +} + +is_pvc_created_by_migration_tool() { + local pvc="$1" ns="$2" + createdby=$(kcmd get pvc "$pvc" -n "$ns" -o go-template="{{ index .metadata.labels \"${CREATED_BY_LABEL_KEY}\" }}" 2>/dev/null || true) + if [[ $createdby == "$MIGRATION_TOOL_ID" ]]; then + echo "true" + else + echo "false" + fi +} + +is_pvc_in_migration() { + local pvc="$1" ns="$2" + inprog=$(kcmd get pvc "$pvc" -n "$ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_INPROGRESS_LABEL_KEY}\" }}" 2>/dev/null || true) + if [[ $inprog == "true" ]]; then + echo "true" + else + echo "false" + fi +} + +get_srcsc_of_sc() { + local sci="$1" + + # if one sc for multiple zones are used, use the incoming storage class which is intermediate based on zone + # else if one sc for multiple zones is false, return the sc name and not source sc [which means one storage class per zone & dont need naming in storage class] + if [[ "$ONE_SC_FOR_MULTIPLE_ZONES" == "true" ]]; then + printf '%s' "$sci" + return 0 + fi + + if [[ -n ${SC_CACHE[$sci]:-} ]]; then + printf '%s' "${SC_CACHE[$sci]}" + return 0 + fi + + sc=$(kcmd get sc "$sci" -o jsonpath="{.metadata.annotations.${ZONE_SC_ANNOTATION_KEY//./\\.}}" 2>/dev/null || true) + if [[ -z $sc ]]; then + SC_CACHE["$sci"]="$sci" + printf '%s' "$sci" + else + SC_CACHE["$sci"]="$sc" + printf '%s' "$sc" + fi +} + +get_sc_of_pvc() { + local pvc="$1" ns="$2" + + if [[ -n ${PVC_SC_CACHE["$ns/$pvc"]:-} ]]; then + printf '%s' "${PVC_SC_CACHE["$ns/$pvc"]}" + return 0 + fi + + sc=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath="{.metadata.annotations.${ZONE_SC_ANNOTATION_KEY//./\\.}}" 2>/dev/null || true) + if [[ -z $sc ]]; then + sc=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.storageClassName}' 2>/dev/null || true) + fi + PVC_SC_CACHE["$ns/$pvc"]="$sc" + printf '%s' "$sc" +} + +create_csi_pv_pvc() { + local pvc="$1" ns="$2" pv="$3" size="$4" mode="$5" sc="$6" diskURI="$7" inplace="${8:-false}" + local csi_pv csi_pvc + local encoded_spec encoded_pv + local pv_before + + # check if MIGRATION_LABEL_KEY exists on source pvc and its value is MIGRATION_LABEL_VALUE + if kcmd get pvc "$pvc" -n "$ns" -o json | jq -e --arg key "$MIGRATION_LABEL_KEY" --arg value "$MIGRATION_LABEL_VALUE" '.metadata.labels[$key]==$value' >/dev/null; then + migration_label_exists=true + fi + + csi_pv="$(name_csi_pv "$pv")" + csi_pvc="$(name_csi_pvc "$pvc")" + if $inplace; then + csi_pvc="$pvc" + fi + if kcmd get pvc "$csi_pvc" -n "$ns" >/dev/null 2>&1; then + kcmd get pvc "$csi_pvc" -n "$ns" -o json | jq -e --arg key "$CREATED_BY_LABEL_KEY" --arg tool "$MIGRATION_TOOL_ID" '.metadata.labels[$key]==$tool' >/dev/null \ + && { info "Intermediate PVC $ns/$csi_pvc exists"; return; } + if ! $inplace; then + warn "Intermediate PVC $ns/$csi_pvc exists but missing label" + return + fi + fi + encoded_spec="" + encoded_pv="" + pv_before="" + if [[ $inplace == true ]]; then + encoded_pv=$(get_pv_encoded_json "$pvc" $ns) + pv_before=$(get_pv_of_pvc "$ns" "$pvc") + + backup_pvc "$pvc" "$ns" || { + warn "PVC backup failed $ns/$pvc" + } + # Base64 encode sanitized JSON for rollback annotation + encoded_spec=$(get_pvc_encoded_json "$pvc" "$ns") + ensure_reclaim_policy_retain "$pv" + if ! kcmd delete pvc "$csi_pvc" -n "$ns" --wait=true >/dev/null 2>&1; then + warn "Deleted preexisting PVC $ns/$csi_pvc for inplace recreation" + audit_add "PersistentVolumeClaim" "$csi_pvc" "$ns" "delete" "N/A" "inplace=true reason=preexisting" + return 1 + fi + audit_add "PersistentVolumeClaim" "$csi_pvc" "$ns" "delete" "kubectl create -f <(echo \"$encoded_spec\" | base64 --decode) " "inplace=true reason=replace" + + if ! kcmd patch pv "$pv" -p '{"spec":{"claimRef":null}}' >/dev/null 2>&1; then + warn "Clear claimRef failed for PV $pv" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-clear-claimref-failed" "kubectl describe pv $pv" "reason=patchError" + return 1 + fi + info "Recreating inplace CSI (from intree) PV/PVC $pv / $csi_pvc -> $csi_pv / $csi_pvc" + else + info "Creating intermediate PV/PVC $csi_pv / $csi_pvc" + fi + + if ! kapply_retry </dev/null 2>&1 || true + kcmd annotate pvc "$csi_pvc" -n "$ns" "${ROLLBACK_ORIG_PV_ANN}-" --overwrite >/dev/null 2>&1 || true + fi + fi + + if wait_pvc_bound "$ns" "$csi_pvc" "$BIND_TIMEOUT_SECONDS"; then + ok "PVC $ns/$csi_pvc bound" + audit_add "PersistentVolumeClaim" "$csi_pvc" "$ns" "bound" "kubectl describe pvc $csi_pvc -n $ns" "csi=true" + return 0 + fi + + warn "Intermediate PVC $ns/$csi_pvc not bound within timeout (${BIND_TIMEOUT_SECONDS}s)" + audit_add "PersistentVolumeClaim" "$csi_pvc" "$ns" "bind-timeout" "kubectl describe pvc $csi_pvc -n $ns" "csi=true timeout=${BIND_TIMEOUT_SECONDS}s" + return 2 +} + +create_pvc_from_snapshot() { + local pvc="$1" ns="$2" pv="$3" size="$4" mode="$5" sc="$6" destpvc="$7" snapshot="$8" + local inplace + local encoded_spec + local migration_label_exists + local encoded_spec encoded_pv + local pv_before + + if [[ "$destpvc" != "$pvc" ]]; then + inplace=false + else + inplace=true + fi + + # check if MIGRATION_LABEL_KEY exists on source pvc and its value is MIGRATION_LABEL_VALUE + if kcmd get pvc "$pvc" -n "$ns" -o json | jq -e --arg key "$MIGRATION_LABEL_KEY" --arg value "$MIGRATION_LABEL_VALUE" '.metadata.labels[$key]==$value' >/dev/null; then + migration_label_exists=true + fi + + if kcmd get pvc "$destpvc" -n "$ns" >/dev/null 2>&1; then + kcmd get pvc "$destpvc" -n "$ns" -o json | jq -e --arg key "$CREATED_BY_LABEL_KEY" --arg tool "$MIGRATION_TOOL_ID" '.metadata.labels[$key]==$tool' >/dev/null \ + && { info "PVC $ns/$destpvc exists"; return; } + if ! $inplace; then + warn "PVC $ns/$destpvc exists but missing label" + return + fi + fi + + encoded_spec="" + encoded_pv="" + pv_before="" + if [[ "$destpvc" == "$pvc" ]]; then + inplace=true + pv_before=$(get_pv_of_pvc "$ns" "$pvc") + + backup_pvc "$pvc" "$ns" || { + warn "PVC (snapshot creation path) backup failed $ns/$pvc" + } + + encoded_spec=$(get_pvc_encoded_json "$pvc" "$ns") + ensure_reclaim_policy_retain "$pv" + + if ! kcmd delete pvc "$pvc" -n "$ns" --wait=true >/dev/null 2>&1; then + warn "Deleted preexisting PVC $ns/$pvc for inplace recreation" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "delete-failed" "N/A" "inplace=true reason=preexisting" + return 1 + fi + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "delete" "kubectl create -f <(echo \"$encoded_spec\" | base64 --decode) " "inplace=true reason=preexisting" + + if ! kcmd patch pv "$pv" -p '{"spec":{"claimRef":null}}' >/dev/null 2>&1; then + warn "Clear claimRef failed for PV $pv" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-clear-claimref-failed" "kubectl describe pv $pv" "reason=patchError" + return 1 + fi + info "Recreating inplace CSI (from snapshot) PV/PVC $snapshot -> $pv / $destpvc" + fi + + info "Creating PVC $ns/$destpvc" + if ! kapply_retry </dev/null 2>&1; then + [[ "$RBAC_EXTRA_VERBOSE" == "true" ]] && info "RBAC OK: $verb $res $*" + return 0 + fi + return 1 + } + + for entry in "${cluster_checks[@]}"; do + [[ -z "$entry" ]] && continue + total=$((total+1)); read -r verb res <<<"$entry" + if rbac_can "$verb" "$res"; then passed=$((passed+1)); else + failures+=("cluster: $verb $res"); warn "RBAC missing: cluster: $verb $res" + [[ "$RBAC_FAIL_FAST" == "true" ]] && { err "RBAC fail-fast."; return 1; } + fi + done + + if [[ -n "$NAMESPACE" ]]; then + for entry in "${ns_checks[@]}"; do + total=$((total+1)); read -r verb res <<<"$entry" + if rbac_can "$verb" "$res" -n "$NAMESPACE"; then passed=$((passed+1)); else + failures+=("namespace:$NAMESPACE: $verb $res"); warn "RBAC missing: namespace:$NAMESPACE: $verb $res" + [[ "$RBAC_FAIL_FAST" == "true" ]] && { err "RBAC fail-fast."; return 1; } + fi + done + else + info "Cluster-wide run: ensure namespace-scoped verbs present in each target namespace." + fi + + if (( ${#failures[@]} > 0 )); then + err "RBAC preflight failed (${#failures[@]} missing)." + printf 'Missing:\n'; printf ' %s\n' "${failures[@]}" + return 1 + fi + ok "RBAC preflight success ($passed/$total checks)." + return 0 +} + +# ------------- Helper Functions ------------- +check_premium_lrs() { + local pvc="$1" ns="$2" sc sku sat + sc=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.storageClassName}' 2>/dev/null || true) + [[ -z "$sc" ]] && return 1 + sku=$(kcmd get sc "$sc" -o jsonpath='{.parameters.skuName}' 2>/dev/null || true) + sat=$(kcmd get sc "$sc" -o jsonpath='{.parameters.storageaccounttype}' 2>/dev/null || true) + [[ -z "$sku" && -z "$sat" ]] && return 1 + { [[ -z "$sku" || "$sku" == "Premium_LRS" ]] && [[ -z "$sat" || "$sat" == "Premium_LRS" ]]; } +} + +check_premiumv2_lrs() { + local ns="$1" pvc="$2" + local val vac + vac="$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeAttributesClassName}' || true)" + if [[ ! -z "$vac" ]]; then + val="$(kcmd get volumeattributesclass.storage.k8s.io "$vac" -o jsonpath='{.parameters.skuName}')" + [[ "$val" == "PremiumV2_LRS" ]] && return 0 + fi + return 1 +} + +# ---------- Snapshot Class & StorageClass Variant Helpers ---------- +ensure_snapshot_class() { + if kcmd get volumesnapshotclass "$SNAPSHOT_CLASS" >/dev/null 2>&1; then + info "VolumeSnapshotClass '$SNAPSHOT_CLASS' exists" + return + fi + info "Creating VolumeSnapshotClass '$SNAPSHOT_CLASS'" + if ! kapply_retry </dev/null 2>&1; then + if [[ "$ATTR_CLASS_FORCE_RECREATE" == "true" ]]; then + info "Recreating VolumeAttributesClass ${ATTR_CLASS_NAME}" + kcmd delete volumeattributesclass "${ATTR_CLASS_NAME}" --wait=true || true + else + info "VolumeAttributesClass ${ATTR_CLASS_NAME} present" + return 0 + fi + fi + info "Creating VolumeAttributesClass ${ATTR_CLASS_NAME} (sku=${TARGET_SKU})" + if ! kapply_retry << EOF +apiVersion: ${ATTR_CLASS_API_VERSION} +kind: VolumeAttributesClass +metadata: + name: ${ATTR_CLASS_NAME} + labels: + ${CREATED_BY_LABEL_KEY}: ${MIGRATION_TOOL_ID} +driverName: disk.csi.azure.com +parameters: + skuName: ${TARGET_SKU} +EOF + then + audit_add "VolumeAttributesClass" "${ATTR_CLASS_NAME}" "" "create-failed" "N/A" "sku=${TARGET_SKU} reason=applyFailure" + exit 1 + else + audit_add "VolumeAttributesClass" "${ATTR_CLASS_NAME}" "" "create" "kubectl delete volumeattributesclass ${ATTR_CLASS_NAME}" "sku=${TARGET_SKU}" + fi +} + +apply_storage_class_variant() { + local orig_name="$1" sc_name="$2" sku="$3" + if kcmd get sc "$sc_name" >/dev/null 2>&1; then + info "StorageClass $sc_name exists" + return + fi + local params_json params_filtered allowed_topologies + params_json=$(kcmd get sc "$orig_name" -o json 2>/dev/null || true) + [[ -z "$params_json" ]] && { warn "Cannot fetch base SC $orig_name; skipping variant"; return; } + params_filtered=$(echo "$params_json" | jq -r ' + .parameters + | to_entries + | map(select(.key != "skuName" + and .key != "storageaccounttype" + and .key != "cachingMode" + and (.key | test("encryption";"i") | not))) + | map(" " + .key + ": \"" + (.value|tostring) + "\"") + | join("\n") + ') + + # Extract allowedTopologies if present + allowed_topologies_yaml=$(echo "$params_json" | jq -r ' + if .allowedTopologies then + "allowedTopologies:" + + (.allowedTopologies | + map("\n- " + + (.matchLabelExpressions // [] | + if length > 0 then + "matchLabelExpressions:" + + (map("\n - key: " + .key + "\n values: [" + (.values | map("\"" + . + "\"") | join(", ")) + "]") | join("")) + else "" + end + ) + ) | join("")) + else + "" + end + ') + + # Extract original labels (excluding system labels, add zone-specific ones) + orig_labels=$(echo "$params_json" | jq -r ' + .metadata.labels // {} + | to_entries + | map(select(.key | test("^(kubernetes\\.io/|k8s\\.io/|pv\\.kubernetes\\.io/|volume\\.kubernetes\\.io/|app\\.kubernetes\\.io/managed-by|storageclass\\.kubernetes\\.io/is-default-class)") | not)) + | map(" " + .key + ": \"" + (.value|tostring) + "\"") + | join("\n") + ') + + # Extract original annotations (excluding system annotations) + orig_annotations=$(echo "$params_json" | jq -r --arg zone_key "$ZONE_SC_ANNOTATION_KEY" ' + .metadata.annotations // {} + | to_entries + | map(select(.key | test("^(kubernetes\\.io/|k8s\\.io/|pv\\.kubernetes\\.io/|volume\\.kubernetes\\.io/|kubectl\\.kubernetes\\.io/|deployment\\.kubernetes\\.io/|control-plane\\.|storageclass\\.kubernetes\\.io/is-default-class)") | not)) + | map(select(.key != $zone_key)) + | map(" " + .key + ": \"" + (.value|tostring|gsub("\\n"; "\\\\n")|gsub("\\\""; "\\\\\"")) + "\"") + | join("\n") + ') + + if [[ -z "$allowed_topologies_yaml" ]]; then + err "Base StorageClass $orig_name has no allowedTopologies; skipping variant $sc_name" + return 1 + fi + + info "Creating StorageClass $sc_name (skuName=$sku)" + if ! kapply_retry </dev/null || true) + sat=$(kcmd get sc "$sc" -o jsonpath='{.parameters.storageaccounttype}' 2>/dev/null || true) + if [[ -n "$sku" || -n "$sat" ]]; then + if ! { [[ -z "$sku" || "$sku" == "Premium_LRS" ]] && [[ -z "$sat" || "$sat" == "Premium_LRS" ]]; }; then + warn "Source SC $sc not Premium_LRS (skuName=$sku storageaccounttype=$sat) – skipping variants" + continue + fi + fi + pv2_sc=$(name_pv2_sc "$sc") + apply_storage_class_variant "$sc" "$pv2_sc" "PremiumV2_LRS" + pv1_sc=$(name_pv1_sc "$sc") + apply_storage_class_variant "$sc" "$pv1_sc" "Premium_LRS" + done +} + +# ---------- Workload & Event Helpers ---------- +list_pods_using_pvc() { + local ns="$1" pvc="$2" + kcmd get pods -n "$ns" \ + --field-selector spec.volumes.persistentVolumeClaim.claimName="$pvc" \ + -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,NODE:.spec.nodeName' 2>/dev/null || true +} + +wait_for_workload_detach() { + local pv="$1" pvc="$2" ns="$3" poll="${POLL_INTERVAL:-60}" + local WORKLOAD_DETACH_TIMEOUT_MINUTES="${WORKLOAD_DETACH_TIMEOUT_MINUTES:-0}" + local timeout_deadline=0 + if (( WORKLOAD_DETACH_TIMEOUT_MINUTES > 0 )); then + timeout_deadline=$(( $(date +%s) + WORKLOAD_DETACH_TIMEOUT_MINUTES * 60 )) + fi + info "Waiting for workload detach from PV $pv (PVC $ns/$pvc)" + while true; do + local attachments + attachments=$(kcmd get volumeattachment -o jsonpath="{range .items[?(@.spec.source.persistentVolumeName=='$pv')]}{.metadata.name}{'\n'}{end}" 2>/dev/null || true) + if [[ -z "$attachments" ]]; then + ok "No VolumeAttachment for $pv" + return 0 + fi + warn "Still attached (VolumeAttachment present) PV=$pv:" + echo "$attachments" | sed 's/^/ - /' + echo "Pods referencing PVC:" + list_pods_using_pvc "$ns" "$pvc" | sed 's/^/ /' || true + info "Scale down workloads then retrying in ${poll}s..." + if (( timeout_deadline > 0 )) && (( $(date +%s) >= timeout_deadline )); then + warn "Detach wait timeout for $ns/$pvc" + if [[ "$EXIT_ON_WORKLOAD_DETACH_TIMEOUT" == "true" ]]; then + err "Exiting due to detach wait timeout." + exit 1 + else + warn "Continuing despite detach wait timeout." + fi + return 1 + fi + sleep "$poll" + done +} + +extract_event_reason() { + local ns="$1" pvc_name="$2" + kcmd get events -n "$ns" -o json 2>/dev/null \ + | jq -r --arg name "$pvc_name" ' + [ .items[] + | select(.involvedObject.name==$name + and (.reason|test("^SKUMigration(Completed|Progress|Started)$"))) + | {reason, ts:(.lastTimestamp // .eventTime // .deprecatedLastTimestamp // .metadata.creationTimestamp // "")} + ] as $arr + | if ($arr|length)>0 + then ($arr|sort_by(.ts)|last|.reason) + else "" + end' +} + +detect_generic_pv2_mode() { + if [[ "${#MIG_PVCS[@]}" -gt 0 ]]; then + for pvc_entry in "${MIG_PVCS[@]}"; do + local _ns="${pvc_entry%%|*}" + local _name="${pvc_entry##*|}" + local _pv="$(kcmd get pvc "$_name" -n "$_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)" + [[ -z "$_pv" ]] && continue + local _zone="$(extract_zone_from_pv_nodeaffinity "$_pv" || true)" + if [[ -n "$_zone" ]]; then + _UNIQUE_ZONES["$_zone"]=1 + fi + + is_created_by_migrator=$(is_pvc_in_migration "$_name" "$_ns") + if [[ $is_created_by_migrator == "true" ]]; then + info "skipping detect_generic_pv2_mode as the migration have already kicked in" + return + fi + + is_migration=$(is_pvc_created_by_migration_tool "$_name" "$_ns") + if [[ $is_migration == "true" ]]; then + info "skipping detect_generic_pv2_mode as there are pvcs created by migration tool" + return + fi + done + fi + local _unique_zone_count="${#_UNIQUE_ZONES[@]}" + if [[ "$_unique_zone_count" == "1" && "$SINGLE_ZONE_USE_GENERIC_PV2_SC" == "true" ]]; then + info "Single zone detected across all PVCs; will use generic pv2 naming (-pv2) instead of zone suffix." + GENERIC_PV2_SC_MODE=1 + else + GENERIC_PV2_SC_MODE=0 + fi +} + +populate_pvcs() { + if [[ -z "$NAMESPACE" ]]; then + mapfile -t MIG_PVCS < <(kcmd get pvc --all-namespaces -l "$MIGRATION_LABEL" -o jsonpath='{range .items[*]}{.metadata.namespace}{"|"}{.metadata.name}{"\n"}{end}') + else + mapfile -t MIG_PVCS < <(kcmd get pvc -n "$NAMESPACE" -l "$MIGRATION_LABEL" -o jsonpath='{range .items[*]}{.metadata.namespace}{"|"}{.metadata.name}{"\n"}{end}') + fi + total_found=${#MIG_PVCS[@]} + if (( total_found == 0 )); then + warn "No PVCs found with label $MIGRATION_LABEL" + exit 0 + fi + IFS=$'\n' MIG_PVCS=($(printf '%s\n' "${MIG_PVCS[@]}" | sort)) + if (( total_found > MAX_PVCS )); then + warn "Found $total_found PVCs; processing only first $MAX_PVCS" + MIG_PVCS=("${MIG_PVCS[@]:0:MAX_PVCS}") + fi + + # Size filtering (< MAX_PVC_CAPACITY_GIB) + local filtered=() skipped_large=0 + for entry in "${MIG_PVCS[@]}"; do + local ns="${entry%%|*}" pvc="${entry##*|}" + local pv size gi + pv=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + if [[ -z "$pv" ]]; then + warn "Skipping $ns/$pvc (not bound, no PV yet)" + continue + fi + size=$(kcmd get pv "$pv" -o jsonpath='{.spec.capacity.storage}' 2>/dev/null || true) + gi=$(size_to_gi_ceiling "$size") + if (( gi >= MAX_PVC_CAPACITY_GIB )); then + warn "Skipping $ns/$pvc size=$size (~${gi}Gi) >= threshold ${MAX_PVC_CAPACITY_GIB}Gi" + skipped_large=$((skipped_large+1)) + continue + fi + filtered+=("$entry") + done + MIG_PVCS=("${filtered[@]}") + + if (( ${#MIG_PVCS[@]} == 0 )); then + warn "After filtering, no PVCs under ${MAX_PVC_CAPACITY_GIB}Gi remain (skipped_large=$skipped_large)" + exit 0 + fi + + info "Processing PVCs (< ${MAX_PVC_CAPACITY_GIB}Gi threshold; skipped_large=$skipped_large):" + printf ' %s\n' "${MIG_PVCS[@]}" +} + +create_unique_storage_classes() { + for entry in "${MIG_PVCS[@]}"; do + ns="${entry%%|*}" pvc="${entry##*|}" + # if $pvc has ZONE_SC_ANNOTATION_KEY annotation, use it otherwise use .spec.storageClassName + sc=$(get_sc_of_pvc "$pvc" "$ns") + [[ -n "$sc" ]] && SC_SET["$sc"]=1 + done + SOURCE_SCS=("${!SC_SET[@]}") + info "Source StorageClasses:" + printf ' %s\n' "${SOURCE_SCS[@]}" + create_variants_for_sources "${SOURCE_SCS[@]}" +} + +# -------- AttrClass Feature Gate Confirmation (additional pre-req) -------- +attrclass_feature_gate_confirm() { + [[ "$MODE" != "attrclass" ]] && return 0 + + # Skip entirely if user explicitly wants to skip the question (they may manage this externally) + if [[ "${ATTRCLASS_SKIP_FEATURE_GATE_PROMPT:-false}" == "true" ]]; then + info "Skipping VolumeAttributesClass feature-gate prompt (ATTRCLASS_SKIP_FEATURE_GATE_PROMPT=true)." + return 0 + fi + + # Auto-accept path for CI / automation + if [[ "${ATTRCLASS_ASSUME_FEATURE_GATES_YES:-false}" == "true" ]]; then + info "Assuming VolumeAttributesClass feature gates & runtime-config are enabled (ATTRCLASS_ASSUME_FEATURE_GATES_YES=true)." + return 0 + fi + + local ref_url="https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/deploy/example/modifyvolume/README.md" + local prompt_msg + prompt_msg=$'\nThe AttrClass migration mode requires that ALL of the following are already in place:\n\n' + prompt_msg+=$'- kube-apiserver started with feature gate: --feature-gates=...,VolumeAttributesClass=true\n' + prompt_msg+=$'- kube-controller-manager feature gate: --feature-gates=...,VolumeAttributesClass=true\n' + prompt_msg+=$'- external-provisioner (Azure Disk CSI) has: --feature-gates=VolumeAttributesClass=true (if required by its version)\n' + prompt_msg+=$'- external-resizer (Azure Disk CSI) has: --feature-gates=VolumeAttributesClass=true (if required)\n' + prompt_msg+=$'- API version ${ATTR_CLASS_API_VERSION} for VolumeAttributesClass is enabled (runtime-config if still beta), e.g.\n' + prompt_msg+=$' --runtime-config=storage.k8s.io/v1beta1=true (adjust if GA -> storage.k8s.io/v1)\n\n' + prompt_msg+=$'Confirm ALL of the above are configured cluster-wide? (y/N): ' + + local ans="" + # Try /dev/tty to remain interactive even if piped; fall back to stdin if tty not available + if [[ -t 0 ]]; then + read -r -p "$prompt_msg" ans + elif [[ -r /dev/tty ]]; then + read -r -p "$prompt_msg" ans < /dev/tty + else + warn "Non-interactive session; cannot prompt for VolumeAttributesClass feature gate confirmation." + PREREQ_ISSUES+=("VolumeAttributesClass feature gates / runtime-config not confirmed (non-interactive and no ATTRCLASS_ASSUME_FEATURE_GATES_YES)") + PREREQ_ISSUES+=("See reference (Prerequisites): $ref_url") + return 0 + fi + + case "${ans,,}" in + y|yes) + info "Feature gate & runtime-config confirmation accepted." + ;; + *) + err "User did NOT confirm required VolumeAttributesClass feature gates/runtime-config." + PREREQ_ISSUES+=("Missing or unconfirmed VolumeAttributesClass feature gates / runtime-config (apiserver/controller-manager/provisioner/resizer / apiVersion ${ATTR_CLASS_API_VERSION})") + PREREQ_ISSUES+=("Enable per (Prerequisites): $ref_url") + ;; + esac +} + +run_prerequisites_checks() { + info "Running migration pre-requisites validation..." + (( ${#MIG_PVCS[@]} > 50 )) && PREREQ_ISSUES+=("Selected PVC count (${#MIG_PVCS[@]}) > recommended batch (50)") + declare -A _SC_JSON_CACHE + for ENTRY in "${MIG_PVCS[@]}"; do + local ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" pv sc size zone sc_json + pv=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ -z "$pv" ]] && { PREREQ_ISSUES+=("PVC/$ns/$pvc not bound"); continue; } + sc=$(get_sc_of_pvc "$pvc" "$ns") + size=$(kcmd get pv "$pv" -o jsonpath='{.spec.capacity.storage}' 2>/dev/null || true) + [[ -z "$sc" ]] && PREREQ_ISSUES+=("PVC/$ns/$pvc missing storageClassName") + [[ -z "$size" ]] && PREREQ_ISSUES+=("PV/$pv capacity missing") + if [[ -n "$sc" ]]; then + if [[ -z "${_SC_JSON_CACHE[$sc]:-}" ]]; then + _SC_JSON_CACHE[$sc]=$(kcmd get sc "$sc" -o json 2>/dev/null || echo "") + fi + sc_json="${_SC_JSON_CACHE[$sc]}" + if [[ -n "$sc_json" ]]; then + local cachingMode enableBursting diskEncryptionType logicalSector perfProfile + cachingMode=$(echo "$sc_json" | jq -r '.parameters.cachingMode // empty') + enableBursting=$(echo "$sc_json" | jq -r '.parameters.enableBursting // empty') + diskEncryptionType=$(echo "$sc_json" | jq -r '.parameters.diskEncryptionType // empty') + logicalSector=$(echo "$sc_json" | jq -r '.parameters.LogicalSectorSize // empty') + perfProfile=$(echo "$sc_json" | jq -r '.parameters.perfProfile // empty') + [[ -n "$cachingMode" && "${cachingMode,,}" != "none" ]] && PREREQ_ISSUES+=("SC/$sc cachingMode=$cachingMode (must be none)") + [[ -n "$enableBursting" && "${enableBursting,,}" != "false" ]] && PREREQ_ISSUES+=("SC/$sc enableBursting=$enableBursting") + [[ "$diskEncryptionType" == "EncryptionAtRestWithPlatformAndCustomerKeys" ]] && PREREQ_ISSUES+=("SC/$sc double encryption unsupported") + [[ -n "$logicalSector" && "$logicalSector" != "512" ]] && PREREQ_ISSUES+=("SC/$sc LogicalSectorSize=$logicalSector (expected 512)") + [[ -n "$perfProfile" && "${perfProfile,,}" != "basic" ]] && PREREQ_ISSUES+=("SC/$sc perfProfile=$perfProfile (must be basic/empty)") + else + PREREQ_ISSUES+=("SC/$sc not retrievable") + fi + fi + local intree + intree=$(kcmd get pv "$pv" -o jsonpath='{.spec.azureDisk.diskURI}' 2>/dev/null || true) + if [[ -z "$intree" ]]; then + local drv + drv=$(kcmd get pv "$pv" -o jsonpath='{.spec.csi.driver}' 2>/dev/null || true) + [[ "$drv" != "disk.csi.azure.com" ]] && PREREQ_ISSUES+=("PV/$pv unknown provisioning type") + fi + done + attrclass_feature_gate_confirm + info "NOTE: PremiumV2 quota not auto-checked." +} + +# ensure_volume_snapshot +# Env/config used (with safe defaults if unset): +# SNAPSHOT_CLASS (required) +# SNAPSHOT_MAX_AGE_SECONDS (default 0 = disable age staleness) +# SNAPSHOT_RECREATE_ON_STALE (default false) +# Behavior: +# - Reuse existing ready snapshot if not stale +# - Detect staleness by age or source PVC resourceVersion drift +# - Optionally recreate if staleness + SNAPSHOT_RECREATE_ON_STALE == true +# Return codes: 0 success/ready, 1 failure (not ready), 2 recreate failed +ensure_volume_snapshot() { + local ns="$1" source_pvc="$2" snap="$3" + + local max_age="${SNAPSHOT_MAX_AGE_SECONDS:-0}" + local recreate_on_stale="${SNAPSHOT_RECREATE_ON_STALE:-false}" + + local exists=false stale=false recreated=false reasons=() + if kcmd get volumesnapshot "$snap" -n "$ns" >/dev/null 2>&1; then + exists=true + # Gather current snapshot metadata + local prev_rv creation_ts creation_epoch now_epoch current_rv + prev_rv=$(kcmd get volumesnapshot "$snap" -n "$ns" -o jsonpath='{.metadata.annotations.disk\.csi\.azure\.com/source-pvc-rv}' 2>/dev/null || true) + creation_ts=$(kcmd get volumesnapshot "$snap" -n "$ns" -o jsonpath='{.metadata.creationTimestamp}' 2>/dev/null || true) + current_rv=$(kcmd get pvc "$source_pvc" -n "$ns" -o jsonpath='{.metadata.resourceVersion}' 2>/dev/null || true) + + if [[ -n "$creation_ts" && $max_age -gt 0 ]]; then + creation_epoch=$(date -d "$creation_ts" +%s 2>/dev/null || echo 0) + now_epoch=$(date +%s) + if (( now_epoch - creation_epoch > max_age )); then + stale=true; reasons+=("age>${max_age}") + fi + fi + if [[ -n "$prev_rv" && -n "$current_rv" && "$prev_rv" != "$current_rv" ]]; then + stale=true; reasons+=("resourceVersionChanged") + fi + + if [[ "$stale" == "true" && "$recreate_on_stale" == "true" ]]; then + warn "Deleting stale snapshot $ns/$snap (${reasons[*]})" + kcmd delete volumesnapshot "$snap" -n "$ns" --ignore-not-found + audit_add "VolumeSnapshot" "$snap" "$ns" "delete" "N/A" "reason=stale(${reasons[*]})" + # Wait until gone + for _ in {1..12}; do + if ! kcmd get volumesnapshot "$snap" -n "$ns" >/dev/null 2>&1; then + exists=false; recreated=true; break + fi + sleep 5 + done + fi + + if [[ "$exists" == "true" && "$stale" == "false" ]]; then + local ready + ready=$(kcmd get volumesnapshot "$snap" -n "$ns" -o jsonpath='{.status.readyToUse}' 2>/dev/null || true) + if [[ "$ready" == "true" ]]; then + info "Snapshot $ns/$snap ready (reused)" + return 0 + fi + info "Waiting on existing snapshot $ns/$snap" + fi + fi + + if [[ "$exists" == "false" ]]; then + info "Creating snapshot $ns/$snap from $source_pvc" + local source_rv + source_rv=$(kcmd get pvc "$source_pvc" -n "$ns" -o jsonpath='{.metadata.resourceVersion}' 2>/dev/null || true) + if ! kapply_retry </dev/null || true) + [[ "$ready" == "true" ]] && { break; } + sleep 5 + done + if [[ "$ready" == "true" ]]; then + ok "Snapshot $ns/$snap ready"; + continue + fi + audit_add "VolumeSnapshot" "$snap" "$ns" "not-ready" "N/A" "sourcePVC=$source_pvc reason=timeout" + warn "Snapshot $ns/$snap not ready after timeout" + return 1 + done +} + +print_migration_cleanup_report() { + local mode="${MODE:-dual}" + local success_header_printed=false + local failed_header_printed=false + local any=false + + if (( ${#MIG_PVCS[@]} == 0 )); then + warn "print_migration_cleanup_report: MIG_PVCS empty (nothing to report)." + return 0 + fi + + info "Generating migration cleanup / investigation report (mode=${mode})..." + + # Cache Released PVs that (a) still reference a claimRef and (b) are PremiumV2_LRS in CSI volumeAttributes. + # Output columns (TSV): + # namespace pvcName pvName reclaimPolicy storageClass capacity skuName + local released_pv_lines + released_pv_lines="$(kcmd get pv -o json 2>/dev/null | jq -r ' + .items[] + | select(.status.phase=="Released" + and .spec.claimRef + and .spec.claimRef.namespace!=null + and .spec.claimRef.name!=null + and .spec.csi!=null + ) + | . as $pv + | ( + $pv.spec.csi.volumeAttributes.skuName + // $pv.spec.csi.volumeAttributes.skuname + // "" + ) as $sku + | select($sku=="PremiumV2_LRS") + | [ + .spec.claimRef.namespace, + .spec.claimRef.name, + .metadata.name, + (.spec.persistentVolumeReclaimPolicy // ""), + (.spec.storageClassName // ""), + (.spec.capacity.storage // ""), + $sku + ] | @tsv + ' 2>/dev/null || true)" + + # For attrclass mode only: + # Identify original in-tree PVs (azureDisk) now phase=Available (claimRef cleared), + # that have a CSI twin PV (same diskURI) created by this tool (endswith -$MIG_SUFFIX, labeled created-by). + # Output columns (TSV): + # origPV origSC origSize diskURI twinCsiPV twinSC + local available_intree_lines="" + if [[ "$mode" == "attrclass" ]]; then + available_intree_lines="$(kcmd get pv -o json 2>/dev/null | jq -r \ + --arg suf "-${MIG_SUFFIX}" \ + --arg tool "${MIGRATION_TOOL_ID}" ' + .items as $all + | [ + .items[] + | select( + .status.phase=="Available" + and .spec.azureDisk.diskURI!=null + ) + | . as $orig + | ($orig.spec.azureDisk.diskURI) as $disk + | ( + $all[] + | select( + .spec.csi!=null + and .spec.csi.volumeHandle==$disk + and (.metadata.labels["disk.csi.azure.com/created-by"]==$tool) + and (.metadata.name|endswith($suf)) + ) + ) as $csi + | select($csi!=null) + | [ + $orig.metadata.name, + ($orig.spec.storageClassName // ""), + ($orig.spec.capacity.storage // ""), + $disk, + $csi.metadata.name, + ($csi.spec.storageClassName // ""), + ($orig.spec.persistentVolumeReclaimPolicy // "") + ] | @tsv + ] + | .[] + ' 2>/dev/null || true)" + fi + + for ENTRY in "${MIG_PVCS[@]}"; do + local ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + local done lbl pv + lbl=$(kcmd get pvc "$pvc" -n "$ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" 2>/dev/null || true) + pv=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ "$lbl" == "$MIGRATION_DONE_LABEL_VALUE" ]] && done=true || done=false + + local snap="" int_pvc="" int_pv="" pv2_pvc="" + if [[ -n "$pv" ]]; then + snap="$(name_snapshot "$pv")" + int_pv="$(name_csi_pv "$pv")" + fi + if [[ "$mode" == "inplace" || "$mode" == "attrclass" ]]; then + # get from dataSourceRef.name from spec of PVC + snap="$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.dataSourceRef.name}' 2>/dev/null || true)" + fi + int_pvc="$(name_csi_pvc "$pvc")" + pv2_pvc="$(name_pv2_pvc "$pvc")" + + if [[ "$done" == "true" ]]; then + $success_header_printed || { + echo + ok "Cleanup candidates (successful migrations)" + echo " (Only resources that still exist and are labeled by this tool are listed)" + success_header_printed=true + } + any=true + + if [[ "$mode" == "dual" ]]; then + echo " Source PVC: $ns/$pvc" + if kcmd get pvc "$int_pvc" -n "$ns" >/dev/null 2>&1 && \ + kcmd get pvc "$int_pvc" -n "$ns" -o json | jq -e --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" '.metadata.labels[$k]==$v' >/dev/null; then + echo " - delete intermediate PVC: kubectl delete pvc $int_pvc -n $ns" + fi + if kcmd get pv "$int_pv" >/dev/null 2>&1 && \ + kcmd get pv "$int_pv" -o json | jq -e --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" '.metadata.labels[$k]==$v' >/dev/null; then + echo " - delete intermediate PV: kubectl delete pv $int_pv" + fi + if kcmd get pvc "$pv2_pvc" -n "$ns" >/dev/null 2>&1 && \ + kcmd get pvc "$pv2_pvc" -n "$ns" -o json | jq -e --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" '.metadata.labels[$k]==$v' >/dev/null; then + echo " - (optional) pv2 PVC present: $pv2_pvc (KEEP unless decommissioning)" + fi + else + if kcmd get pvc "$int_pvc" -n "$ns" >/dev/null 2>&1 && \ + kcmd get pvc "$int_pvc" -n "$ns" -o json | jq -e --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" '.metadata.labels[$k]==$v' >/dev/null; then + echo " Source PVC: $ns/$pvc" + echo " - delete intermediate PVC: kubectl delete pvc $int_pvc -n $ns" + fi + fi + if [[ -n "$snap" ]] && kcmd get volumesnapshot "$snap" -n "$ns" >/dev/null 2>&1 && \ + kcmd get volumesnapshot "$snap" -n "$ns" -o json | jq -e --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" '.metadata.labels[$k]==$v' >/dev/null; then + echo " - delete snapshot: kubectl delete volumesnapshot $snap -n $ns" + fi + if [[ "$mode" == "dual" && -n "$pv" ]]; then + echo " - (review) original PV: $pv" + fi + else + $failed_header_printed || { + echo + warn "Artifacts for incomplete/pending migrations" + echo " (Review before deletion; may be needed for retry/rollback)" + failed_header_printed=true + } + any=true + echo " Incomplete PVC: $ns/$pvc" + + if [[ "$mode" == "dual" ]]; then + [[ "$(kcmd get pvc "$int_pvc" -n "$ns" -o name 2>/dev/null || true)" ]] && \ + echo " - intermediate PVC exists: $int_pvc" + [[ "$(kcmd get pv "$int_pv" -o name 2>/dev/null || true)" ]] && \ + echo " - intermediate PV exists: $int_pv" + [[ "$(kcmd get pvc "$pv2_pvc" -n "$ns" -o name 2>/dev/null || true)" ]] && \ + echo " - pv2 PVC (target) exists: $pv2_pvc" + else + [[ "$(kcmd get pvc "$pvc" -n "$ns" -o name 2>/dev/null || true)" ]] && \ + echo " - current PVC phase: $(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.status.phase}' 2>/dev/null || true)" + fi + if [[ -n "$snap" ]] && kcmd get volumesnapshot "$snap" -n "$ns" >/dev/null 2>&1; then + echo " - snapshot exists: $snap" + fi + [[ -n "$pv" ]] && echo " - source PV: $pv" + echo " - retry guidance: leave artifacts intact; script will reuse them." + fi + + # PremiumV2 Released PVs referencing this claim (leftover pv2 PVs) + if [[ -n "$released_pv_lines" ]]; then + local had_rel=false + while IFS=$'\t' read -r rns rpvc rpv rreclaim rsc rcap rsku; do + [[ -z "$rns" ]] && continue + if [[ "$rns" == "$ns" && "$rpvc" == "$pvc" && "$rpv" != "$pv" ]]; then + $had_rel || { echo " - released PremiumV2 PV(s) associated (not currently) with claim:"; had_rel=true; } + echo " * $rpv (sku=$rsku reclaim=${rreclaim:-?} sc=${rsc:-?} size=${rcap:-?})" + echo " inspect: kubectl describe pv $rpv" + echo " delete : kubectl delete pv $rpv # after verifying data & rollback success" + fi + done <<< "$released_pv_lines" + fi + done + + # AttrClass extra: list original in-tree PVs now Available with CSI twin + if [[ "$mode" == "attrclass" && -n "$available_intree_lines" ]]; then + echo + ok "Original in-tree PVs (phase=Available) with CSI replacement present" + echo " (These are the original in-tree PVs; after verifying successful migration & no rollback need, you may delete them.)" + while IFS=$'\t' read -r origPV origSC origSize diskURI twinPV twinSC origRECLAIM; do + [[ -z "$origPV" ]] && continue + echo " - $origPV (sc=${origSC:-?} size=${origSize:-?}) diskURI=$diskURI" + echo " twin CSI PV: $twinPV (sc=${twinSC:-?})" + echo " inspect (Reclaim: $origRECLAIM) : kubectl describe pv $origPV" + echo " delete : kubectl delete pv $origPV # non-destructive if reclaimPolicy=Retain" + done <<< "$available_intree_lines" + fi + + if [[ "$any" == "false" ]]; then + info "No PVC entries to report." + else + echo + info "Report complete. (No actions performed.)" + fi +} + +run_without_errexit() { + LAST_RUN_WITHOUT_ERREXIT_RC=0 + local errexit_was_set=false + [[ $- == *e* ]] && errexit_was_set=true + + set +e + "$@" + + LAST_RUN_WITHOUT_ERREXIT_RC=$? + $errexit_was_set && set -e +} + +require_bins \ No newline at end of file diff --git a/hack/premium-to-premiumv2-migrator-dualpvc.sh b/hack/premium-to-premiumv2-migrator-dualpvc.sh new file mode 100755 index 0000000000..60198a984c --- /dev/null +++ b/hack/premium-to-premiumv2-migrator-dualpvc.sh @@ -0,0 +1,339 @@ +#!/usr/bin/env bash +# shellcheck source=./lib-premiumv2-migration-common.sh +set -euo pipefail +IFS=$'\n\t' + +SCRIPT_START_TS="$(date +'%Y-%m-%dT%H:%M:%S')" +SCRIPT_START_EPOCH="$(date +%s)" +MODE=dual + +# Declarations +declare -a MIG_PVCS +declare -a PREREQ_ISSUES +declare -a CONFLICT_ISSUES +declare -a PV2_CREATE_FAILURES +declare -a PV2_BIND_TIMEOUTS +declare -a NON_DETACHED_PVCS # PVCs we skipped because workloads still attached +declare -A NON_DETACHED_SET # Fast membership check: key = "ns|pvc" + +MIG_PVCS=() +PREREQ_ISSUES=() +CONFLICT_ISSUES=() +PV2_CREATE_FAILURES=() +PV2_BIND_TIMEOUTS=() +NON_DETACHED_PVCS=() +NON_DETACHED_SET=() + +cleanup_on_error() { + finalize_audit_summary "$SCRIPT_START_TS" "$SCRIPT_START_EPOCH" || true + err "Script failed (exit=$rc); cleanup_on_error ran." +} +trap 'rc=$?; if [ $rc -ne 0 ]; then cleanup_on_error $rc; fi' EXIT + +# (after array zeroing, before ensure_no_foreign_conflicts) +safe_array_len() { + local name="$1" + # If array not declared (future refactor), return 0 instead of tripping set -u + declare -p "$name" &>/dev/null || { echo 0; return; } + eval "echo \${#${name}[@]}" +} + +# Zonal-aware helper lib +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ZONAL_HELPER_LIB="${SCRIPT_DIR}/premium-to-premiumv2-zonal-aware-helper.sh" +[[ -f "$ZONAL_HELPER_LIB" ]] || { echo "[ERROR] Missing lib: $ZONAL_HELPER_LIB" >&2; exit 1; } +# shellcheck disable=SC1090 +. "$ZONAL_HELPER_LIB" + +# Run RBAC preflight (mode=dual) +MODE=$MODE migration_rbac_check || { err "Aborting due to insufficient permissions."; exit 1; } + +info "Migration label: $MIGRATION_LABEL" +info "Target snapshot class: $SNAPSHOT_CLASS" +info "Max PVCs: $MAX_PVCS MIG_SUFFIX=$MIG_SUFFIX WAIT_FOR_WORKLOAD=$WAIT_FOR_WORKLOAD" + +# Snapshot, SC variant, workload detach, event helpers now in lib +ensure_snapshot_class # from common lib + +# ---------------- Discover Tagged PVCs ---------------- +MODE=$MODE process_pvcs_for_zone_preparation + +# --- Conflict / prerequisite logic reused only here (kept local) --- +ensure_no_foreign_conflicts() { + info "Checking for pre-existing conflicting objects not created by this tool..." + if kcmd get volumesnapshotclass "$SNAPSHOT_CLASS" >/dev/null 2>&1; then + if ! kcmd get volumesnapshotclass "$SNAPSHOT_CLASS" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" \ + '.metadata.labels[$k]==$v' >/dev/null; then + CONFLICT_ISSUES+=("VolumeSnapshotClass/$SNAPSHOT_CLASS (missing label)") + fi + fi + for ENTRY in "${MIG_PVCS[@]}"; do + local ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + local pv2_pvc pv diskuri int_pvc int_pv snap + pv2_pvc="$(name_pv2_pvc "$pvc")" + if kcmd get pvc "$pv2_pvc" -n "$ns" >/dev/null 2>&1; then + if ! kcmd get pvc "$pv2_pvc" -n "$ns" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" \ + '.metadata.labels[$k]==$v' >/dev/null; then + CONFLICT_ISSUES+=("PVC/$ns/$pv2_pvc (pv2) missing label") + fi + fi + pv=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ -z "$pv" ]] && continue + diskuri=$(kcmd get pv "$pv" -o jsonpath='{.spec.azureDisk.diskURI}' 2>/dev/null || true) + if [[ -n "$diskuri" ]]; then + int_pvc="$(name_csi_pvc "$pvc")" + int_pv="$(name_csi_pv "$pv")" + if kcmd get pvc "$int_pvc" -n "$ns" >/dev/null 2>&1; then + kcmd get pvc "$int_pvc" -n "$ns" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" \ + '.metadata.labels[$k]==$v' >/dev/null || \ + CONFLICT_ISSUES+=("PVC/$ns/$int_pvc (intermediate) missing label") + fi + if kcmd get pv "$int_pv" >/dev/null 2>&1; then + kcmd get pv "$int_pv" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" \ + '.metadata.labels[$k]==$v' >/dev/null || \ + CONFLICT_ISSUES+=("PV/$int_pv (intermediate) missing label") + fi + fi + snap="$(name_snapshot "$pv")" + if kcmd get volumesnapshot "$snap" -n "$ns" >/dev/null 2>&1; then + kcmd get volumesnapshot "$snap" -n "$ns" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" \ + '.metadata.labels[$k]==$v' >/dev/null || \ + CONFLICT_ISSUES+=("VolumeSnapshot/$ns/$snap missing label") + fi + done + if (( $(safe_array_len CONFLICT_ISSUES) > 0 )); then + err "Conflict check failed ($(safe_array_len CONFLICT_ISSUES) items)." + printf ' - %s\n' "${CONFLICT_ISSUES[@]}" + exit 1 + fi + ok "No conflicting pre-existing objects detected." +} + +# ------------- Pre-Req & Conflicts ------------- +print_combined_validation_report_and_exit_if_needed() { + local prereq_count + prereq_count=$(safe_array_len PREREQ_ISSUES) + local conflict_count + conflict_count=$(safe_array_len CONFLICT_ISSUES) + if (( prereq_count==0 && conflict_count==0 )); then + ok "Pre-req & conflict checks passed." + return + fi + echo + err "Pre-run validation failed:" + (( prereq_count>0 )) && { echo " Pre-requisite issues ($prereq_count):"; printf ' - %s\n' "${PREREQ_ISSUES[@]}"; } + (( conflict_count>0 )) && { echo " Naming conflicts ($conflict_count):"; printf ' - %s\n' "${CONFLICT_ISSUES[@]}"; } + echo + err "Resolve issues and re-run. (No mutations performed.)" + exit 1 +} + +run_prerequisites_and_conflicts() { + run_prerequisites_checks + ensure_no_foreign_conflicts + print_combined_validation_report_and_exit_if_needed +} + +run_prerequisites_and_conflicts + +# ------------- Collect Unique Source SCs ------------- +create_unique_storage_classes + +# ------------- Main Migration Loop ------------- +declare -A PVC_SNAPSHOTS +for ENTRY in "${MIG_PVCS[@]}"; do + pvc_ns="${ENTRY%%|*}" + pvc="${ENTRY##*|}" + + if ! check_premium_lrs "$pvc" "$pvc_ns"; then + info "PVC $pvc_ns/$pvc not Premium_LRS -> skip" + continue + fi + + DONE_LABEL=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" 2>/dev/null || true) + [[ "$DONE_LABEL" == "$MIGRATION_DONE_LABEL_VALUE" ]] && { info "Already migrated $pvc_ns/$pvc"; continue; } + + pv=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ -z "$pv" ]] && { warn "PVC $pvc_ns/$pvc not bound; skip"; continue; } + + if [[ "$WAIT_FOR_WORKLOAD" == "true" ]]; then + attachments=$(kcmd get volumeattachment -o jsonpath="{range .items[?(@.spec.source.persistentVolumeName=='$pv')]}{.metadata.name}{'\n'}{end}" 2>/dev/null || true) + if [[ -n "$attachments" ]]; then + if ! wait_for_workload_detach "$pv" "$pvc" "$pvc_ns"; then + warn "Workload still attached -> deferring migration for $pvc_ns/$pvc (recording in NON_DETACHED_PVCS; no labels changed)." + NON_DETACHED_PVCS+=("${pvc_ns}|${pvc}") + NON_DETACHED_SET["${pvc_ns}|${pvc}"]=1 + continue + fi + fi + fi + + ensure_reclaim_policy_retain "$pv" + + mode=$(kcmd get pv "$pv" -o jsonpath='{.spec.volumeMode}' 2>/dev/null || true) + sc=$(get_sc_of_pvc "$pvc" "$pvc_ns") + size=$(kcmd get pv "$pv" -o jsonpath='{.spec.capacity.storage}' 2>/dev/null || true) + diskuri=$(kcmd get pv "$pv" -o jsonpath='{.spec.azureDisk.diskURI}' 2>/dev/null || true) + scpv2="$(name_pv2_sc "$sc")" + + csi_driver=$(kcmd get pv "$pv" -o jsonpath='{.spec.csi.driver}' 2>/dev/null || true) + if [[ -n "$diskuri" ]]; then + if [[ -z "$sc" || -z "$size" ]]; then warn "Missing sc/size for in-tree $pvc_ns/$pvc"; continue; fi + scpv1="$(name_pv1_sc "$sc")" + create_csi_pv_pvc "$pvc" "$pvc_ns" "$pv" "$size" "$mode" "${scpv1}" "$diskuri" + snapshot_source_pvc="$(name_csi_pvc "$pvc")" + else + [[ "$csi_driver" != "disk.csi.azure.com" ]] && { warn "Unknown PV driver for $pv"; continue; } + if [[ -z "$scpv2" || -z "$size" ]]; then warn "Missing sc/size for CSI $pvc_ns/$pvc"; continue; fi + kcmd get sc "${scpv2}" >/dev/null 2>&1 || { warn "Missing ${scpv2}"; continue; } + snapshot_source_pvc="$pvc" + fi + + snapshot="$(name_snapshot "$pv")" + create_snapshot "$snapshot" "$snapshot_source_pvc" "$pvc_ns" "$pv" || { warn "Snapshot failed $pvc_ns/$pvc"; continue; } + PVC_SNAPSHOTS+=("${pvc_ns}|${snapshot}|${pvc}") +done + +wait_for_snapshots_ready + +SOURCE_SNAPSHOTS=("${!PVC_SNAPSHOTS[@]}") +for ENTRY in "${SOURCE_SNAPSHOTS[@]}"; do + IFS='|' read -r pvc_ns snapshot pvc <<< "$ENTRY" + pv=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + size=$(kcmd get pv "$pv" -o jsonpath='{.spec.capacity.storage}' 2>/dev/null || true) + mode=$(kcmd get pv "$pv" -o jsonpath='{.spec.volumeMode}' 2>/dev/null || true) + sc=$(get_sc_of_pvc "$pvc" "$pvc_ns") + scpv2="$(name_pv2_sc "$sc")" + pv2_pvc="$(name_pv2_pvc "$pvc")" + + run_without_errexit create_pvc_from_snapshot "$pvc" "$pvc_ns" "$pv" "$size" "$mode" "$scpv2" "$pv2_pvc" "$snapshot" + case "$LAST_RUN_WITHOUT_ERREXIT_RC" in + 0) ;; # success + 1) PV2_CREATE_FAILURES+=("${pvc_ns}/${pvc}") ;; + 2) PV2_BIND_TIMEOUTS+=("${pvc_ns}/${pvc}") ;; + esac +done + +# ------------- Monitoring Loop ------------- +deadline=$(( $(date +%s) + MONITOR_TIMEOUT_MINUTES*60 )) +info "Monitoring migrations (timeout ${MONITOR_TIMEOUT_MINUTES}m)..." + +while true; do + ALL_DONE=true + for ENTRY in "${MIG_PVCS[@]}"; do + pvc_ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" pv2_pvc="$(name_pv2_pvc "$pvc")" + + # Skip monitoring entirely for PVCs we never started due to attachment + if [[ ${NON_DETACHED_SET["${pvc_ns}|${pvc}"]+x} ]]; then + continue + fi + + lbl=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" || true) + if [[ "$lbl" == "$MIGRATION_DONE_LABEL_VALUE" || "$lbl" == "$MIGRATION_DONE_LABEL_VALUE_FALSE" ]]; then + continue + fi + + STATUS=$(kcmd get pvc "$pv2_pvc" -n "$pvc_ns" -o jsonpath='{.status.phase}' 2>/dev/null || true) + [[ "$STATUS" != "Bound" ]] && ALL_DONE=false + reason=$(extract_event_reason "$pvc_ns" "$pv2_pvc") + case "$reason" in + SKUMigrationCompleted) + mark_source_done "$pvc_ns" "$pvc" + ok "Completed $pvc_ns/$pvc" + continue + ;; + SKUMigrationProgress|SKUMigrationStarted) + mark_source_in_progress "$pvc_ns" "$pvc" + info "$reason $pvc_ns/$pv2_pvc"; ALL_DONE=false ;; + ReasonSKUMigrationTimeout) + mark_source_notdone "$pvc_ns" "$pvc" + warn "$reason $pvc_ns/$pv2_pvc"; ALL_DONE=false ;; + "") + info "No migration events yet for $pvc_ns/$pv2_pvc"; ALL_DONE=false ;; + esac + if [[ "$STATUS" == "Bound" && -z "$reason" && $MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES -gt 0 ]]; then + orig_pv=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ -z "$orig_pv" ]] && continue + inprog_val=$(kcmd get pv "$orig_pv" -o go-template="{{ index .metadata.labels \"${MIGRATION_INPROGRESS_LABEL_KEY}\" }}" 2>/dev/null || true) + if [[ "$inprog_val" != "$MIGRATION_INPROGRESS_LABEL_VALUE" ]]; then + cts=$(kcmd get pvc "$pv2_pvc" -n "$pvc_ns" -o jsonpath='{.metadata.creationTimestamp}' 2>/dev/null || true) + if [[ -n "$cts" ]]; then + creation_epoch=$(date -d "$cts" +%s 2>/dev/null || echo 0) + now_epoch=$(date +%s) + if (( now_epoch - creation_epoch >= MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES * 60 )); then + warn "Forcing ${MIGRATION_INPROGRESS_LABEL_KEY}=${MIGRATION_INPROGRESS_LABEL_VALUE} on PV $orig_pv (no events)" + kcmd label pv "$orig_pv" "${MIGRATION_INPROGRESS_LABEL_KEY}=${MIGRATION_INPROGRESS_LABEL_VALUE}" --overwrite + audit_add "PersistentVolume" "$orig_pv" "" "label" \ + "kubectl label pv $orig_pv ${MIGRATION_INPROGRESS_LABEL_KEY}-" \ + "forced=true reason=noEventsAfter${MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES}m" + fi + fi + fi + fi + done + if [[ "$ALL_DONE" == "true" ]]; then + ok "All processed PVCs migrated or already labeled." + break + fi + if (( $(date +%s) >= deadline )); then + warn "Monitor timeout reached." + for ENTRY in "${MIG_PVCS[@]}"; do + pvc_ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + # Ignore non-detached deferred PVCs + if [[ ${NON_DETACHED_SET["${pvc_ns}|${pvc}"]+x} ]]; then + continue + fi + kcmd get pvc "$pvc" -n "$pvc_ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" | \ + grep -q "^${MIGRATION_DONE_LABEL_VALUE}\$" && continue + orig_pv=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ -z "$orig_pv" ]] && continue + inprog_val=$(kcmd get pv "$orig_pv" -o go-template="{{ index .metadata.labels \"${MIGRATION_INPROGRESS_LABEL_KEY}\" }}" 2>/dev/null || true) + if [[ "$inprog_val" != "$MIGRATION_INPROGRESS_LABEL_VALUE" ]]; then + warn "Timeout fallback: labeling PV $orig_pv with ${MIGRATION_INPROGRESS_LABEL_KEY}=${MIGRATION_INPROGRESS_LABEL_VALUE}" + kcmd label pv "$orig_pv" "${MIGRATION_INPROGRESS_LABEL_KEY}=${MIGRATION_INPROGRESS_LABEL_VALUE}" --overwrite + audit_add "PersistentVolume" "$orig_pv" "" "label" \ + "kubectl label pv $orig_pv ${MIGRATION_INPROGRESS_LABEL_KEY}-" \ + "forced=true reason=monitorTimeout" + fi + done + break + fi + sleep "$POLL_INTERVAL" +done + +# ------------- Summary & Audit ------------- +echo +info "Summary:" +if (( ${#PV2_CREATE_FAILURES[@]} > 0 )); then + echo + warn "pv2 PVC creation failures (no resource or create error):" + printf ' - %s\n' "${PV2_CREATE_FAILURES[@]}" +fi +if (( ${#PV2_BIND_TIMEOUTS[@]} > 0 )); then + echo + warn "pv2 PVC bind timeouts:" + printf ' - %s\n' "${PV2_BIND_TIMEOUTS[@]}" +fi + +for entry in "${MIG_PVCS[@]}"; do + ns="${entry%%|*}" pvc="${entry##*|}" + if [[ ${NON_DETACHED_SET["${ns}|${pvc}"]+x} ]]; then + echo " - ${ns}/${pvc} deferred (workload still attached)" + continue + fi + lbl=$(kcmd get pvc "$pvc" -n "$ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" 2>/dev/null || true) + if [[ "$lbl" == "$MIGRATION_DONE_LABEL_VALUE" ]]; then + echo " ✓ $ns/$pvc migrated" + else + echo " • $ns/$pvc NOT completed" + fi +done + +MODE=$MODE print_migration_cleanup_report +finalize_audit_summary "$SCRIPT_START_TS" "$SCRIPT_START_EPOCH" +ok "Script finished." \ No newline at end of file diff --git a/hack/premium-to-premiumv2-migrator-inplace.sh b/hack/premium-to-premiumv2-migrator-inplace.sh new file mode 100755 index 0000000000..0f9612ed98 --- /dev/null +++ b/hack/premium-to-premiumv2-migrator-inplace.sh @@ -0,0 +1,386 @@ +#!/usr/bin/env bash +# shellcheck source=./lib-premiumv2-migration-common.sh +set -euo pipefail +IFS=$'\n\t' + +SCRIPT_START_TS="$(date +'%Y-%m-%dT%H:%M:%S')" +SCRIPT_START_EPOCH="$(date +%s)" +MODE=inplace + +# Declarations +declare -a MIG_PVCS +declare -a PREREQ_ISSUES +declare -a CONFLICT_ISSUES +declare -a ROLLBACK_FAILURES +declare -a NON_DETACHED_PVCS # PVCs skipped because workload still attached +declare -A NON_DETACHED_SET # membership map ns|pvc -> 1 + +# Explicit zeroing (defensive; avoids 'unbound variable' under set -u even if declarations are edited later) +MIG_PVCS=() +PREREQ_ISSUES=() +CONFLICT_ISSUES=() +ROLLBACK_FAILURES=() +NON_DETACHED_PVCS=() +NON_DETACHED_SET=() + +cleanup_on_error() { + finalize_audit_summary "$SCRIPT_START_TS" "$SCRIPT_START_EPOCH" || true + err "Script failed (exit=$rc); cleanup_on_error ran." +} +trap 'rc=$?; if [ $rc -ne 0 ]; then cleanup_on_error $rc; fi' EXIT + +# Helper: safe length (avoids accidental nounset trip if future refactor removes a declare) +safe_array_len() { + # usage: safe_array_len arrayName + local name="$1" + declare -p "$name" &>/dev/null || { echo 0; return; } + # indirect expansion + eval "echo \${#${name}[@]}" +} + +# Zonal-aware helper lib +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ZONAL_HELPER_LIB="${SCRIPT_DIR}/premium-to-premiumv2-zonal-aware-helper.sh" +[[ -f "$ZONAL_HELPER_LIB" ]] || { echo "[ERROR] Missing lib: $ZONAL_HELPER_LIB" >&2; exit 1; } +# shellcheck disable=SC1090 +. "$ZONAL_HELPER_LIB" + +# Run RBAC preflight (mode=inplace) +MODE=$MODE migration_rbac_check || { err "Aborting due to insufficient permissions."; exit 1; } + +info "Migration label: $MIGRATION_LABEL" +info "Target snapshot class: $SNAPSHOT_CLASS" +info "Max PVCs: $MAX_PVCS MIG_SUFFIX=$MIG_SUFFIX WAIT_FOR_WORKLOAD=$WAIT_FOR_WORKLOAD" + +# Snapshot, SC variant, workload detach, event helpers now in lib +ensure_snapshot_class # from common lib + +# --- Conflict / prerequisite logic reused only here (kept local) --- +ensure_no_foreign_conflicts() { + info "Checking for pre-existing conflicting objects not created by this tool..." + if kcmd get volumesnapshotclass "$SNAPSHOT_CLASS" >/dev/null 2>&1; then + if ! kcmd get volumesnapshotclass "$SNAPSHOT_CLASS" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" \ + '.metadata.labels[$k]==$v' >/dev/null; then + CONFLICT_ISSUES+=("VolumeSnapshotClass/$SNAPSHOT_CLASS (missing label)") + fi + fi + for ENTRY in "${MIG_PVCS[@]}"; do + local ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + local pv snap + pv=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ -z "$pv" ]] && continue + snap="$(name_snapshot "$pv")" + if kcmd get volumesnapshot "$snap" -n "$ns" >/dev/null 2>&1; then + kcmd get volumesnapshot "$snap" -n "$ns" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" \ + '.metadata.labels[$k]==$v' >/dev/null || \ + CONFLICT_ISSUES+=("VolumeSnapshot/$ns/$snap missing label") + fi + done + if (( $(safe_array_len CONFLICT_ISSUES) > 0 )); then + err "Conflict check failed ($(safe_array_len CONFLICT_ISSUES) items)." + printf ' - %s\n' "${CONFLICT_ISSUES[@]}" + exit 1 + fi + ok "No conflicting pre-existing objects detected." +} + +# ---------------- Discover Tagged PVCs ---------------- +MODE=$MODE process_pvcs_for_zone_preparation + +# ---------------- Pre-Requisite Validation (mirrors dual script) ---------------- +print_combined_validation_report_and_exit_if_needed() { + local prereq_count + prereq_count=$(safe_array_len PREREQ_ISSUES) + local conflict_count + conflict_count=$(safe_array_len CONFLICT_ISSUES) + if (( prereq_count==0 && conflict_count==0 )); then + ok "Pre-req & conflict checks passed." + return + fi + echo + err "Pre-run validation failed:" + (( prereq_count>0 )) && { echo " Pre-requisite issues ($prereq_count):"; printf ' - %s\n' "${PREREQ_ISSUES[@]}"; } + (( conflict_count>0 )) && { echo " Naming conflicts ($conflict_count):"; printf ' - %s\n' "${CONFLICT_ISSUES[@]}"; } + echo + err "Resolve issues and re-run. (No mutations performed.)" + exit 1 +} + +run_prerequisites_and_conflicts() { + run_prerequisites_checks + ensure_no_foreign_conflicts + print_combined_validation_report_and_exit_if_needed +} + +run_prerequisites_and_conflicts + +# ------------- Collect Unique Source SCs ------------- +create_unique_storage_classes + +rollback_inplace() { + local ns="$1" pvc="$2" + warn "Rollback for $ns/$pvc" + local encoded + encoded=$(kcmd get pvc "$pvc" -n "$ns" -o go-template="{{ index .metadata.annotations \"${ROLLBACK_PVC_YAML_ANN}\" }}" 2>/dev/null || true) + if [[ -z "$encoded" ]]; then + warn "No rollback annotation" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-skip" "N/A" "reason=noAnnotation" + ROLLBACK_FAILURES+=("${ns}/${pvc}:no-annotation") + return 1 + fi + local spec_doc + spec_doc=$(printf '%s' "$encoded" | b64d) + + if [[ -z "$spec_doc" ]]; then + warn "Empty rollback spec" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-empty-spec" "N/A" "reason=emptySpec" + ROLLBACK_FAILURES+=("${ns}/${pvc}:empty-spec") + return 1 + fi + + # Delete pv2 PVC (best-effort) + encoded_spec=$(get_pvc_encoded_json "$pvc" "$ns") + if ! kcmd delete pvc "$pvc" -n "$ns" --wait=true; then + warn "Rollback delete failed (continuing) for $ns/$pvc" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-delete-failed" "N/A" "reason=deleteError" + # Continue – apply may still succeed if object was already gone + else + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-delete" "kubectl create -f <(echo \"$encoded_spec\" | base64 --decode) " "inplace=true reason=rollbackreplace" + fi + + # Clear the claimRef on the original PV to allow re-binding + local orig_pv + orig_pv=$(printf '%s' "$spec_doc" | jq -r '.spec.volumeName // empty') + if [[ -z "$orig_pv" ]]; then + warn "Rollback missing original PV name in spec" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-missing-pv" "N/A" "reason=missingPV" + ROLLBACK_FAILURES+=("${ns}/${pvc}:missing-pv") + return 1 + fi + if ! kcmd patch pv "$orig_pv" -p '{"spec":{"claimRef":null}}' >/dev/null 2>&1; then + warn "Rollback clear claimRef failed for PV $orig_pv" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-clear-claimref-failed" "kubectl describe pv $orig_pv" "reason=patchError" + ROLLBACK_FAILURES+=("${ns}/${pvc}:clear-claimref-failed") + return 1 + fi + + # Recreate original PVC using retrying apply + if ! printf '%s\n' "$spec_doc" | kapply_retry; then + err "Rollback apply failed for $ns/$pvc" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-restore-failed" "kubectl get pvc $pvc -n $ns -o yaml" "original=true reason=applyError" + ROLLBACK_FAILURES+=("${ns}/${pvc}:apply-failed") + return 1 + fi + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-restore" "kubectl delete pvc $pvc -n $ns" "original=true" + + if wait_pvc_bound "$ns" "$pvc" 600; then + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-bound" "kubectl describe pvc $pvc -n $ns" "" + return 0 + else + warn "Rollback PVC $ns/$pvc not yet bound (600s timeout)" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "rollback-bind-timeout" "kubectl describe pvc $pvc -n $ns" "timeout=600s" + ROLLBACK_FAILURES+=("${ns}/${pvc}:bind-timeout") + return 1 + fi +} + +# ------------- Main Migration Loop ------------- +declare -A PVC_SNAPSHOTS +for ENTRY in "${MIG_PVCS[@]}"; do + pvc_ns="${ENTRY%%|*}" + pvc="${ENTRY##*|}" + + if check_premiumv2_lrs "$pvc_ns" "$pvc"; then + info "PVC $pvc_ns/$pvc not PremiumV2_LRS -> skip" + continue + fi + + DONE_LABEL=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" 2>/dev/null || true) + [[ "$DONE_LABEL" == "$MIGRATION_DONE_LABEL_VALUE" ]] && { info "Already migrated $pvc_ns/$pvc"; continue; } + + pv=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ -z "$pv" ]] && { warn "PVC $pvc_ns/$pvc not bound; skip"; continue; } + + if [[ "$WAIT_FOR_WORKLOAD" == "true" ]]; then + attachments=$(kcmd get volumeattachment -o jsonpath="{range .items[?(@.spec.source.persistentVolumeName=='$pv')]}{.metadata.name}{'\n'}{end}" 2>/dev/null || true) + if [[ -n "$attachments" ]]; then + if ! wait_for_workload_detach "$pv" "$pvc" "$pvc_ns"; then + warn "Workload still attached -> deferring migration of $pvc_ns/$pvc (no labels changed)" + NON_DETACHED_PVCS+=("${pvc_ns}|${pvc}") + NON_DETACHED_SET["${pvc_ns}|${pvc}"]=1 + continue + fi + fi + fi + + ensure_reclaim_policy_retain "$pv" + + mode=$(kcmd get pv "$pv" -o jsonpath='{.spec.volumeMode}' 2>/dev/null || true) + sc=$(get_sc_of_pvc "$pvc" "$pvc_ns") + size=$(kcmd get pv "$pv" -o jsonpath='{.spec.capacity.storage}' 2>/dev/null || true) + diskuri=$(kcmd get pv "$pv" -o jsonpath='{.spec.azureDisk.diskURI}' 2>/dev/null || true) + scpv2="$(name_pv2_sc "$sc")" + + csi_driver=$(kcmd get pv "$pv" -o jsonpath='{.spec.csi.driver}' 2>/dev/null || true) + if [[ -n "$diskuri" ]]; then + if [[ -z "$sc" || -z "$size" ]]; then warn "Missing sc/size for in-tree $pvc_ns/$pvc"; continue; fi + scpv1="$(name_pv1_sc "$sc")" + kcmd get sc "${scpv1}" >/dev/null 2>&1 || { warn "Missing ${scpv1}"; continue; } + create_csi_pv_pvc "$pvc" "$pvc_ns" "$pv" "$size" "$mode" "${scpv1}" "$diskuri" || { warn "Failed to create CSI PV/PVC for $pvc_ns/$pvc"; continue; } + snapshot_source_pvc="$(name_csi_pvc "$pvc")" + else + [[ "$csi_driver" != "disk.csi.azure.com" ]] && { warn "Unknown PV driver for $pv"; continue; } + if [[ -z "$scpv2" || -z "$size" ]]; then warn "Missing sc/size for CSI $pvc_ns/$pvc"; continue; fi + kcmd get sc "${scpv2}" >/dev/null 2>&1 || { warn "Missing ${scpv2}"; continue; } + snapshot_source_pvc="$pvc" + fi + + snapshot="$(name_snapshot "$pv")" + create_snapshot "$snapshot" "$snapshot_source_pvc" "$pvc_ns" "$pv" || { warn "Snapshot failed $pvc_ns/$pvc"; continue; } + + PVC_SNAPSHOTS+=("${pvc_ns}|${snapshot}|${pvc}") +done + +wait_for_snapshots_ready + +# ------------- Main Migration Loop ------------- +SOURCE_SNAPSHOTS=("${!PVC_SNAPSHOTS[@]}") +for ENTRY in "${SOURCE_SNAPSHOTS[@]}"; do + IFS='|' read -r pvc_ns snapshot pvc <<< "$ENTRY" + pv=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + + mode=$(kcmd get pv "$pv" -o jsonpath='{.spec.volumeMode}' 2>/dev/null || true) + sc=$(get_sc_of_pvc "$pvc" "$pvc_ns") + size=$(kcmd get pv "$pv" -o jsonpath='{.spec.capacity.storage}' 2>/dev/null || true) + diskuri=$(kcmd get pv "$pv" -o jsonpath='{.spec.azureDisk.diskURI}' 2>/dev/null || true) + scpv2="$(name_pv2_sc "$sc")" + + run_without_errexit create_pvc_from_snapshot "$pvc" "$pvc_ns" "$pv" "$size" "$mode" "$scpv2" "$pvc" "$snapshot" + rc=$LAST_RUN_WITHOUT_ERREXIT_RC + if [[ $rc -eq 0 ]]; then + ok "PV2 creation success $pvc_ns/$pvc" + audit_add "PersistentVolumeClaim" "$pvc" "$pvc_ns" "migrated" "kubectl describe pvc $pvc -n $pvc_ns" "mode=inplace" + elif [[ $rc -eq 2 ]]; then + warn "Timeout $pvc_ns/$pvc" + audit_add "PersistentVolumeClaim" "$pvc" "$pvc_ns" "migrate-timeout" "kubectl describe pvc $pvc -n $pvc_ns" "mode=inplace" + if [[ "$ROLLBACK_ON_TIMEOUT" == "true" ]]; then + rollback_inplace "$pvc_ns" "$pvc" || true + fi + else + warn "Migration failure rc=$rc $pvc_ns/$pvc" + audit_add "PersistentVolumeClaim" "$pvc" "$pvc_ns" "migrate-failed" "kubectl describe pvc $pvc -n $pvc_ns" "mode=inplace rc=$rc" + if [[ "$ROLLBACK_ON_TIMEOUT" == "true" ]]; then + rollback_inplace "$pvc_ns" "$pvc" || true + fi + fi +done + +# ------------- Monitoring Loop ------------- +deadline=$(( $(date +%s) + MONITOR_TIMEOUT_MINUTES*60 )) +info "Monitoring migrations (timeout ${MONITOR_TIMEOUT_MINUTES}m)..." + +while true; do + ALL_DONE=true + for ENTRY in "${MIG_PVCS[@]}"; do + pvc_ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + + # Skip monitoring for PVCs we never started due to attachment + if [[ ${NON_DETACHED_SET["${pvc_ns}|${pvc}"]+x} ]]; then + continue + fi + + lbl=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" || true) + if [[ "$lbl" == "$MIGRATION_DONE_LABEL_VALUE" || "$lbl" == "$MIGRATION_DONE_LABEL_VALUE_FALSE" ]]; then + continue + fi + + STATUS=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o jsonpath='{.status.phase}' 2>/dev/null || true) + [[ "$STATUS" != "Bound" ]] && ALL_DONE=false + reason=$(extract_event_reason "$pvc_ns" "$pvc") + case "$reason" in + SKUMigrationCompleted) + mark_source_done "$pvc_ns" "$pvc" + ok "Completed $pvc_ns/$pvc" + continue + ;; + SKUMigrationProgress|SKUMigrationStarted) + mark_source_in_progress "$pvc_ns" "$pvc" + info "$reason $pvc_ns/$pvc"; ALL_DONE=false ;; + ReasonSKUMigrationTimeout) + rollback_inplace "$pvc_ns" "$pvc" || true + mark_source_notdone "$pvc_ns" "$pvc" + warn "$reason $pvc_ns/$pvc"; ALL_DONE=false ;; + "") + info "No migration events yet for $pvc_ns/$pvc"; ALL_DONE=false ;; + esac + if [[ "$STATUS" == "Bound" && -z "$reason" && $MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES -gt 0 ]]; then + orig_pv=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ -z "$orig_pv" ]] && continue + inprog_val=$(kcmd get pv "$orig_pv" -o go-template="{{ index .metadata.labels \"${MIGRATION_INPROGRESS_LABEL_KEY}\" }}" 2>/dev/null || true) + if [[ "$inprog_val" != "$MIGRATION_INPROGRESS_LABEL_VALUE" ]]; then + cts=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o jsonpath='{.metadata.creationTimestamp}' 2>/dev/null || true) + if [[ -n "$cts" ]]; then + creation_epoch=$(date -d "$cts" +%s 2>/dev/null || echo 0) + now_epoch=$(date +%s) + if (( now_epoch - creation_epoch >= MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES * 60 )); then + warn "Forcing ${MIGRATION_INPROGRESS_LABEL_KEY}=${MIGRATION_INPROGRESS_LABEL_VALUE} on PV $orig_pv (no events)" + kcmd label pv "$orig_pv" "${MIGRATION_INPROGRESS_LABEL_KEY}=${MIGRATION_INPROGRESS_LABEL_VALUE}" --overwrite + audit_add "PersistentVolume" "$orig_pv" "" "label" \ + "kubectl label pv $orig_pv ${MIGRATION_INPROGRESS_LABEL_KEY}-" \ + "forced=true reason=noEventsAfter${MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES}m" + fi + fi + fi + fi + done + if [[ "$ALL_DONE" == "true" ]]; then + ok "All processed PVCs migrated or already labeled." + break + fi + if (( $(date +%s) >= deadline )); then + warn "Monitor timeout reached." + for ENTRY in "${MIG_PVCS[@]}"; do + pvc_ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + # Ignore non-detached deferred PVCs + if [[ ${NON_DETACHED_SET["${pvc_ns}|${pvc}"]+x} ]]; then + continue + fi + kcmd get pvc "$pvc" -n "$pvc_ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" | \ + grep -q "^${MIGRATION_DONE_LABEL_VALUE}\$" && continue + warn "Timeout rollback $pvc_ns/$pvc" + rollback_inplace "$pvc_ns" "$pvc" || true + done + break + fi + sleep "$POLL_INTERVAL" +done + +# ------------- Summary & Audit ------------- +echo +info "Summary:" +if (( ${#ROLLBACK_FAILURES[@]} > 0 )); then + echo + warn "Rollback failures:" + printf ' - %s\n' "${ROLLBACK_FAILURES[@]}" +fi + +for entry in "${MIG_PVCS[@]}"; do + ns="${entry%%|*}" pvc="${entry##*|}" + if [[ ${NON_DETACHED_SET["${ns}|${pvc}"]+x} ]]; then + echo " - ${ns}/${pvc} deferred (workload still attached)" + continue + fi + lbl=$(kcmd get pvc "$pvc" -n "$ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" 2>/dev/null || true) + if [[ "$lbl" == "$MIGRATION_DONE_LABEL_VALUE" ]]; then + echo " ✓ $ns/$pvc migrated" + else + echo " • $ns/$pvc NOT completed" + fi +done + +MODE=$MODE print_migration_cleanup_report +finalize_audit_summary "$SCRIPT_START_TS" "$SCRIPT_START_EPOCH" + +ok "Script finished." \ No newline at end of file diff --git a/hack/premium-to-premiumv2-migrator-vac.sh b/hack/premium-to-premiumv2-migrator-vac.sh new file mode 100755 index 0000000000..f7e59b39b1 --- /dev/null +++ b/hack/premium-to-premiumv2-migrator-vac.sh @@ -0,0 +1,384 @@ +#!/usr/bin/env bash +# shellcheck source=./lib-premiumv2-migration-common.sh +# shellcheck source=./premium-to-premiumv2-zonal-aware-helper.sh + +set -euo pipefail +IFS=$'\n\t' + +SCRIPT_START_TS="$(date +'%Y-%m-%dT%H:%M:%S')" +SCRIPT_START_EPOCH="$(date +%s)" +MODE=attrclass + +# Environment overrides specific to this mode (others inherited from lib): +ATTR_CLASS_NAME="${ATTR_CLASS_NAME:-azuredisk-premiumv2}" +ATTR_CLASS_API_VERSION="${ATTR_CLASS_API_VERSION:-storage.k8s.io/v1beta1}" # Update to v1 when GA. +TARGET_SKU="${TARGET_SKU:-PremiumV2_LRS}" +ATTR_CLASS_FORCE_RECREATE="${ATTR_CLASS_FORCE_RECREATE:-false}" +PV_POLL_INTERVAL_SECONDS="${PV_POLL_INTERVAL_SECONDS:-10}" +SKU_UPDATE_TIMEOUT_MINUTES="${SKU_UPDATE_TIMEOUT_MINUTES:-60}" +CSI_BASELINE_SC="${CSI_BASELINE_SC:-}" + +# Declarations (parity) +declare -a MIG_PVCS +declare -a PREREQ_ISSUES +declare -a CONFLICT_ISSUES +declare -a NON_DETACHED_PVCS # PVCs skipped because workload still attached +declare -A NON_DETACHED_SET # membership map ns|pvc -> 1 +PREREQ_ISSUES=() +CONFLICT_ISSUES=() +MIG_PVCS=() +NON_DETACHED_PVCS=() +NON_DETACHED_SET=() + +cleanup_on_error() { + local exit_code="$1" + finalize_audit_summary "$SCRIPT_START_TS" "$SCRIPT_START_EPOCH" || true + err "Script failed (exit=$exit_code); cleanup_on_error ran." +} +trap 'rc=$?; if [ $rc -ne 0 ]; then cleanup_on_error $rc; fi' EXIT + +# (after array zeroing, before ensure_no_foreign_conflicts) +safe_array_len() { + local name="$1" + # If array not declared (future refactor), return 0 instead of tripping set -u + declare -p "$name" &>/dev/null || { echo 0; return; } + eval "echo \${#${name}[@]}" +} + +# Zonal-aware helper lib +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ZONAL_HELPER_LIB="${SCRIPT_DIR}/premium-to-premiumv2-zonal-aware-helper.sh" +[[ -f "$ZONAL_HELPER_LIB" ]] || { echo "[ERROR] Missing lib: $ZONAL_HELPER_LIB" >&2; exit 1; } +# shellcheck disable=SC1090 +. "$ZONAL_HELPER_LIB" + +# Run RBAC preflight (mode=inplace) +MODE=$MODE migration_rbac_check || { err "Aborting due to insufficient permissions."; exit 1; } + +info "Migration label: $MIGRATION_LABEL" +info "Target snapshot class: $SNAPSHOT_CLASS" +info "ATTR_CLASS_NAME=${ATTR_CLASS_NAME} TARGET_SKU=${TARGET_SKU}" +info "Max PVCs: $MAX_PVCS MIG_SUFFIX=$MIG_SUFFIX WAIT_FOR_WORKLOAD=$WAIT_FOR_WORKLOAD" + +# Snapshot, VAC, SC variant, workload detach, event helpers now in lib +ensure_snapshot_class # from common lib +ensure_volume_attributes_class + +# --- Conflict / prerequisite logic reused only here (kept local) --- +ensure_no_foreign_conflicts() { + info "Checking for pre-existing conflicting objects not created by this tool..." + if kcmd get volumesnapshotclass "$SNAPSHOT_CLASS" >/dev/null 2>&1; then + if ! kcmd get volumesnapshotclass "$SNAPSHOT_CLASS" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" \ + '.metadata.labels[$k]==$v' >/dev/null; then + CONFLICT_ISSUES+=("VolumeSnapshotClass/$SNAPSHOT_CLASS (missing label)") + fi + fi + + # VolumeAttributesClass ownership check + if kcmd get volumeattributesclass "${ATTR_CLASS_NAME}" >/dev/null 2>&1; then + if ! kcmd get volumeattributesclass "${ATTR_CLASS_NAME}" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" '.metadata.labels[$k]==$v' >/dev/null; then + CONFLICT_ISSUES+=("VolumeAttributesClass/${ATTR_CLASS_NAME} (exists without label ${CREATED_BY_LABEL_KEY}=${MIGRATION_TOOL_ID})") + fi + fi + + for ENTRY in "${MIG_PVCS[@]}"; do + local ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + local pv snap + pv=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + [[ -z "$pv" ]] && continue + snap="$(name_snapshot "$pv")" + if kcmd get volumesnapshot "$snap" -n "$ns" >/dev/null 2>&1; then + kcmd get volumesnapshot "$snap" -n "$ns" -o json | jq -e \ + --arg k "$CREATED_BY_LABEL_KEY" --arg v "$MIGRATION_TOOL_ID" \ + '.metadata.labels[$k]==$v' >/dev/null || \ + CONFLICT_ISSUES+=("VolumeSnapshot/$ns/$snap missing label") + fi + local current_attr + current_attr=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeAttributesClassName}' 2>/dev/null || true) + if [[ -n "$current_attr" && "$current_attr" != "$ATTR_CLASS_NAME" ]]; then + CONFLICT_ISSUES+=("PVC/${ns}/${pvc} has volumeAttributesClassName=${current_attr} (expected empty or ${ATTR_CLASS_NAME})") + fi + done + + if (( $(safe_array_len CONFLICT_ISSUES) > 0 )); then + err "Conflict check failed ($(safe_array_len CONFLICT_ISSUES) items)." + printf ' - %s\n' "${CONFLICT_ISSUES[@]}" + exit 1 + fi + ok "No conflicting pre-existing objects detected." +} + +# ---------------- Pre-Requisite Validation (mirrors dual script) ---------------- +print_combined_validation_report_and_exit_if_needed() { + local prereq_count + prereq_count=$(safe_array_len PREREQ_ISSUES) + local conflict_count + conflict_count=$(safe_array_len CONFLICT_ISSUES) + if (( prereq_count==0 && conflict_count==0 )); then + ok "Pre-req & conflict checks passed." + return + fi + echo + err "Pre-run validation failed:" + (( prereq_count>0 )) && { echo " Pre-requisite issues ($prereq_count):"; printf ' - %s\n' "${PREREQ_ISSUES[@]}"; } + (( conflict_count>0 )) && { echo " Naming conflicts ($conflict_count):"; printf ' - %s\n' "${CONFLICT_ISSUES[@]}"; } + echo + err "Resolve issues and re-run. (No mutations performed.)" + exit 1 +} + +run_prerequisites_and_conflicts() { + run_prerequisites_checks + ensure_no_foreign_conflicts + print_combined_validation_report_and_exit_if_needed +} + +run_prerequisites_and_conflicts + +# ---------------- Discover Tagged PVCs ---------------- +MODE=$MODE process_pvcs_for_zone_preparation + +# ------------- Collect Unique Source SCs ------------- +create_unique_storage_classes + +migrate_pvc_attributes_class() { + local ns="$1" pvc="$2" + info "Processing PVC $ns/$pvc" + local pv + pv="$(get_pv_of_pvc "$ns" "$pvc")" + if [[ -z "$pv" ]]; then + warn "Skip $ns/$pvc (no PV yet)" + return + fi + + ensure_reclaim_policy_retain "$pv" + ensure_volume_attributes_class + + local cur_attr + cur_attr="$(kubectl get pvc "$pvc" -n "$ns" -o jsonpath='{.spec.volumeAttributesClassName}' 2>/dev/null || true)" + if [[ "$cur_attr" != "$ATTR_CLASS_NAME" ]]; then + info "Patching volumeAttributesClassName=${ATTR_CLASS_NAME}" + kubectl patch pvc "$pvc" -n "$ns" --type=merge -p "{\"spec\":{\"volumeAttributesClassName\":\"${ATTR_CLASS_NAME}\"}}" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "patch" \ + "kubectl patch pvc $pvc -n $ns --type=json -p '[{\"op\":\"remove\",\"path\":\"/spec/volumeAttributesClassName\"}]'" \ + "attrclass=${ATTR_CLASS_NAME}" + else + info "PVC already references ${ATTR_CLASS_NAME}" + fi + + # CHANGED (monitor handles completion): do NOT block here waiting for sku update. + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "attrclass-applied" "kubectl describe pvc $pvc -n $ns" "pv=$pv targetSku=${TARGET_SKU}" +} + +trigger_snapshot() { + local ns="$1" pvc="$2" + info "Processing PVC $ns/$pvc" + local pv + pv="$(get_pv_of_pvc "$ns" "$pvc")" + if [[ -z "$pv" ]]; then + warn "Skip $ns/$pvc (no PV yet)" + return + fi + + # Mark in-progress early + kubectl label pvc "$pvc" -n "$ns" --overwrite "${MIGRATION_INPROGRESS_LABEL_KEY}=${MIGRATION_INPROGRESS_LABEL_VALUE}" >/dev/null 2>&1 || true + + if is_in_tree_pv "$pv"; then + mode=$(kcmd get pv "$pv" -o jsonpath='{.spec.volumeMode}' 2>/dev/null || true) + sc=$(get_sc_of_pvc "$pvc" "$ns") + diskuri=$(kcmd get pv "$pv" -o jsonpath='{.spec.azureDisk.diskURI}' 2>/dev/null || true) + scpv1="$(name_pv1_sc "$sc")" + + info "In-tree PV ($pv) -> converting" + create_csi_pv_pvc "$pvc" "$ns" "$pv" "$size" "$mode" "${scpv1}" "$diskuri" true || { + err "Failed to create CSI PV/PVC for $ns/$pvc" + audit_add "PersistentVolumeClaim" "$pvc" "$ns" "convert-failed" "N/A" "phase=intree-create-csi" + return + } + pv="$(get_pv_of_pvc "$ns" "$pvc")" + else + backup_pvc "$pvc" "$ns" || { + warn "PVC backup failed $ns/$pvc" + } + fi + + snapshot="$(name_snapshot "$pv")" + create_snapshot "$snapshot" "$pvc" "$pvc_ns" "$pv" || { + warn "Snapshot failed $pvc_ns/$pvc" + return + } +} + + +info "Candidate PVCs:" +printf ' %s\n' "${MIG_PVCS[@]}" + +# -------- Initial Mutation Pass (apply attr class / conversion) = migration loop -------- +declare -A PVC_SNAPSHOTS +for ENTRY in "${MIG_PVCS[@]}"; do + pvc_ns="${ENTRY%%|*}" + pvc="${ENTRY##*|}" + lbl=$(kcmd get pvc "$pvc" -n "$pvc_ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" 2>/dev/null || true) + [[ "$lbl" == "$MIGRATION_DONE_LABEL_VALUE" ]] && { info "Already migrated $pvc_ns/$pvc"; continue; } + + pv="$(get_pv_of_pvc "$pvc_ns" "$pvc")" + if [[ -z "$pv" ]]; then + warn "PVC lost PV binding? $pvc_ns/$pvc. Skipping migration for now." + continue + fi + + check_premiumv2_lrs "$pvc_ns" "$pvc" && { + ok "Already PremiumV2: $pvc_ns/$pvc" + continue + } + + if [[ "$WAIT_FOR_WORKLOAD" == "true" ]]; then + attachments=$(kcmd get volumeattachment -o jsonpath="{range .items[?(@.spec.source.persistentVolumeName=='$pv')]}{.metadata.name}{'\n'}{end}" 2>/dev/null || true) + if [[ -n "$attachments" ]]; then + if ! wait_for_workload_detach "$pv" "$pvc" "$pvc_ns"; then + warn "Workload still attached -> deferring migration for $pvc_ns/$pvc (no labels changed)" + NON_DETACHED_PVCS+=("${pvc_ns}|${pvc}") + NON_DETACHED_SET["${pvc_ns}|${pvc}"]=1 + continue + fi + fi + fi + + ensure_reclaim_policy_retain "$pv" + + mode=$(kcmd get pv "$pv" -o jsonpath='{.spec.volumeMode}' 2>/dev/null || true) + sc=$(kcmd get pv "$pv" -o jsonpath='{.spec.storageClassName}' 2>/dev/null || true) + size=$(kcmd get pv "$pv" -o jsonpath='{.spec.capacity.storage}' 2>/dev/null || true) + diskuri=$(kcmd get pv "$pv" -o jsonpath='{.spec.azureDisk.diskURI}' 2>/dev/null || true) + + trigger_snapshot "$pvc_ns" "$pvc" + PVC_SNAPSHOTS+=("${pvc_ns}|${pvc}") +done + +wait_for_snapshots_ready + +SOURCE_SNAPSHOTS=("${!PVC_SNAPSHOTS[@]}") +for ENTRY in "${SOURCE_SNAPSHOTS[@]}"; do + ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + migrate_pvc_attributes_class "$ns" "$pvc" +done + +# -------- Monitoring Loop (events + sku poll) -------- +# MONITOR LOOP ADDED +deadline=$(( $(date +%s) + MONITOR_TIMEOUT_MINUTES * 60 )) +info "Monitoring attrclass migrations (timeout ${MONITOR_TIMEOUT_MINUTES}m)..." +monitor_start_epoch=$(date +%s) + +while true; do + ALL_DONE=true + for ENTRY in "${MIG_PVCS[@]}"; do + ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + + # Skip monitoring for deferred PVCs (never mutated) + if [[ ${NON_DETACHED_SET["${ns}|${pvc}"]+x} ]]; then + continue + fi + + # Already done? + lbl=$(kcmd get pvc "$pvc" -n "$ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" || true) + if [[ "$lbl" == "$MIGRATION_DONE_LABEL_VALUE" || "$lbl" == "$MIGRATION_DONE_LABEL_VALUE_FALSE" ]]; then + continue + fi + + pv="$(get_pv_of_pvc "$ns" "$pvc")" + if [[ -z "$pv" ]]; then + warn "PVC lost PV binding? $ns/$pvc" + continue + fi + + # Fast path: sku already updated + if ! check_premiumv2_lrs "$ns" "$pvc"; then + kubectl label pvc "$pvc" -n "$ns" "${MIGRATION_INPROGRESS_LABEL_KEY}-" 2>/dev/null 2>&1 || true + continue + fi + + reason=$(extract_event_reason "$ns" "$pvc") + case "$reason" in + SKUMigrationCompleted) + mark_source_done "$ns" "$pvc" + ok "Completed (event) $ns/$pvc" + continue + ;; + SKUMigrationProgress|SKUMigrationStarted) + mark_source_in_progress "$ns" "$pvc" + info "$reason $ns/$pvc" + ALL_DONE=false + ;; + ReasonSKUMigrationTimeout) + mark_source_notdone "$ns" "$pvc" + warn "$reason $ns/$pv2_pvc" + ALL_DONE=false + ;; + "") + info "No migration events yet for $ns/$pvc" + ALL_DONE=false + ;; + esac + + # Force in-progress label on PV after threshold if no events + if [[ -n "$pv" && $MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES -gt 0 ]]; then + inprog=$(kcmd get pv "$pv" -o go-template="{{ index .metadata.labels \"${MIGRATION_INPROGRESS_LABEL_KEY}\" }}" 2>/dev/null || true) + if [[ "$inprog" != "$MIGRATION_INPROGRESS_LABEL_VALUE" ]]; then + cts=$(kcmd get pvc "$pvc" -n "$ns" -o jsonpath='{.metadata.creationTimestamp}' 2>/dev/null || true) + if [[ -n "$cts" ]]; then + creation_epoch=$(date -d "$cts" +%s 2>/dev/null || echo 0) + now_epoch=$(date +%s) + if (( now_epoch - creation_epoch >= MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES * 60 )); then + if (( now_epoch - monitor_start_epoch < MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES * 60 )); then + # Avoid false positive for csi volumes modified to pv2 + continue + fi + warn "Forcing in-progress label on PV $pv (no events after ${MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES}m)" + kcmd label pv "$pv" "${MIGRATION_INPROGRESS_LABEL_KEY}=${MIGRATION_INPROGRESS_LABEL_VALUE}" --overwrite + audit_add "PersistentVolume" "$pv" "" "label" \ + "kubectl label pv $pv ${MIGRATION_INPROGRESS_LABEL_KEY}-" \ + "forced=true reason=noEventsAfter${MIGRATION_FORCE_INPROGRESS_AFTER_MINUTES}m" + fi + fi + fi + fi + done + + if [[ "$ALL_DONE" == "true" ]]; then + info "All migrations completed." + break + fi + + if (( $(date +%s) > deadline )); then + err "Monitoring timeout reached" + break + fi + + sleep "$POLL_INTERVAL" +done + +# ------------- Summary ------------- +echo +info "Summary:" +for ENTRY in "${MIG_PVCS[@]}"; do + ns="${ENTRY%%|*}" pvc="${ENTRY##*|}" + if [[ ${NON_DETACHED_SET["${ns}|${pvc}"]+x} ]]; then + echo " - ${ns}/${pvc} deferred (workload still attached)" + continue + fi + lbl=$(kcmd get pvc "$pvc" -n "$ns" -o go-template="{{ index .metadata.labels \"${MIGRATION_DONE_LABEL_KEY}\" }}" 2>/dev/null || true) + if [[ "$lbl" == "$MIGRATION_DONE_LABEL_VALUE" ]]; then + echo " ✓ $ns/$pvc migrated" + else + echo " • $ns/$pvc NOT completed" + fi +done + +info "Cleanup / leftover report:" +MODE=$MODE print_migration_cleanup_report + +finalize_audit_summary "$SCRIPT_START_TS" "$SCRIPT_START_EPOCH" + +ok "AttrClass migration script finished." \ No newline at end of file diff --git a/hack/premium-to-premiumv2-zonal-aware-helper.sh b/hack/premium-to-premiumv2-zonal-aware-helper.sh new file mode 100755 index 0000000000..3b6e42a264 --- /dev/null +++ b/hack/premium-to-premiumv2-zonal-aware-helper.sh @@ -0,0 +1,718 @@ +#!/usr/bin/env bash +# shellcheck source=./lib-premiumv2-migration-common.sh + +# Zone-aware migration helper for Azure Disk CSI Driver +# This script ensures PremiumV2 LRS disks are created in correct zones +set -euo pipefail +IFS=$'\n\t' + +# Source the common migration library +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib-premiumv2-migration-common.sh" + +# ---------- Zone-aware Configuration ---------- +ZONE_MAPPING_FILE="${ZONE_MAPPING_FILE:-disk-zone-mapping.txt}" + +# ---------- Zone Detection Functions ---------- + +# Parse the zone mapping file +# Format: = (e.g., /subscriptions/.../disks/mydisk=uksouth-1) +load_zone_mapping() { + local mapping_file="$1" + declare -g -A DISK_ZONE_MAP + + if [[ ! -f "$mapping_file" ]]; then + warn "Zone mapping file not found: $mapping_file" + return 1 + fi + + info "Loading zone mapping from: $mapping_file" + local line_count=0 + while IFS='=' read -r arm_id zone || [[ -n "$arm_id" ]]; do + [[ -z "$arm_id" || "$arm_id" =~ ^[[:space:]]*# ]] && continue # Skip empty lines and comments + [[ -z "$zone" ]] && { warn "Invalid mapping line: $arm_id (missing zone)"; continue; } + + # Normalize ARM ID (remove leading/trailing whitespace) + arm_id=$(echo "$arm_id" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | tr '[:upper:]' '[:lower:]') + zone=$(echo "$zone" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | tr '[:upper:]' '[:lower:]') + + DISK_ZONE_MAP["$arm_id"]="$zone" + line_count=$((line_count + 1)) + done < "$mapping_file" + + info "Loaded $line_count disk-to-zone mappings" + return 0 +} + +# Extract zone from StorageClass allowedTopologies +# Returns the zone if exactly one is found, empty string otherwise +extract_zone_from_storageclass() { + local sc_name="$1" + local sc_json zones_json zone_count + + sc_json=$(kcmd get sc "$sc_name" -o json 2>/dev/null || true) + [[ -z "$sc_json" ]] && return 1 + + zones_json=$(echo "$sc_json" | jq -r ' + .allowedTopologies // [] + | map(.matchLabelExpressions // []) + | flatten + | map(.values // []) + | flatten + | unique + ') + + zone_count=$(echo "$zones_json" | jq 'length') + + if [[ "$zone_count" == "1" ]]; then + echo "$zones_json" | jq -r '.[0]' + return 0 + elif [[ "$zone_count" == "0" ]]; then + info "No zone constraints in StorageClass $sc_name, checking PV nodeAffinity" + return 1 + else + info "Multiple zones in StorageClass $sc_name, need to determine specific zone" + return 2 + fi +} + +# Extract zone from PV nodeAffinity +# Returns the zone if exactly one matching zone expression is found +# Only considers recognized zone keys to avoid accidentally pulling region or other topology labels. +extract_zone_from_pv_nodeaffinity() { + local pv_name="$1" + local pv_json zone_count + + pv_json=$(kcmd get pv "$pv_name" -o json 2>/dev/null || true) + [[ -z "$pv_json" ]] && return 1 + + # Extract zones from nodeAffinity restricted to zone keys + local zones_json + zones_json=$(echo "$pv_json" | jq -r ' + .spec.nodeAffinity.required.nodeSelectorTerms // [] + | map(.matchExpressions // []) + | flatten + | map(select(.key == "topology.disk.csi.azure.com/zone")) + | map(.values // []) + | flatten + | unique + ') + + zone_count=$(echo "$zones_json" | jq 'length') + + if [[ "$zone_count" == "1" ]]; then + echo "$zones_json" | jq -r '.[0]' + return 0 + else + return 1 + fi +} + +# Get disk URI from PV (handles both in-tree and CSI) +get_disk_uri_from_pv() { + local pv_name="$1" + local disk_uri + + # Try in-tree first + disk_uri=$(kcmd get pv "$pv_name" -o jsonpath='{.spec.azureDisk.diskURI}' 2>/dev/null || true) + + if [[ -z "$disk_uri" ]]; then + # Try CSI + disk_uri=$(kcmd get pv "$pv_name" -o jsonpath='{.spec.csi.volumeHandle}' 2>/dev/null || true) + fi + + echo "$disk_uri" +} + +# Determine zone for a PVC (updated priority: PV nodeAffinity first) +# Exit codes: +# 0 -> zone from PV nodeAffinity +# 1 -> zone from StorageClass allowedTopologies +# 2 -> zone from mapping file +# 3 -> failed to determine zone +determine_zone_for_pvc() { + local pvc_name="$1" pvc_ns="$2" + local sc_name pv_name zone disk_uri rc + + sc_name=$(kcmd get pvc "$pvc_name" -n "$pvc_ns" -o jsonpath='{.spec.storageClassName}' 2>/dev/null || true) + pv_name=$(kcmd get pvc "$pvc_name" -n "$pvc_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + + [[ -z "$sc_name" ]] && { warn "PVC $pvc_ns/$pvc_name has no storageClassName"; return 3; } + [[ -z "$pv_name" ]] && { warn "PVC $pvc_ns/$pvc_name has no bound PV"; return 3; } + + # Priority Step 1: PV nodeAffinity (most authoritative actual placement) + zone=$(extract_zone_from_pv_nodeaffinity "$pv_name"); rc=$? + if [[ $rc -eq 0 && -n "$zone" ]]; then + printf '%s\n' "$zone" + return 0 + fi + + # Priority Step 2: StorageClass allowedTopologies (design intent) + zone=$(extract_zone_from_storageclass "$sc_name"); rc=$? + if [[ $rc -eq 0 && -n "$zone" ]]; then + printf '%s\n' "$zone" + return 1 + fi + + # Priority Step 3: Mapping file fallback + disk_uri=$(get_disk_uri_from_pv "$pv_name") + disk_uri=$(echo "$disk_uri" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | tr '[:upper:]' '[:lower:]') + if [[ -n "$disk_uri" && -n "${DISK_ZONE_MAP[$disk_uri]:-}" ]]; then + zone="${DISK_ZONE_MAP[$disk_uri]}" + echo "$zone" + return 2 + fi + + warn "Unable to determine zone for PVC $pvc_ns/$pvc_name (SC: $sc_name, PV: $pv_name, Disk: ${disk_uri:-unknown})" + return 3 +} + +# Create zone-specific StorageClass with comprehensive property preservation +# Returns the name of the created/existing StorageClass +create_zone_specific_storageclass() { + local orig_sc="$1" zone="$2" sku="${3:-Premium_LRS}" zone_sc_name=${4:-"${orig_sc}-${zone}"} + + if kcmd get sc "$zone_sc_name" >/dev/null 2>&1; then + # Check if this StorageClass was created by our migration script + local created_by existing_sku existing_zone + created_by=$(kcmd get sc "$zone_sc_name" -o jsonpath="{.metadata.labels.${CREATED_BY_LABEL_KEY//./\\.}}" 2>/dev/null || true) + + if [[ "$created_by" == "$MIGRATION_TOOL_ID" ]]; then + # Verify it matches our expected configuration + existing_sku=$(kcmd get sc "$zone_sc_name" -o jsonpath='{.parameters.skuName}' 2>/dev/null || true) + existing_zone=$(extract_zone_from_storageclass "$zone_sc_name") + + if [[ "$existing_sku" == "$sku" && "$existing_zone" == "$zone" ]]; then + info "Zone-specific StorageClass $zone_sc_name already exists (created by migration script, sku=$existing_sku, zone=$existing_zone)" + return 0 + else + warn "Zone-specific StorageClass $zone_sc_name exists but has different configuration:" + warn " Expected: sku=$sku, zone=$zone" + warn " Existing: sku=$existing_sku, zone=$existing_zone" + warn " Recreating to match expected configuration..." + + # Delete and recreate with correct configuration + if ! kcmd delete sc "$zone_sc_name" --wait=true 2>/dev/null; then + err "Failed to delete existing zone StorageClass $zone_sc_name for recreation" + return 1 + fi + audit_add "StorageClass" "$zone_sc_name" "" "delete" "N/A" "reason=mismatchedConfig expectedSku=$sku expectedZone=$zone existingSku=$existing_sku existingZone=$existing_zone" + info "Deleted existing zone StorageClass $zone_sc_name with mismatched configuration" + fi + else + # StorageClass exists but wasn't created by our script + if [[ -z "$created_by" ]]; then + err "Zone-specific StorageClass $zone_sc_name already exists but was not created by migration script" + err " Missing label: ${CREATED_BY_LABEL_KEY}=${MIGRATION_TOOL_ID}" + else + err "Zone-specific StorageClass $zone_sc_name already exists but was created by different tool: $created_by" + fi + err " Please rename or delete the existing StorageClass, or choose a different zone naming scheme" + err " Current StorageClass details:" + kcmd describe sc "$zone_sc_name" | head -20 >&2 || true + return 1 + fi + fi + + local params_json + params_json=$(kcmd get sc "$orig_sc" -o json 2>/dev/null || true) + if [[ -z "$params_json" ]]; then + err "Cannot fetch base StorageClass $orig_sc" + return 1 + fi + + # Validate that we have a valid StorageClass + local sc_kind + sc_kind=$(echo "$params_json" | jq -r '.kind // ""') + if [[ "$sc_kind" != "StorageClass" ]]; then + err "Invalid StorageClass data for $orig_sc" + return 1 + fi + + info "Analyzing StorageClass $orig_sc for zone-specific variant creation (zone=$zone, sku=$sku)" + + # Extract all components with detailed preservation + local orig_labels orig_annotations params_filtered provisioner + local reclaim_policy allow_volume_expansion volume_binding_mode mount_options_yaml + + # Extract original labels (excluding system labels, add zone-specific ones) + orig_labels=$(echo "$params_json" | jq -r ' + .metadata.labels // {} + | to_entries + | map(select(.key | test("^(kubernetes\\.io/|k8s\\.io/|pv\\.kubernetes\\.io/|volume\\.kubernetes\\.io/|app\\.kubernetes\\.io/managed-by|storageclass\\.kubernetes\\.io/is-default-class)") | not)) + | map(" " + .key + ": \"" + (.value|tostring) + "\"") + | join("\n") + ') + + # Extract original annotations (excluding system annotations) + orig_annotations=$(echo "$params_json" | jq -r ' + .metadata.annotations // {} + | to_entries + | map(select(.key | test("^(kubernetes\\.io/|k8s\\.io/|pv\\.kubernetes\\.io/|volume\\.kubernetes\\.io/|kubectl\\.kubernetes\\.io/|deployment\\.kubernetes\\.io/|control-plane\\.|storageclass\\.kubernetes\\.io/is-default-class)") | not)) + | map(" " + .key + ": \"" + (.value|tostring|gsub("\\n"; "\\\\n")|gsub("\\\""; "\\\\\"")) + "\"") + | join("\n") + ') + + # Extract provisioner (ensure it's CSI-compatible) + provisioner=$(echo "$params_json" | jq -r '.provisioner // "disk.csi.azure.com"') + if [[ "$provisioner" == "kubernetes.io/azure-disk" ]]; then + provisioner="disk.csi.azure.com" + info " Provisioner: Converted in-tree to CSI provisioner" + else + info " Provisioner: $provisioner" + fi + + # Extract parameters with zone-specific migration filtering + params_filtered=$(echo "$params_json" | jq -r ' + .parameters // {} + | to_entries + | map(select( + .key != "cachingMode" + and (.key | test("^(diskEncryption|encryption)"; "i") | not) + and (.key | test("^(enableBursting|perfProfile)$") | not) + and (.key | test("^(LogicalSectorSize)$") | not) + )) + | map(" " + .key + ": \"" + (.value|tostring) + "\"") + | join("\n") + ') + + # Log parameter analysis + local param_count filtered_count + param_count=$(echo "$params_json" | jq -r '.parameters // {} | keys | length') + filtered_count=$(echo "$params_json" | jq -r ' + .parameters // {} + | to_entries + | map(select( + .key != "cachingMode" + and (.key | test("^(diskEncryption|encryption)"; "i") | not) + and (.key | test("^(enableBursting|perfProfile)$") | not) + and (.key | test("^(LogicalSectorSize)$") | not) + )) + | length + ') + info " Parameters: $param_count total, $filtered_count preserved (zone-safe filtering applied)" + + # Extract storage class properties + reclaim_policy=$(echo "$params_json" | jq -r '.reclaimPolicy // "Retain"') + allow_volume_expansion=$(echo "$params_json" | jq -r '.allowVolumeExpansion // true') + volume_binding_mode=$(echo "$params_json" | jq -r '.volumeBindingMode // "WaitForFirstConsumer"') + + # For zone-specific StorageClasses, prefer WaitForFirstConsumer to ensure proper zone placement + if [[ "$volume_binding_mode" == "Immediate" ]]; then + volume_binding_mode="WaitForFirstConsumer" + info " VolumeBindingMode: Changed from Immediate to WaitForFirstConsumer for zone awareness" + else + info " VolumeBindingMode: $volume_binding_mode (zone-compatible)" + fi + + info " ReclaimPolicy: $reclaim_policy" + info " AllowVolumeExpansion: $allow_volume_expansion" + + # Extract mountOptions if present + local mount_options_count + mount_options_count=$(echo "$params_json" | jq -r '.mountOptions // [] | length') + mount_options_yaml=$(echo "$params_json" | jq -r ' + if .mountOptions and (.mountOptions | length > 0) then + "mountOptions:" + + (.mountOptions | map("\n- \"" + . + "\"") | join("")) + else + "" + end + ') + + if [[ $mount_options_count -gt 0 ]]; then + info " MountOptions: $mount_options_count options preserved" + fi + + # Create zone-specific allowedTopologies (override any existing topology constraints) + local allowed_topologies_yaml + allowed_topologies_yaml="allowedTopologies: +- matchLabelExpressions: + - key: topology.kubernetes.io/zone + values: [\"$zone\"]" + + info " AllowedTopologies: Zone-specific constraint set to $zone" + + + info "Creating zone-specific StorageClass $zone_sc_name with comprehensive property preservation" + + # Build the complete zone-specific StorageClass YAML + local sc_yaml + sc_yaml=$(cat </dev/null 2>&1; then + + warn "Failed to annotate PVC $pvc_ns/$pvc_name" + audit_add "PersistentVolumeClaim" "$pvc_name" "$pvc_ns" "annotate-failed" \ + "kubectl annotate pvc $pvc_name -n $pvc_ns ${ZONE_SC_ANNOTATION_KEY}-" \ + "zoneStorageClass=$zone_sc zone=$zone reason=kubectlError" + return 1 + fi + + audit_add "PersistentVolumeClaim" "$pvc_name" "$pvc_ns" "annotate" \ + "kubectl annotate pvc $pvc_name -n $pvc_ns ${ZONE_SC_ANNOTATION_KEY}-" \ + "zoneStorageClass=$zone_sc zone=$zone" + + ok "PVC $pvc_ns/$pvc_name annotated with zone-specific StorageClass" + return 0 +} + +# Process a single PVC for zone-aware migration +process_pvc_for_zone_migration() { + local pvc_entry="$1" # Format: namespace|pvcname + local pvc_ns="${pvc_entry%%|*}" + local pvc_name="${pvc_entry##*|}" + + info "Processing PVC $pvc_ns/$pvc_name for zone-aware migration" + + local existing_annotation + existing_annotation=$(kcmd get pvc "$pvc_name" -n "$pvc_ns" -o jsonpath="{.metadata.annotations.${ZONE_SC_ANNOTATION_KEY//./\\.}}" 2>/dev/null || true) + if [[ -n "$existing_annotation" ]]; then + info "PVC $pvc_ns/$pvc_name already annotated with zone-specific StorageClass: $existing_annotation (skipping)" + return 0 + fi + + local zone + zone=$(determine_zone_for_pvc "$pvc_name" "$pvc_ns") + local zone_src_rc=$? + case $zone_src_rc in + 0) + info "Zone determined from PV nodeAffinity for PVC $pvc_ns/$pvc_name: $zone" + ;; + 1) + info "Zone inferred from StorageClass allowedTopologies for PVC $pvc_ns/$pvc_name: $zone" + return 1 + ;; + 2) + info "Zone resolved from mapping file for PVC $pvc_ns/$pvc_name: $zone" + ;; + *) + warn "Failed to determine zone for PVC $pvc_ns/$pvc_name (rc=$zone_src_rc)" + return 2 + ;; + esac + + local orig_sc + orig_sc=$(kcmd get pvc "$pvc_name" -n "$pvc_ns" -o jsonpath='{.spec.storageClassName}' 2>/dev/null || true) + [[ -z "$orig_sc" ]] && { err "PVC $pvc_ns/$pvc_name has no storageClassName"; return 2; } + + local zone_sc + if [[ "${GENERIC_PV2_SC_MODE:-0}" == "1" ]]; then + # Reuse generic naming, keep zone constraint inside the SC spec + zone_sc="$(name_pv2_sc "$orig_sc")" + else + zone_sc="${orig_sc}-${zone}" + fi + + if ! create_zone_specific_storageclass "$orig_sc" "$zone" "Premium_LRS" "$zone_sc"; then + err "Failed to create zone-specific StorageClass $zone_sc (zone=$zone orig=$orig_sc)" + return 2 + fi + + if ! annotate_pvc_with_zone_storageclass "$pvc_name" "$pvc_ns" "$zone_sc" "$zone"; then + err "Failed to annotate PVC $pvc_ns/$pvc_name with zone SC $zone_sc" + return 2 + fi + + ok "PVC $pvc_ns/$pvc_name zone=$zone (source=$zone_src_rc) annotated with $zone_sc" + return 0 +} + +# Process PVCs for zone-aware preparation without full migration +process_pvcs_for_zone_preparation() { + local start_ts start_epoch + start_ts="$(date +'%Y-%m-%dT%H:%M:%S')" + start_epoch="$(date +%s)" + + info "Starting zone-aware PVC processing (preparation only)" + + # Load zone mapping if file exists + declare -g -A DISK_ZONE_MAP + if [[ -f "$ZONE_MAPPING_FILE" ]]; then + if ! load_zone_mapping "$ZONE_MAPPING_FILE"; then + err "Failed to load zone mapping file: $ZONE_MAPPING_FILE" + exit 1 + fi + else + warn "Zone mapping file not found: $ZONE_MAPPING_FILE (will rely on StorageClass and PV nodeAffinity only)" + fi + + # Check prerequisites + if is_direct_exec; then + migration_rbac_check || exit 1 + fi + + # Populate PVCs marked for migration + populate_pvcs + detect_generic_pv2_mode + + if is_direct_exec; then + run_prerequisites_checks + else + info "Skipping run_prerequisites_checks (script sourced)" + fi + + local processed=0 skipped=0 failed=0 + for pvc_entry in "${MIG_PVCS[@]}"; do + local pvc_ns="${pvc_entry%%|*}" + local pvc_name="${pvc_entry##*|}" + + is_created_by_migrator=$(is_pvc_in_migration "$pvc_name" "$pvc_ns") + if [[ $is_created_by_migrator == "true" ]]; then + info "Skipping PVC $pvc_ns/$pvc_name (created by migration tool)" + skipped=$((skipped + 1)) + continue + fi + + is_migration=$(is_pvc_created_by_migration_tool "$pvc_name" "$pvc_ns") + if [[ $is_migration == "true" ]]; then + info "Skipping PVC $pvc_ns/$pvc_name (already in migration)" + skipped=$((skipped + 1)) + continue + fi + + # Check if PVC already has zone-specific StorageClass annotation + local existing_annotation + existing_annotation=$(kcmd get pvc "$pvc_name" -n "$pvc_ns" -o jsonpath="{.metadata.annotations.${ZONE_SC_ANNOTATION_KEY//./\\.}}" 2>/dev/null || true) + if [[ -n "$existing_annotation" ]]; then + info "Skipping PVC $pvc_ns/$pvc_name (already has zone annotation: $existing_annotation)" + skipped=$((skipped + 1)) + continue + fi + + run_without_errexit process_pvc_for_zone_migration "$pvc_entry" + case $LAST_RUN_WITHOUT_ERREXIT_RC in + 1) + skipped=$((skipped + 1)) + ;; + 0) + processed=$((processed + 1)) + ;; + *) + failed=$((failed + 1)) + ;; + esac + done + + if is_direct_exec; then + echo + ok "Zone-aware PVC processing complete:" + echo " Processed: $processed" + echo " Skipped: $skipped" + echo " Failed: $failed" + echo " Total: ${#MIG_PVCS[@]}" + finalize_audit_summary "$start_ts" "$start_epoch" + else + if [[ $failed -gt 0 ]]; then + err "Zone-aware PVC processing complete with failures:" + echo " Processed: $processed" + echo " Skipped: $skipped" + echo " Failed: $failed" + echo " Total: ${#MIG_PVCS[@]}" + return 1 + else + ok "Zone-aware PVC processing complete:" + echo " Processed: $processed" + echo " Skipped: $skipped" + echo " Failed: $failed" + echo " Total: ${#MIG_PVCS[@]}" + return 0 + fi + fi +} + +# Generate zone mapping template +generate_zone_mapping_template() { + local output_file="${1:-disk-zone-mapping-template.txt}" + + info "Generating zone mapping template: $output_file" + + { + echo "# Azure Disk Zone Mapping File" + echo "# Format: =" + echo "# Example zones: uksouth-1, uksouth-2, uksouth-3" + echo "# " + echo "# Examples:" + echo "# /subscriptions/12345678-1234-1234-1234-123456789012/resourceGroups/myRG/providers/Microsoft.Compute/disks/myDisk1=uksouth-1" + echo "# /subscriptions/12345678-1234-1234-1234-123456789012/resourceGroups/myRG/providers/Microsoft.Compute/disks/myDisk2=uksouth-2" + echo "" + + # Generate entries for existing PVCs + if [[ ${#MIG_PVCS[@]} -gt 0 ]]; then + echo "# Generated from current PVCs marked for migration:" + for pvc_entry in "${MIG_PVCS[@]}"; do + local pvc_ns="${pvc_entry%%|*}" + local pvc_name="${pvc_entry##*|}" + local pv_name disk_uri + + pv_name=$(kcmd get pvc "$pvc_name" -n "$pvc_ns" -o jsonpath='{.spec.volumeName}' 2>/dev/null || true) + if [[ -n "$pv_name" ]]; then + disk_uri=$(get_disk_uri_from_pv "$pv_name") + if [[ -n "$disk_uri" ]]; then + echo "# PVC: $pvc_ns/$pvc_name, PV: $pv_name" + echo "$disk_uri= # Replace with actual zone" + fi + fi + done + fi + } > "$output_file" + + ok "Zone mapping template created: $output_file" + echo "Edit this file and set ZONE_MAPPING_FILE=$output_file" +} + +# Show help +show_help() { + cat << 'EOF' +Zone-Aware Azure Disk Migration Helper + +This script prepares PVCs for zone-aware migration from Premium_LRS to PremiumV2_LRS +by ensuring disks are created in the correct Azure availability zones. + +USAGE: + premium-to-premiumv2-zonal-aware-helper.sh [OPTIONS] MODE COMMAND + +MODE: + dual Prepare for migration while keeping existing disks + inplace In-place migration mode - migrate disks without keeping existing ones + attrclass Attribute class migration mode - migrate disks based on their attributes + +COMMANDS: + process Process PVCs and create zone-specific StorageClasses (no migration) + generate-template Generate a zone mapping template file + help Show this help + +OPTIONS: + Environment variables (set before running): + + ZONE_MAPPING_FILE Path to disk-zone mapping file + Default: disk-zone-mapping.txt + Format: /subscriptions/.../disks/mydisk=uksouth-1 + + MIGRATION_LABEL Label selector for PVCs to migrate + Default: disk.csi.azure.com/pv2migration=true + + MAX_PVCS Maximum PVCs to process in one run + Default: 50 + +ZONE DETECTION LOGIC: + 1. Check StorageClass allowedTopologies - if single zone, use it + 2. If multiple zones or no constraints, check PV nodeAffinity + 3. If no nodeAffinity or multiple zones, consult zone mapping file + 4. Error if zone cannot be determined + +EXAMPLES: + # Generate template + ./premium-to-premiumv2-zonal-aware-helper.sh generate-template + + # Process all labeled PVCs for zone preparation + ./premium-to-premiumv2-zonal-aware-helper.sh process + + # Edit the template file and run processing + ZONE_MAPPING_FILE=disk-zone-mapping.txt ./premium-to-premiumv2-zonal-aware-helper.sh process + +OUTPUT: + - Creates zone-specific StorageClasses (e.g., premium-ssd-uksouth-1) + - Annotates PVCs with zone-specific StorageClass names + - Audit log for all actions taken + +ANNOTATIONS ADDED: + disk.csi.azure.com/zone-specific-storageclass: +EOF +} + +# Main entry point +main() { + local mode="${1:-help}" + local cmd="${2:-help}" + + case "$mode" in + dual|inplace|attrclass) + export MODE="$mode" + ;; + help|--help|-h) + show_help + exit 0 + ;; + *) + err "Unknown mode: $mode" + echo + show_help + exit 1 + ;; + esac + + case "$cmd" in + process) + process_pvcs_for_zone_preparation + ;; + generate-template) + generate_zone_mapping_template "${2:-}" + ;; + help|--help|-h) + show_help + ;; + *) + err "Unknown command: $cmd" + echo + show_help + exit 1 + ;; + esac +} + +# Run if executed directly +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main "$@" +fi \ No newline at end of file