Skip to content

OADP-3307: Add Azure and GCP CloudStorage API provider implementations #1828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 19, 2025
Merged

OADP-3307: Add Azure and GCP CloudStorage API provider implementations #1828

merged 1 commit into from
Aug 19, 2025
+8,308 −376

Conversation

kaovilai
Copy link
Member

@kaovilai kaovilai commented Jul 11, 2025 •
edited
Loading

Why the changes were made

Related issue: OADP-3307

How to test the changes made

Prerequisites

  1. An OpenShift cluster with Azure Workload Identity enabled
  2. GCP cluster with Workload Identity Federation configured (for GCP testing)
  3. Proper identity setup using the helper scripts in dotfiles repository

Testing Azure CloudStorage with Workload Identity

  1. Set up Azure identity for both service accounts:
# Use the updated helper function that now creates federated credentials for both service accounts
create-velero-identity-for-azure-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: azure-test-storage
  namespace: openshift-adp
spec:
  name: velero
  provider: azure
  creationSecret:
    name: cloud-credentials-azure
    key: azurekey  # Required key for Azure credentials
  config:
    storageAccount: "<your-storage-account>"

Note: The creationSecret.key must be azurekey for Azure provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource above
oc apply -f cloudstorage.yaml

# Check that the resource was created and container exists in Azure
oc get cloudstorage azure-test-storage -n openshift-adp
az storage container list --account-name <storage-account> --auth-mode login

Verify with DataProtectionApplication:

# Create a DPA with Azure configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
      - azure
      - openshift
  backupLocations:
  - velero:
      provider: azure
      default: true
      credential:
        name: cloud-credentials-azure
        key: azurekey
      config:
        resourceGroup: <resource-group>
        storageAccount: <storage-account>
      objectStorage:
        bucket: velero
        prefix: <cluster-name>

Test Backup and Restore:

# Create a test namespace with an application
oc create namespace test-backup-azure
oc create deployment nginx --image=bitnami/nginx:latest -n test-backup-azure
oc wait --for=condition=available deployment/nginx -n test-backup-azure --timeout=60s

# Create a backup
velero backup create azure-test-backup --include-namespaces=test-backup-azure

# Wait and verify backup completed
velero backup describe azure-test-backup

# Delete the namespace
oc delete namespace test-backup-azure

# Restore from backup
velero restore create azure-test-restore --from-backup azure-test-backup

# Verify namespace was restored
oc get namespace test-backup-azure

Verify Azure Workload Identity is working:

# Check Velero pod has correct environment variables
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_CLIENT_ID")]}' | jq .
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_FEDERATED_TOKEN_FILE")]}' | jq .

# Check service account annotation
oc get sa velero -n openshift-adp -o jsonpath='{.metadata.annotations.azure\.workload\.identity/client-id}'

Test error handling with invalid storage account:

# Create CloudStorage with non-existent storage account
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: azure-invalid-storage
  namespace: openshift-adp
spec:
  name: test-container
  provider: azure
  creationSecret:
    name: cloud-credentials-azure
    key: azurekey
  config:
    storageAccount: "nonexistentstorage123"
EOF

# Add deletion annotation and delete (should succeed with the fix)
oc annotate cloudstorage azure-invalid-storage -n openshift-adp oadp.openshift.io/cloudstorage-delete=true --overwrite
oc delete cloudstorage azure-invalid-storage -n openshift-adp --timeout=60s

Testing GCP CloudStorage with Workload Identity Federation

  1. Set up GCP identity for both service accounts:
# Use the updated helper function that now creates workload identity bindings for both service accounts
create-velero-identity-for-gcp-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: gcp-test-storage
  namespace: openshift-adp
spec:
  name: <bucket-name>
  provider: gcp
  region: us-central1
  creationSecret:
    name: cloud-credentials-gcp
    key: service_account.json  # Required key for GCP credentials

Note: The creationSecret.key must be service_account.json for GCP provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource
oc apply -f cloudstorage-gcp.yaml

# Verify bucket exists in GCS
gsutil ls -p <project-id> | grep "gs://<bucket-name>/"

Test with DataProtectionApplication:

# Create DPA with GCP configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa-gcp
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - gcp
        - openshift
        - csi
  backupLocations:
    - name: default
      velero:
        provider: gcp
        default: true
        credential:
          name: cloud-credentials-gcp
          key: service_account.json
        objectStorage:
          bucket: <bucket-name>
          prefix: velero
  snapshotLocations:
    - name: default
      velero:
        provider: gcp
        credential:
          name: cloud-credentials-gcp
          key: service_account.json
        config:
          project: <project-id>
          snapshotLocation: us-central1

Verify backup/restore works:

# Create test resources
oc create namespace test-backup-gcp
oc create deployment hello-openshift --image=openshift/hello-openshift:latest -n test-backup-gcp
oc wait --for=condition=available deployment/hello-openshift -n test-backup-gcp --timeout=60s

# Backup and restore
velero backup create gcp-test-backup --include-namespaces=test-backup-gcp
velero backup describe gcp-test-backup
gsutil ls -r gs://<bucket-name>/velero/backups/

# Delete and restore
oc delete namespace test-backup-gcp
velero restore create gcp-test-restore --from-backup gcp-test-backup
oc get all -n test-backup-gcp

Comprehensive Test Cases

Azure Test Cases

Test Case 1 - Basic CloudStorage Resource Creation

This test verifies that the CloudStorage controller can successfully create Azure storage containers using the new provider implementation.

# Set up environment variables
export STORAGE_ACCOUNT_NAME="<your-storage-account>"

# Create CloudStorage resource
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: azure-test-storage
  namespace: openshift-adp
spec:
  name: velero
  provider: azure
  creationSecret:
    name: cloud-credentials-azure
    key: azurekey
  config:
    storageAccount: "${STORAGE_ACCOUNT_NAME}"
EOF

# Wait for resource creation
sleep 10

# Verify the CloudStorage resource was created
oc get cloudstorage azure-test-storage -n openshift-adp

# Check the status phase (should eventually show 'Available' or similar)
oc get cloudstorage azure-test-storage -n openshift-adp -o jsonpath='{.status.phase}'

# Verify the container exists in Azure
az storage container list --account-name ${STORAGE_ACCOUNT_NAME} --auth-mode login --query "[?name=='velero']"

Expected outcome: CloudStorage resource is created successfully, and the Azure storage container is provisioned.

Test Case 2 - DataProtectionApplication Integration

This test ensures that the CloudStorage provider works correctly with OADP's DataProtectionApplication, allowing Velero to use Azure storage for backups.

# Set up environment variables
export AZURE_RESOURCE_GROUP="<your-resource-group>"
export STORAGE_ACCOUNT_NAME="<your-storage-account>"
export CONTAINER_NAME="velero"
export CLUSTER_NAME="<your-cluster-name>"

# Create DPA with Azure configuration
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
      - azure
      - openshift
  backupLocations:
  - velero:
      provider: azure
      default: true
      credential:
        name: cloud-credentials-azure
        key: azurekey
      config:
        resourceGroup: ${AZURE_RESOURCE_GROUP}
        storageAccount: ${STORAGE_ACCOUNT_NAME}
      objectStorage:
        bucket: ${CONTAINER_NAME}
        prefix: ${CLUSTER_NAME}
EOF

# Wait for DPA to reconcile
oc wait --for=condition=Reconciled dpa/dpa -n openshift-adp --timeout=300s

# Verify Velero pod is running
oc get pods -n openshift-adp -l app.kubernetes.io/name=velero

# Check Velero can access the backup location
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc exec -n openshift-adp ${VELERO_POD} -- /velero backup-location get

Expected outcome: DPA is reconciled successfully, Velero pod is running, and backup location is accessible.

Test Case 3 - Backup and Restore Operations

This test validates end-to-end backup and restore functionality using Azure storage as the backend.

# Clean up any existing backups/restores
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc exec -n openshift-adp ${VELERO_POD} -- /velero backup delete --all --confirm 2>/dev/null || true
oc exec -n openshift-adp ${VELERO_POD} -- /velero restore delete --all --confirm 2>/dev/null || true

# Create test namespace and application
oc create namespace test-backup-azure || true
oc create deployment nginx --image=bitnami/nginx:latest -n test-backup-azure
oc expose deployment nginx --port=8080 --target-port=8080 -n test-backup-azure

# Wait for deployment to be ready
oc wait --for=condition=available deployment/nginx -n test-backup-azure --timeout=60s

# Create backup
velero backup create azure-test-backup --include-namespaces=test-backup-azure

# Wait for backup to complete (check status)
sleep 30
velero backup describe azure-test-backup

# Delete the test namespace to simulate data loss
oc delete namespace test-backup-azure

# Wait for namespace deletion
sleep 10

# Restore from backup
velero restore create azure-test-restore --from-backup azure-test-backup

# Wait for restore to complete
sleep 30

# Verify namespace and resources were restored
oc get namespace test-backup-azure
oc get all -n test-backup-azure

Expected outcome: Backup completes successfully, namespace is deleted and then restored with all resources intact.

Test Case 4 - Azure Workload Identity Authentication

This test verifies that the Azure Workload Identity authentication is properly configured for both the Velero pod and the controller manager.

# Get Velero pod name
POD_NAME=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')

# Check AZURE_CLIENT_ID environment variable
echo "Checking AZURE_CLIENT_ID..."
oc get pod ${POD_NAME} -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_CLIENT_ID")]}' | jq .

# Check AZURE_FEDERATED_TOKEN_FILE environment variable
echo "Checking AZURE_FEDERATED_TOKEN_FILE..."
oc get pod ${POD_NAME} -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_FEDERATED_TOKEN_FILE")]}' | jq .

# Check service account annotations for Velero
echo "Velero service account annotations:"
oc get sa velero -n openshift-adp -o jsonpath='{.metadata.annotations}' | jq .

# Check service account annotations for controller manager
echo "Controller manager service account annotations:"
oc get sa openshift-adp-controller-manager -n openshift-adp -o jsonpath='{.metadata.annotations}' | jq .

# Verify federated token file is mounted
oc get pod ${POD_NAME} -n openshift-adp -o jsonpath='{.spec.volumes[?(@.name=="azure-identity-token")]}' | jq .

Expected outcome: Both service accounts have proper Azure Workload Identity annotations, and environment variables are correctly set.

Test Case 5 - Error Handling with Invalid Storage Account

This test verifies that the CloudStorage controller handles errors gracefully when given invalid configuration, and that resources can be deleted even when the underlying storage doesn't exist (idempotent deletion).

# Create CloudStorage with non-existent storage account
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: azure-invalid-storage
  namespace: openshift-adp
spec:
  name: test-container
  provider: azure
  creationSecret:
    name: cloud-credentials-azure
    key: azurekey
  config:
    storageAccount: "nonexistentstorage123"
EOF

# Wait and check status
sleep 10
oc describe cloudstorage azure-invalid-storage -n openshift-adp

# Check controller logs for error handling
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "azure-invalid-storage" | tail -10

# Test deletion with annotation (should succeed with the fix)
echo "Testing deletion with invalid storage account..."
oc annotate cloudstorage azure-invalid-storage -n openshift-adp oadp.openshift.io/cloudstorage-delete=true --overwrite

# Delete the resource (should complete within timeout)
if oc delete cloudstorage azure-invalid-storage -n openshift-adp --timeout=60s; then
    echo "✓ CloudStorage deleted successfully (idempotent behavior working)"
else
    echo "✗ CloudStorage deletion failed"
    # Force cleanup if needed
    oc patch cloudstorage azure-invalid-storage -n openshift-adp --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' 2>/dev/null || true
fi

# Check events for deletion process
oc get events -n openshift-adp --field-selector involvedObject.name=azure-invalid-storage --sort-by='.lastTimestamp' | tail -5

Expected outcome: CloudStorage resource shows error status but can be deleted successfully without hanging.

Test Case 6 - Container Lifecycle Management

This test verifies that containers can be created with unique names and properly cleaned up when CloudStorage resources are deleted.

# Generate unique container name
CONTAINER_NAME="velero-test-$(date +%Y%m%d%H%M%S)"
echo "Creating container: ${CONTAINER_NAME}"

# Create CloudStorage with unique container name
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: azure-new-container
  namespace: openshift-adp
spec:
  name: ${CONTAINER_NAME}
  provider: azure
  creationSecret:
    name: cloud-credentials-azure
    key: azurekey
  config:
    storageAccount: "${STORAGE_ACCOUNT_NAME}"
EOF

# Wait for container creation
sleep 15

# Verify container was created in Azure
echo "Checking if container ${CONTAINER_NAME} was created..."
az storage container list --account-name ${STORAGE_ACCOUNT_NAME} --auth-mode login --query "[?name=='${CONTAINER_NAME}']" -o table

# Test container deletion
oc annotate cloudstorage azure-new-container -n openshift-adp oadp.openshift.io/cloudstorage-delete=true --overwrite
oc delete cloudstorage azure-new-container -n openshift-adp --timeout=60s

# Verify container was deleted from Azure
sleep 10
echo "Checking if container ${CONTAINER_NAME} was deleted..."
az storage container list --account-name ${STORAGE_ACCOUNT_NAME} --auth-mode login --query "[?name=='${CONTAINER_NAME}']" -o table

Expected outcome: Container is created in Azure when CloudStorage is created and deleted when CloudStorage is deleted.

Test Case 7 - Role Assignment Validation

This test ensures that the Azure managed identity has all the required roles for OADP operations.

# Set up environment variables
export AZURE_RESOURCE_GROUP="<your-resource-group>"
export IDENTITY_NAME="<your-managed-identity-name>"

# Get managed identity principal ID
PRINCIPAL_ID=$(az identity show -g "${AZURE_RESOURCE_GROUP}" -n "${IDENTITY_NAME}" --query principalId -o tsv)
echo "Principal ID: ${PRINCIPAL_ID}"

# List all role assignments for the identity
echo "Role assignments for the managed identity:"
az role assignment list --assignee ${PRINCIPAL_ID} --all --query "[].roleDefinitionName" -o table

# Check for specific required roles
ROLES=$(az role assignment list --assignee ${PRINCIPAL_ID} --all --query "[].roleDefinitionName" -o tsv)

echo "Checking for required roles..."
for role in "Contributor" "Storage Blob Data Contributor" "Disk Snapshot Contributor"; do
    if echo "${ROLES}" | grep -q "${role}"; then
        echo "✓ ${role} is assigned"
    else
        echo "✗ ${role} is NOT assigned"
    fi
done

# Check scope of role assignments
echo "Role assignment scopes:"
az role assignment list --assignee ${PRINCIPAL_ID} --all --query "[].{Role:roleDefinitionName, Scope:scope}" -o table

Expected outcome: All three required roles are assigned to the managed identity at the appropriate scope.

GCP Test Cases

Test Case 1 - Basic CloudStorage Resource Creation

This test verifies that the CloudStorage controller can successfully create GCS buckets using the new provider implementation.

# Set up environment variables
export GCP_PROJECT_ID="<your-project-id>"
export BUCKET_NAME="<your-bucket-name>"

# Create CloudStorage resource
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: gcp-test-storage
  namespace: openshift-adp
spec:
  name: ${BUCKET_NAME}
  provider: gcp
  region: us-central1
  creationSecret:
    name: cloud-credentials-gcp
    key: service_account.json
EOF

# Wait for resource creation
sleep 5
oc get cloudstorage gcp-test-storage -n openshift-adp
oc describe cloudstorage gcp-test-storage -n openshift-adp

# Wait for bucket creation in GCS (retry up to 30 times)
echo "Waiting for bucket creation in GCS..."
for i in {1..30}; do
    if gsutil ls -p ${GCP_PROJECT_ID} | grep -q "gs://${BUCKET_NAME}/"; then
        echo "✅ Bucket ${BUCKET_NAME} exists in GCS"
        break
    else
        echo "  Waiting for bucket creation... ($i/30)"
        sleep 10
    fi
done

# Check CloudStorage status if bucket wasn't created
if [ $i -eq 30 ]; then
    echo "❌ Timeout waiting for bucket creation"
    oc get cloudstorage gcp-test-storage -n openshift-adp -o jsonpath='{.status}' | jq .
fi

Expected outcome: CloudStorage resource is created and GCS bucket is provisioned successfully.

Test Case 2 - DataProtectionApplication Integration

This test ensures that the CloudStorage provider works correctly with OADP's DataProtectionApplication for GCP.

# Set up environment variables
export GCP_PROJECT_ID="<your-project-id>"
export BUCKET_NAME="<your-bucket-name>"

# Create DPA with GCP configuration
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa-gcp
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - gcp
        - openshift
        - csi
  backupLocations:
    - name: default
      velero:
        provider: gcp
        default: true
        credential:
          name: cloud-credentials-gcp
          key: service_account.json
        objectStorage:
          bucket: ${BUCKET_NAME}
          prefix: velero
  snapshotLocations:
    - name: default
      velero:
        provider: gcp
        credential:
          name: cloud-credentials-gcp
          key: service_account.json
        config:
          project: ${GCP_PROJECT_ID}
          snapshotLocation: us-central1
EOF

# Wait for DPA to be ready
oc wait --for=condition=Reconciled dpa/dpa-gcp -n openshift-adp --timeout=300s

# Verify Velero deployment
oc get pods -n openshift-adp -l app.kubernetes.io/name=velero

# Wait for Velero pod to be ready
oc wait --for=condition=ready pod -l app.kubernetes.io/name=velero -n openshift-adp --timeout=300s

# Check backup storage location status
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc exec -n openshift-adp $VELERO_POD -- /velero backup-location get

Expected outcome: DPA is reconciled, Velero pod is running, and backup location is accessible.

Test Case 3 - Backup and Restore Operations

This test validates end-to-end backup and restore functionality using GCS as the backend.

# Clean up existing backups/restores
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')

# Delete all existing backups
oc exec -n openshift-adp $VELERO_POD -- /velero backup delete --all --confirm || true

# Wait for backup CRs to be deleted
while [ $(oc get backups.velero.io -n openshift-adp --no-headers 2>/dev/null | wc -l) -gt 0 ]; do
    echo "Waiting for backup CRs to be deleted..."
    sleep 5
done

# Delete all existing restores
oc exec -n openshift-adp $VELERO_POD -- /velero restore delete --all --confirm || true

# Wait for restore CRs to be deleted
while [ $(oc get restores.velero.io -n openshift-adp --no-headers 2>/dev/null | wc -l) -gt 0 ]; do
    echo "Waiting for restore CRs to be deleted..."
    sleep 5
done

# Create test namespace with deployment
oc create namespace test-backup-gcp || echo "Namespace might already exist"

cat << EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-openshift
  namespace: test-backup-gcp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello-openshift
  template:
    metadata:
      labels:
        app: hello-openshift
    spec:
      containers:
      - name: hello-openshift
        image: openshift/hello-openshift:latest
        ports:
        - containerPort: 8080
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
---
apiVersion: v1
kind: Service
metadata:
  name: hello-openshift
  namespace: test-backup-gcp
spec:
  selector:
    app: hello-openshift
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
EOF

# Wait for deployment
oc wait --for=condition=available deployment/hello-openshift -n test-backup-gcp --timeout=60s

# Create backup
velero backup create gcp-test-backup --include-namespaces=test-backup-gcp

# Monitor backup progress
sleep 10
velero backup describe gcp-test-backup --details

# Verify backup in GCS bucket
gsutil ls -r gs://${BUCKET_NAME}/velero/backups/

# Delete test namespace
oc delete namespace test-backup-gcp
sleep 20

# Restore from backup
velero restore create gcp-test-restore --from-backup gcp-test-backup
sleep 15

# Verify restoration
oc get all -n test-backup-gcp

Expected outcome: Backup is stored in GCS, namespace is deleted and restored successfully with all resources.

Test Case 4 - Error Handling with Invalid Credentials

This test verifies that the CloudStorage controller handles authentication errors gracefully.

# Create CloudStorage with non-existent secret
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: gcp-invalid-creds
  namespace: openshift-adp
spec:
  name: test-invalid-bucket
  provider: gcp
  region: us-central1
  creationSecret:
    name: non-existent-secret
    key: service_account.json
EOF

# Check CloudStorage status
sleep 5
oc describe cloudstorage gcp-invalid-creds -n openshift-adp

# Check controller logs for error messages
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "gcp-invalid-creds" | tail -10

# Clean up
oc delete cloudstorage gcp-invalid-creds -n openshift-adp

Expected outcome: CloudStorage shows error status indicating the secret doesn't exist.

Validation Steps

After running tests, validate the implementation:

# Check operator logs for errors
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "error" | tail -20

# Check CloudStorage controller specific logs
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "cloudstorage" | tail -20

# For Azure - verify federated credentials exist for both service accounts
az identity federated-credential list \
  --identity-name <identity-name> \
  --resource-group <resource-group> \
  --query "[].subject"

# For GCP - verify workload identity bindings
gcloud iam service-accounts get-iam-policy <service-account-email> \
  --format=json | jq '.bindings[].members[]' | grep "openshift-adp"

# Check CloudStorage status for all resources
oc get cloudstorage -n openshift-adp

# Verify no authentication errors in logs
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "unable to determine if bucket exists" || echo "No auth errors found"

Key Changes

  1. Azure: Added federated credential creation for openshift-adp-controller-manager service account
  2. GCP: Added workload identity bindings for openshift-adp-controller-manager service account
  3. Code: Made SharedCredentialsFileFromSecret function generic to support both AWS and GCP
  4. Documentation: Added comprehensive comments explaining Azure's credential handling approach

Required Secret Key Format by Provider

Provider Secret Key Example
AWS credentials AWS credentials file format
Azure azurekey Azure environment variables format
GCP service_account.json GCP service account JSON

This ensures that the CloudStorage controller can authenticate properly regardless of which service account it runs under.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 11, 2025
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Jul 11, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 11, 2025
@kaovilai kaovilai changed the title Add Azure and GCP CloudStorage provider implementations OADP-3307: Add Azure and GCP CloudStorage provider implementations Jul 11, 2025
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 11, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jul 11, 2025 •
edited by openshift-ci bot
Loading

@kaovilai: This pull request references OADP-3307 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the feature request to target the "4.20.0" version, but no target version was set.

In response to this:

Why the changes were made

This PR implements support for Azure Blob Storage and Google Cloud Storage as CloudStorage providers in OADP, extending beyond the existing AWS S3 support. This addresses the need for multi-cloud backup storage options and enables users to leverage their existing cloud storage accounts across different providers.

Related to: OADP-3307

Key changes:

  • Implemented Azure Blob Storage client with full container lifecycle management
  • Implemented Google Cloud Storage client with bucket operations support
  • Extended CloudStorage API to accept 'azure' and 'gcp' as valid providers
  • Added provider-specific configuration options through a new 'config' field
  • Added comprehensive unit tests for both implementations

How to test the changes made

  1. Azure CloudStorage testing:
# Create a CloudStorage CR for Azure
kubectl apply -f - <<EOF
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: azure-test
spec:
  name: my-container
  provider: azure
  creationSecret:
    name: azure-credentials
  config:
    storageAccount: "mystorageaccount"
EOF
  1. GCP CloudStorage testing:
# Create a CloudStorage CR for GCP
kubectl apply -f - <<EOF
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: gcp-test
spec:
  name: my-bucket
  provider: gcp
  region: us-central1
  creationSecret:
    name: gcp-credentials
EOF
  1. Run unit tests:
go test ./pkg/bucket/... -v
  1. Verify CRD updates:
kubectl explain cloudstorage.spec.provider
kubectl explain cloudstorage.spec.config

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Jul 11, 2025 •
edited by openshift-ci bot
Loading

@kaovilai: This pull request references OADP-3307 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the feature request to target the "4.20.0" version, but no target version was set.

In response to this:

Why the changes were made

This PR implements support for Azure Blob Storage and Google Cloud Storage as CloudStorage providers in OADP, extending beyond the existing AWS S3 support. This addresses the need for multi-cloud backup storage options and enables users to leverage their existing cloud storage accounts across different providers.

Related to: OADP-3307

Key changes:

  • Implemented Azure Blob Storage client with full container lifecycle management
  • Implemented Google Cloud Storage client with bucket operations support
  • Extended CloudStorage API to accept 'azure' and 'gcp' as valid providers
  • Added provider-specific configuration options through a new 'config' field
  • Added comprehensive unit tests for both implementations

How to test the changes made

  1. Azure CloudStorage testing:
# Create a CloudStorage CR for Azure
kubectl apply -f - <<EOF
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: azure-test
spec:
  name: my-container
  provider: azure
  creationSecret:
    name: azure-credentials
  config:
    storageAccount: "mystorageaccount"
EOF
  1. GCP CloudStorage testing:
# Create a CloudStorage CR for GCP
kubectl apply -f - <<EOF
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
  name: gcp-test
spec:
  name: my-bucket
  provider: gcp
  region: us-central1
  creationSecret:
    name: gcp-credentials
EOF
  1. Run unit tests:
go test ./pkg/bucket/... -v
  1. Verify CRD updates:
kubectl explain cloudstorage.spec.provider
kubectl explain cloudstorage.spec.config

🤖 Generated with Claude Code

This will need to rebase when #1712 merges.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@kaovilai kaovilai force-pushed the OADP-3307 branch 2 times, most recently from b197ddd to 150c818 Compare July 12, 2025 04:22
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 18, 2025
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 31, 2025
@kaovilai kaovilai changed the title OADP-3307: Add Azure and GCP CloudStorage provider implementations OADP-3307: Add Azure and GCP CloudStorage API provider implementations Aug 1, 2025
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 1, 2025
@kaovilai kaovilai force-pushed the OADP-3307 branch 2 times, most recently from 14184a7 to e2c38f5 Compare August 1, 2025 02:48
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 1, 2025
@kaovilai kaovilai force-pushed the OADP-3307 branch 6 times, most recently from 317a215 to 2a289d0 Compare August 1, 2025 06:35
@kaovilai
Copy link
Member Author

kaovilai commented Aug 1, 2025

Azure CloudStorage Test Results ✅

Manual Test Execution Results:

Test Case 1: Basic CloudStorage Resource Creation ✅
Test Case 2: DataProtectionApplication Integration ✅
Test Case 3: Backup and Restore Operations ✅
Test Case 4: Azure Workload Identity Authentication ✅
Test Case 5: Error Handling - Invalid Storage Account ✅
Test Case 6: Role Assignment Validation ✅

@kaovilai
Copy link
Member Author

kaovilai commented Aug 1, 2025

✅ GCP CloudStorage API Test Results

Successfully tested CloudStorage API functionality for GCP with Workload Identity Federation on OpenShift.

Test Environment:

  • Cluster: tkaovila-20250731-wif
  • GCP Project: gc-acm-demo
  • OADP Version: v99.0.0

Summary: All test cases PASSED ✅

Test Execution Details

Test Case 1: Basic CloudStorage Resource Creation ✅

  • CloudStorage CR created successfully
  • Bucket velero-gc-acm-demo-tkaovila-20250731-wif-20250801025545 automatically created in GCS
  • Bucket verified in us-central1 region
  • CloudStorage controller properly handling GCP WIF authentication

Test Case 2: DataProtectionApplication Integration ✅

  • DPA successfully reconciled with CloudStorage
  • Velero pod deployed and running
  • Backup storage location validated and marked as Available
  • GCP credentials properly configured via WIF

Test Case 3: Backup and Restore Operations ✅

  • Created test namespace with OpenShift-compatible workload
  • Backup completed successfully (22 items backed up)
  • Backup artifacts stored in GCS bucket:
    • gcp-test-backup.tar.gz
    • Resource lists, logs, and metadata files
  • Restore completed successfully
  • All resources recreated (2 pods, 1 service, 1 deployment)

Test Case 4: Error Handling - Invalid Credentials ✅

  • CloudStorage CR created with non-existent secret
  • Controller properly handled the error case
  • Resource remained in pending state as expected

Validation Results:

  • ✅ Service account has required permissions (roles/storage.admin)
  • ✅ Workload Identity Federation configured correctly
  • ✅ Bucket IAM policies properly set
  • ✅ CloudStorage controller creating and managing buckets
  • ✅ Backup/restore operations working with GCP storage

Notes:

  • Bucket deletion with annotation oadp.openshift.io/cloudstorage-delete=true verified in manual testing

The CloudStorage API is working correctly with GCP WIF authentication. The operator successfully:

  1. Creates GCS buckets using workload identity credentials
  2. Integrates with DPA for backup operations
  3. Handles error cases appropriately
  4. Manages bucket lifecycle with proper cleanup

@kaovilai kaovilai marked this pull request as ready for review August 1, 2025 08:16
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 1, 2025
@openshift-ci openshift-ci bot requested a review from mrnold August 1, 2025 08:16
@openshift-ci openshift-ci bot requested a review from sseago August 1, 2025 08:16
@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 1, 2025 •
edited by openshift-ci bot
Loading

@kaovilai: This pull request references OADP-3307 which is a valid jira issue.

In response to this:

Why the changes were made

This PR fixes Azure Workload Identity authentication failures for the OADP CloudStorage controller by ensuring both service accounts (velero and openshift-adp-controller-manager) have proper federated identity credentials configured.

The issue was that while the velero service account had federated identity credentials, the openshift-adp-controller-manager service account (used by the CloudStorage controller) lacked these credentials, causing authentication failures when attempting to manage Azure storage containers.

Related issue: OADP-3307

How to test the changes made

Prerequisites

  1. An OpenShift cluster with Azure Workload Identity enabled
  2. GCP cluster with Workload Identity Federation configured (for GCP testing)
  3. Proper identity setup using the helper scripts in dotfiles repository

Testing Azure CloudStorage with Workload Identity

  1. Set up Azure identity for both service accounts:
# Use the updated helper function that now creates federated credentials for both service accounts
create-velero-identity-for-azure-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-test-storage
 namespace: openshift-adp
spec:
 name: velero
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey  # Required key for Azure credentials
 config:
   storageAccount: "<your-storage-account>"

Note: The creationSecret.key must be azurekey for Azure provider.

  1. Run the full test suite:
./run-azure-test-cases.zsh

Testing GCP CloudStorage with Workload Identity Federation

  1. Set up GCP identity for both service accounts:
# Use the updated helper function that now creates workload identity bindings for both service accounts
create-velero-identity-for-gcp-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: gcp-test-storage
 namespace: openshift-adp
spec:
 name: <bucket-name>
 provider: gcp
 region: us-central1
 creationSecret:
   name: cloud-credentials-gcp
   key: service_account.json  # Required key for GCP credentials

Note: The creationSecret.key must be service_account.json for GCP provider.

  1. Run the full test suite:
./run-gcp-test-cases.zsh

Verifying the Fix

  1. Check that both service accounts can authenticate:
# Check CloudStorage controller logs (should not show authentication errors)
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "unable to determine if bucket exists"

# For Azure, verify federated credentials exist for both service accounts
az identity federated-credential list \
 --identity-name <identity-name> \
 --resource-group <resource-group> \
 --query "[].subject"

# For GCP, verify workload identity bindings
gcloud iam service-accounts get-iam-policy <service-account-email> \
 --format=json | jq '.bindings[].members[]' | grep "openshift-adp"
  1. Verify CloudStorage operations work:
  • Container/bucket creation
  • Container/bucket deletion (with proper annotation)
  • Status updates in CloudStorage CR

Key Changes

  1. Azure: Added federated credential creation for openshift-adp-controller-manager service account
  2. GCP: Added workload identity bindings for openshift-adp-controller-manager service account
  3. Code: Made SharedCredentialsFileFromSecret function generic to support both AWS and GCP
  4. Documentation: Added comprehensive comments explaining Azure's credential handling approach

Required Secret Key Format by Provider

Provider Secret Key Example
AWS credentials AWS credentials file format
Azure azurekey Azure environment variables format
GCP service_account.json GCP service account JSON

This ensures that the CloudStorage controller can authenticate properly regardless of which service account it runs under.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

1 similar comment
@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 1, 2025 •
edited by openshift-ci bot
Loading

@kaovilai: This pull request references OADP-3307 which is a valid jira issue.

In response to this:

Why the changes were made

This PR fixes Azure Workload Identity authentication failures for the OADP CloudStorage controller by ensuring both service accounts (velero and openshift-adp-controller-manager) have proper federated identity credentials configured.

The issue was that while the velero service account had federated identity credentials, the openshift-adp-controller-manager service account (used by the CloudStorage controller) lacked these credentials, causing authentication failures when attempting to manage Azure storage containers.

Related issue: OADP-3307

How to test the changes made

Prerequisites

  1. An OpenShift cluster with Azure Workload Identity enabled
  2. GCP cluster with Workload Identity Federation configured (for GCP testing)
  3. Proper identity setup using the helper scripts in dotfiles repository

Testing Azure CloudStorage with Workload Identity

  1. Set up Azure identity for both service accounts:
# Use the updated helper function that now creates federated credentials for both service accounts
create-velero-identity-for-azure-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-test-storage
 namespace: openshift-adp
spec:
 name: velero
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey  # Required key for Azure credentials
 config:
   storageAccount: "<your-storage-account>"

Note: The creationSecret.key must be azurekey for Azure provider.

  1. Run the full test suite:
./run-azure-test-cases.zsh

Testing GCP CloudStorage with Workload Identity Federation

  1. Set up GCP identity for both service accounts:
# Use the updated helper function that now creates workload identity bindings for both service accounts
create-velero-identity-for-gcp-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: gcp-test-storage
 namespace: openshift-adp
spec:
 name: <bucket-name>
 provider: gcp
 region: us-central1
 creationSecret:
   name: cloud-credentials-gcp
   key: service_account.json  # Required key for GCP credentials

Note: The creationSecret.key must be service_account.json for GCP provider.

  1. Run the full test suite:
./run-gcp-test-cases.zsh

Verifying the Fix

  1. Check that both service accounts can authenticate:
# Check CloudStorage controller logs (should not show authentication errors)
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "unable to determine if bucket exists"

# For Azure, verify federated credentials exist for both service accounts
az identity federated-credential list \
 --identity-name <identity-name> \
 --resource-group <resource-group> \
 --query "[].subject"

# For GCP, verify workload identity bindings
gcloud iam service-accounts get-iam-policy <service-account-email> \
 --format=json | jq '.bindings[].members[]' | grep "openshift-adp"
  1. Verify CloudStorage operations work:
  • Container/bucket creation
  • Container/bucket deletion (with proper annotation)
  • Status updates in CloudStorage CR

Key Changes

  1. Azure: Added federated credential creation for openshift-adp-controller-manager service account
  2. GCP: Added workload identity bindings for openshift-adp-controller-manager service account
  3. Code: Made SharedCredentialsFileFromSecret function generic to support both AWS and GCP
  4. Documentation: Added comprehensive comments explaining Azure's credential handling approach

Required Secret Key Format by Provider

Provider Secret Key Example
AWS credentials AWS credentials file format
Azure azurekey Azure environment variables format
GCP service_account.json GCP service account JSON

This ensures that the CloudStorage controller can authenticate properly regardless of which service account it runs under.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

kaovilai
kaovilai commented Aug 1, 2025
View reviewed changes
docs/config/aws/oadp-aws-sts-cloud-authentication.adoc Outdated
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adoc format so its easier on doc team to integrate with openshift-docs

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++ we should follow this practice more often
cc: @weshayutin

shubham-pampattiwar
shubham-pampattiwar reviewed Aug 1, 2025
View reviewed changes
@kaovilai
Copy link
Member Author

kaovilai commented Aug 1, 2025

/cherry-pick oadp-1.5

@openshift-cherrypick-robot
Copy link
Contributor

@kaovilai: once the present PR merges, I will cherry-pick it on top of oadp-1.5 in a new PR and assign it to you.

In response to this:

/cherry-pick oadp-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kaovilai kaovilai force-pushed the OADP-3307 branch 3 times, most recently from 4b6041a to 155c91e Compare August 2, 2025 03:03
@weshayutin
Copy link
Contributor

./run-azure-test-cases.zsh ? where does this file live?

@weshayutin
Copy link
Contributor

unit test?

internal/controller/bsl_test.go:1692:1: File is not properly formatted (gci)
									"s3ForcePathStyle":     "true",
^
internal/controller/bsl_test.go:4658:2: import-shadowing: The name 'scheme' shadows an import name (revive)
	scheme, err := getSchemeForFakeClient()
	^
2 issues:
* gci: 1
* revive: 1
make[1]: *** [Makefile:167: lint] Error 1
make[1]: Leaving directory '/go/src/github.com/openshift/oadp-operator'
make: *** [Makefile:144: test] Error 2

@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 4, 2025 •
edited by openshift-ci bot
Loading

@kaovilai: This pull request references OADP-3307 which is a valid jira issue.

In response to this:

Why the changes were made

This PR fixes Azure Workload Identity authentication failures for the OADP CloudStorage controller by ensuring both service accounts (velero and openshift-adp-controller-manager) have proper federated identity credentials configured.

The issue was that while the velero service account had federated identity credentials, the openshift-adp-controller-manager service account (used by the CloudStorage controller) lacked these credentials, causing authentication failures when attempting to manage Azure storage containers.

Related issue: OADP-3307

How to test the changes made

Prerequisites

  1. An OpenShift cluster with Azure Workload Identity enabled
  2. GCP cluster with Workload Identity Federation configured (for GCP testing)
  3. Proper identity setup using the helper scripts in dotfiles repository

Testing Azure CloudStorage with Workload Identity

  1. Set up Azure identity for both service accounts:
# Use the updated helper function that now creates federated credentials for both service accounts
create-velero-identity-for-azure-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-test-storage
 namespace: openshift-adp
spec:
 name: velero
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey  # Required key for Azure credentials
 config:
   storageAccount: "<your-storage-account>"

Note: The creationSecret.key must be azurekey for Azure provider.

  1. Run the full test suite:

Test Suite Overview:

  • Test Case 1: Basic CloudStorage Resource Creation - Verifies basic CloudStorage creation and status
  • Test Case 2: DataProtectionApplication Integration - Tests DPA creation with Azure BSL configuration
  • Test Case 3: Backup and Restore Operations - Validates end-to-end backup/restore functionality
  • Test Case 4: Azure Workload Identity Authentication - Confirms proper environment variables and service account annotations
  • Test Case 5: Error Handling - Tests behavior with invalid storage accounts
  • Test Case 6: Container Lifecycle Management - Tests container creation and deletion
  • Test Case 7: Role Assignment Validation - Verifies required Azure roles are assigned

Testing GCP CloudStorage with Workload Identity Federation

  1. Set up GCP identity for both service accounts:
# Use the updated helper function that now creates workload identity bindings for both service accounts
create-velero-identity-for-gcp-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: gcp-test-storage
 namespace: openshift-adp
spec:
 name: <bucket-name>
 provider: gcp
 region: us-central1
 creationSecret:
   name: cloud-credentials-gcp
   key: service_account.json  # Required key for GCP credentials

Note: The creationSecret.key must be service_account.json for GCP provider.

  1. Run the full test suite:

Test Suite Overview:

  • Test Case 1: Basic CloudStorage Resource Creation - Verifies bucket creation in GCS
  • Test Case 2: DataProtectionApplication Integration - Tests DPA with GCP BSL and VSL configuration
  • Test Case 3: Backup and Restore Operations - Full backup/restore cycle with hello-openshift deployment
  • Test Case 4: Error Handling - Tests behavior with invalid credentials

Verifying the Fix

  1. Check that both service accounts can authenticate:
# Check CloudStorage controller logs (should not show authentication errors)
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "unable to determine if bucket exists"

# For Azure, verify federated credentials exist for both service accounts
az identity federated-credential list \
 --identity-name <identity-name> \
 --resource-group <resource-group> \
 --query "[].subject"

# For GCP, verify workload identity bindings
gcloud iam service-accounts get-iam-policy <service-account-email> \
 --format=json | jq '.bindings[].members[]' | grep "openshift-adp"
  1. Verify CloudStorage operations work:
  • Container/bucket creation
  • Container/bucket deletion (with proper annotation)
  • Status updates in CloudStorage CR

Key Changes

  1. Azure: Added federated credential creation for openshift-adp-controller-manager service account
  2. GCP: Added workload identity bindings for openshift-adp-controller-manager service account
  3. Code: Made SharedCredentialsFileFromSecret function generic to support both AWS and GCP
  4. Documentation: Added comprehensive comments explaining Azure's credential handling approach

Required Secret Key Format by Provider

Provider Secret Key Example
AWS credentials AWS credentials file format
Azure azurekey Azure environment variables format
GCP service_account.json GCP service account JSON

This ensures that the CloudStorage controller can authenticate properly regardless of which service account it runs under.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 4, 2025 •
edited by openshift-ci bot
Loading

@kaovilai: This pull request references OADP-3307 which is a valid jira issue.

In response to this:

Why the changes were made

This PR fixes Azure Workload Identity authentication failures for the OADP CloudStorage controller by ensuring both service accounts (velero and openshift-adp-controller-manager) have proper federated identity credentials configured.

The issue was that while the velero service account had federated identity credentials, the openshift-adp-controller-manager service account (used by the CloudStorage controller) lacked these credentials, causing authentication failures when attempting to manage Azure storage containers.

Related issue: OADP-3307

How to test the changes made

Prerequisites

  1. An OpenShift cluster with Azure Workload Identity enabled
  2. GCP cluster with Workload Identity Federation configured (for GCP testing)
  3. Proper identity setup using the helper scripts in dotfiles repository

Testing Azure CloudStorage with Workload Identity

  1. Set up Azure identity for both service accounts:
# Use the updated helper function that now creates federated credentials for both service accounts
create-velero-identity-for-azure-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-test-storage
 namespace: openshift-adp
spec:
 name: velero
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey  # Required key for Azure credentials
 config:
   storageAccount: "<your-storage-account>"

Note: The creationSecret.key must be azurekey for Azure provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource above
oc apply -f cloudstorage.yaml

# Check that the resource was created and container exists in Azure
oc get cloudstorage azure-test-storage -n openshift-adp
az storage container list --account-name <storage-account> --auth-mode login

Verify with DataProtectionApplication:

# Create a DPA with Azure configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
     - azure
     - openshift
 backupLocations:
 - velero:
     provider: azure
     default: true
     credential:
       name: cloud-credentials-azure
       key: azurekey
     config:
       resourceGroup: <resource-group>
       storageAccount: <storage-account>
     objectStorage:
       bucket: velero
       prefix: <cluster-name>

Test Backup and Restore:

# Create a test namespace with an application
oc create namespace test-backup-azure
oc create deployment nginx --image=bitnami/nginx:latest -n test-backup-azure
oc wait --for=condition=available deployment/nginx -n test-backup-azure --timeout=60s

# Create a backup
velero backup create azure-test-backup --include-namespaces=test-backup-azure

# Wait and verify backup completed
velero backup describe azure-test-backup

# Delete the namespace
oc delete namespace test-backup-azure

# Restore from backup
velero restore create azure-test-restore --from-backup azure-test-backup

# Verify namespace was restored
oc get namespace test-backup-azure

Verify Azure Workload Identity is working:

# Check Velero pod has correct environment variables
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_CLIENT_ID")]}' | jq .
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_FEDERATED_TOKEN_FILE")]}' | jq .

# Check service account annotation
oc get sa velero -n openshift-adp -o jsonpath='{.metadata.annotations.azure\.workload\.identity/client-id}'

Test error handling with invalid storage account:

# Create CloudStorage with non-existent storage account
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-invalid-storage
 namespace: openshift-adp
spec:
 name: test-container
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey
 config:
   storageAccount: "nonexistentstorage123"
EOF

# Add deletion annotation and delete (should succeed with the fix)
oc annotate cloudstorage azure-invalid-storage -n openshift-adp oadp.openshift.io/cloudstorage-delete=true --overwrite
oc delete cloudstorage azure-invalid-storage -n openshift-adp --timeout=60s

Testing GCP CloudStorage with Workload Identity Federation

  1. Set up GCP identity for both service accounts:
# Use the updated helper function that now creates workload identity bindings for both service accounts
create-velero-identity-for-gcp-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: gcp-test-storage
 namespace: openshift-adp
spec:
 name: <bucket-name>
 provider: gcp
 region: us-central1
 creationSecret:
   name: cloud-credentials-gcp
   key: service_account.json  # Required key for GCP credentials

Note: The creationSecret.key must be service_account.json for GCP provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource
oc apply -f cloudstorage-gcp.yaml

# Verify bucket exists in GCS
gsutil ls -p <project-id> | grep "gs://<bucket-name>/"

Test with DataProtectionApplication:

# Create DPA with GCP configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa-gcp
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
       - gcp
       - openshift
       - csi
 backupLocations:
   - name: default
     velero:
       provider: gcp
       default: true
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       objectStorage:
         bucket: <bucket-name>
         prefix: velero
 snapshotLocations:
   - name: default
     velero:
       provider: gcp
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       config:
         project: <project-id>
         snapshotLocation: us-central1

Verify backup/restore works:

# Create test resources
oc create namespace test-backup-gcp
oc create deployment hello-openshift --image=openshift/hello-openshift:latest -n test-backup-gcp
oc wait --for=condition=available deployment/hello-openshift -n test-backup-gcp --timeout=60s

# Backup and restore
velero backup create gcp-test-backup --include-namespaces=test-backup-gcp
velero backup describe gcp-test-backup
gsutil ls -r gs://<bucket-name>/velero/backups/

# Delete and restore
oc delete namespace test-backup-gcp
velero restore create gcp-test-restore --from-backup gcp-test-backup
oc get all -n test-backup-gcp

Verifying the Fix

  1. Check that both service accounts can authenticate:
# Check CloudStorage controller logs (should not show authentication errors)
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "unable to determine if bucket exists"

# For Azure, verify federated credentials exist for both service accounts
az identity federated-credential list \
 --identity-name <identity-name> \
 --resource-group <resource-group> \
 --query "[].subject"

# For GCP, verify workload identity bindings
gcloud iam service-accounts get-iam-policy <service-account-email> \
 --format=json | jq '.bindings[].members[]' | grep "openshift-adp"
  1. Verify CloudStorage operations work:
  • Container/bucket creation
  • Container/bucket deletion (with proper annotation)
  • Status updates in CloudStorage CR

Key Changes

  1. Azure: Added federated credential creation for openshift-adp-controller-manager service account
  2. GCP: Added workload identity bindings for openshift-adp-controller-manager service account
  3. Code: Made SharedCredentialsFileFromSecret function generic to support both AWS and GCP
  4. Documentation: Added comprehensive comments explaining Azure's credential handling approach

Required Secret Key Format by Provider

Provider Secret Key Example
AWS credentials AWS credentials file format
Azure azurekey Azure environment variables format
GCP service_account.json GCP service account JSON

This ensures that the CloudStorage controller can authenticate properly regardless of which service account it runs under.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 12, 2025 •
edited by openshift-ci bot
Loading

@kaovilai: This pull request references OADP-3307 which is a valid jira issue.

In response to this:

Why the changes were made

Related issue: OADP-3307

How to test the changes made

Prerequisites

  1. An OpenShift cluster with Azure Workload Identity enabled
  2. GCP cluster with Workload Identity Federation configured (for GCP testing)
  3. Proper identity setup using the helper scripts in dotfiles repository

Testing Azure CloudStorage with Workload Identity

  1. Set up Azure identity for both service accounts:
# Use the updated helper function that now creates federated credentials for both service accounts
create-velero-identity-for-azure-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-test-storage
 namespace: openshift-adp
spec:
 name: velero
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey  # Required key for Azure credentials
 config:
   storageAccount: "<your-storage-account>"

Note: The creationSecret.key must be azurekey for Azure provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource above
oc apply -f cloudstorage.yaml

# Check that the resource was created and container exists in Azure
oc get cloudstorage azure-test-storage -n openshift-adp
az storage container list --account-name <storage-account> --auth-mode login

Verify with DataProtectionApplication:

# Create a DPA with Azure configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
     - azure
     - openshift
 backupLocations:
 - velero:
     provider: azure
     default: true
     credential:
       name: cloud-credentials-azure
       key: azurekey
     config:
       resourceGroup: <resource-group>
       storageAccount: <storage-account>
     objectStorage:
       bucket: velero
       prefix: <cluster-name>

Test Backup and Restore:

# Create a test namespace with an application
oc create namespace test-backup-azure
oc create deployment nginx --image=bitnami/nginx:latest -n test-backup-azure
oc wait --for=condition=available deployment/nginx -n test-backup-azure --timeout=60s

# Create a backup
velero backup create azure-test-backup --include-namespaces=test-backup-azure

# Wait and verify backup completed
velero backup describe azure-test-backup

# Delete the namespace
oc delete namespace test-backup-azure

# Restore from backup
velero restore create azure-test-restore --from-backup azure-test-backup

# Verify namespace was restored
oc get namespace test-backup-azure

Verify Azure Workload Identity is working:

# Check Velero pod has correct environment variables
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_CLIENT_ID")]}' | jq .
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_FEDERATED_TOKEN_FILE")]}' | jq .

# Check service account annotation
oc get sa velero -n openshift-adp -o jsonpath='{.metadata.annotations.azure\.workload\.identity/client-id}'

Test error handling with invalid storage account:

# Create CloudStorage with non-existent storage account
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-invalid-storage
 namespace: openshift-adp
spec:
 name: test-container
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey
 config:
   storageAccount: "nonexistentstorage123"
EOF

# Add deletion annotation and delete (should succeed with the fix)
oc annotate cloudstorage azure-invalid-storage -n openshift-adp oadp.openshift.io/cloudstorage-delete=true --overwrite
oc delete cloudstorage azure-invalid-storage -n openshift-adp --timeout=60s

Testing GCP CloudStorage with Workload Identity Federation

  1. Set up GCP identity for both service accounts:
# Use the updated helper function that now creates workload identity bindings for both service accounts
create-velero-identity-for-gcp-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: gcp-test-storage
 namespace: openshift-adp
spec:
 name: <bucket-name>
 provider: gcp
 region: us-central1
 creationSecret:
   name: cloud-credentials-gcp
   key: service_account.json  # Required key for GCP credentials

Note: The creationSecret.key must be service_account.json for GCP provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource
oc apply -f cloudstorage-gcp.yaml

# Verify bucket exists in GCS
gsutil ls -p <project-id> | grep "gs://<bucket-name>/"

Test with DataProtectionApplication:

# Create DPA with GCP configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa-gcp
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
       - gcp
       - openshift
       - csi
 backupLocations:
   - name: default
     velero:
       provider: gcp
       default: true
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       objectStorage:
         bucket: <bucket-name>
         prefix: velero
 snapshotLocations:
   - name: default
     velero:
       provider: gcp
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       config:
         project: <project-id>
         snapshotLocation: us-central1

Verify backup/restore works:

# Create test resources
oc create namespace test-backup-gcp
oc create deployment hello-openshift --image=openshift/hello-openshift:latest -n test-backup-gcp
oc wait --for=condition=available deployment/hello-openshift -n test-backup-gcp --timeout=60s

# Backup and restore
velero backup create gcp-test-backup --include-namespaces=test-backup-gcp
velero backup describe gcp-test-backup
gsutil ls -r gs://<bucket-name>/velero/backups/

# Delete and restore
oc delete namespace test-backup-gcp
velero restore create gcp-test-restore --from-backup gcp-test-backup
oc get all -n test-backup-gcp

Verifying the Fix

  1. Check that both service accounts can authenticate:
# Check CloudStorage controller logs (should not show authentication errors)
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "unable to determine if bucket exists"

# For Azure, verify federated credentials exist for both service accounts
az identity federated-credential list \
 --identity-name <identity-name> \
 --resource-group <resource-group> \
 --query "[].subject"

# For GCP, verify workload identity bindings
gcloud iam service-accounts get-iam-policy <service-account-email> \
 --format=json | jq '.bindings[].members[]' | grep "openshift-adp"
  1. Verify CloudStorage operations work:
  • Container/bucket creation
  • Container/bucket deletion (with proper annotation)
  • Status updates in CloudStorage CR

Key Changes

  1. Azure: Added federated credential creation for openshift-adp-controller-manager service account
  2. GCP: Added workload identity bindings for openshift-adp-controller-manager service account
  3. Code: Made SharedCredentialsFileFromSecret function generic to support both AWS and GCP
  4. Documentation: Added comprehensive comments explaining Azure's credential handling approach

Required Secret Key Format by Provider

Provider Secret Key Example
AWS credentials AWS credentials file format
Azure azurekey Azure environment variables format
GCP service_account.json GCP service account JSON

This ensures that the CloudStorage controller can authenticate properly regardless of which service account it runs under.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 12, 2025 •
edited by openshift-ci bot
Loading

@kaovilai: This pull request references OADP-3307 which is a valid jira issue.

In response to this:

Why the changes were made

Related issue: OADP-3307

How to test the changes made

Prerequisites

  1. An OpenShift cluster with Azure Workload Identity enabled
  2. GCP cluster with Workload Identity Federation configured (for GCP testing)
  3. Proper identity setup using the helper scripts in dotfiles repository

Testing Azure CloudStorage with Workload Identity

  1. Set up Azure identity for both service accounts:
# Use the updated helper function that now creates federated credentials for both service accounts
create-velero-identity-for-azure-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-test-storage
 namespace: openshift-adp
spec:
 name: velero
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey  # Required key for Azure credentials
 config:
   storageAccount: "<your-storage-account>"

Note: The creationSecret.key must be azurekey for Azure provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource above
oc apply -f cloudstorage.yaml

# Check that the resource was created and container exists in Azure
oc get cloudstorage azure-test-storage -n openshift-adp
az storage container list --account-name <storage-account> --auth-mode login

Verify with DataProtectionApplication:

# Create a DPA with Azure configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
     - azure
     - openshift
 backupLocations:
 - velero:
     provider: azure
     default: true
     credential:
       name: cloud-credentials-azure
       key: azurekey
     config:
       resourceGroup: <resource-group>
       storageAccount: <storage-account>
     objectStorage:
       bucket: velero
       prefix: <cluster-name>

Test Backup and Restore:

# Create a test namespace with an application
oc create namespace test-backup-azure
oc create deployment nginx --image=bitnami/nginx:latest -n test-backup-azure
oc wait --for=condition=available deployment/nginx -n test-backup-azure --timeout=60s

# Create a backup
velero backup create azure-test-backup --include-namespaces=test-backup-azure

# Wait and verify backup completed
velero backup describe azure-test-backup

# Delete the namespace
oc delete namespace test-backup-azure

# Restore from backup
velero restore create azure-test-restore --from-backup azure-test-backup

# Verify namespace was restored
oc get namespace test-backup-azure

Verify Azure Workload Identity is working:

# Check Velero pod has correct environment variables
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_CLIENT_ID")]}' | jq .
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_FEDERATED_TOKEN_FILE")]}' | jq .

# Check service account annotation
oc get sa velero -n openshift-adp -o jsonpath='{.metadata.annotations.azure\.workload\.identity/client-id}'

Test error handling with invalid storage account:

# Create CloudStorage with non-existent storage account
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-invalid-storage
 namespace: openshift-adp
spec:
 name: test-container
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey
 config:
   storageAccount: "nonexistentstorage123"
EOF

# Add deletion annotation and delete (should succeed with the fix)
oc annotate cloudstorage azure-invalid-storage -n openshift-adp oadp.openshift.io/cloudstorage-delete=true --overwrite
oc delete cloudstorage azure-invalid-storage -n openshift-adp --timeout=60s

Testing GCP CloudStorage with Workload Identity Federation

  1. Set up GCP identity for both service accounts:
# Use the updated helper function that now creates workload identity bindings for both service accounts
create-velero-identity-for-gcp-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: gcp-test-storage
 namespace: openshift-adp
spec:
 name: <bucket-name>
 provider: gcp
 region: us-central1
 creationSecret:
   name: cloud-credentials-gcp
   key: service_account.json  # Required key for GCP credentials

Note: The creationSecret.key must be service_account.json for GCP provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource
oc apply -f cloudstorage-gcp.yaml

# Verify bucket exists in GCS
gsutil ls -p <project-id> | grep "gs://<bucket-name>/"

Test with DataProtectionApplication:

# Create DPA with GCP configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa-gcp
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
       - gcp
       - openshift
       - csi
 backupLocations:
   - name: default
     velero:
       provider: gcp
       default: true
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       objectStorage:
         bucket: <bucket-name>
         prefix: velero
 snapshotLocations:
   - name: default
     velero:
       provider: gcp
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       config:
         project: <project-id>
         snapshotLocation: us-central1

Verify backup/restore works:

# Create test resources
oc create namespace test-backup-gcp
oc create deployment hello-openshift --image=openshift/hello-openshift:latest -n test-backup-gcp
oc wait --for=condition=available deployment/hello-openshift -n test-backup-gcp --timeout=60s

# Backup and restore
velero backup create gcp-test-backup --include-namespaces=test-backup-gcp
velero backup describe gcp-test-backup
gsutil ls -r gs://<bucket-name>/velero/backups/

# Delete and restore
oc delete namespace test-backup-gcp
velero restore create gcp-test-restore --from-backup gcp-test-backup
oc get all -n test-backup-gcp

Running Comprehensive Test Suites

For detailed test execution, follow these steps:

Azure Test Suite

  1. Set up environment variables:
export CLUSTER_NAME="<your-cluster-name>"
export AZURE_SUBSCRIPTION_ID="<your-subscription-id>"
export AZURE_TENANT_ID="<your-tenant-id>"
export AZURE_CLIENT_ID="<your-client-id>"
export AZURE_RESOURCE_GROUP="<your-resource-group>"
export STORAGE_ACCOUNT_NAME="<your-storage-account>"
export CONTAINER_NAME="velero"
export IDENTITY_NAME="<your-managed-identity-name>"
  1. Run test cases sequentially:
  • Test Case 1: Basic CloudStorage Creation (see commands above)
  • Test Case 2: DataProtectionApplication Integration
  • Test Case 3: Backup and Restore Operations
  • Test Case 4: Azure Workload Identity Authentication
  • Test Case 5: Error Handling with Invalid Storage
  • Test Case 6: Container Lifecycle Management
  • Test Case 7: Role Assignment Validation

Each test case includes verification steps to ensure proper functionality.

GCP Test Suite

  1. Set up environment variables:
export CLUSTER_NAME="<your-cluster-name>"
export GCP_PROJECT_ID="<your-project-id>"
export GCP_REGION="us-central1"
export SERVICE_ACCOUNT_EMAIL="<your-service-account-email>"
export BUCKET_NAME="<your-bucket-name>"
  1. Run test cases:
  • Test Case 1: Basic CloudStorage Creation
  • Test Case 2: DataProtectionApplication Integration
  • Test Case 3: Backup and Restore Operations
  • Test Case 4: Error Handling with Invalid Credentials

Validation Steps

After running tests, validate the implementation:

# Check operator logs for errors
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "error"

# For Azure - verify federated credentials
az identity federated-credential list \
 --identity-name <identity-name> \
 --resource-group <resource-group> \
 --query "[].subject"

# For GCP - verify workload identity bindings
gcloud iam service-accounts get-iam-policy <service-account-email> \
 --format=json | jq '.bindings[].members[]' | grep "openshift-adp"

# Check CloudStorage status
oc get cloudstorage -n openshift-adp

Key Changes

  1. Azure: Added federated credential creation for openshift-adp-controller-manager service account
  2. GCP: Added workload identity bindings for openshift-adp-controller-manager service account
  3. Code: Made SharedCredentialsFileFromSecret function generic to support both AWS and GCP
  4. Documentation: Added comprehensive comments explaining Azure's credential handling approach

Required Secret Key Format by Provider

Provider Secret Key Example
AWS credentials AWS credentials file format
Azure azurekey Azure environment variables format
GCP service_account.json GCP service account JSON

This ensures that the CloudStorage controller can authenticate properly regardless of which service account it runs under.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Aug 12, 2025 •
edited by openshift-ci bot
Loading

@kaovilai: This pull request references OADP-3307 which is a valid jira issue.

In response to this:

Why the changes were made

Related issue: OADP-3307

How to test the changes made

Prerequisites

  1. An OpenShift cluster with Azure Workload Identity enabled
  2. GCP cluster with Workload Identity Federation configured (for GCP testing)
  3. Proper identity setup using the helper scripts in dotfiles repository

Testing Azure CloudStorage with Workload Identity

  1. Set up Azure identity for both service accounts:
# Use the updated helper function that now creates federated credentials for both service accounts
create-velero-identity-for-azure-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-test-storage
 namespace: openshift-adp
spec:
 name: velero
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey  # Required key for Azure credentials
 config:
   storageAccount: "<your-storage-account>"

Note: The creationSecret.key must be azurekey for Azure provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource above
oc apply -f cloudstorage.yaml

# Check that the resource was created and container exists in Azure
oc get cloudstorage azure-test-storage -n openshift-adp
az storage container list --account-name <storage-account> --auth-mode login

Verify with DataProtectionApplication:

# Create a DPA with Azure configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
     - azure
     - openshift
 backupLocations:
 - velero:
     provider: azure
     default: true
     credential:
       name: cloud-credentials-azure
       key: azurekey
     config:
       resourceGroup: <resource-group>
       storageAccount: <storage-account>
     objectStorage:
       bucket: velero
       prefix: <cluster-name>

Test Backup and Restore:

# Create a test namespace with an application
oc create namespace test-backup-azure
oc create deployment nginx --image=bitnami/nginx:latest -n test-backup-azure
oc wait --for=condition=available deployment/nginx -n test-backup-azure --timeout=60s

# Create a backup
velero backup create azure-test-backup --include-namespaces=test-backup-azure

# Wait and verify backup completed
velero backup describe azure-test-backup

# Delete the namespace
oc delete namespace test-backup-azure

# Restore from backup
velero restore create azure-test-restore --from-backup azure-test-backup

# Verify namespace was restored
oc get namespace test-backup-azure

Verify Azure Workload Identity is working:

# Check Velero pod has correct environment variables
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_CLIENT_ID")]}' | jq .
oc get pod $VELERO_POD -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_FEDERATED_TOKEN_FILE")]}' | jq .

# Check service account annotation
oc get sa velero -n openshift-adp -o jsonpath='{.metadata.annotations.azure\.workload\.identity/client-id}'

Test error handling with invalid storage account:

# Create CloudStorage with non-existent storage account
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-invalid-storage
 namespace: openshift-adp
spec:
 name: test-container
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey
 config:
   storageAccount: "nonexistentstorage123"
EOF

# Add deletion annotation and delete (should succeed with the fix)
oc annotate cloudstorage azure-invalid-storage -n openshift-adp oadp.openshift.io/cloudstorage-delete=true --overwrite
oc delete cloudstorage azure-invalid-storage -n openshift-adp --timeout=60s

Testing GCP CloudStorage with Workload Identity Federation

  1. Set up GCP identity for both service accounts:
# Use the updated helper function that now creates workload identity bindings for both service accounts
create-velero-identity-for-gcp-cluster <cluster-name>
  1. Create the CloudStorage resource with the required secret format:
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: gcp-test-storage
 namespace: openshift-adp
spec:
 name: <bucket-name>
 provider: gcp
 region: us-central1
 creationSecret:
   name: cloud-credentials-gcp
   key: service_account.json  # Required key for GCP credentials

Note: The creationSecret.key must be service_account.json for GCP provider.

  1. Verify the functionality:

Basic CloudStorage Creation:

# Apply the CloudStorage resource
oc apply -f cloudstorage-gcp.yaml

# Verify bucket exists in GCS
gsutil ls -p <project-id> | grep "gs://<bucket-name>/"

Test with DataProtectionApplication:

# Create DPA with GCP configuration
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa-gcp
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
       - gcp
       - openshift
       - csi
 backupLocations:
   - name: default
     velero:
       provider: gcp
       default: true
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       objectStorage:
         bucket: <bucket-name>
         prefix: velero
 snapshotLocations:
   - name: default
     velero:
       provider: gcp
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       config:
         project: <project-id>
         snapshotLocation: us-central1

Verify backup/restore works:

# Create test resources
oc create namespace test-backup-gcp
oc create deployment hello-openshift --image=openshift/hello-openshift:latest -n test-backup-gcp
oc wait --for=condition=available deployment/hello-openshift -n test-backup-gcp --timeout=60s

# Backup and restore
velero backup create gcp-test-backup --include-namespaces=test-backup-gcp
velero backup describe gcp-test-backup
gsutil ls -r gs://<bucket-name>/velero/backups/

# Delete and restore
oc delete namespace test-backup-gcp
velero restore create gcp-test-restore --from-backup gcp-test-backup
oc get all -n test-backup-gcp

Comprehensive Test Cases

Azure Test Cases

Test Case 1 - Basic CloudStorage Resource Creation

This test verifies that the CloudStorage controller can successfully create Azure storage containers using the new provider implementation.

# Set up environment variables
export STORAGE_ACCOUNT_NAME="<your-storage-account>"

# Create CloudStorage resource
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-test-storage
 namespace: openshift-adp
spec:
 name: velero
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey
 config:
   storageAccount: "${STORAGE_ACCOUNT_NAME}"
EOF

# Wait for resource creation
sleep 10

# Verify the CloudStorage resource was created
oc get cloudstorage azure-test-storage -n openshift-adp

# Check the status phase (should eventually show 'Available' or similar)
oc get cloudstorage azure-test-storage -n openshift-adp -o jsonpath='{.status.phase}'

# Verify the container exists in Azure
az storage container list --account-name ${STORAGE_ACCOUNT_NAME} --auth-mode login --query "[?name=='velero']"

Expected outcome: CloudStorage resource is created successfully, and the Azure storage container is provisioned.

Test Case 2 - DataProtectionApplication Integration

This test ensures that the CloudStorage provider works correctly with OADP's DataProtectionApplication, allowing Velero to use Azure storage for backups.

# Set up environment variables
export AZURE_RESOURCE_GROUP="<your-resource-group>"
export STORAGE_ACCOUNT_NAME="<your-storage-account>"
export CONTAINER_NAME="velero"
export CLUSTER_NAME="<your-cluster-name>"

# Create DPA with Azure configuration
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
     - azure
     - openshift
 backupLocations:
 - velero:
     provider: azure
     default: true
     credential:
       name: cloud-credentials-azure
       key: azurekey
     config:
       resourceGroup: ${AZURE_RESOURCE_GROUP}
       storageAccount: ${STORAGE_ACCOUNT_NAME}
     objectStorage:
       bucket: ${CONTAINER_NAME}
       prefix: ${CLUSTER_NAME}
EOF

# Wait for DPA to reconcile
oc wait --for=condition=Reconciled dpa/dpa -n openshift-adp --timeout=300s

# Verify Velero pod is running
oc get pods -n openshift-adp -l app.kubernetes.io/name=velero

# Check Velero can access the backup location
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc exec -n openshift-adp ${VELERO_POD} -- /velero backup-location get

Expected outcome: DPA is reconciled successfully, Velero pod is running, and backup location is accessible.

Test Case 3 - Backup and Restore Operations

This test validates end-to-end backup and restore functionality using Azure storage as the backend.

# Clean up any existing backups/restores
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc exec -n openshift-adp ${VELERO_POD} -- /velero backup delete --all --confirm 2>/dev/null || true
oc exec -n openshift-adp ${VELERO_POD} -- /velero restore delete --all --confirm 2>/dev/null || true

# Create test namespace and application
oc create namespace test-backup-azure || true
oc create deployment nginx --image=bitnami/nginx:latest -n test-backup-azure
oc expose deployment nginx --port=8080 --target-port=8080 -n test-backup-azure

# Wait for deployment to be ready
oc wait --for=condition=available deployment/nginx -n test-backup-azure --timeout=60s

# Create backup
velero backup create azure-test-backup --include-namespaces=test-backup-azure

# Wait for backup to complete (check status)
sleep 30
velero backup describe azure-test-backup

# Delete the test namespace to simulate data loss
oc delete namespace test-backup-azure

# Wait for namespace deletion
sleep 10

# Restore from backup
velero restore create azure-test-restore --from-backup azure-test-backup

# Wait for restore to complete
sleep 30

# Verify namespace and resources were restored
oc get namespace test-backup-azure
oc get all -n test-backup-azure

Expected outcome: Backup completes successfully, namespace is deleted and then restored with all resources intact.

Test Case 4 - Azure Workload Identity Authentication

This test verifies that the Azure Workload Identity authentication is properly configured for both the Velero pod and the controller manager.

# Get Velero pod name
POD_NAME=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')

# Check AZURE_CLIENT_ID environment variable
echo "Checking AZURE_CLIENT_ID..."
oc get pod ${POD_NAME} -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_CLIENT_ID")]}' | jq .

# Check AZURE_FEDERATED_TOKEN_FILE environment variable
echo "Checking AZURE_FEDERATED_TOKEN_FILE..."
oc get pod ${POD_NAME} -n openshift-adp -o jsonpath='{.spec.containers[0].env[?(@.name=="AZURE_FEDERATED_TOKEN_FILE")]}' | jq .

# Check service account annotations for Velero
echo "Velero service account annotations:"
oc get sa velero -n openshift-adp -o jsonpath='{.metadata.annotations}' | jq .

# Check service account annotations for controller manager
echo "Controller manager service account annotations:"
oc get sa openshift-adp-controller-manager -n openshift-adp -o jsonpath='{.metadata.annotations}' | jq .

# Verify federated token file is mounted
oc get pod ${POD_NAME} -n openshift-adp -o jsonpath='{.spec.volumes[?(@.name=="azure-identity-token")]}' | jq .

Expected outcome: Both service accounts have proper Azure Workload Identity annotations, and environment variables are correctly set.

Test Case 5 - Error Handling with Invalid Storage Account

This test verifies that the CloudStorage controller handles errors gracefully when given invalid configuration, and that resources can be deleted even when the underlying storage doesn't exist (idempotent deletion).

# Create CloudStorage with non-existent storage account
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-invalid-storage
 namespace: openshift-adp
spec:
 name: test-container
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey
 config:
   storageAccount: "nonexistentstorage123"
EOF

# Wait and check status
sleep 10
oc describe cloudstorage azure-invalid-storage -n openshift-adp

# Check controller logs for error handling
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "azure-invalid-storage" | tail -10

# Test deletion with annotation (should succeed with the fix)
echo "Testing deletion with invalid storage account..."
oc annotate cloudstorage azure-invalid-storage -n openshift-adp oadp.openshift.io/cloudstorage-delete=true --overwrite

# Delete the resource (should complete within timeout)
if oc delete cloudstorage azure-invalid-storage -n openshift-adp --timeout=60s; then
   echo "✓ CloudStorage deleted successfully (idempotent behavior working)"
else
   echo "✗ CloudStorage deletion failed"
   # Force cleanup if needed
   oc patch cloudstorage azure-invalid-storage -n openshift-adp --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]' 2>/dev/null || true
fi

# Check events for deletion process
oc get events -n openshift-adp --field-selector involvedObject.name=azure-invalid-storage --sort-by='.lastTimestamp' | tail -5

Expected outcome: CloudStorage resource shows error status but can be deleted successfully without hanging.

Test Case 6 - Container Lifecycle Management

This test verifies that containers can be created with unique names and properly cleaned up when CloudStorage resources are deleted.

# Generate unique container name
CONTAINER_NAME="velero-test-$(date +%Y%m%d%H%M%S)"
echo "Creating container: ${CONTAINER_NAME}"

# Create CloudStorage with unique container name
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: azure-new-container
 namespace: openshift-adp
spec:
 name: ${CONTAINER_NAME}
 provider: azure
 creationSecret:
   name: cloud-credentials-azure
   key: azurekey
 config:
   storageAccount: "${STORAGE_ACCOUNT_NAME}"
EOF

# Wait for container creation
sleep 15

# Verify container was created in Azure
echo "Checking if container ${CONTAINER_NAME} was created..."
az storage container list --account-name ${STORAGE_ACCOUNT_NAME} --auth-mode login --query "[?name=='${CONTAINER_NAME}']" -o table

# Test container deletion
oc annotate cloudstorage azure-new-container -n openshift-adp oadp.openshift.io/cloudstorage-delete=true --overwrite
oc delete cloudstorage azure-new-container -n openshift-adp --timeout=60s

# Verify container was deleted from Azure
sleep 10
echo "Checking if container ${CONTAINER_NAME} was deleted..."
az storage container list --account-name ${STORAGE_ACCOUNT_NAME} --auth-mode login --query "[?name=='${CONTAINER_NAME}']" -o table

Expected outcome: Container is created in Azure when CloudStorage is created and deleted when CloudStorage is deleted.

Test Case 7 - Role Assignment Validation

This test ensures that the Azure managed identity has all the required roles for OADP operations.

# Set up environment variables
export AZURE_RESOURCE_GROUP="<your-resource-group>"
export IDENTITY_NAME="<your-managed-identity-name>"

# Get managed identity principal ID
PRINCIPAL_ID=$(az identity show -g "${AZURE_RESOURCE_GROUP}" -n "${IDENTITY_NAME}" --query principalId -o tsv)
echo "Principal ID: ${PRINCIPAL_ID}"

# List all role assignments for the identity
echo "Role assignments for the managed identity:"
az role assignment list --assignee ${PRINCIPAL_ID} --all --query "[].roleDefinitionName" -o table

# Check for specific required roles
ROLES=$(az role assignment list --assignee ${PRINCIPAL_ID} --all --query "[].roleDefinitionName" -o tsv)

echo "Checking for required roles..."
for role in "Contributor" "Storage Blob Data Contributor" "Disk Snapshot Contributor"; do
   if echo "${ROLES}" | grep -q "${role}"; then
       echo "✓ ${role} is assigned"
   else
       echo "✗ ${role} is NOT assigned"
   fi
done

# Check scope of role assignments
echo "Role assignment scopes:"
az role assignment list --assignee ${PRINCIPAL_ID} --all --query "[].{Role:roleDefinitionName, Scope:scope}" -o table

Expected outcome: All three required roles are assigned to the managed identity at the appropriate scope.

GCP Test Cases

Test Case 1 - Basic CloudStorage Resource Creation

This test verifies that the CloudStorage controller can successfully create GCS buckets using the new provider implementation.

# Set up environment variables
export GCP_PROJECT_ID="<your-project-id>"
export BUCKET_NAME="<your-bucket-name>"

# Create CloudStorage resource
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: gcp-test-storage
 namespace: openshift-adp
spec:
 name: ${BUCKET_NAME}
 provider: gcp
 region: us-central1
 creationSecret:
   name: cloud-credentials-gcp
   key: service_account.json
EOF

# Wait for resource creation
sleep 5
oc get cloudstorage gcp-test-storage -n openshift-adp
oc describe cloudstorage gcp-test-storage -n openshift-adp

# Wait for bucket creation in GCS (retry up to 30 times)
echo "Waiting for bucket creation in GCS..."
for i in {1..30}; do
   if gsutil ls -p ${GCP_PROJECT_ID} | grep -q "gs://${BUCKET_NAME}/"; then
       echo "✅ Bucket ${BUCKET_NAME} exists in GCS"
       break
   else
       echo "  Waiting for bucket creation... ($i/30)"
       sleep 10
   fi
done

# Check CloudStorage status if bucket wasn't created
if [ $i -eq 30 ]; then
   echo "❌ Timeout waiting for bucket creation"
   oc get cloudstorage gcp-test-storage -n openshift-adp -o jsonpath='{.status}' | jq .
fi

Expected outcome: CloudStorage resource is created and GCS bucket is provisioned successfully.

Test Case 2 - DataProtectionApplication Integration

This test ensures that the CloudStorage provider works correctly with OADP's DataProtectionApplication for GCP.

# Set up environment variables
export GCP_PROJECT_ID="<your-project-id>"
export BUCKET_NAME="<your-bucket-name>"

# Create DPA with GCP configuration
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
 name: dpa-gcp
 namespace: openshift-adp
spec:
 configuration:
   velero:
     defaultPlugins:
       - gcp
       - openshift
       - csi
 backupLocations:
   - name: default
     velero:
       provider: gcp
       default: true
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       objectStorage:
         bucket: ${BUCKET_NAME}
         prefix: velero
 snapshotLocations:
   - name: default
     velero:
       provider: gcp
       credential:
         name: cloud-credentials-gcp
         key: service_account.json
       config:
         project: ${GCP_PROJECT_ID}
         snapshotLocation: us-central1
EOF

# Wait for DPA to be ready
oc wait --for=condition=Reconciled dpa/dpa-gcp -n openshift-adp --timeout=300s

# Verify Velero deployment
oc get pods -n openshift-adp -l app.kubernetes.io/name=velero

# Wait for Velero pod to be ready
oc wait --for=condition=ready pod -l app.kubernetes.io/name=velero -n openshift-adp --timeout=300s

# Check backup storage location status
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')
oc exec -n openshift-adp $VELERO_POD -- /velero backup-location get

Expected outcome: DPA is reconciled, Velero pod is running, and backup location is accessible.

Test Case 3 - Backup and Restore Operations

This test validates end-to-end backup and restore functionality using GCS as the backend.

# Clean up existing backups/restores
VELERO_POD=$(oc get pods -n openshift-adp -l app.kubernetes.io/name=velero -o jsonpath='{.items[0].metadata.name}')

# Delete all existing backups
oc exec -n openshift-adp $VELERO_POD -- /velero backup delete --all --confirm || true

# Wait for backup CRs to be deleted
while [ $(oc get backups.velero.io -n openshift-adp --no-headers 2>/dev/null | wc -l) -gt 0 ]; do
   echo "Waiting for backup CRs to be deleted..."
   sleep 5
done

# Delete all existing restores
oc exec -n openshift-adp $VELERO_POD -- /velero restore delete --all --confirm || true

# Wait for restore CRs to be deleted
while [ $(oc get restores.velero.io -n openshift-adp --no-headers 2>/dev/null | wc -l) -gt 0 ]; do
   echo "Waiting for restore CRs to be deleted..."
   sleep 5
done

# Create test namespace with deployment
oc create namespace test-backup-gcp || echo "Namespace might already exist"

cat << EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
 name: hello-openshift
 namespace: test-backup-gcp
spec:
 replicas: 2
 selector:
   matchLabels:
     app: hello-openshift
 template:
   metadata:
     labels:
       app: hello-openshift
   spec:
     containers:
     - name: hello-openshift
       image: openshift/hello-openshift:latest
       ports:
       - containerPort: 8080
         protocol: TCP
       securityContext:
         allowPrivilegeEscalation: false
         capabilities:
           drop:
           - ALL
         runAsNonRoot: true
         seccompProfile:
           type: RuntimeDefault
---
apiVersion: v1
kind: Service
metadata:
 name: hello-openshift
 namespace: test-backup-gcp
spec:
 selector:
   app: hello-openshift
 ports:
 - port: 8080
   targetPort: 8080
   protocol: TCP
EOF

# Wait for deployment
oc wait --for=condition=available deployment/hello-openshift -n test-backup-gcp --timeout=60s

# Create backup
velero backup create gcp-test-backup --include-namespaces=test-backup-gcp

# Monitor backup progress
sleep 10
velero backup describe gcp-test-backup --details

# Verify backup in GCS bucket
gsutil ls -r gs://${BUCKET_NAME}/velero/backups/

# Delete test namespace
oc delete namespace test-backup-gcp
sleep 20

# Restore from backup
velero restore create gcp-test-restore --from-backup gcp-test-backup
sleep 15

# Verify restoration
oc get all -n test-backup-gcp

Expected outcome: Backup is stored in GCS, namespace is deleted and restored successfully with all resources.

Test Case 4 - Error Handling with Invalid Credentials

This test verifies that the CloudStorage controller handles authentication errors gracefully.

# Create CloudStorage with non-existent secret
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: CloudStorage
metadata:
 name: gcp-invalid-creds
 namespace: openshift-adp
spec:
 name: test-invalid-bucket
 provider: gcp
 region: us-central1
 creationSecret:
   name: non-existent-secret
   key: service_account.json
EOF

# Check CloudStorage status
sleep 5
oc describe cloudstorage gcp-invalid-creds -n openshift-adp

# Check controller logs for error messages
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "gcp-invalid-creds" | tail -10

# Clean up
oc delete cloudstorage gcp-invalid-creds -n openshift-adp

Expected outcome: CloudStorage shows error status indicating the secret doesn't exist.

Validation Steps

After running tests, validate the implementation:

# Check operator logs for errors
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "error" | tail -20

# Check CloudStorage controller specific logs
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "cloudstorage" | tail -20

# For Azure - verify federated credentials exist for both service accounts
az identity federated-credential list \
 --identity-name <identity-name> \
 --resource-group <resource-group> \
 --query "[].subject"

# For GCP - verify workload identity bindings
gcloud iam service-accounts get-iam-policy <service-account-email> \
 --format=json | jq '.bindings[].members[]' | grep "openshift-adp"

# Check CloudStorage status for all resources
oc get cloudstorage -n openshift-adp

# Verify no authentication errors in logs
oc logs -n openshift-adp deployment/oadp-operator-controller-manager | grep -i "unable to determine if bucket exists" || echo "No auth errors found"

Key Changes

  1. Azure: Added federated credential creation for openshift-adp-controller-manager service account
  2. GCP: Added workload identity bindings for openshift-adp-controller-manager service account
  3. Code: Made SharedCredentialsFileFromSecret function generic to support both AWS and GCP
  4. Documentation: Added comprehensive comments explaining Azure's credential handling approach

Required Secret Key Format by Provider

Provider Secret Key Example
AWS credentials AWS credentials file format
Azure azurekey Azure environment variables format
GCP service_account.json GCP service account JSON

This ensures that the CloudStorage controller can authenticate properly regardless of which service account it runs under.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Aug 19, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kaovilai, shubham-pampattiwar, sseago

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [kaovilai,shubham-pampattiwar,sseago]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kaovilai kaovilai added the lgtm Indicates that a PR is ready to be merged. label Aug 19, 2025
@openshift-merge-bot openshift-merge-bot bot merged commit f871f7a into openshift:oadp-dev Aug 19, 2025
10 of 11 checks passed
@openshift-cherrypick-robot
Copy link
Contributor

@kaovilai: new pull request created: #1918

In response to this:

/cherry-pick oadp-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Aug 19, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sseago sseago sseago approved these changes

@shubham-pampattiwar shubham-pampattiwar shubham-pampattiwar approved these changes

@mrnold mrnold Awaiting requested review from mrnold

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@kaovilai @openshift-ci-robot @openshift-cherrypick-robot @weshayutin @sseago @shubham-pampattiwar @openshift-merge-robot