removing secondary storage: Cannot delete image store with active templates backup

[sbrueseke]

ACS: 4.19.1.2
I want to delete a secondary storage and get the following error: Cannot delete image store with active templates backup!

There are no Volume Snapshots on this secondary storage and it has been switched to read-only.
I see that there are 2 systemvm templates still mapped to this secondary storage, but those templates are also mapped to the other secondary storage in the same zone. I am unable to delete those templates from just the secondary storage I want to delete. Same with ISOs.

In the database I see that the secondary storage I want to delete has id 1 in image_store table.
In template_store_ref there are several templates in state Ready mapped to store_id 1. The amount is identically with the number of Templates and ISOs still on this storage.
In volume_store_ref is no volume mapped to store_id 1 with state not Destroyed, so this should be okay.
Same with snapshot_store_ref.

Is this a known issue with Cloudstack when deleting a secondary storage? I found some old mails in the mailinglist, but it looked like those issues were solved in versions before 4.19.1.2.

As far as I understand, it should be possible to delete a secondary storage without editing the database. Can someone confirm this?

[DaanHoogland]

@sbrueseke at the moment the following checks are done:

        // Verify that there are no live snapshot, template, volume on the image
        // store to be deleted
        List<SnapshotDataStoreVO> snapshots = _snapshotStoreDao.listByStoreId(storeId, DataStoreRole.Image);
        if (snapshots != null && snapshots.size() > 0) {
            throw new InvalidParameterValueException("Cannot delete image store with active snapshots backup!");
        }
        List<VolumeDataStoreVO> volumes = _volumeStoreDao.listByStoreId(storeId);
        if (volumes != null && volumes.size() > 0) {
            throw new InvalidParameterValueException("Cannot delete image store with active volumes backup!");
        }

        // search if there are user templates stored on this image store, excluding system, builtin templates
        List<TemplateJoinVO> templates = _templateViewDao.listActiveTemplates(storeId);
        if (templates != null && templates.size() > 0) {
            throw new InvalidParameterValueException("Cannot delete image store with active templates backup!");
        }

so no, you can not delete an secstror if it contains templates (or ISOs) or Volume - or snapshot backups.

I think what you need is an API call (or extra parameter) to remove a item from a specific store.

I think at the moment a snapshot can only be deleted from a zone, not form a specific store: https://cloudstack.apache.org/api/apidocs-4.20/apis/deleteSnapshot.html

[sbrueseke]

Thx, Daan! Do we need to do a migrateSecondaryStorageData even if all Templates and ISOs are already on the new secondary storage?
Only to be sure: All volumes, spanpshots, ISOs and Templates are already on the new secondary storage. The only data left on the storage we want to delete are duplicates of Templates and ISOs, because those files are stored on each secondary storage.

What does the workflow looks like when you want to delete a secondary storage or is this just not working at the moment in ACS?

[DaanHoogland]

I am not sure if that will work, but it is worth a shot. It should (force) migrate without doing anything. (Warranty until I hit Reply on this message!!!)

Use a test env first. I see no condition for when the target already exists (should be there though) What might go wrong is that a migration is started overwriting a good image but failing. You should still have the original on the store to remove at that point.

[DaanHoogland]

@sbrueseke , are you still having to deal with this? (or can you give a summary of what works for you?)

[sbrueseke]

Looks like we fixed the issue. We were unable to reproduce this issue in our testlab. We were able to delete a secondary storage without any issues in out testlab. So I still do not know what was getting us into this state.
This is what we did to fix it:

• Check which templates are still connected to the secondary storage: select * from template_store_ref where store_id=1 and state="Ready"
• Check which templates are effected: select * from vm_template where id=<template_id>;
• Update db for all templates from above: UPDATE template_store_ref SET state = 'Destroyed', destroyed = 1 WHERE id = ;

After that we were able to delete the unneeded secondary storage.
@DaanHoogland thx for your help, pushing us in the right direction.