Volume Stuck in Snapshotting

[boubouX]

Hello,

I have a volume stuck in ”Snapshotting” for several days.

This volume also has a daily snapshot schedule, which I would like to delete, but the WebUI does not offer me that option, likely due to the “Snapshotting” status.

I have removed all my other scheduled volume snapshots; now, I see only one entry in the snapshot_schedule and snapshot_policy tables.

Can I delete this record from both tables?

I also found the entry in the snapshot_view table.

This is a 24/7 VM with heavy usage, and I cannot shut it down for another two days.

Does anyone have any suggestions for a resolution?

Thank you,
Antoine

[prashanthr2]

@boubouX I wouldn't make changes to DB unless it's absolutely necessary and there is no other way.

If volume is stuck in snapshotting, I think you should try to fix this first and do a clean delete of the snapshot schedule from WebUI. Do you know if why this happened and everything looks good with the volume from hypervisor end? Also, what is the hypervisor here?

[boubouX]

Hello prashanthr2,

Thank you for your response.

My best guess is that the issue occurred because the account or domain temporarily ran out of secondary storage capacity quota, which has since been resolved.

Here is the current status:

1. The VM is working as expected.
2. A snapshot of its volume is in the Creating state, with a DAILY interval type, created on October 19.
3. The VDI is 250 GB, stored locally on the KVM host in QCOW2 format. It is not shown as snapshotted when checked via qemu-img info.
4. A manual reverse database lookup from the VDI path returns the unexpected VM name “ABTWDCAPP01” instead of the expected “TS02”. “ABTWDCAPP01” does not appear in the web UI.
5. In the web UI, the volume state is “Snapshotting.” It shows no metrics collected, which is unusual, and a SNAPSHOT.CREATED event is logged every 5 minutes.
6. Every 5 minutes, the Management Server logs the following error:

2025-10-28 06:35:24,718 ERROR [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-7fdc17a1) (logid:d13be60e) The scheduling of snapshot [6763e09b-dd3b-4ff4-b1fd-1046d0dbb681] for volume [147] failed due to [com.cloud.exception.InvalidParameterValueException: VolumeId: 147 is not in Ready state but Snapshotting. Cannot take snapshot.].

7. The environment is running Apache CloudStack 4.19.3, and the Management Server was last rebooted about a day ago.
8. I can shut down the VM in about 16 hours for further investigation.

Do you have any suggestions for the next steps?

Best regards

[prashanthr2]

Hello @boubouX Thanks for details on the issue.

Since you can stop the VM in next few hours I would recommend the following

1. Stop the VM and check if the state of the Volume changes to Ready (TBH I doubt if this happens)
2. If the volume remains in Snapshotting state. Update the DB to reflect the state of the affected VM to Ready. Once done you should be able to cancel the snapshot schedule from Web interface.

Note: Please take the backup of your DB before making any changes.

[boubouX]

Hello @prashanthr2, thanks again for your reply,

I ended up having to change the volume state to Ready by manual DB editing, and I was then able to delete the snapshot schedule.

Unfortunately, I was not able to migrate the VM, but this may be another blocking issue. #11933
I will report back once I resolve it, but I still. suspect issues.

-Regards

[boubouX]

The volume appears as Ready after I manually changed its status, but migration is still blocked because a snapshot has been “Creating” for over 10 days. I was hoping it would eventually time out, but that doesn’t seem to be happening.

I’m considering making a copy of the VDI on storage, backing up the database, and then updating the snapshot table entry to set the state to "Destroyed" and the removed field with the current timestamp.

Does anyone have a safer or better approach to handle this situation?

[prashanthr2]

@boubouX Unfortunately the only option left is updating the DB as you suggested.

Please make sure you also take care of entries in snapshot_store_ref and snapshot_zone_ref for the snapshot with the issue.

Also, if you are using cloudstack-usage , you may need to take care of usage event for the snapshot with the issue here. Ignore below if you are not concerned or using usage service.

Check if the usage event is created for the snapshot in the cloud.usage_events. If you see snapshot usage events for the snapshot, you can mark the snapshot status to 'Backedup' in the snapshot table and delete it from UI. This will create a snapshot delete usage event.

[boubouX]

Thank you, @prashanthr2. Unfortunately, there is no snapshot entry in the event_usage table. I'm not too concerned with it, if the events eventually fall off.

I have identified the specific record in snapshot and snapshot_store_ref, but I cannot find any related record in snapshot_zone_ref.

Does the modification sequence matter? snapshot, then snapshot_store_ref?

Regards

[prashanthr2]

@boubouX modification sequence shouldn't matter ideally. But just to be safe. update snapshot_store_ref first and then snapshot (this is the order in which the tables are updated when we remove the snapshot through GUI or API).

[daviftorres]

Hi everyone, can we confirm that this issue is related to:

• Timeout snapshot #11923
• Timeout snapshot #11893

It also appears to share the same root cause and symptoms as the NAS backup issue I reported earlier:

• NAS Backup - Raise Event for Succeeded / Failed / Timed Out #11727

In summary, long-running jobs such as snapshots or backups (both copying data to NFS storage) eventually time out, and the job state never progresses (Snapshotting and BackingUp respectively).

In my testing, the backup operations themselves do not fail, CloudStack simply times out on checking for completion. I was able to manually restore from the copied .qcow2 files, and everything was intact (including a 7 TB volume that was ~5 TB utilized).

For context, during the tests we tried to increase the timeout settings:

• backup.snapshot.wait from the default 21600 to 216000 (from 1 hour to 10 hours)
  • No changes.
• commands.timeout from the default empty to CreateObjectCommand=14400 (4 hours)
  • Only the frontend stopped timing out.