Removal/cleanup hooks not implemented

[nvinuesa]

While investigating the Juju issue juju/juju#21860 I found out that what was actually happening is that the postgresql (16/edge) charm was never releasing the storage (looks like one of the snaps is still running). This prevents the correct storage removal which itself prevents the machine removal which prevents the graceful model removal. This works of course if we run destroy-model --force ....
The storage of the model after running model destroy is the following:

  Storage Unit  Storage ID  Type        Status     Message                                                
                data/10     filesystem  detaching  umount failed: umount:                                 
  /var/snap/charmed-postgresql/common/var/lib/postgresql: target is busy.: exit status 32                 
...
                logs/3      filesystem  detaching  umount failed: umount:                               
  /var/snap/charmed-postgresql/common/data/logs: target is busy.: exit status 32                          

After discussing with @marceloneppel we came to the conclusion that the removal-related hooks handlers are not implemented yet in the postgresql charm (namely stop or storage-detaching).

Note: One important implementation detail is that the storage detaching and removal in Juju happens in a concurrent worker with retries. This means that even if the snaps are definitely stopped during the stop hook (which will run after storage-detaching) it will not block the eventual storage removal.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-9685.

This message was autogenerated

[skatsaounis]

I think I hit something similar with Juju 3.6. The units were removed from the machine, the machine went in STOPPED state in LXD, but the storage was left back.

I had to remove the storage manually with the Juju CLI and try lxd delete the machine. However, I was getting errors:

juju remove-unit postgresql/0
...
juju storage
Unit          Storage ID  Type        Pool    Size     Status    Message
              archive/0   filesystem  rootfs  18 GiB   attached  
              data/1      filesystem  rootfs  18 GiB   attached  
              logs/2      filesystem  rootfs  18 GiB   attached  
              temp/3      filesystem  rootfs  18 GiB   attached

juju remove-storage archive/0
juju remove-storage data/1
juju remove-storage logs/2
juju remove-storage temp/3

lxc delete et3-microcloud:juju-20162c-2
Error: Failed deleting instance "juju-20162c-2" in project "prod-maas-labs": Error deleting storage volume: Failed to run: zfs destroy -r local/virtual-machines/prod-maas-labs_juju-20162c-2.block: exit status 1 (cannot destroy 'local/virtual-machines/prod-maas-labs_juju-20162c-2.block': dataset is busy)

I had to ask my MicroCloud administrator to stop the lingering qemu process on the LXD host and delete the machine. Then it got removed from the model

[wallyworld]

RE: #1550 (comment)

This remove unit leaving storage is expected behavior. The default behaviour for remove application and remove unit is to detach the storage and leave it in the model so it can be re-attached to a different unit if needed. ie we don't delete people's data.

There's a --destroy-storage option the user can supply if the storage is to be removed.

Note: the above only applies to model scoped storage (eg ebs volume). For machine scoped storage (eg rootfs), that always gets removed as it can't be detached from the machine.