rm: cannot remove '/home/forge/site.com/releases/78': Directory not empty

[jasperf]

Get an error on rollback being told directory is not empty:

dep rollback production
➤ Executing task rollback
In Client.php line 103:
  The command "cd /home/forge/site.com && (rm -rf /home/forge/site.com/releases/78)" failed.  
  Exit Code: 1 (General error)                                                                      
  Host Name: production                                                                             
  ================                                                                                  
  rm: cannot remove '/home/forge/site.com/releases/78': Directory not empty                      
rollback [-p|--parallel] [-l|--limit LIMIT] [--no-hooks] [--log LOG] [--roles ROLES] [--hosts HOSTS] [-o|--option OPTION] [--] [<stage>]

Does the rollback did work. Release 78 is not symlinked to anymore:

forge@smt-prod-w-2:~/smart48.com$ ll
total 20
drwxrwxr-x  5 forge forge 4096 Oct  6 06:45 ./
drwxr-xr-x 10 forge forge 4096 Sep 20 09:51 ../
drwxrwxr-x  2 forge forge 4096 Oct  6 06:04 .dep/
lrwxrwxrwx  1 forge forge   11 Oct  6 06:45 current -> releases/77/
drwxrwxr-x 12 forge forge 4096 Oct  6 06:04 releases/
drwxrwxr-x  4 forge forge 4096 Oct  6 06:02 shared/

But really odd so need to get to the bottom of this. How is this possible?

I do see that 78 still has content, I think our shared storage directory:

forge@smt-prod-w-2:~/domain.com/releases$ cd 78/
forge@smt-prod-w-2:~/domain.com.com/releases/78$ ll
total 12
drwxrwxr-x  3 forge forge 4096 Oct  6 06:45 ./
drwxrwxr-x 12 forge forge 4096 Oct  6 06:04 ../
drwxr-xr-x  4 forge forge 4096 Oct  6 06:45 storage/
forge@smt-prod-w-2:~/domain.com.com/releases/78$ cd storage/
forge@smt-prod-w-2:~/domain.com domain.com com/releases/78/storage$ ll
total 16
drwxr-xr-x 4 forge forge 4096 Oct  6 06:45 ./
drwxrwxr-x 3 forge forge 4096 Oct  6 06:45 ../
drwxr-xr-x 3 forge forge 4096 Oct  6 06:45 framework/
drwxr-xr-x 2 forge forge 4096 Oct  6 06:45 logs/

and permissions seem fine

forge@smt-prod-w-2:~/domain.com/releases/78/storage$ stat -c "%a %n" *
755 framework
755 logs
forge@smt-prod-w-2:~/domain.com.com/releases/78/storage$ cd ..
forge@smt-prod-w-2:~/domain.com.com/releases/78$ stat -c "%a %n" *
755 storage

and all directories and files are owned by forge:forge which is the user being used.

Wonder if it is an issue with the cache files...

forge@smt-prod-w-2:~/domain.com/releases/78/storage/framework/views$ ll
total 24
drwxr-xr-x 2 forge forge 4096 Oct  6 06:45 ./
drwxr-xr-x 3 forge forge 4096 Oct  6 06:45 ../
-rw-r--r-- 1 forge forge  439 Oct  6 06:45 48baec34afd6d744c03286e6970be7890ba04554.php
-rw-r--r-- 1 forge forge 6872 Oct  6 06:45 80a34a0d38971cc7918214d22820f032dceb2878.php
-rw-r--r-- 1 forge forge  445 Oct  6 06:45 fa19840f5e7c3cb37835e052ddbef9434dc6afc4.php

These are generated on deployment by Laravel and that was 30 minutes ago here..at the time of the rollback perhaps.. that would be odd also however.

[jasperf]

On new deployment I was successful again but was told that release 78 was not a local repository and skipped:

ep deploy production
✈︎ Deploying master on production
✔ Executing task deploy:prepare
✔ Executing task deploy:lock
✔ Executing task deploy:release
➤ Executing task deploy:update_code
Cloning into '/home/forge/smart48.com/releases/79'...
fatal: reference repository '/home/forge/domain.com/releases/78' is not a local repository.
Connection to smart48.com closed.
Cloning into '/home/forge/domain.com.com/releases/79'...
remote: Enumerating objects: 57199, done.
remote: Counting objects: 100% (8324/8324), done.
remote: Compressing objects: 100% (2803/2803), done.
remote: Total 57199 (delta 5931), reused 7640 (delta 5396), pack-reused 48875
Receiving objects: 100% (57199/57199), 448.79 MiB | 17.94 MiB/s, done.
Resolving deltas: 100% (38339/38339), done.
Updating files: 100% (7303/7303), done.
Connection to domain.com closed.

Any ideas who to deal with this?

[jasperf]

Actually storage/framework/views should be completely shared so storage/framework/views cache and storage/logs should be there. So why are they in the project root folder?

forge@smt-prod-w-2:~/domain.com/releases/79/storage$ ll
total 28
drwxrwxr-x  7 forge forge 4096 Oct  6 07:26 ./
drwxrwxr-x 16 forge forge 4096 Oct  6 07:26 ../
drwxrwxr-x  3 forge forge 4096 Oct  6 07:27 app/
drwxrwxr-x  2 forge forge 4096 Oct  6 07:26 debugbar/
drwxrwxr-x  2 forge forge 4096 Oct  6 07:26 fonts/
drwxrwxr-x  5 forge forge 4096 Oct  6 07:26 framework/
drwxrwxr-x  3 forge forge 4096 Oct  6 07:26 invoices/
lrwxrwxrwx  1 forge forge   28 Oct  6 07:26 logs -> ../../../shared/storage/logs/
lrwxrwxrwx  1 forge forge   27 Oct  6 07:26 tls -> ../../../shared/storage/tls/

[jasperf]

perhaps logs cause laravel.log is still generated inside local storage as well as view cache.. no complaints about fonts, app, debugbar

[antonmedv]

From https://unix.stackexchange.com/questions/12616/whats-the-reason-that-rm-rf-a-b-says-a-b-is-not-empty

Very possibly. It might be that rm -rf first removes all files, then removes all directories. Under the hood, the rmdir system call for removing a directory will fail if the directory is not empty. You might have a race condition going on, where rm -rf checks that the directory is empty, and it is; another process then creates a new file; finally rm -rf calls rmdir, but another process creates a file beforehand.

We can try to remove symlinks to shared dir before performing deletion of release.

Can you test if it will work? Thanks.

PS Also in v7 there is no deletion of broken release at rollback task.

[jasperf]

The removal of a symlink to shared director before deleting a release would perhaps be a solution to avoid the race condition. But how can I do this in the current Deployer 6 deploy.php setup @antonmedv ? What setting is there for this order change?

[antonmedv]

There is no such feature.

[jasperf]

That is too bad. Well, moving more to storage/shared to avoid the race issue. Added a question in another card on that. Thanks for the feedback!

[jasperf]

Well, as stated deployment did work and we are working on moving all in storage to shared. So will close this thread as deployment did work and an improvement is underway.