MB-68168: Handle exdev rename error when retiring keys

It may happen that data dir and retired keys dir are located on
different filesystems. In this case, the rename operation will
fail with an exdev error, so we should fall back to copy+delete
the file instead of renaming it.

Key files are small so the performance impact of this fallback
should be negligible.

Change-Id: I26a71c4aaa4e65d9a098b167276835708a4bce31
Reviewed-on: https://review.couchbase.org/c/ns_server/+/232659
Well-Formed: Build Bot <[email protected]>
Tested-by: Build Bot <[email protected]>
Tested-by: Timofey Barmin <[email protected]>
Reviewed-by: Navdeep S Boparai <[email protected]>

Author: timofey-barmin

apps/ns_server/src/encryption_service.erl

@@ -978,6 +978,28 @@ retire_key(Kind, Filename) ->
         ok ->
             case misc:atomic_rename(FromPath, ToPath) of
                 ok -> ok;
+                {error, exdev} ->
+                    %% Cross-filesystem rename, fall back to copy + delete
+                    ?log_debug("Cross-filesystem rename failed, "
+                               "will try copy+delete ~p to ~p",
+                               [FromPath, ToPath]),
+                    case file:copy(FromPath, ToPath) of
+                        {ok, _} -> ok;
+                        {error, CopyReason} ->
+                            %% Copy failed, but it is not critical enough to
+                            %% fail the whole operation, as the reason we keep
+                            %% the keys in retired_keys_dir is "just in case".
+                            %% The system can work without it.
+                            ?log_error("Copy failed ~p to ~p: ~p",
+                                       [FromPath, ToPath, CopyReason])
+                    end,
+                    case file:delete(FromPath) of
+                        ok -> ok;
+                        {error, DeleteReason} ->
+                            ?log_error("Failed to delete file ~p: ~p",
+                                       [FromPath, DeleteReason]),
+                            {error, DeleteReason}
+                    end;
                 {error, Reason} ->
                     ?log_error("Failed to retire ~p key ~p (~p): ~p",
                                 [Kind, Filename, FromPath, Reason]),