ws_restore Weka quota problems

[planetmija]

ws_restore does not work with Weka directory quotas. The following error occurs:
rename("/work/...testexpired-1756138302", "/work/...testexpired-1756138302") = -1 EXDEV (Invalid cross-device link)

After reversing this change it works for us using SUID:

hpc-workspace/src/ws.cpp

Line 953 in 97dcd0e

// int ret = mv(wssourcename.c_str(), targetwsdir.c_str()); // does not work with capabilities

[holgerBerger]

ok that comes unexpected, I did not expect that an FS with that limitation would show up again.
DNE1 in lustre and NEC scatefs had similar issues.

Moving files (or directories) between two directories with quotas, into a directory with a quota, or outside a directory with a quota is not supported. The WekaFS filesystem returns EXDEV in such a case, which is usually converted by the operating system to copy and delete but is OS-dependent.

is documented in https://docs.weka.io/3.14/fs/quota-management

can you explain a bit which directory has the quota here?
Do you set quotas to workspaces after creation?

[planetmija]

Each workspace is automatically assigned a default quota. Therefore, restoring always involves moving to or from a directory with a quota. I can see that this could cause problems as well. All workspaces get a default quota of 5 TB. However, we can sometimes allow higher quotas. Therefore, moving data from a directory with more than 5 TB to a directory with a quota of 5 TB would fail at some point.

[holgerBerger]

I see, that sounds reasonable... and unavoidable
bad from the WEKA side, so we probably need

• a fallback when rename fails
• that fallback should be somehow fail prove, that mv could take some time, and it could get interrupted by the user
  because that workaround was bit of a mess it was removed in the first place, i was glad to get rid of it.

[holgerBerger]

i would expect the ws_expirer to fail as well, does this not happen?

[planetmija]

ws_expirer works. This moves the data to a path that is not restricted by a quota. ws_restore fails as it probably involves moving an expired workspace with a quota to a new workspace with a quota.

[holgerBerger]

this is surprising as the WEKA manual says into or out of quota-enforced directories

https://docs.weka.io/weka-filesystems-and-object-stores/quota-management

File movement:

The rename() operation, when implemented by link() and unlink(), behaves like an atomic file move across filesystems.

Moving files into or out of quota-enforced directories triggers EXDEV (cross-device link error).

[planetmija]

When a directory expires, we move the quota-enforced directory to the .removed directory, which has no quota. We have specified that every newly created directory (workspace) has a quota. However, the main directory in which the workspaces and the .removed directory are located does not have a quota. Therefore, it is only when moving to a new workspace that problems arise. One solution would be to restore the workspace to its previous location. However, you would then need to ensure that the user has not already created a workspace with the same name.

[holgerBerger]

ah... yes.. now i get it.

[holgerBerger]

should be fixed in v1 and v2