Skip to content

Commit e11fe82

Browse files
authored
Fix bug with new minio behavior (#4725)
New behavior doesn't delete masked objects. So check after the first deletion if we need to remove anything that was uncovered. --- TYPE: BUG DESC: Fix bug with new minio behavior
1 parent b4de1c3 commit e11fe82

File tree

1 file changed

+27
-0
lines changed
  • tiledb/sm/filesystem

1 file changed

+27
-0
lines changed

tiledb/sm/filesystem/s3.cc

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -443,6 +443,33 @@ void S3::remove_dir(const URI& uri) const {
443443

444444
std::vector<std::string> paths;
445445
throw_if_not_ok(ls(uri, &paths, ""));
446+
447+
// Bail early if we don't have anything to delete.
448+
if (paths.empty()) {
449+
return;
450+
}
451+
452+
throw_if_not_ok(
453+
parallel_for(vfs_thread_pool_, 0, paths.size(), [&](size_t i) {
454+
throw_if_not_ok(remove_object(URI(paths[i])));
455+
return Status::Ok();
456+
}));
457+
458+
// Minio changed their delete behavior when an object masks another object
459+
// with the same prefix. Previously, minio would delete any object with
460+
// a matching prefix. The new behavior is to only delete the object masking
461+
// the "directory" of objects below. To handle this we just run a second
462+
// ls to see if we still have paths to remove, and remove them if so.
463+
464+
paths.clear();
465+
throw_if_not_ok(ls(uri, &paths, ""));
466+
467+
// We got everything on the first pass.
468+
if (paths.empty()) {
469+
return;
470+
}
471+
472+
// Delete the uncovered object prefixes.
446473
throw_if_not_ok(
447474
parallel_for(vfs_thread_pool_, 0, paths.size(), [&](size_t i) {
448475
throw_if_not_ok(remove_object(URI(paths[i])));

0 commit comments

Comments
 (0)