Investigate why some Ceph object lock tests can't delete bucket at the end #529
Description
It appears some Ceph object lock tests fail to clean up the test bucket, because we return a "bucket not empty" error:
- https://build.dev.storj.tools/job/gateway-st-gerrit/2651/testReport/junit/s3tests_boto3.functional/test_s3/Integration___test_object_lock_changing_mode_from_compliance/
- https://build.dev.storj.tools/job/gateway-st-gerrit/2651/testReport/junit/s3tests_boto3.functional/test_s3/Integration___test_object_lock_changing_mode_from_governance_with_bypass/
- https://build.dev.storj.tools/job/gateway-st-gerrit/2651/testReport/junit/s3tests_boto3.functional/test_s3/Integration___test_object_lock_changing_mode_from_governance_without_bypass/
The three failing tests are "changing mode" and at the end don't expicitly call DeleteObject to clean up the object, where other tests are doing this and passing correctly. Instead, these changing mode tests seem to be relying on the test runner to try and clean things up before it deletes the bucket. See nuke_bucket function here: https://github.com/ceph/s3-tests/blob/d245097771698774d36a0283e9ef69bf67f3ba76/s3tests_boto3/functional/__init__.py#L98.
We should investigate if there's something wrong here, like it couldn't delete some of the leftover object versions. It seems like it's supposed to wait until the retention period has expired before cleaning things up, but it wasn't clear by the trace in the test if it actually waited or not.