Empty, closed container with no replicas or datanodes associated is never deleted by SCM

[silvanias]

Hey all, in my Ozone cluster I have noticed that I accrued a large amount of empty containers that only exist on the SCM, and are never cleaned up. See example output from one of these containers:

Container id: 5
Pipeline id: 5b837d3e-a2df-4eb1-a987-2f56b698f563
Write PipelineId: b68448ac-e3c6-40c8-ab04-525c6a6d7497
Write Pipeline State: CLOSED
Container State: CLOSED
Datanodes: []
Replicas: []

These containers stack up over time and take up space (currently 70gb on this cluster). Looking at the source code for the container deletion logic:
/workspaces/ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/health/EmptyContainerHandler.java

Containers are only considered Empty and Closed if there are not any replicas associated with it. Thus the example container will never be deleted with the current logic. What is the reasoning behind this decision? Is there anyway to circumvent it with a cli command?

[silvanias]

Would it be possible to get a response to this? I am happy to make a PR to change the logic, and add a conditional for this specific case. But I would like to understand the original purpose of the logic check.

[ivandika3]

cc: @sodonnel @ashishkumar50

[ivandika3]

But I would like to understand the original purpose of the logic check.

I think EmptyContainerCheckHandler already contains the purpose

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/health/EmptyContainerHandler.java

Lines 80 to 85 in 9a95424

	// If the container is empty and has no replicas, it is possible it was
	// a container which stuck in the closing state which never got any
	// replicas created on the datanodes. In this case, we don't have enough
	// information to delete the container, so we just log it as EMPTY,
	// leaving it as CLOSED and return true, otherwise, it will end up marked
	// as missing in the replication check handlers.

[silvanias]

Thanks for pointing me to those comments. Do you mean that here we can't participate in the standard deletion flow because we don't have any replicas to send delete commands to?

If so, could we not instead take closed containers with no keys and no replicas, and skip straight to deleting the container from SCM metadata directly? Since there are no replicas on any datanodes, there's nothing to clean up there, so we could bypass the DELETING state entirely.

I'm happy to put together a PR for this if the approach sounds reasonable.

[ivandika3]

@silvanias From #5504 which also tries to delete the container, it seems there might be some edge cases that might cause data loss and it has been decided to leave these containers to be empty to prevent the data loss. Since it has been a while, I'm not sure whether the issue is still valid so might require the people more familiar with it to validate (@sodonnel @errose28 @siddhantsangwan).

You mentioned that there is a 70GB space usage for storing the containers. Are you talking about the SCM DB (RocksDB) size or the cluster size (as mentioned in #9685)? Our cluster have >10 millions containers and our SCM DB is less than 50GB.

Anyway, you can try to raise a PR and tag the related people to initiate more discussion.