GC: Data may remain after garbage collecting everything

[schomatis]

Regarding a future transition to Badger, its garbage collection process bails out in the case there is only one value log file, and by default the size of a value log file is set to 1GB. This translates to the scenario where a user adds data and then "deletes" it (e.g., unpinning and garbage collecting it) and it will still have up to 1GB of disk taken with these values marked as deleted but not reclaimed by Badger's GC.

This is not an error, and for big players 1GB may be negligible, but it will raise a normal end user's attention (specially because now everything is deleted and the disk space gets freed), and should be taken in consideration (maybe in the form of clear documentation clarifying that, or catching the ErrNoRewrite error and printing a warning to the user when appropriate).

[schomatis]

As a simple PoC,

ipfs init --profile=badgerds
du -hc  ~/.ipfs
# 84K	total
ipfs add -r --pin=false --silent ~/go/src/github.com/ipfs/go-ipfs
ipfs repo stat
# NumObjects: 5563
# RepoSize:   140915008
du -hc  ~/.ipfs
# 135M	total
ipfs repo gc -q
ipfs repo stat
# NumObjects: 13
# RepoSize:   143023447
du -hc  ~/.ipfs
# 137M	total

(compare with running the same commands without --profile=badgerds.)

[schomatis]

Opened issue dgraph-io/badger#442 to track the minimum value log file size allowed as a possible workaround for the user who wishes to minimize the used disk space after garbage collection.

[leerspace]

I might be mistaken, but I don't think that badger's GC is triggered yet during IPFS' GC: #4300. Or, at least it doesn't seem to be yet on 0.4.14-rc2.

[schomatis]

@leerspace Thanks for pointing me to that PR which gave me more context on the matter. Badger's GC is now linked to IPFS GC through #4578 where vlogFileSize is set to 1 GB. I'm attaching a more detailed script where it can be seen that the number of value log files are never reduced to zero (GC doesn't act when there is only one of them) which causes the effect described above.

For full GC there is an offline tool being developed in dgraph-io/badger#400, which will likely help with the cases where the online badger GC doesn't help

The offline GC tool that is mentioned in that PR seems to have the same constraint so I don't think it will provide a solution to this issue.

Badger GC PoC script

set -x
set -e

export IPFS_PATH=~/.ipfs_badger_test
ipfs init --profile=badgerds
du -hsc  $IPFS_PATH
# 84K	total
ipfs add -r --pin=false --silent -Q ~/go/src/github.com/ipfs/go-ipfs
ipfs repo stat
# NumObjects: 5560
# RepoSize:   140914167
du -hsc  $IPFS_PATH
# 135M	total
ls -lah $IPFS_PATH/badgerds
# 134M mar 19 15:47 000000.vlog  <- Single log file.

ipfs repo gc -q > /dev/null # Not really that quiet.
ipfs repo stat
# NumObjects: 13
# RepoSize:   143021466
du -hsc  $IPFS_PATH
# 137M	total

# Repeat again but setting the value log size to 100 MB.
rm -rv $IPFS_PATH # WARNING! Check the env.var to not delete the actual IPFS root.
ipfs init --profile=badgerds

# Set the 'vlogFileSize' configuration to 100 MB. (May need to install jq.)
cp $IPFS_PATH/config $IPFS_PATH/config.back
cat $IPFS_PATH/config.back | jq  '.Datastore.Spec.child.vlogFileSize = "100000000"' > $IPFS_PATH/config

ipfs add -r --pin=false --silent -Q ~/go/src/github.com/ipfs/go-ipfs
ipfs repo stat
# NumObjects: 5560
# RepoSize:   140918856

ls -lah $IPFS_PATH/badgerds
# 96M mar 19 15:47 000000.vlog  <- First log file cut short at 100 MB (will be GC'ed).
# 39M mar 19 15:47 000001.vlog  <- Second log file (will be left behind in the GC process).

ipfs repo gc -q > /dev/null # Not really that quiet.
ipfs repo stat
# NumObjects: 13
# RepoSize:   143023447
ls -lah $IPFS_PATH/badgerds
# 40M mar 19 15:47 000001.vlog
du -hsc  $IPFS_PATH
# 41M	total

# Multiple succesive calls to GC won't change this.
ipfs repo gc -q > /dev/null
ipfs repo gc -q > /dev/null
ipfs repo gc -q > /dev/null

ls -lah $IPFS_PATH/badgerds
# 40M mar 19 15:47 000001.vlog
du -hsc  $IPFS_PATH
# 41M	total

So does badger do any GC at all, or is it just that it holds on to a bit of diskspace?

[schomatis]

@lgierth Yes, it does GC, it's just that it waits for certain conditions before doing it (like having more than one value log file and having those log files filled with deleted keys passed a certain percentage threshold). So if those conditions aren't met when the user calls ipfs repo gc which will in turn request Badger's GC nothing will happen and the user will not perceive any disk space being freed (even though there may be much unpinned content).

This is expected as Badger's architecture is more complex than flatfs which simply stores keys as separate files (where GC is just the process of deleting those files) so it makes sense that Badger's GC waits for the situation "to be worth it" before doing any actual work (which would include scanning files and searching keys) that may obstruct the rest of the DB operations.

I think a simplified version of what I just explained should be clearly documented for users to prepare their disk-freeing expectations to the Badger scenario (which will definitely be different from what they're accustomed to with the current flatfs default datastore).

So when will it free diskspace then? I've just had these two new gateways run out of diskspace and it refuses to restart (Error: runtime error: slice bounds out of range).

So right now I have no choice other than rm -rf $IPFS_PATH and switch the new gateways back to flatfs.

It would be good if an explicit ipfs repo gc it would tell badger to nevermind what it thinks is best and just do its cleanup routine.

Ah I see: https://github.com/dgraph-io/badger#garbage-collection

This does sound like we can do online garbage collection at a time of our choice.

[schomatis]

I've just had these two new gateways run out of diskspace

In that case Badger should already be performing GC when running ipfs repo gc. I'm assuming that there are many more than just one value log file and the percentage of discarded keys is 10% to trigger a GC (that is, if in a value log file more than 10% of it's keys are marked as deleted it will rewrite the value log file to a smaller size without those keys).

@Stebalien cleared this up for me on IRC:

badger needs to have a special gc function called to actually gc. The new datastore interfaces expose this but they're still in the process of being piped through.

I've put the gateways back on flatfs for the time being, I'll be on research retreat and then vacation for the next 4 weeks -- otherwise I'd be up for riding the edge :)

[magik6k]

@Stebalien cleared this up for me on IRC:

badger needs to have a special gc function called to actually gc. The new datastore interfaces expose this but they're still in the process of being piped through.

GC is called on badger - see https://github.com/ipfs/go-ipfs/blob/master/pin/gc/gc.go#L115-L125 / https://github.com/ipfs/go-ds-badger/blob/master/datastore.go#L258

[Stebalien]

Got it. I hadn't realized that had been bubbled up yet.

[schomatis]

@lgierth Could you document here more about the server environment in which you had the GC problem that we discussed in IRC please? I think it may be an interesting test case to have under consideration (big database, almost no space left, and to reclaim it it was necessary to write delete entries before actually running Badger's GC).

[leerspace]

I'm not seeing any reclaimed free space after garbage collection using badger with v0.4.19, but maybe someone can tell whether this is "expected" behavior.

For example (NumObjects and RepoSize from ipfs repo stat):

$ ipfs init --profile=badgerds
# NumObjects: 28
# RepoSize:   49962
$ ipfs daemon &
$ ipfs get QmSnuWmxptJZdLJpKRarxBMS2Ju2oANVrgbr2xWbie9b2D #stopped just after 3.5 GB
# NumObjects: 16118
# RepoSize:   3798130246
$ ipfs repo gc
# NumObjects: 13
# RepoSize:   3798130246
$ ipfs shutdown
# NumObjects: 93
# RepoSize:   3806508960
$ ipfs repo gc # offline garbage collection still gets me no space reclaimed
# NumObjects: 13
# RepoSize:   3805143019