Memory balloons when compacting

[Stebalien]

While we've gotten most of the memory issues under control (see #17 and #83), we're still using the default value log size of 1GiB. Unfortunately, it looks like badger loads the entire value log into memory when compacting (garbage collecting). This happens whenever we delete more than 20% of the values in any given value log.

We can set the default value log size to be smaller (e.g., 64MiB?) but then we'll end up with more files open. As far as I know, badger never closes files.

[Stebalien]

   1.04GB     1.04GB (flat, cum) 93.24% of Total
         .          .    390:		// NOTE: It might be possible that the entry read from the LSM Tree points to
         .          .    391:		// an older vlog file. See the comments in the else part.
         .          .    392:		if vp.Fid == f.fid && vp.Offset == e.offset {
         .          .    393:			moved++
         .          .    394:			// This new entry only contains the key, and a pointer to the value.
  512.04kB   512.04kB    395:			ne := new(Entry)
         .          .    396:			ne.meta = 0 // Remove all bits. Different keyspace doesn't need these bits.
         .          .    397:			ne.UserMeta = e.UserMeta
         .          .    398:			ne.ExpiresAt = e.ExpiresAt
         .          .    399:
         .          .    400:			// Create a new key in a separate keyspace, prefixed by moveKey. We are not
         .          .    401:			// allowed to rewrite an older version of key in the LSM tree, because then this older
         .          .    402:			// version would be at the top of the LSM tree. To work correctly, reads expect the
         .          .    403:			// latest versions to be at the top, and the older versions at the bottom.
         .          .    404:			if bytes.HasPrefix(e.Key, badgerMove) {
  512.05kB   512.05kB    405:				ne.Key = append([]byte{}, e.Key...)
         .          .    406:			} else {
         .          .    407:				ne.Key = make([]byte, len(badgerMove)+len(e.Key))
         .          .    408:				n := copy(ne.Key, badgerMove)
         .          .    409:				copy(ne.Key[n:], e.Key)
         .          .    410:			}
         .          .    411:
    1.03GB     1.03GB    412:			ne.Value = append([]byte{}, e.Value...)
         .          .    413:			es := int64(ne.estimateSize(vlog.opt.ValueThreshold))
         .          .    414:			// Ensure length and size of wb is within transaction limits.
         .          .    415:			if int64(len(wb)+1) >= vlog.opt.maxBatchCount ||
         .          .    416:				size+es >= vlog.opt.maxBatchSize {
         .          .    417:				tr.LazyPrintf("request has %d entries, size %d", len(wb), size)

[Stebalien]

Note: this is valueLog.rewrite.

[Kubuxu]

I think this is resolved by dgraph-io/badger#1292 but it is in the master branch of badger v2.

[lidel]

Could this explain memory issues from ipfs/distributed-wikipedia-mirror#85 ?

[Stebalien]

Yes.

[lidel]

Did some spelunking and the fix from dgraph-io/badger#1292 is already in v1.6.2 (was backported in: dgraph-io/badger#1505)
So its something else.