fragmentation metric

[telsasoft]

I learned in #17024 that fragmentation is a measurement of size of free space.
I thought our usage pattern would have allowed for a quite low frag metric, but nothing I've tried has allowed improving it more than temporarily. It seems the fragmentation (or the metric, at least) is very tightly connected to the pool capacity.

I did a bunch of experiments like this:
time dd if=/dev/zero bs=1M count=1k of=/newzfs3/zero0

I'm okay with it that when the pool is 51% full, the fragmentation says 2%.
But when the pool is 65% full, it's 46% fragmented, and when it's 79% full, it shows as 68% fragmented.
I found that with recordsize=1M, when the pool is 79% full, it shows 26% fragmented, so that's an improvement.
I tried different "bs=" sizes, tried adding a ZIL, tried changing several module params, none of these had any effect.

This is disappointing and surprising. Is it really expected that 6 files written sequentially would end up with 26-68% fragmentation just by filling the pool to 80%? Notably, most of those writes were done while the pool was less than half full. I could believe that data written to a pool that's 80-90% full might tend to be stored in a non-optimal/fragmented manner, depending on the usage patterns. But why would such ideal input be written so poorly ?

[amotin]

If at some point you've significantly filled your pool, deleting something later, that resulted in some metaslabs remain almost full, I guess you could be affected by the same issue this PR of mine should fix: #17037 .

[telsasoft]

I've kept all our pools under 85%. But often around 80%. Typically they're built with a 128 GB vdev, and then have other vdevs added if/when they approach 85%.

Here's an example. In this case, the pool has 1 vdev, and has never had more than one, and it was probably built as 128 GB and never resized.

NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfs         127G  85.7G  41.3G        -         -    60%    67%  1.00x    ONLINE  -

I've never used zdb before, but I think this shows a histogram of our pool fragmentation. Everything between 40% to 67% except for one oddball at 0%...

$ time sudo zdb zfs -mm |grep -E 'fragmentation|^Metaslabs:|^space map object' |awk '/^space map/{l=$0} /fragmentation/{a[l]=$NF} END{for (i in a){print a[i]}}' |sort -nr |sort |uniq -c |sort -nr
     14 59
     14 58
     11 61
      9 66
      9 63
      8 65
      8 64
      7 62
      7 60
      6 57
      5 67
      4 56
      3 55
      3 53
      3 48
      2 68
      2 54
      2 52
      2 51
      2 50
      2 49
      1 47
      1 43
      1 40
      1 0

The pool is like this:

$ time sudo zdb zfs -M
        pool zfs        fragmentation    59%
                          9:  62797 **********
                         10: 154851 ***********************
                         11: 275838 ****************************************
                         12: 261505 **************************************
                         13: 184386 ***************************
                         14: 171282 *************************
                         15: 170010 *************************
                         16: 109219 ****************
                         17:  51497 ********
                         18:  16755 ***
                         19:   2427 *
                         20:    337 *
                         21:     14 *
                         22:      1 *

I applied your patch, built new .debs, installed them, exported the pool, removed and reinstalled the module, and now see frag=60% instead of frag=59%... Edit: I also just applied commit 4049651 (Expand fragmentation table..) and now see frag=70%. Note that the datasets in this pool mostly use recordsize=1M.

I think my complaint is different than what your patch tries to address. I'm not concerned with how the frag metric is computed. I'm concerned with how the data is actually written. ISTM that initially filling a pool to 80% in the most optimal possible way should not result in moderately high fragmentation. I thought there would be plenty of large allocations left.

[amotin]

If we trust that histogram, then the pool is really fragmented. Whatever you are doing to it leaves plenty of small holes. I would not expect that from sequential writes of identical 1MB blocks. The chunks left are too small.

[telsasoft]

That was from a production system, and not the synthetic "dd" case I started with. Sorry for the confusion.

I don't doubt that our production systems are fragmented. But I still wonder why they end up that way. Which is why I looked at what sequential reads would do.

[amotin]

When you are randomly writing and deleting the data, especially with blocks of different sizes (due to small files or different compressability), fragmentation is just a part of life. That is why it was recommended to not fill pools with such workloads more than ~50%.

[telsasoft]

The thing is, our data is primarily append-only data tables, partitioned by time, and deleted in bulk. Some of those tables are also rewritten in batch during maintenance. I would've thought that our workload would tend to have specifically low fragmentation, which is why this is bugging me now..

[amotin]

If the appends are short and not aligned to the record size, they may each time leave random-sized hole where the previous record version resided.

[telsasoft]

Yes - I've been thinking about how to improve on that - thanks for mentioning the alignment aspect, which I hadn't considered.

I just learned about 'special' vdevs (related: special_small_blocks). It seems to help a lot. If I use a 7GB vdev plus a 1GB 'special' vdev, and then write 5 GB of data, the main storage does not end up with high fragmentation, even if I later remove the special vdev.

I don't know if we'll end up (needing to) use special vdevs forever, but it looks like it'll at least be helpful as a transitional measure to get our existing fragmentation under control.

[justinpryzby]

I've noticed that zfs-remove rewrites the evacuated data to the remaining vdevs with minimal fragmentation. I haven't looked at the implementation, but that's great.

That leaves behind an 'indirect' vdev, which is kind of ugly, especially after you've done a few removals... I wish they were not shown in zpool list -v. Also, I wonder whether there is a potential performance problem with a large number of removals. Imagine a pool with 4 vdevs which ends up with perhaps a few hundred removals and associated indirect vdevs. I doubt that's the intended use, nor well tested. Is there any way to get rid of the indirect vdevs?

That leaves me with another question. It's widely stated that "defragmentation is neither supported nor needed". But ... it seems it's sometimes wanted. And it seems like a reasonable way to effect it is to 1) optionally add a new vdev; 2) remove one of the pre-existing vdevs; 3) re-add the removed vdev; 4) optionally repeat with other pre-existing vdevs; 5) optionally remove the new vdev added in step 1.

Also, I wonder if there's any reason it wouldn't be possible, instead of fully removing a vdev, to instead just evacuate its data. That would avoid leaving behind an 'indirect' vdev, and seem to implement a kind of poor-man's defragmentation.

[amotin]

Device removal does not work with individual blocks, but rather sequential chunks of up to 16MB, reduced to a neatest free space border. I would not call it de-fragmentation. It does not significant affect fragmentation, but adds a mess of indirection tables that likely stay with the pool forever.