Need help/advice determining the size of the special vdev

[jumbi77]

Hi guys,

we currently running raidz2 with 8x 18TB Seagate Exos X18 (ST18000NM000J) - most using for large video files and accessing it with samba with different clients. We will shorty migrate the data to a new server and planning raidz2 but with special vdev (ZFS metadata special device). We want to use 2x SATA Samsung SM883 SSDs in mirror and the open questions currently are (A) what size to choose for the SSDs (B) what is the best value for special_small_blocks in our setup (C) optionally/generally: would you advice against 2x SATA SSDs (SM883) and better using 2x NVMe? Will this be a noticeably difference?

Regarding (A) and (B) I followed level1techs article and this are the results:

Block Size Histogram

  block   psize                lsize                asize
   size   Count   Size   Cum.  Count   Size   Cum.  Count   Size   Cum.
    512:   144K  72.2M  72.2M   144K  72.2M  72.2M      0      0      0
     1K:  28.5K  32.8M   105M  28.5K  32.8M   105M      0      0      0
     2K:  13.5K  34.8M   140M  13.5K  34.8M   140M      0      0      0
     4K:   219K   881M  1021M  9.41K  51.4M   191M      0      0      0
     8K:  40.5K   414M  1.40G  8.25K  93.3M   284M    66K   792M   792M
    16K:  94.4K  1.65G  3.05G   117K  1.92G  2.20G   348K  8.15G  8.92G
    32K:  52.0K  2.44G  5.49G  27.0K  1.29G  3.48G   128K  5.98G  14.9G
    64K:  65.5K  5.95G  11.4G  51.5K  4.74G  8.23G  70.6K  6.62G  21.5G
   128K:  98.6K  18.1G  29.5G   305K  42.6G  50.9G  96.8K  17.9G  39.4G
   256K:   105K  37.5G  67.0G  60.2K  20.8G  71.7G   107K  38.9G  78.3G
   512K:   135K  96.8G   164G  39.0K  27.2G  98.9G   119K  88.9G   167G
     1M:  1.78M  1.81T  1.97T  1.90M  1.90T  2.00T  1.82M  2.42T  2.58T
     2M:   126K   367G  2.32T  2.53K  7.51G  2.00T  99.0K   318G  2.89T
     4M:  19.8M  79.2T  81.5T  20.0M  80.0T  82.0T  19.9M   106T   109T
     8M:      0      0  81.5T      0      0  82.0T      0      0   109T
    16M:      0      0  81.5T      0      0  82.0T      0      0   109T

Reading this histogram - I am right that (A) 2x 1,92TB SDDs are more than enough (no 3,84TB needed) if I choose (B) special_small_blocks=512K ? As far as I understand if setting the parameter to 512K the space taking up will be around aprox. 167GB for special vdev right? And going higher (to 1M) will be a big jump to 2.6 TB, which is too much for the selected SSDs.

Just asking here to get sure before buying/migrating. Much thanks in advance.

[amotin]

First I would start calculations from estimating the amount of metadata that will go onto the special vdev in any case. You may run zdb -bb {pool_name} and look for PSIZE and ASIZE on Metadata Total line. PSIZE does not account for multiple metadata copies, so will probably be underestimated about 2x. ASIZE on the other hand accounts for RAIDZ overheads of the current pool (should be better for mirrors), so will be some higher that it should. At very least you'd need about a double of that capacity, including expected growth as a special vdev.

Second, I would look on your expected pool topology, and particular RAIDZ2 vdev width minus parity. I'd consider more or less space-efficient storage starting from ~16KB of data per data disk in vdev, which for 8 disks give ~128KB blocks. That is where I would target special_small_blocks value. Using that as a base you can use the above histogram to estimate the required special vdev capacity. If the size you get this way will look too small, you can go higher, but likely with diminishing return. You do not want your regular disks to remain idle while SSDs are bottle-necked.

The last point is somewhat related to the required special vdevs throughput. If there will be only metadata and very small data, SATA may be fine, but the higher you'll get on special_small_blocks, the more throughput demands may appear for special vdevs, which may get unacceptably limited by SATA interface.

[jumbi77]

Thanks for the quick feedback.

At very least you'd need about a double of that capacity, including expected growth as a special vdev.

I understand the expected growth, but can you explain why I need at least double of that capacity (from ASIZE)?

The PSIZE and ASIZE on Metadata Total is only around 3.27 resp. 15.8GB (full output of zdb -Lbbbs {pool_name} see below).

Blocks	LSIZE	PSIZE	ASIZE	  avg	 comp	%Total	Type
  462K	30.0G	3.27G	15.8G	35.1K	 9.17	  0.01	Metadata Total

So just to hold metadata of that pool/config I do not need really big SSDs: You recommend at least special_small_blocks=128k, which will take up around ASIZE Cum=39.4G.

Thanks for the notice regarding throughput (NVMe > SATA), we will rethink switching to NVMe if we use special_small_blocks > 128k because we do not want that the special vdev will be the bottleneck.

If throughput of special vdev will be not a concern (thanks to NVMe), do you recommend going higher than special_small_blocks=128k, like 256K or 512K for that pool config?

Details of zdb -Lbbbs {pool_name}

``` root@debian:/# cat /root/zdb_data.txt

Traversing all blocks ...

bp count:              23783874
ganged count:                 0
bp logical:      90140813862912      avg: 3789997
bp physical:     89652105278464      avg: 3769449     compression:   1.01
bp allocated:    119786479153152      avg: 5036457     compression:   0.75
bp deduped:                   0    ref>1:      0   deduplication:   1.00
bp cloned:                    0    count:      0
Normal class:    119789540659200     used: 83.21%
Embedded log class         712704     used:  0.00%

additional, non-pointer bps of type 0:        214
 number of (compressed) bytes:  number of bps
		 27:      2 **
		 28:     14 **************
		 29:     33 *********************************
		 30:      0
		 31:      1 *
		 32:      6 ******
		 33:     20 ********************
		 34:      0
		 35:      0
		 36:      0
		 37:      0
		 38:      0
		 39:      2 **
		 40:      0
		 41:      2 **
		 42:      3 ***
		 43:      2 **
		 44:     13 *************
		 45:      1 *
		 46:      0
		 47:      0
		 48:      0
		 49:      1 *
		 50:      2 **
		 51:      0
		 52:      0
		 53:      0
		 54:      3 ***
		 55:      4 ****
		 56:      2 **
		 57:      1 *
		 58:      0
		 59:      0
		 60:      1 *
		 61:      3 ***
		 62:      0
		 63:      2 **
		 64:      7 *******
		 65:      6 ******
		 66:      4 ****
		 67:      2 **
		 68:      1 *
		 69:      0
		 70:      0
		 71:      0
		 72:      0
		 73:      1 *
		 74:      1 *
		 75:      2 **
		 76:      0
		 77:      0
		 78:      0
		 79:      0
		 80:      0
		 81:      0
		 82:      0
		 83:      2 **
		 84:      1 *
		 85:      3 ***
		 86:      9 *********
		 87:      0
		 88:      2 **
		 89:      2 **
		 90:      2 **
		 91:      2 **
		 92:      2 **
		 93:     13 *************
		 94:      1 *
		 95:      1 *
		 96:      3 ***
		 97:      9 *********
		 98:      0
		 99:      1 *
		100:      0
		101:      1 *
		102:      1 *
		103:      0
		104:      1 *
		105:      2 **
		106:      2 **
		107:      2 **
		108:      0
		109:      5 *****
		110:      2 **
		111:      2 **
		112:      1 *
Dittoed blocks on same vdev: 473010
Dittoed blocks in same metaslab: 1

Blocks LSIZE PSIZE ASIZE avg comp %Total Type
- - - - - - - unallocated
2 32K 8K 72K 36K 4.00 0.00 object directory
1 128K 12K 72K 72K 10.67 0.00 L1 object array
131 65.5K 65.5K 4.61M 36K 1.00 0.00 L0 object array
132 194K 77.5K 4.68M 36.3K 2.50 0.00 object array
1 16K 4K 36K 36K 4.00 0.00 packed nvlist
- - - - - - - packed nvlist size
24 3M 96K 864K 36K 32.00 0.00 L1 bpobj
397 49.6M 5.69M 30.3M 78.3K 8.72 0.00 L0 bpobj
421 52.6M 5.79M 31.2M 75.8K 9.10 0.00 bpobj
- - - - - - - bpobj header
- - - - - - - SPA space map header
172 2.69M 688K 6.05M 36K 4.00 0.00 L1 SPA space map
8.69K 1.09G 108M 635M 73.0K 10.27 0.00 L0 SPA space map
8.86K 1.09G 109M 641M 72.3K 10.23 0.00 SPA space map
24 660K 660K 1.05M 45K 1.00 0.00 ZIL intent log
87 10.9M 348K 2.04M 24K 32.00 0.00 L5 DMU dnode
87 10.9M 348K 2.04M 24K 32.00 0.00 L4 DMU dnode
87 10.9M 348K 2.04M 24K 32.00 0.00 L3 DMU dnode
88 11M 352K 2.07M 24.1K 32.00 0.00 L2 DMU dnode
208 26M 7.72M 24.0M 118K 3.37 0.00 L1 DMU dnode
63.3K 1012M 985M 2.93G 47.4K 1.03 0.00 L0 DMU dnode
63.8K 1.06G 994M 2.96G 47.5K 1.09 0.00 DMU dnode
99 396K 396K 2.33M 24.1K 1.00 0.00 DMU objset
- - - - - - - DSL directory
29 16K 2K 36K 1.24K 8.00 0.00 DSL directory child map
27 44K 42.5K 864K 32K 1.04 0.00 DSL dataset snap map
51 710K 180K 1.69M 33.9K 3.94 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS V0 ACL
3.79K 484M 15.1M 90.9M 24.0K 31.99 0.00 L2 ZFS plain file
193K 24.1G 1.85G 7.52G 40.0K 13.05 0.01 L1 ZFS plain file
22.2M 82.0T 81.5T 109T 4.90M 1.01 99.99 L0 ZFS plain file
22.4M 82.0T 81.5T 109T 4.86M 1.01 99.99 ZFS plain file
19.8K 2.48G 79.4M 476M 24.0K 31.99 0.00 L1 ZFS directory
168K 725M 247M 3.99G 24.3K 2.94 0.00 L0 ZFS directory
188K 3.19G 326M 4.45G 24.3K 10.01 0.00 ZFS directory
26 13K 13K 624K 24K 1.00 0.00 ZFS master node
57 61.5K 52.5K 1.12M 20.2K 1.17 0.00 ZFS delete queue
- - - - - - - zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
- - - - - - - persistent error log
1 128K 44K 180K 180K 2.91 0.00 L1 SPA history
669 83.6M 7.82M 47.0M 71.9K 10.69 0.00 L0 SPA history
670 83.8M 7.86M 47.2M 72.1K 10.65 0.00 SPA history
- - - - - - - SPA history offsets
- - - - - - - Pool properties
- - - - - - - DSL permissions
- - - - - - - ZFS ACL
- - - - - - - ZFS SYSACL
- - - - - - - FUID table
- - - - - - - FUID table size
1 2K 2K 36K 36K 1.00 0.00 DSL dataset next clones
- - - - - - - scan work queue
195 100K 87.5K 3.96M 20.8K 1.15 0.00 ZFS user/group/project used
- - - - - - - ZFS user/group/project quota
- - - - - - - snapshot refcount tags
- - - - - - - DDT ZAP algorithm
- - - - - - - DDT statistics
2.75K 3.02M 3.02M 66.1M 24K 1.00 0.00 System attributes
26 13K 11K 528K 20.3K 1.18 0.00 SA master node
26 39K 39K 624K 24K 1.00 0.00 SA attr registration
58 928K 232K 1.36M 24K 4.00 0.00 SA attr layouts
- - - - - - - scan translations
- - - - - - - deduplicated block
194 254K 140K 3.38M 17.8K 1.81 0.00 DSL deadlist map
- - - - - - - DSL deadlist map hdr
1 2K 2K 36K 36K 1.00 0.00 DSL dir clones
10 1.25M 40K 360K 36K 32.00 0.00 bpobj subobj
- - - - - - - deferred free
- - - - - - - dedup ditto
25 3.12M 100K 900K 36K 32.00 0.00 L1 other
365 2.71M 751K 12.4M 34.7K 3.70 0.00 L0 other
390 5.84M 851K 13.3M 34.8K 7.03 0.00 other
87 10.9M 348K 2.04M 24K 32.00 0.00 L5 Total
87 10.9M 348K 2.04M 24K 32.00 0.00 L4 Total
87 10.9M 348K 2.04M 24K 32.00 0.00 L3 Total
3.87K 496M 15.5M 92.9M 24.0K 31.99 0.00 L2 Total
213K 26.6G 1.93G 8.01G 38.5K 13.77 0.01 L1 Total
22.5M 82.0T 81.5T 109T 4.85M 1.01 99.99 L0 Total
22.7M 82.0T 81.5T 109T 4.80M 1.01 100.00 Total
462K 30.0G 3.27G 15.8G 35.1K 9.17 0.01 Metadata Total

Block Size Histogram

block psize lsize asize
size Count Size Cum. Count Size Cum. Count Size Cum.
512: 144K 72.2M 72.2M 144K 72.2M 72.2M 0 0 0
1K: 28.5K 32.8M 105M 28.5K 32.8M 105M 0 0 0
2K: 13.5K 34.8M 140M 13.5K 34.8M 140M 0 0 0
4K: 219K 881M 1021M 9.41K 51.4M 191M 0 0 0
8K: 40.5K 414M 1.40G 8.25K 93.3M 284M 66K 792M 792M
16K: 94.4K 1.65G 3.05G 117K 1.92G 2.20G 348K 8.15G 8.92G
32K: 52.0K 2.44G 5.49G 27.0K 1.29G 3.48G 128K 5.98G 14.9G
64K: 65.5K 5.95G 11.4G 51.5K 4.74G 8.23G 70.6K 6.62G 21.5G
128K: 98.6K 18.1G 29.5G 305K 42.6G 50.9G 96.8K 17.9G 39.4G
256K: 105K 37.5G 67.0G 60.2K 20.8G 71.7G 107K 38.9G 78.3G
512K: 135K 96.8G 164G 39.0K 27.2G 98.9G 119K 88.9G 167G
1M: 1.78M 1.81T 1.97T 1.90M 1.90T 2.00T 1.82M 2.42T 2.58T
2M: 126K 367G 2.32T 2.53K 7.51G 2.00T 99.0K 318G 2.89T
4M: 19.8M 79.2T 81.5T 20.0M 80.0T 82.0T 19.9M 106T 109T
8M: 0 0 81.5T 0 0 82.0T 0 0 109T
16M: 0 0 81.5T 0 0 82.0T 0 0 109T

</p>
</details> 

[amotin]

Double size I meant to not use more than 50-75% of special vdev at any time to reduce its fragmentation and possible use of gang blocks. But I am not saying it is double of ASIZE. As I have told above, ASIZE you see right now is based on RAIDZ overheads, and it will be smaller on mirrored special. So with PSIZE of 3.27G and ASIZE of 15.8G I would guess something about 7-8G for mirrors actually (due to most of metadata having 2 copies), and about double of that for vdev capacity.

16KB per disk (128KB) is where you get decent space efficiency and processing overhead from RAIDZ. Space efficiency in your case should not be critical, since you don't have so many metadata and small blocks. Optimum performance point for HDDs may be higher due to large seek times, that is why (in addition to better compression, etc) you have recordsize increased to 4MB, as I understand, which is reasonable (may be a bit on higher side) for sequential I/O and gives each disk ~512KB of sequential data even in case of pool fragmentation, which should be more than enough. You may set special_small_blocks higher than 128KB, but I am not sure what blocks do you expect fit into that range and whether you benefit from faster access to them. If those blocks are just random better-compressed ranges or the last blocks of files, putting them on special vdevs may not give much benefits (if any).