Replies: 14 comments 30 replies
-
|
Hi, did you find out what the problem was? What did you end up doing? I am in the process of setting up new Debian 11 system and might need the QAT for compression, I will test with zstd algorithm to see how it goes first though... |
Beta Was this translation helpful? Give feedback.
-
|
Hi @AGI-chandler, No, I gave up. But a couple of fresh eyes on this discussion might bring some new ideas. Keep me informed, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
In looking for the QAT drivers to install, they have a newer release 4.20.0-00001 now. Also, did you try any of the |
Beta Was this translation helpful? Give feedback.
-
As written in my first post, I used the following options: I'm not sure what other options in this list would be needed... |
Beta Was this translation helpful? Give feedback.
-
Me neither, tbh. I couldn't find any documentation on the options, frankly. I wonder why? 🤔 |
Beta Was this translation helpful? Give feedback.
-
Oh! Those are configure options for QAT... What options did you use for ZFS? At a minimum, I believe you need to point ZFS to the driver code as well, with |
Beta Was this translation helpful? Give feedback.
-
|
Some details about my choices:
But anyway, the QAT part seems good, as I wrote more than a year ago:
About ZFS setup:
I export Maybe there was some setup mistake on my ZFS pool? I hope this could help you move some steps ahead. Keep me informed. |
Beta Was this translation helpful? Give feedback.
-
|
After carefully re-reading your OP, I noticed: That's a Debian version of the module. Plus you mentioned You need to download ZFS source, properly configure it, build it, build the debs, then install the debs. Before that, you need to export all your ZFS pools so they are unmounted/deactivated, uninstall the Debian zfs-dkms and any other related packages, and You'll also need the Debian I successfully did this a few years ago on CentOS and Debian, but am struggling to get it updated on the Debian machine, you can see my other issue here, I think I must have messed up the kernel source or something. In looking in my command history on the CentOS machine, it's hilarious. I must have wiped out the ZFS source tree a dozen times while I figured out the correct way to make everyone play together. Unfortunately, on the Debian machine, my history there is gone. But, in configuring ZFS, you'll at least want to use |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I found nothing about Debian disabling compression with ZFS/QAT. At least, there's no Debian patch in the zfs-linux package (from which the zfs-dkms is built): https://salsa.debian.org/zfsonlinux-team/zfs/-/tree/master/debian/patches. I would be interested about any pointer about that... It could be useful to improve ZFS doc around QAT/Debian with such info. Moreover, I suppose there should be some public discussions about this on the Debian side... I'll give your suggestion a try anyway. Mor than a year has spend since my last try anyway, the versions are different now, so who knows? I subscribed to your other issue to follow your adventure with ZFS/QAT :) Thank you very much for your feedback and suggestion. |
Beta Was this translation helpful? Give feedback.
-
|
Well, I've honestly never used distribution sources for anything, so am learning as well... However, when I did this a few years ago, the documentation out there basically had one way to do it, and that was to download/install the Intel drivers and then download/configure/install the ZFS source code... after a bit of trial and error, I got that working perfectly in both CentOS Linux 2.6 and Debian Linux 4.19, and they've been running great for years. The ability for ZFS to do compression, encryption, and checksums is standard, and that remains enabled. However, in order to offload those tasks from the CPU to the QAT, some extra You can check the Build log for zfs-linux (2.1.7-2) on amd64 for example and the only mention of Those That seems straightforward enough, but your QAT is somehow handling the ZFS encryption.... er wait, now that I'm connecting all the details, you only mentioned
Now I just noticed that the Still, the fact that you have Well that was a fun little learning experience and hope if was helpful and informative. I think you'll find if you configure the ZFS source yourself, then it'll start using your QAT hardware! |
Beta Was this translation helpful? Give feedback.
-
Debian does not provide the module packaged. I suppose the only compilation it does is for CLIs. Note that dkms actually rebuild the module locally, from source provided by Debian. Setting I wonder for the
My tests with cryptsetup was to show that the QAT module compiled and loaded in kernel actually work with some other tools. However, cryptsetup is userland where ZFS is in kernel space. The fact that my zfs module was having some QAT stats shows that it was compiled with "QAT aware" code... I'll try again when I'll find some time... |
Beta Was this translation helpful? Give feedback.
-
Yes it does, doesn't it? See for example the
Oh that's neat! Thanks for showing that! I did write to them, though, at
Yes, this is what the
Are you sure? I don't see that. I am pretty tired, though. The
Good luck! You will be happy when you get it, and so will your CPU, as it will drop to nearly idle usage with the QAT is handling all that work. |
Beta Was this translation helpful? Give feedback.
-
You are mixing up the QAT module compilation and the ZFS one :)
Well, during intensive writes (external backup to the ZFS storage), |
Beta Was this translation helpful? Give feedback.
-
|
Yo @ioguix! I'm happy to report I finally got things working on my end! and your idea was right! The Debian What ended up working for me was adding to the kernel command I notice in your initial are you using this computer for virtualization tasks? If not, then try turning those options off. From my understanding, the QAT can only be used for virtual tasks (by guest services) or physical tasks (by host services), but not both. That's why I had to add Also might want to drop |
Beta Was this translation helpful? Give feedback.
-
I already knew dkms and zfs-dkms was quite noticeable. About the ICP_ROOT to export, it's documented in either QAT doc and ZFS one. Another tip, I think you can just
Well, the sample module code seems to work correctly, so I suspect this one is fine. |
Beta Was this translation helpful? Give feedback.
-
|
Well @ioguix the fun continues. After rebooting the computer, ZFS would NOT integrate with the QAT any longer. I spent all afternoon trying to figure out why. I think it had to do with the ZFS module loading before the QAT was initialized and could load its module. I tried the systemd override you mentioned in your OP but it was not good enough. Nothing was. I must have tried 10 different things, various other overrides, adding the QAT modules to /etc/modules, blacklisting zfs modules, adding the QAT modules to the initramfs, and so on. No matter what I did that sneaky ZFS module would always insert itself before the QAT was ready, even before the systemd "Load the ZFS modules" service had started. For example a message like this would always show up: So I went back to the source and tried again and finally was able to get that running, hopefully it survives a reboot! Here are the commands I used that worked in case you want to try:
|
Beta Was this translation helpful? Give feedback.
-
and of course it did not! The culprit is definitely the the ZFS module getting loaded before the So I completely uninstalled and blew out the zfs-2.1.9 source installation and went back to the The next trick is to totally block the ZFS module from getting loaded early in the boot process. I was able to accomplish that by adding this to Basically, it tells After rebooting, there were some slight issues with dependency loops, so I further had to make these changes: To Then run Not sure why I'm having to go through any of this, I suppose people like us using QAT are rare and so these conditions are not fully accounted for. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @AGI-chandler , I've got my hands dirty tonight, messing with this issue. It's not finished yet, but bellow my thoughts. kernel spaceSo, it seems the motherboard capability AND features enabled in the kernel have some kind of influence about how the QAT module are build and/or setup by default. Deactivating the MB capabilities was not enough, I had to rebuild the QAT module after reboot. The sample code module wasn't working after disabling After adding the kernel parameter export ICP_ROOT=/root/qat-4.20/
cd $ICP_ROOT
make -j4 samples-install
insmod /usr/local/bin/cpa_sample_code.ko runTests=1
journalctl -k # should show various test and benchmarks
rmmod cpa_sample_codeHere is a sample of the log written by the sample module: So, at this point, I know the QAT module is working fine from the kernel space API now. I rebuilt the ZFS module, but the counters in May I ask what is your checksum and encryption setup for ZFS ? root=> zfs get checksum tank/data
NAME PROPERTY VALUE SOURCE
tank/data checksum sha256 local
root=> zfs get encryption tank/data
NAME PROPERTY VALUE SOURCE
tank/data encryption aes-256-gcm -User landIn fact, the qat counters in What surprise me is that this activity is quite CPU intensive, with a high load, no matter what... Worse: testing with root=> service qat_service stop
root=> cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1 836185 iterations per second for 256-bit key
PBKDF2-sha256 1281877 iterations per second for 256-bit key
PBKDF2-sha512 432580 iterations per second for 256-bit key
PBKDF2-ripemd160 376103 iterations per second for 256-bit key
PBKDF2-whirlpool 205442 iterations per second for 256-bit key
argon2i 4 iterations, 795418 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id 4 iterations, 706485 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
# Algorithm | Key | Encryption | Decryption
aes-cbc 128b 442.1 MiB/s 920.6 MiB/s
serpent-cbc 128b 43.5 MiB/s 102.1 MiB/s
twofish-cbc 128b 91.3 MiB/s 104.9 MiB/s
aes-cbc 256b 336.1 MiB/s 754.7 MiB/s
serpent-cbc 256b 43.5 MiB/s 102.1 MiB/s
twofish-cbc 256b 91.2 MiB/s 104.9 MiB/s
aes-xts 256b 809.0 MiB/s 808.4 MiB/s
serpent-xts 256b 97.9 MiB/s 98.0 MiB/s
twofish-xts 256b 100.0 MiB/s 100.0 MiB/s
aes-xts 512b 675.1 MiB/s 674.8 MiB/s
serpent-xts 512b 97.9 MiB/s 98.0 MiB/s
twofish-xts 512b 100.0 MiB/s 100.0 MiB/s
root=> service qat_service start
root=> cat /sys/kernel/debug/qat_c3xxx_0000\:01\:00.0/fw_counters
+------------------------------------------------+
| FW Statistics for Qat Device |
+------------------------------------------------+
| Firmware Requests [AE 0]: 205 |
| Firmware Responses[AE 0]: 205 |
| RAS Events [AE 0]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 1]: 0 |
| Firmware Responses[AE 1]: 0 |
| RAS Events [AE 1]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 2]: 0 |
| Firmware Responses[AE 2]: 0 |
| RAS Events [AE 2]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 3]: 204 |
| Firmware Responses[AE 3]: 204 |
| RAS Events [AE 3]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 4]: 0 |
| Firmware Responses[AE 4]: 0 |
| RAS Events [AE 4]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 5]: 0 |
| Firmware Responses[AE 5]: 0 |
| RAS Events [AE 5]: 0 |
+------------------------------------------------+
root=> cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1 826952 iterations per second for 256-bit key
PBKDF2-sha256 1274089 iterations per second for 256-bit key
PBKDF2-sha512 432580 iterations per second for 256-bit key
PBKDF2-ripemd160 372893 iterations per second for 256-bit key
PBKDF2-whirlpool 205120 iterations per second for 256-bit key
argon2i 4 iterations, 786472 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id 4 iterations, 798208 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
# Algorithm | Key | Encryption | Decryption
aes-cbc 128b 322.5 MiB/s 323.0 MiB/s
serpent-cbc 128b 43.5 MiB/s 102.0 MiB/s
twofish-cbc 128b 91.2 MiB/s 104.9 MiB/s
aes-cbc 256b 254.7 MiB/s 255.2 MiB/s
serpent-cbc 256b 43.5 MiB/s 102.1 MiB/s
twofish-cbc 256b 91.2 MiB/s 104.9 MiB/s
aes-xts 256b 322.5 MiB/s 322.5 MiB/s
serpent-xts 256b 97.9 MiB/s 98.0 MiB/s
twofish-xts 256b 100.0 MiB/s 100.1 MiB/s
aes-xts 512b 255.7 MiB/s 255.6 MiB/s
serpent-xts 512b 97.9 MiB/s 98.0 MiB/s
twofish-xts 512b 100.0 MiB/s 100.0 MiB/s
root=> cat /sys/kernel/debug/qat_c3xxx_0000\:01\:00.0/fw_counters
+------------------------------------------------+
| FW Statistics for Qat Device |
+------------------------------------------------+
| Firmware Requests [AE 0]: 18669 |
| Firmware Responses[AE 0]: 18669 |
| RAS Events [AE 0]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 1]: 0 |
| Firmware Responses[AE 1]: 0 |
| RAS Events [AE 1]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 2]: 0 |
| Firmware Responses[AE 2]: 0 |
| RAS Events [AE 2]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 3]: 18668 |
| Firmware Responses[AE 3]: 18668 |
| RAS Events [AE 3]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 4]: 0 |
| Firmware Responses[AE 4]: 0 |
| RAS Events [AE 4]: 0 |
+------------------------------------------------+
| Firmware Requests [AE 5]: 0 |
| Firmware Responses[AE 5]: 0 |
| RAS Events [AE 5]: 0 |
+------------------------------------------------+Maybe this benchmark is not the best suited to test the QAT performance/load impact (I don't know if it's parallelized)... root=> /usr/local/bin/cpa_sample_code runTests=1
[...]
Cipher AES128-CBC
Direction Encrypt
API Traditional
Packet Size 4096
Number of Threads 6
Total Submissions 600000
Total Responses 600000
Total Retries 954978
CPU Frequency(kHz) 2200076
Throughput(Mbps) 19200
[...]Do you have some similar results? |
Beta Was this translation helpful? Give feedback.
-
|
Heya. Yeah you'll want/need to rebuild the QAT drivers when making BIOS or kernel changes like that, and then rebuild/reinstall zfs(-dkms) as well. I know with all the testing and reconfiguring and retesting sometimes it's hard to remember if all the modules have been built properly so I just uninstall and rebuild everything again to be safe. I don't currently use encryption. Checksum is "on" and compression is "gzip": I will try some benchmarks too, we have the 8970 card which is pretty beefy and 2x 5222 cpu's, but here is the one you ran:
Performance might be different with Intel Atom and qat_c3xxx. I only got the 8970 card for gzip compression and without it the system load is 17-18 but with it the load is 1-2, so it makes a huge difference there. You might want to open a case with Intel just to make sure all your bases are covered. Let me try cryptsetup... Also it looks weird that only 2 of your Click to expand
|
Beta Was this translation helpful? Give feedback.
-
Hey @AGI-chandler , just wanted to say thanks for sharing the CPU impact (and lol, I wonder if you're the chandler I remember from 15-20 years ago on some IRC channel, ceph maybe or so... anyway, hf and thanks again!) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for the discussion/documentation, helped tremendiously during setup. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, What is your setup for encryption, compression and checksum? Your CPU/card ? Do you have any before/after benchmark? |
Beta Was this translation helpful? Give feedback.
-
|
Checksum and compression only, running on an Intel(R) Xeon(R) D-2146NT CPU @ 2.30GHz. with 12 HDDs at 4xraidz + slog + l2arc performancewise, this is where I ended up using a fio benchmark: This is perfectly fine as the primary backup target for veeam. If you want to know anything else, let me know, I'll try to help. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @volkermauel ! Thanks for your feedback, nice beefy disk setup. Having numbers without QAT enabled, especially about CPU load, would be very interesting. I'll try to do some benchmark using fio on my little setup with encryption, but so far I had no luck with QAT :/ |
Beta Was this translation helpful? Give feedback.
-
|
oh and also remember to |
Beta Was this translation helpful? Give feedback.
-
I admit I built this ZFS pool long time ago and forgot how I set it up exactly. So this gave me some hope, but nope, my recordsize if 128k. Thanks for the spark of hope anyway ;) # zfs get recordsize
NAME PROPERTY VALUE SOURCE
tank recordsize 128K default
tank/data recordsize 128K default
...I'll recheck the whole boot process, modules and services order in a week or two, but according to my first message, I already tried to handled it using service setup: # cat /etc/systemd/system/zfs-load-module.service.d/override.conf
[Unit]
After=qat_serviceAnd as I wrote, I'm fairly sure I tried to load everything by hands as well... Thanks for your help and fresh idea! |
Beta Was this translation helpful? Give feedback.
-
|
can you post a dmesg kernel log? maybe we're able to debug it that way |
Beta Was this translation helpful? Give feedback.
-
With QAT for encryption (+checksum + compression (both inherited)): Checking QAT: So QAT is being used from what it looks like. Load average: 20.73 and now only checksum, but no compression + encryption: and now the other way around: Encryption + Checksum, but no compression |
Beta Was this translation helpful? Give feedback.
-
|
Worse performance with the QAT? |
Beta Was this translation helpful? Give feedback.
-
|
I am trying to get QAT working on debian 11, this time on arm64. Is there any way to have ZFS use the built-in drivers instead of the rather old source-code? At least the built-in drivers build fine on arm64. My priority is getting crypto acceleration working as I have another accelerator for (de)compression. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @ayakael, Well, look at last @volkermauel message here: #12723 (reply in thread) I'm now doubtful QAT crypto acceleration worth the setup and maintenance effort with actual ZFS code. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I saw that. Anything would be better than what I have now, as arm64 crypto extensions aren't used by ZFS. I have a 96 core server, and crypto operations are capped at 500 mb/s. |
Beta Was this translation helpful? Give feedback.
-
|
What rather old source code are you referring to? I'm pretty sure the built-in kernel QAT support only extends to... other kernel modules like |
Beta Was this translation helpful? Give feedback.
-
|
You'd probably be better served extending the ZFS crypto code to leverage aarch64's crypto extensions than trying to get QAT running. But YMMV. |
Beta Was this translation helpful? Give feedback.
-
|
Hey everyone! I got checksum and compression working on Debian 12, but as soon as I try to create an encrypted ZFS I get this kernel panic, any ideas? I only have some lines of the panic as my connection for video isn't fast enough: Also saw this (is my CPU too old? I'm trying this on an AMD E-350): 
|
Beta Was this translation helpful? Give feedback.
-
|
Since QAT limits the level of gzip compression, compare its achievable compression ratio with zstd or lz4. There's no significant difference between |
Beta Was this translation helpful? Give feedback.
-
|
I don't know that the QAT limits it, I think that is set at the application level. You can specify We have 4 different servers with ZFS volumes here ranging in age from 3-10 years, so have progressed over the years testing and using a variety of setups. GZIP has historically compressed the best, particularly well with our genetic data, but required huge amounts of CPU to accomplish. QAT is able take nearly all this overhead away from the CPU's and do it faster as well. Nowadays, zStandard has surpassed GZIP, providing better compression, and requiring much less CPU. Take for example using two of our newer systems: Sample FASTQ data: 178,653,182,026 bytes Disk Usage on ZFS with gzip compression (with QAT): 51,738,035,712 bytes
Disk Usage on ZFS with zstd compression (without QAT): 44,410,388,992 bytes
zStandard cut out more than 7 gigabytes more than GZIP and took about the same time and CPU resources, no QAT even needed. What about good ol' ZIP? 😛 It can compress it even further: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, You're right, the QAT doc shows it supports a higher gzip compression level, but it also has some limitations. My question is, why not buy more CPU resources and allocate them separately for compression? With the price of QAT, we could upgrade to a higher CPU model. In reality, the CPU overhead for IO is not substantial, and multi-core requirements are not demanding. A significant portion of the overhead is attributed to resource contention and system scheduling Therefore, I believe that dividing CPU resources according to NUMA nodes will yield better results. |
Beta Was this translation helpful? Give feedback.
-
|
Well the CPU overhead has diminished over time, but there was a time for a while when it was significant... our old 12-core Intel Xeon E5-2620v2 server could not handle it, 100% load constantly if we tried to use ZFS with gzip compression. Your idea sounds smart and worth investigation and testing... the QAT may still provide significant benefits in certain applications where there is a constant, high throughput of data--I'm thinking like ISP backbone type servers where both compression and encryption could be utilized... the QAT 8970 card for example boasts "Data Rate Per Port: Up to 100G" I'm guessing that's bits/s? |
Beta Was this translation helpful? Give feedback.
-
|
I can answer for the limit, just read that a few moments ago.
|
Beta Was this translation helpful? Give feedback.
-
|
@AGI-chandler even if the CPUs can now handle the load there's a reason why Intel is cramming more and more offload engines into them and all SmartNIC style hardware still has them: latency. it's a double-win, you have more free CPU resources but also your total latency goes down with any kind of hardware offload.
Just adding CPU ressources is the same arguments as for software raid years ago. Totally valid for a system in non-degraded state and without substantial load. If you didn't look you also didn't see you got 20% of IOPS. I'm in horror how raw the QAT software ecosys is, considering QAT 2.0 lacks all support and they are 'in deliberation' of whether to do anything about this. But a clear community push would likely help them not ruin their own product. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
During a (borg) backup reading a whole ZFS encrypted file system, I noticed the
z_rd_intthreads were eating 30-50 of %sys CPU. Theperf topcommand was showing a gcm function (I did not record the exact name) on top of the list.I was using LVM pool on Luks for this partition on the same server in the past, and never experienced such a load. The disk is now 100% ZFS without Luks beneath it.
I was expecting some more load when moving to ZFS, and I'm not sure how large it was supposed to be, but I wonder if I missed something on the way. Since then, I try to enable QAT for ZFS with no success.
The OS is Debian 11 bullseye.
Here are some details about my kernel, CPU and its QAT:
I compiled and installed QAT 4.15 using:
So QAT is available from kernel space and the kernel crypto framework.
It looks to work fine from the crypto side. The kernel module expose various algos, and the QAT counters are moving when using
cryptsetup benchmark:The
ICP_ROOTenvironment variable has been exported before compiling the ZFS module using dkms (apt-get install zfs-dkmsordpkg-reconfigure zfs-dkms).The ZFS module seems to have QAT enabled, it detects it (no
*disabledparameters enabled), but the QAT counters are not moving at all:I added a service drop-in file to make sure the ZFS module is loaded after qat_service is started:
Here is my ZFS setup:
I tried to enable gzip compression on
tank/dataand wrote some data, but no more luck with compression, nothing moves.Also, I started from zfs 2.0.3 available on Debian repos and just upgraded to 2.0.6 available from Debian backports since a few days.
Do I miss something in my setup? Should I report anything else to help with some more diagnostics?
Thanks in advance!
Beta Was this translation helpful? Give feedback.
All reactions