Merge pull request ceph#58641 from hualongfeng/qat_doc

doc/radosgw/qat-accel: Update and Add QATlib information

Reviewed-by: Zac Dover <[email protected]>

Author: zdover23

doc/radosgw/qat-accel.rst

@@ -33,13 +33,20 @@ QAT Environment Setup
    encryption and compression services. And QAT driver in kernel space have to
    be loaded to drive the hardware.
 
-The driver package can be downloaded from `Intel Quickassist Technology`_.
+The out-of-tree QAT driver package can be downloaded from `Intel Quickassist
+Technology`_.
 
-2. The implementation for QAT based encryption is directly base on QAT API which
-   is included the driver package. But QAT support for compression depends on
-   QATzip project, which is a user space library which builds on top of the QAT
-   API. Currently, QATzip speeds up gzip compression and decompression at the
-   time of writing.
+The QATlib can be downloaded from `qatlib`_, which is used for the in-tree QAT
+driver.
+
+   .. note::
+      The out-of-tree QAT driver is gradually turning to intree driver+QATlib.
+
+2. The implementation of QAT-based encryption is directly based on the QAT API,
+   which is included the driver package. However, QAT support for compression
+   depends on the QATzip project, which is a userspace library that builds on
+   top of the QAT API. At the time of writing (July 2024), QATzip speeds up
+   gzip compression and decompression.
 
 See `QATzip`_.
 
@@ -48,36 +55,40 @@ Implementation
 1. QAT based Encryption for RGW 
 
 `OpenSSL support for RGW encryption`_ has been merged into Ceph, and Intel also
-provides one `QAT Engine`_ for OpenSSL. So, theoretically speaking, QAT based
-encryption in Ceph can be directly supported through OpenSSl+QAT Engine.
+provides one `QAT Engine`_ for OpenSSL. Theoretically, QAT-based encryption in
+Ceph can be directly supported through OpenSSl+QAT Engine.
 
-But the QAT Engine for OpenSSL currently supports chained operations only, and
-so Ceph will not be able to utilize QAT hardware feature for crypto operations
-based on OpenSSL crypto plugin. As a result, one QAT plugin based on native
-QAT API is added into crypto framework.
+However, the QAT Engine for OpenSSL currently supports only chained operations,
+which means that Ceph will not be able to utilize QAT hardware features for
+crypto operations based on the OpenSSL crypto plugin. As a result, one QAT plugin
+based on native QAT API is added into the crypto framework.
 
 2. QAT Support for Compression
 
-As mentioned above, QAT support for compression is based on QATzip library in
-user space, which is designed to take full advantage of the performance provided
-by QuickAssist Technology. Unlike QAT based encryption, QAT based compression
-is supported through a tool class for QAT acceleration rather than a compressor
-plugin. The common tool class can transparently accelerate the existing compression
-types, but only zlib compressor can be supported at the time of writing. So
-user is allowed to use it to speed up zlib compressor as long as the QAT
-hardware is available and QAT is capable to handle it.
+As mentioned above, QAT support for compression is based on the QATzip library
+in user space, which is designed to take full advantage of the performance that
+QuickAssist Technology provides. Unlike QAT-based encryption, QAT-based
+compression is supported through a tool class for QAT acceleration rather than
+a compressor plugin. This common tool class can transparently accelerate the
+existing compression types, but only zlib compressor is supported at the
+time of writing. This means that the user can use this tool class to speed up
+zlib compressor if the QAT hardware is available and if QAT is capable of
+handling it.
 
 Configuration
 =============
 #. Prerequisites
 
-   Make sure the QAT driver with version v1.7.L.4.14.0 or higher has been installed.
-   Remember to set an environment variable "ICP_ROOT" for your QAT driver package
-   root directory. 
+   **For out-of-tree QAT**
 
-   To enable the QAT based encryption and compression, user needs to modify the QAT
-   configuration files. For example, for Intel QuickAssist Adapter 8970 product, revise 
-   c6xx_dev0/1/2.conf in the directory ``/etc/`` and keep them the same, e.g.:
+   Make sure the out-of-tree QAT driver with version v1.7.L.4.14.0 or higher
+   has been installed.  Remember to set an environment variable ``ICP_ROOT``
+   for your QAT driver package root directory. 
+
+   To enable the QAT based encryption and compression, the user must modify the
+   QAT configuration files. For example, for the Intel QuickAssist Adapter 8970
+   product, revise ``c6xx_dev0/1/2.conf`` in the directory ``/etc/`` and keep them
+   the same. For example:
 
    .. code-block:: ini
         
@@ -101,55 +112,131 @@ Configuration
       # List of core affinities
       Dc0CoreAffinity = 0
 
-#. QAT based Encryption for RGW 
+   **For in-tree QAT**
+
+   There are some prerequisites for using QATlib. Make sure that your system
+   meets the `QATlib System Requirements`_ .
+
+   * To properly use the QATlib library, the Intel VT-d and SR-IOV parameters
+     must be enabled in the platform BIOS.
+   * Some qatlib features require a recent kernel driver or firmware version.
+     See `QATlib Kernel Driver Releases`_.
+   * The supported platform contains a 4xxx Intel Communications device or
+     newer.
+   * The ``intel_iommu`` parameter must be enabled. Verify that this setting is
+     enabled by running the following commands:
+
+     .. prompt:: bash $
 
-   The CMake option ``WITH_QAT=ON`` must be configured. If you build Ceph from
+        cat /proc/cmdline | grep intel_iommu=on
+        sudo sh -c 'echo "@qat - memlock 204800" >> /etc/security/limits.conf'
+        sudo su -l $USER
+
+   For configuration and Tuning see `QATlib Configuration and Tuning`_.
+
+#. QAT-based Encryption for RGW 
+
+   The CMake option ``WITH_QATDRV=ON`` must be set. If you build Ceph from
    source code (see: :ref:`build-ceph`), navigate to your cloned Ceph repository 
    and execute the following:
 
    .. prompt:: bash $ 
 
       cd ceph
-      ./do_cmake.sh -DWITH_QAT=ON
+      ./do_cmake.sh -DWITH_QATDRV=ON
       cd build
       ininja
 
-   .. note::
-     The section name of the QAT configuration files must be ``CEPH`` since 
-     the section name is set as "CEPH" in Ceph crypto source code.
+   .. note:: The section name of the QAT configuration files must be ``CEPH``,
+      because the section name is set as ``CEPH`` in the Ceph crypto source code.
 
-   Then, edit the Ceph configuration file to make use of QAT based crypto plugin::
+   Edit the Ceph configuration file (usually ``ceph.conf``) to make use of the
+   QAT-based crypto plugin::
 
       plugin crypto accelerator = crypto_qat
 
 #. QAT Support for Compression
 
-   Before starting, make sure both QAT driver and `QATzip`_  have been installed. Besides 
-   "ICP_ROOT", remember to set the environment variable "QZ_ROOT" for the root directory
-   of your QATzip source tree.
+   **For out-of-tree QAT**
 
-   The following CMake options have to be configured to trigger QAT based compression
-   when building Ceph:
+   For the out-of-tree QAT driver package, before starting ensure that both QAT
+   driver and `QATzip`_  have been installed. Besides ``ICP_ROOT``, remember to
+   set the environment variable ``QZ_ROOT`` to the root directory of your QATzip
+   source tree.
+
+   The following CMake options have to be configured to trigger QAT-based
+   compression when building Ceph:
 
    .. prompt:: bash $
 
-      ./do_cmake.sh -DWITH_QAT=ON -DWITH_QATZIP=ON
+      ./do_cmake.sh -DWITH_QATDRV=ON -DWITH_QATZIP=ON -DWITH_SYSTEM_QATZIP=ON -DWITH_QATLIB=OFF
 
-   Then, set an environment variable to clarify the section name of User Process Instance
-   Section in QAT configuration files, e.g.:
+   Set an environment variable to clarify the section name of User Process
+   Instance Section in the QAT configuration files. For example: 
 
    .. prompt:: bash $
 
       export QAT_SECTION_NAME=CEPH
 
-   Next, edit the Ceph configuration file to enable QAT support for compression::
+   **For in-tree QAT**
+
+   For in-tree QAT, make sure that your system meets the `QATlib System
+   Requirements`_.  QATlib can be installed from pre-built packages or from
+   source code.  See `QATlib Installation`_ . After QATlib is installed, you
+   can run ``cpa_sample_code`` to check if the QAT environment is OK.
+
+   If you are using QATlib source code, Ceph cmake build has the compilation
+   options of qatlib and qatzip enabled by default. Our normal compilation
+   already includes QAT-compressor-related code.
+
+   .. prompt:: bash $
+
+      ./do_cmake.sh
+
+   If you are using pre-built packages installed on the system, the following
+   CMake options have to be configured when building Ceph:
+
+   .. prompt:: bash $
+
+      ./do_cmake.sh -DWITH_SYSTEM_QATLIB=ON -DWITH_SYSTEM_QATZIP=ON
+
+
+   **For both out-of-tree QAT and in-tree QAT**
+
+   Edit the Ceph configuration file (usually ``ceph.conf``) to enable QAT
+   support for *zlib* compression::
 
       qat compressor enabled=true
 
+   Set the RGW compression method:
+
+   .. prompt:: bash $
+
+      # for storage class(STANDARD)
+      radosgw-admin zone placement modify --rgw-zone=default --placement-id=default-placement --compression=zlib
+      # or create a new storage class(COLD) and define data pool(default.rgw.cold.data)
+      radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id default-placement --storage-class COLD
+      radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class COLD --compression zlib --data-pool default.rgw.cold.data
+
+CONFIG REFERENCE
+================
+The following QAT-related settings can be added to the Ceph configuration file
+(usually `ceph.conf`) under the ``[client.rgw.{instance-name}]`` section.
+
+.. confval:: qat_compressor_session_max_number
+.. confval:: qat_compressor_busy_polling
+
+
 
 .. _QAT Support for Compression: https://github.com/ceph/ceph/pull/19714
 .. _QAT based Encryption for RGW: https://github.com/ceph/ceph/pull/19386
 .. _Intel Quickassist Technology: https://01.org/intel-quickassist-technology
 .. _QATzip: https://github.com/intel/QATzip
 .. _OpenSSL support for RGW encryption: https://github.com/ceph/ceph/pull/15168
 .. _QAT Engine: https://github.com/intel/QAT_Engine
+.. _qatlib: https://github.com/intel/qatlib
+.. _QATlib User's Guide: https://intel.github.io/quickassist/qatlib/index.html
+.. _QATlib System Requirements: https://intel.github.io/quickassist/qatlib/requirements.html
+.. _QATlib Installation: https://intel.github.io/quickassist/qatlib/install.html
+.. _QATlib Configuration and Tuning: https://intel.github.io/quickassist/qatlib/configuration.html
+.. _QATlib Kernel Driver Releases: https://intel.github.io/quickassist/RN/In-Tree/in_tree_firmware_RN.html#qat-kernel-driver-releases-features