xref: /dpdk/doc/guides/cryptodevs/qat.rst (revision 4b53e9802b6b6040ad5622b1414aaa93d9581d0c) 
1..  SPDX-License-Identifier: BSD-3-Clause
2    Copyright(c) 2015-2019 Intel Corporation.
3
4Intel(R) QuickAssist (QAT) Crypto Poll Mode Driver
5==================================================
6
7QAT documentation consists of three parts:
8
9* Details of the symmetric and asymmetric crypto services below.
10* Details of the :doc:`compression service <../compressdevs/qat_comp>`
11  in the compressdev drivers section.
12* Details of building the common QAT infrastructure and the PMDs to support the
13  above services. See :ref:`building_qat` below.
14
15
16Symmetric Crypto Service on QAT
17-------------------------------
18
19The QAT symmetric crypto PMD (hereafter referred to as `QAT SYM [PMD]`) provides
20poll mode crypto driver support for the following hardware accelerator devices:
21
22* ``Intel QuickAssist Technology DH895xCC``
23* ``Intel QuickAssist Technology C62x``
24* ``Intel QuickAssist Technology C3xxx``
25* ``Intel QuickAssist Technology 200xx``
26* ``Intel QuickAssist Technology D15xx``
27* ``Intel QuickAssist Technology C4xxx``
28* ``Intel QuickAssist Technology 4xxx``
29
30
31Features
32~~~~~~~~
33
34The QAT SYM PMD has support for:
35
36Cipher algorithms:
37
38* ``RTE_CRYPTO_CIPHER_3DES_CBC``
39* ``RTE_CRYPTO_CIPHER_3DES_CTR``
40* ``RTE_CRYPTO_CIPHER_AES128_CBC``
41* ``RTE_CRYPTO_CIPHER_AES192_CBC``
42* ``RTE_CRYPTO_CIPHER_AES256_CBC``
43* ``RTE_CRYPTO_CIPHER_AES128_CTR``
44* ``RTE_CRYPTO_CIPHER_AES192_CTR``
45* ``RTE_CRYPTO_CIPHER_AES256_CTR``
46* ``RTE_CRYPTO_CIPHER_AES_XTS``
47* ``RTE_CRYPTO_CIPHER_SNOW3G_UEA2``
48* ``RTE_CRYPTO_CIPHER_NULL``
49* ``RTE_CRYPTO_CIPHER_KASUMI_F8``
50* ``RTE_CRYPTO_CIPHER_DES_CBC``
51* ``RTE_CRYPTO_CIPHER_AES_DOCSISBPI``
52* ``RTE_CRYPTO_CIPHER_DES_DOCSISBPI``
53* ``RTE_CRYPTO_CIPHER_ZUC_EEA3``
54
55Hash algorithms:
56
57* ``RTE_CRYPTO_AUTH_SHA1``
58* ``RTE_CRYPTO_AUTH_SHA1_HMAC``
59* ``RTE_CRYPTO_AUTH_SHA224``
60* ``RTE_CRYPTO_AUTH_SHA224_HMAC``
61* ``RTE_CRYPTO_AUTH_SHA256``
62* ``RTE_CRYPTO_AUTH_SHA256_HMAC``
63* ``RTE_CRYPTO_AUTH_SHA384``
64* ``RTE_CRYPTO_AUTH_SHA384_HMAC``
65* ``RTE_CRYPTO_AUTH_SHA512``
66* ``RTE_CRYPTO_AUTH_SHA512_HMAC``
67* ``RTE_CRYPTO_AUTH_AES_XCBC_MAC``
68* ``RTE_CRYPTO_AUTH_SNOW3G_UIA2``
69* ``RTE_CRYPTO_AUTH_MD5_HMAC``
70* ``RTE_CRYPTO_AUTH_NULL``
71* ``RTE_CRYPTO_AUTH_KASUMI_F9``
72* ``RTE_CRYPTO_AUTH_AES_GMAC``
73* ``RTE_CRYPTO_AUTH_ZUC_EIA3``
74* ``RTE_CRYPTO_AUTH_AES_CMAC``
75
76Supported AEAD algorithms:
77
78* ``RTE_CRYPTO_AEAD_AES_GCM``
79* ``RTE_CRYPTO_AEAD_AES_CCM``
80* ``RTE_CRYPTO_AEAD_CHACHA20_POLY1305``
81
82Protocol offloads:
83
84* ``RTE_SECURITY_PROTOCOL_DOCSIS``
85
86Supported Chains
87~~~~~~~~~~~~~~~~
88
89All the usual chains are supported and also some mixed chains:
90
91.. table:: Supported hash-cipher chains for wireless digest-encrypted cases
92
93   +------------------+-----------+-------------+----------+----------+
94   | Cipher algorithm | NULL AUTH | SNOW3G UIA2 | ZUC EIA3 | AES CMAC |
95   +==================+===========+=============+==========+==========+
96   | NULL CIPHER      | Y         | 2&3         | 2&3      | Y        |
97   +------------------+-----------+-------------+----------+----------+
98   | SNOW3G UEA2      | 2&3       | 1&2&3       | 2&3      | 2&3      |
99   +------------------+-----------+-------------+----------+----------+
100   | ZUC EEA3         | 2&3       | 2&3         | 2&3      | 2&3      |
101   +------------------+-----------+-------------+----------+----------+
102   | AES CTR          | 1&2&3     | 2&3         | 2&3      | Y        |
103   +------------------+-----------+-------------+----------+----------+
104
105* The combinations marked as "Y" are supported on all QAT hardware versions.
106* The combinations marked as "2&3" are supported on GEN2 and GEN3 QAT hardware only.
107* The combinations marked as "1&2&3" are supported on GEN1, GEN2 and GEN3 QAT hardware only.
108
109
110Limitations
111~~~~~~~~~~~
112
113* Only supports the session-oriented API implementation (session-less APIs are not supported).
114* SNOW 3G (UEA2), KASUMI (F8) and ZUC (EEA3) supported only if cipher length and offset fields are byte-multiple.
115* SNOW 3G (UIA2) and ZUC (EIA3) supported only if hash length and offset fields are byte-multiple.
116* No BSD support as BSD QAT kernel driver not available.
117* ZUC EEA3/EIA3 is not supported by dh895xcc devices
118* Maximum additional authenticated data (AAD) for GCM is 240 bytes long and must be passed to the device in a buffer rounded up to the nearest block-size multiple (x16) and padded with zeros.
119* Queue-pairs are thread-safe on Intel CPUs but Queues are not (that is, within a single
120  queue-pair all enqueues to the TX queue must be done from one thread and all dequeues
121  from the RX queue must be done from one thread, but enqueues and dequeues may be done
122  in different threads.)
123* A GCM limitation exists, but only in the case where there are multiple
124  generations of QAT devices on a single platform.
125  To optimise performance, the GCM crypto session should be initialised for the
126  device generation to which the ops will be enqueued. Specifically if a GCM
127  session is initialised on a GEN2 device, but then attached to an op enqueued
128  to a GEN3 device, it will work but cannot take advantage of hardware
129  optimisations in the GEN3 device. And if a GCM session is initialised on a
130  GEN3 device, then attached to an op sent to a GEN1/GEN2 device, it will not be
131  enqueued to the device and will be marked as failed. The simplest way to
132  mitigate this is to use the PCI allowlist to avoid mixing devices of different
133  generations in the same process if planning to use for GCM.
134* The mixed algo feature on GEN2 is not supported by all kernel drivers. Check
135  the notes under the Available Kernel Drivers table below for specific details.
136* Out-of-place is not supported for combined Crypto-CRC DOCSIS security
137  protocol.
138* ``RTE_CRYPTO_CIPHER_DES_DOCSISBPI`` is not supported for combined Crypto-CRC
139  DOCSIS security protocol.
140* Multi-segment buffers are not supported for combined Crypto-CRC DOCSIS
141  security protocol.
142
143Extra notes on KASUMI F9
144~~~~~~~~~~~~~~~~~~~~~~~~
145
146When using KASUMI F9 authentication algorithm, the input buffer must be
147constructed according to the
148`3GPP KASUMI specification <http://cryptome.org/3gpp/35201-900.pdf>`_
149(section 4.4, page 13). The input buffer has to have COUNT (4 bytes),
150FRESH (4 bytes), MESSAGE and DIRECTION (1 bit) concatenated. After the DIRECTION
151bit, a single '1' bit is appended, followed by between 0 and 7 '0' bits, so that
152the total length of the buffer is multiple of 8 bits. Note that the actual
153message can be any length, specified in bits.
154
155Once this buffer is passed this way, when creating the crypto operation,
156length of data to authenticate "op.sym.auth.data.length" must be the length
157of all the items described above, including the padding at the end.
158Also, offset of data to authenticate "op.sym.auth.data.offset"
159must be such that points at the start of the COUNT bytes.
160
161Asymmetric Crypto Service on QAT
162--------------------------------
163
164The QAT asymmetric crypto PMD (hereafter referred to as `QAT ASYM [PMD]`) provides
165poll mode crypto driver support for the following hardware accelerator devices:
166
167* ``Intel QuickAssist Technology DH895xCC``
168* ``Intel QuickAssist Technology C62x``
169* ``Intel QuickAssist Technology C3xxx``
170* ``Intel QuickAssist Technology D15xx``
171* ``Intel QuickAssist Technology C4xxx``
172* ``Intel QuickAssist Technology 4xxx``
173
174The QAT ASYM PMD has support for:
175
176* ``RTE_CRYPTO_ASYM_XFORM_MODEX``
177* ``RTE_CRYPTO_ASYM_XFORM_MODINV``
178* ``RTE_CRYPTO_ASYM_XFORM_RSA``
179* ``RTE_CRYPTO_ASYM_XFORM_ECDSA``
180* ``RTE_CRYPTO_ASYM_XFORM_ECPM``
181* ``RTE_CRYPTO_ASYM_XFORM_ECDH``
182
183Limitations
184~~~~~~~~~~~
185
186* Big integers longer than 4096 bits are not supported.
187* Queue-pairs are thread-safe on Intel CPUs but Queues are not (that is, within a single
188  queue-pair all enqueues to the TX queue must be done from one thread and all dequeues
189  from the RX queue must be done from one thread, but enqueues and dequeues may be done
190  in different threads.)
191* RSA-2560, RSA-3584 are not supported
192
193.. _building_qat:
194
195Building PMDs on QAT
196--------------------
197
198A QAT device can host multiple acceleration services:
199
200* symmetric cryptography
201* data compression
202* asymmetric cryptography
203
204These services are provided to DPDK applications via PMDs which register to
205implement the corresponding cryptodev and compressdev APIs. The PMDs use
206common QAT driver code which manages the QAT PCI device. They also depend on a
207QAT kernel driver being installed on the platform, see :ref:`qat_kernel` below.
208
209
210Configuring and Building the DPDK QAT PMDs
211~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
212
213
214Further information on configuring, building and installing DPDK is described
215:doc:`here <../linux_gsg/build_dpdk>`.
216
217.. _building_qat_config:
218
219Build Configuration
220~~~~~~~~~~~~~~~~~~~
221
222These are the build configuration options affecting QAT, and their default values:
223
224.. code-block:: console
225
226	RTE_PMD_QAT_MAX_PCI_DEVICES=48
227	RTE_PMD_QAT_COMP_IM_BUFFER_SIZE=65536
228
229Both QAT SYM PMD and QAT ASYM PMD have an external dependency on libcrypto, so are not
230built by default.
231
232The QAT compressdev PMD has no external dependencies, so is built by default.
233
234The number of VFs per PF varies - see table below. If multiple QAT packages are
235installed on a platform then RTE_PMD_QAT_MAX_PCI_DEVICES should be
236adjusted to the number of VFs which the QAT common code will need to handle.
237
238.. Note::
239
240        There are separate config items (not QAT-specific) for max cryptodevs
241        RTE_CRYPTO_MAX_DEVS and max compressdevs RTE_COMPRESS_MAX_DEVS,
242        if necessary these should be adjusted to handle the total of QAT and other
243        devices which the process will use. In particular for crypto, where each
244        QAT VF may expose two crypto devices, sym and asym, it may happen that the
245        number of devices will be bigger than MAX_DEVS and the process will show an error
246        during PMD initialisation. To avoid this problem RTE_CRYPTO_MAX_DEVS may be
247        increased or -a, allow domain:bus:devid:func option may be used.
248
249
250QAT compression PMD needs intermediate buffers to support Deflate compression
251with Dynamic Huffman encoding. RTE_PMD_QAT_COMP_IM_BUFFER_SIZE
252specifies the size of a single buffer, the PMD will allocate a multiple of these,
253plus some extra space for associated meta-data. For GEN2 devices, 20 buffers are
254allocated while for GEN1 devices, 12 buffers are allocated, plus 1472 bytes overhead.
255
256.. Note::
257
258	If the compressed output of a Deflate operation using Dynamic Huffman
259	Encoding is too big to fit in an intermediate buffer, then the
260	operation will be split into smaller operations and their results will
261	be merged afterwards.
262	This is not possible if any checksum calculation was requested - in such
263	case the code falls back to fixed compression.
264	To avoid this less performant case, applications should configure
265	the intermediate buffer size to be larger than the expected input data size
266	(compressed output size is usually unknown, so the only option is to make
267	larger than the input size).
268
269
270Running QAT PMD with minimum threshold for burst size
271~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
272
273If only a small number or packets can be enqueued. Each enqueue causes an expensive MMIO write.
274These MMIO write occurrences can be optimised by setting any of the following parameters:
275
276- qat_sym_enq_threshold
277- qat_asym_enq_threshold
278- qat_comp_enq_threshold
279
280When any of these parameters is set rte_cryptodev_enqueue_burst function will
281return 0 (thereby avoiding an MMIO) if the device is congested and number of packets
282possible to enqueue is smaller.
283To use this feature the user must set the parameter on process start as a device additional parameter::
284
285  -a 03:01.1,qat_sym_enq_threshold=32,qat_comp_enq_threshold=16
286
287All parameters can be used with the same device regardless of order. Parameters are separated
288by comma. When the same parameter is used more than once first occurrence of the parameter
289is used.
290Maximum threshold that can be set is 32.
291
292Running QAT PMD with Intel IPSEC MB library for symmetric precomputes function
293~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
294
295The QAT PMD use Openssl library for partial hash calculation in symmetirc precomputes function by
296default, the following parameter is allow QAT PMD switch over to multi-buffer job API if Intel
297IPSEC MB library installed on system.
298
299- qat_ipsec_mb_lib
300
301To use this feature the user must set the parameter on process start as a device additional parameter::
302
303  -a 03:01.1,qat_ipsec_mb_lib=1
304
305
306Device and driver naming
307~~~~~~~~~~~~~~~~~~~~~~~~
308
309* The qat cryptodev symmetric crypto driver name is "crypto_qat".
310* The qat cryptodev asymmetric crypto driver name is "crypto_qat_asym".
311
312The "rte_cryptodev_devices_get()" returns the devices exposed by either of these drivers.
313
314* Each qat sym crypto device has a unique name, in format
315  "<pci bdf>_<service>", e.g. "0000:41:01.0_qat_sym".
316* Each qat asym crypto device has a unique name, in format
317  "<pci bdf>_<service>", e.g. "0000:41:01.0_qat_asym".
318  This name can be passed to "rte_cryptodev_get_dev_id()" to get the device_id.
319
320.. Note::
321
322	The cryptodev driver name is passed to the dpdk-test-crypto-perf tool in the "-devtype" parameter.
323
324	The qat crypto device name is in the format of the worker parameter passed to the crypto scheduler.
325
326* The qat compressdev driver name is "compress_qat".
327  The rte_compressdev_devices_get() returns the devices exposed by this driver.
328
329* Each qat compression device has a unique name, in format
330  <pci bdf>_<service>, e.g. "0000:41:01.0_qat_comp".
331  This name can be passed to rte_compressdev_get_dev_id() to get the device_id.
332
333.. _qat_kernel:
334
335Dependency on the QAT kernel driver
336~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
337
338To use QAT an SRIOV-enabled QAT kernel driver is required. The VF
339devices created and initialised by this driver will be used by the QAT PMDs.
340
341Instructions for installation are below, but first an explanation of the
342relationships between the PF/VF devices and the PMDs visible to
343DPDK applications.
344
345Each QuickAssist PF device exposes a number of VF devices. Each VF device can
346enable one symmetric cryptodev PMD and/or one asymmetric cryptodev PMD and/or
347one compressdev PMD.
348These QAT PMDs share the same underlying device and pci-mgmt code, but are
349enumerated independently on their respective APIs and appear as independent
350devices to applications.
351
352.. Note::
353
354   Each VF can only be used by one DPDK process. It is not possible to share
355   the same VF across multiple processes, even if these processes are using
356   different acceleration services.
357
358   Conversely one DPDK process can use one or more QAT VFs and can expose both
359   cryptodev and compressdev instances on each of those VFs.
360
361
362Available kernel drivers
363~~~~~~~~~~~~~~~~~~~~~~~~
364
365Kernel drivers for each device for each service are listed in the following table. (Scroll right
366to see the full table)
367
368
369.. _table_qat_pmds_drivers:
370
371.. table:: QAT device generations, devices and drivers
372
373   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
374   | S   | A   | C   | Gen | Device   | Driver/ver    | Kernel Module | Pci Driver | PF Did | #PFs | VF Did | VFs/PF |
375   +=====+=====+=====+=====+==========+===============+===============+============+========+======+========+========+
376   | Yes | No  | No  | 1   | DH895xCC | linux/4.4+    | qat_dh895xcc  | dh895xcc   | 435    | 1    | 443    | 32     |
377   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
378   | Yes | Yes | No  | "   | "        | IDZ/4.12.0+   | "             | "          | "      | "    | "      | "      |
379   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
380   | Yes | Yes | Yes | "   | "        | IDZ/4.13.0+   | "             | "          | "      | "    | "      | "      |
381   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
382   | Yes | No  | No  | 2   | C62x     | linux/4.5+    | qat_c62x      | c6xx       | 37c8   | 3    | 37c9   | 16     |
383   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
384   | Yes | Yes | Yes | "   | "        | IDZ/4.12.0+   | "             | "          | "      | "    | "      | "      |
385   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
386   | Yes | No  | No  | 2   | C3xxx    | linux/4.5+    | qat_c3xxx     | c3xxx      | 19e2   | 1    | 19e3   | 16     |
387   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
388   | Yes | Yes | Yes | "   | "        | IDZ/4.12.0+   | "             | "          | "      | "    | "      | "      |
389   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
390   | Yes | No  | No  | 2   | 200xx    | p             | qat_200xx     | 200xx      | 18ee   | 1    | 18ef   | 16     |
391   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
392   | Yes | No  | No  | 2   | D15xx    | p             | qat_d15xx     | d15xx      | 6f54   | 1    | 6f55   | 16     |
393   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
394   | Yes | No  | No  | 3   | C4xxx    | p             | qat_c4xxx     | c4xxx      | 18a0   | 1    | 18a1   | 128    |
395   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
396   | Yes | No  | No  | 4   | 4xxx     | N/A           | qat_4xxx      | 4xxx       | 4940   | 4    | 4941   | 16     |
397   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
398   | Yes | No  | No  | 4   | 401xxx   | N/A           | qat_401xxx    | 4xxx       | 4942   | 2    | 4943   | 16     |
399   +-----+-----+-----+-----+----------+---------------+---------------+------------+--------+------+--------+--------+
400
401* Note: Symmetric mixed crypto algorithms feature on Gen 2 works only with IDZ driver version 4.9.0+
402
403The first 3 columns indicate the service:
404
405* S = Symmetric crypto service (via cryptodev API)
406* A = Asymmetric crypto service  (via cryptodev API)
407* C = Compression service (via compressdev API)
408
409The ``Driver`` column indicates either the Linux kernel version in which
410support for this device was introduced or a driver available on Intel Developer Zone (IDZ).
411There are both linux in-tree and IDZ kernel drivers available for some
412devices. p = release pending.
413
414If you are running on a kernel which includes a driver for your device, see
415`Installation using kernel.org driver`_ below. Otherwise see
416`Installation using IDZ QAT driver`_.
417
418
419Installation using kernel.org driver
420~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
421
422The examples below are based on the C62x device, if you have a different device
423use the corresponding values in the above table.
424
425In BIOS ensure that SRIOV is enabled and either:
426
427* Disable VT-d or
428* Enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file.
429
430Check that the QAT driver is loaded on your system, by executing::
431
432    lsmod | grep qa
433
434You should see the kernel module for your device listed, e.g.::
435
436    qat_c62x               5626  0
437    intel_qat              82336  1 qat_c62x
438
439Next, you need to expose the Virtual Functions (VFs) using the sysfs file system.
440
441First find the BDFs (Bus-Device-Function) of the physical functions (PFs) of
442your device, e.g.::
443
444    lspci -d:37c8
445
446You should see output similar to::
447
448    1a:00.0 Co-processor: Intel Corporation Device 37c8
449    3d:00.0 Co-processor: Intel Corporation Device 37c8
450    3f:00.0 Co-processor: Intel Corporation Device 37c8
451
452Enable the VFs for each PF by echoing the number of VFs per PF to the pci driver::
453
454     echo 16 > /sys/bus/pci/drivers/c6xx/0000:1a:00.0/sriov_numvfs
455     echo 16 > /sys/bus/pci/drivers/c6xx/0000:3d:00.0/sriov_numvfs
456     echo 16 > /sys/bus/pci/drivers/c6xx/0000:3f:00.0/sriov_numvfs
457
458Check that the VFs are available for use. For example ``lspci -d:37c9`` should
459list 48 VF devices available for a ``C62x`` device.
460
461To complete the installation follow the instructions in
462`Binding the available VFs to the vfio-pci driver`_.
463
464.. Note::
465
466   If the QAT kernel modules are not loaded and you see an error like ``Failed
467   to load MMP firmware qat_895xcc_mmp.bin`` in kernel logs, this may be as a
468   result of not using a distribution, but just updating the kernel directly.
469
470   Download firmware from the `kernel firmware repo
471   <http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/tree/>`_.
472
473   Copy qat binaries to ``/lib/firmware``::
474
475      cp qat_895xcc.bin /lib/firmware
476      cp qat_895xcc_mmp.bin /lib/firmware
477
478   Change to your linux source root directory and start the qat kernel modules::
479
480      insmod ./drivers/crypto/qat/qat_common/intel_qat.ko
481      insmod ./drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko
482
483.. Note::
484
485   If you see the following warning in ``/var/log/messages`` it can be ignored:
486   ``IOMMU should be enabled for SR-IOV to work correctly``.
487
488
489Installation using IDZ QAT driver
490~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
491
492Download the latest QuickAssist Technology Driver from `Intel Developer Zone
493<https://developer.intel.com/quickassist>`_.
494Consult the *Quick Start Guide* at the same URL for further information.
495
496The steps below assume you are:
497
498* Building on a platform with one ``C62x`` device.
499* Using package ``qat1.7.l.4.2.0-000xx.tar.gz``.
500* On Fedora26 kernel ``4.11.11-300.fc26.x86_64``.
501
502In the BIOS ensure that SRIOV is enabled and VT-d is disabled.
503
504Uninstall any existing QAT driver, for example by running:
505
506* ``./installer.sh uninstall`` in the directory where originally installed.
507
508
509Build and install the SRIOV-enabled QAT driver::
510
511    mkdir /QAT
512    cd /QAT
513
514    # Copy the package to this location and unpack
515    tar zxof qat1.7.l.4.2.0-000xx.tar.gz
516
517    ./configure --enable-icp-sriov=host
518    make install
519
520You can use ``cat /sys/kernel/debug/qat<your device type and bdf>/version/fw`` to confirm the driver is correctly installed and is using firmware version 4.2.0.
521You can use ``lspci -d:37c9`` to confirm the presence of the 16 VF devices available per ``C62x`` PF.
522
523Confirm the driver is correctly installed and is using firmware version 4.2.0::
524
525    cat /sys/kernel/debug/qat<your device type and bdf>/version/fw
526
527
528Confirm the presence of 48 VF devices - 16 per PF::
529
530    lspci -d:37c9
531
532
533To complete the installation - follow instructions in
534`Binding the available VFs to the vfio-pci driver`_.
535
536.. Note::
537
538   If using a later kernel and the build fails with an error relating to
539   ``strict_stroul`` not being available apply the following patch:
540
541   .. code-block:: diff
542
543      /QAT/QAT1.6/quickassist/utilities/downloader/Target_CoreLibs/uclo/include/linux/uclo_platform.h
544      + #if LINUX_VERSION_CODE >= KERNEL_VERSION(3,18,5)
545      + #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (kstrtoul((str), (base), (num))) printk("Error strtoull convert %s\n", str); }
546      + #else
547      #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,38)
548      #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (strict_strtoull((str), (base), (num))) printk("Error strtoull convert %s\n", str); }
549      #else
550      #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,25)
551      #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; strict_strtoll((str), (base), (num));}
552      #else
553      #define STR_TO_64(str, base, num, endPtr)                                 \
554           do {                                                               \
555                 if (str[0] == '-')                                           \
556                 {                                                            \
557                      *(num) = -(simple_strtoull((str+1), &(endPtr), (base))); \
558                 }else {                                                      \
559                      *(num) = simple_strtoull((str), &(endPtr), (base));      \
560                 }                                                            \
561           } while(0)
562      + #endif
563      #endif
564      #endif
565
566
567.. Note::
568
569   If the build fails due to missing header files you may need to do following::
570
571      sudo yum install zlib-devel
572      sudo yum install openssl-devel
573      sudo yum install libudev-devel
574
575.. Note::
576
577   If the build or install fails due to mismatching kernel sources you may need to do the following::
578
579      sudo yum install kernel-headers-`uname -r`
580      sudo yum install kernel-src-`uname -r`
581      sudo yum install kernel-devel-`uname -r`
582
583.. Note::
584
585   If the build fails on newer GCC versions (such as GCC 12) with an error relating to
586   ``-lc`` not being found, apply the following patch:
587
588   .. code-block:: diff
589
590      /QAT/quickassist/lookaside/access_layer/src/Makefile
591      cd $(ICP_FINAL_OUTPUT_DIR);\
592      cmd="$(LINKER) $(LIB_SHARED_FLAGS) -o \
593        $(LIB_SHARED) $(ADDITIONAL_OBJECTS) $(ADDITIONAL_LIBS) *.o -lpthread -ludev \
594      - -Bstatic -L$(ADF_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL) \
595      - -ladf_user -L$(OSAL_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL)/ \
596      - -losal -Bdynamic -lc"; \
597      + -Bstatic -L$(ADF_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL) \
598      + -ladf_user -L$(OSAL_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL)/ \
599      + -losal -Bdynamic -L/lib/x86_64-linux-gnu/ -lc"; \
600      echo "$$cmd"; \
601      $$cmd
602
603   Followed by this patch:
604
605   .. code-block:: diff
606
607      /QAT/quickassist/build_system/build_files/OS/linux_common_user_space_rules.mk
608      @echo 'Creating shared library ${LIB_SHARED}'; \
609      cd $($(PROG_ACY)_FINAL_OUTPUT_DIR);\
610      -  echo $(LINKER) $(LIB_SHARED_FLAGS) -o $@  $(OBJECTS) $(ADDITIONAL_OBJECTS) -lc;\
611      -  $(LINKER) $(LIB_SHARED_FLAGS) -o $@  $(OBJECTS) $(ADDITIONAL_OBJECTS) -lc ;
612      +  echo $(LINKER) $(LIB_SHARED_FLAGS) -o $@  $(OBJECTS) $(ADDITIONAL_OBJECTS) \
613      +  -L/lib/x86_64-linux-gnu/ -lc;\
614      +  $(LINKER) $(LIB_SHARED_FLAGS) -o $@  $(OBJECTS) $(ADDITIONAL_OBJECTS) \
615      +  -L/lib/x86_64-linux-gnu/ -lc ;
616
617
618Binding the available VFs to the vfio-pci driver
619~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
620
621Note:
622
623* Please note that due to security issues, the usage of older DPDK igb_uio
624  driver is not recommended. This document shows how to use the more secure
625  vfio-pci driver.
626* If QAT fails to bind to vfio-pci on Linux kernel 5.9+, please see the
627  QATE-39220 and QATE-7495 issues in
628  `IDZ doc <https://cdrdv2.intel.com/v1/dl/getContent/710057?explicitVersion=true>`_
629  which details the constraint about trusted guests and add `disable_denylist=1`
630  to the vfio-pci params to use QAT. See also `this patch description <https://lkml.org/lkml/2020/7/23/1155>`_.
631
632Unbind the VFs from the stock driver so they can be bound to the vfio-pci driver.
633
634For an Intel(R) QuickAssist Technology DH895xCC device
635^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
636
637The unbind command below assumes ``BDFs`` of ``03:01.00-03:04.07``, if your
638VFs are different adjust the unbind command below::
639
640    cd to the top-level DPDK directory
641    for device in $(seq 1 4); do \
642        for fn in $(seq 0 7); do \
643            usertools/dpdk-devbind.py -u 0000:03:0${device}.${fn}; \
644        done; \
645    done
646
647For an Intel(R) QuickAssist Technology C62x device
648^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
649
650The unbind command below assumes ``BDFs`` of ``1a:01.00-1a:02.07``,
651``3d:01.00-3d:02.07`` and ``3f:01.00-3f:02.07``, if your VFs are different
652adjust the unbind command below::
653
654    cd to the top-level DPDK directory
655    for device in $(seq 1 2); do \
656        for fn in $(seq 0 7); do \
657            usertools/dpdk-devbind.py -u 0000:1a:0${device}.${fn}; \
658            usertools/dpdk-devbind.py -u 0000:3d:0${device}.${fn}; \
659            usertools/dpdk-devbind.py -u 0000:3f:0${device}.${fn}; \
660        done; \
661    done
662
663For Intel(R) QuickAssist Technology C3xxx or 200xx or D15xx device
664^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
665
666The unbind command below assumes ``BDFs`` of ``01:01.00-01:02.07``, if your
667VFs are different adjust the unbind command below::
668
669    cd to the top-level DPDK directory
670    for device in $(seq 1 2); do \
671        for fn in $(seq 0 7); do \
672            usertools/dpdk-devbind.py -u 0000:01:0${device}.${fn}; \
673        done; \
674    done
675
676Bind to the vfio-pci driver
677^^^^^^^^^^^^^^^^^^^^^^^^^^^
678
679Load the vfio-pci driver, bind the VF PCI Device id to it using the
680``dpdk-devbind.py`` script then use the ``--status`` option
681to confirm the VF devices are now in use by vfio-pci kernel driver,
682e.g. for the C62x device::
683
684    cd to the top-level DPDK directory
685    modprobe vfio-pci
686    usertools/dpdk-devbind.py -b vfio-pci 0000:03:01.1
687    usertools/dpdk-devbind.py --status
688
689Use ``modprobe vfio-pci disable_denylist=1`` from kernel 5.9 onwards.
690See note in the section `Binding the available VFs to the vfio-pci driver`_
691above.
692
693Testing
694~~~~~~~
695
696QAT SYM crypto PMD can be tested by running the test application::
697
698    cd ./<build_dir>/app/test
699    ./dpdk-test -l1 -n1 -a <your qat bdf>
700    RTE>>cryptodev_qat_autotest
701
702QAT ASYM crypto PMD can be tested by running the test application::
703
704    cd ./<build_dir>/app/test
705    ./dpdk-test -l1 -n1 -a <your qat bdf>
706    RTE>>cryptodev_qat_asym_autotest
707
708QAT compression PMD can be tested by running the test application::
709
710    cd ./<build_dir>/app/test
711    ./dpdk-test -l1 -n1 -a <your qat bdf>
712    RTE>>compressdev_autotest
713
714
715Debugging
716~~~~~~~~~
717
718There are 2 sets of trace available via the dynamic logging feature:
719
720* pmd.qat.dp exposes trace on the data-path.
721* pmd.qat.general exposes all other trace.
722
723pmd.qat exposes both sets of traces.
724They can be enabled using the log-level option (where 8=maximum log level) on
725the process cmdline, e.g. using any of the following::
726
727    --log-level="pmd.qat.general,8"
728    --log-level="pmd.qat.dp,8"
729    --log-level="pmd.qat,8"
730
731.. Note::
732
733    The global RTE_LOG_DP_LEVEL overrides data-path trace so must be set to
734    RTE_LOG_DEBUG to see all the trace. This variable is in config/rte_config.h
735    for meson build.
736    Also the dynamic global log level overrides both sets of trace, so e.g. no
737    QAT trace would display in this case::
738
739	--log-level="7" --log-level="pmd.qat.general,8"
740