1 # Acceleration Framework {#accel_fw} 2 3 SPDK provides a framework for abstracting general acceleration capabilities 4 that can be implemented through plug-in modules and low-level libraries. These 5 plug-in modules include support for hardware acceleration engines such as 6 the Intel(R) I/O Acceleration Technology (IOAT) engine and the Intel(R) Data 7 Streaming Accelerator (DSA) engine. Additionally, a software plug-in module 8 exists to enable use of the framework in environments without hardware 9 acceleration capabilities. ISA/L is used for optimized CRC32C calculation within 10 the software module. 11 12 ## Acceleration Framework Functions {#accel_functions} 13 14 Functions implemented via the framework can be found in the DoxyGen documentation of the 15 framework public header file here [accel.h](https://spdk.io/doc/accel_8h.html) 16 17 ## Acceleration Framework Design Considerations {#accel_dc} 18 19 The general interface is defined by `/include/spdk/accel.h` and implemented 20 in `/lib/accel`. These functions may be called by an SPDK application and in 21 most cases, except where otherwise documented, are asynchronous and follow the 22 standard SPDK model for callbacks with a callback argument. 23 24 If the acceleration framework is started without initializing a hardware module, 25 optimized software implementations of the operations will back the public API. All 26 operations supported by the framework have a backing software implementation in 27 the event that no hardware accelerators have been enabled for that operation. 28 29 When multiple hardware modules are enabled the framework will assign each operation to 30 a module based on the order in which it was initialized. So, for example if two modules are 31 enabled, IOAT and software, the software module will be used for every operation except those 32 supported by IOAT. 33 34 ## Acceleration Low Level Libraries {#accel_libs} 35 36 Low level libraries provide only the most basic functions that are specific to 37 the hardware. Low level libraries are located in the '/lib' directory with the 38 exception of the software implementation which is implemented as part of the 39 framework itself. The software low level library does not expose a public API. 40 Applications may choose to interact directly with a low level library if there are 41 specific needs/considerations not met via accessing the library through the 42 framework/module. Note that when using the low level libraries directly, the 43 framework abstracted interface is bypassed as the application will call the public 44 functions exposed by the individual low level libraries. Thus, code written this 45 way needs to be certain that the underlying hardware exists everywhere that it runs. 46 47 The low level library for IOAT is located in `/lib/ioat`. The low level library 48 for DSA and IAA is in `/lib/idxd` (IDXD stands for Intel(R) Data Acceleration Driver and 49 supports both DSA and IAA hardware accelerators). In `/lib/idxd` folder, SPDK supports the ability 50 to use either user space and kernel space drivers. The following describes each usage scenario: 51 52 Leveraging user space idxd driver: The DSA devices are managed by the SPDK user space 53 driver in a dedicated SPDK process, then the device cannot be shared by another 54 process. The benefit of this usage is no kernel dependency. 55 56 Leveraging kernel space driver: The DSA devices are managed by kernel 57 space drivers. And the Work queues inside the DSA device can be shared among 58 different processes. Naturally, it can be used in cloud native scenario. The drawback of 59 this usage is the kernel dependency, i.e., idxd kernel driver must be supported and loaded 60 in the kernel. 61 62 ## Acceleration Plug-In Modules {#accel_modules} 63 64 Plug-in modules depend on low level libraries to interact with the hardware and 65 add additional functionality such as queueing during busy conditions or flow 66 control in some cases. The framework in turn depends on the modules to provide 67 the complete implementation of the acceleration component. A module must be 68 selected via startup RPC when the application is started. Otherwise, if no startup 69 RPC is provided, the framework is available and will use the software plug-in module. 70 71 ### IOAT Module {#accel_ioat} 72 73 To use the IOAT module, use the RPC [`ioat_scan_accel_module`](https://spdk.io/doc/jsonrpc.html) before starting the application. 74 75 ### DSA Module {#accel_dsa} 76 77 The DSA module supports the DSA hardware and relies on the low level IDXD library. 78 79 To use the DSA module, use the RPC 80 [`dsa_scan_accel_module`](https://spdk.io/doc/jsonrpc.html). By default, this 81 will attempt to load the SPDK user-space idxd driver. To use the built-in 82 kernel driver on Linux, add the `-k` parameter. See the next section for 83 details on using the kernel driver. 84 85 The DSA hardware supports a limited queue depth and channels. This means that 86 only a limited number of `spdk_thread`s will be able to acquire a channel. 87 Design software to deal with the inability to get a channel. 88 89 **Important**: The devices `PCI_DEVICE_ID_INTEL_DSA_SPR0` and `PCI_DEVICE_ID_INTEL_IAX_SPR0` 90 have been added to the vfio_pci denylist. This prevents the use of userspace DSA and IAA drivers 91 from SPDK on systems that are affected. 92 93 **Additionally**, to use the kernel DSA driver from SPDK, users are required to possess 94 the `CAP_SYS_RAWIO` capability. 95 96 #### How to use kernel idxd driver {#accel_idxd_kernel} 97 98 There are several dependencies to leverage the Linux idxd driver for driving DSA devices. 99 100 1 Linux kernel support: You need to have a Linux kernel with the `idxd` driver 101 loaded. Further, add the following command line options to the kernel boot 102 commands: 103 104 ```bash 105 intel_iommu=on,sm_on 106 ``` 107 108 2 User library dependency: Users need to install the developer version of the 109 `accel-config` library. This is often packaged, but the source is available on 110 [GitHub](https://github.com/intel/idxd-config). After the library is installed, 111 users can use the `accel-config` command to configure the work queues(WQs) of 112 the idxd devices managed by the kernel with the following steps: 113 114 Note: this library must be installed before you run `configure` 115 116 ```bash 117 accel-config disable-wq dsa0/wq0.1 118 accel-config disable-device dsa0 119 accel-config config-wq --group-id=0 --mode=dedicated --wq-size=128 --type=user --name="MyApp1" 120 --priority=10 --block-on-fault=1 dsa0/wq0.1 121 accel-config config-engine dsa0/engine0.0 --group-id=0 122 accel-config config-engine dsa0/engine0.1 --group-id=0 123 accel-config config-engine dsa0/engine0.2 --group-id=0 124 accel-config config-engine dsa0/engine0.3 --group-id=0 125 accel-config enable-device dsa0 126 accel-config enable-wq dsa0/wq0.1 127 ``` 128 129 DSA can be configured in many ways, but the above configuration is needed for use with SPDK. 130 Before you can run using the kernel driver you need to make sure that the hardware is bound 131 to the kernel driver and not VFIO. By default when you run `setup.sh` DSA devices will be 132 bound to VFIO. To exclude DSA devices, pass a whitespace separated list of DSA devices BDF 133 using the PCI_BLOCKED parameter as shown below. 134 135 ```bash 136 sudo PCI_BLOCKED="0000:04:00.0 0000:05:00.0" ./setup.sh 137 ``` 138 139 Note: you might need to run `sudo ./setup.sh reset` to unbind all drivers before performing 140 the step above. 141 142 ### Software Module {#accel_sw} 143 144 The software module is enabled by default. If no hardware module is explicitly 145 enabled via startup RPC as discussed earlier, the software module will use ISA-L 146 if available for functions such as CRC32C. Otherwise, standard glibc calls are 147 used to back the framework API. 148 149 ### dpdk_cryptodev {#accel_dpdk_cryptodev} 150 151 The dpdk_cryptodev module uses DPDK CryptoDev API to implement crypto operations. 152 The following ciphers and PMDs are supported: 153 154 - AESN-NI Multi Buffer Crypto Poll Mode Driver: RTE_CRYPTO_CIPHER_AES128_CBC 155 - Intel(R) QuickAssist (QAT) Crypto Poll Mode Driver: RTE_CRYPTO_CIPHER_AES128_CBC, 156 RTE_CRYPTO_CIPHER_AES128_XTS 157 (Note: QAT is functional however is marked as experimental until the hardware has 158 been fully integrated with the SPDK CI system.) 159 - MLX5 Crypto Poll Mode Driver: RTE_CRYPTO_CIPHER_AES256_XTS, RTE_CRYPTO_CIPHER_AES512_XTS 160 - UADK Crypto Poll Mode Driver: RTE_CRYPTO_CIPHER_AES256_XTS, RTE_CRYPTO_CIPHER_AES512_XTS 161 162 To enable this module, use [`dpdk_cryptodev_scan_accel_module`](https://spdk.io/doc/jsonrpc.html), 163 this RPC is available in STARTUP state and the SPDK application needs to be run with `--wait-for-rpc` 164 CLI parameter. To select a specific PMD, use [`dpdk_cryptodev_set_driver`](https://spdk.io/doc/jsonrpc.html) 165 166 ### Module to Operation Code Assignment {#accel_assignments} 167 168 When multiple modules are initialized, the accel framework will assign op codes to 169 modules by first assigning all op codes to the Software Module and then overriding 170 op code assignments to Hardware Modules in the order in which they were initialized. 171 The RPC `accel_get_opc_assignments` can be used at any time to see the current 172 assignment map including the names of valid operations. The RPC `accel_assign_opc` 173 can be used after initializing the desired Hardware Modules but before starting the 174 framework in the event that a specific override is desired. Note that to start an 175 application and send startup RPC's use the `--wait-for-rpc` parameter and then use the 176 `framework_start_init` RPC to continue. For example, assume the DSA Module is initialized 177 but for some reason the desire is to have the Software Module handle copies instead. 178 The following RPCs would accomplish the copy override: 179 180 ```bash 181 ./scripts/rpc.py dsa_scan_accel_module 182 ./scripts/rpc.py accel_assign_opc -o copy -m software 183 ./scripts/rpc.py framework_start_init 184 ./scripts/rpc.py accel_get_opc_assignments 185 { 186 "copy": "software", 187 "fill": "dsa", 188 "dualcast": "dsa", 189 "compare": "dsa", 190 "crc32c": "dsa", 191 "copy_crc32c": "dsa", 192 "compress": "software", 193 "decompress": "software" 194 } 195 ``` 196 197 To determine the name of available modules and their supported operations use the 198 RPC `accel_get_module_info`. 199