1# Acceleration Framework {#accel_fw} 2 3SPDK provides a framework for abstracting general acceleration capabilities 4that can be implemented through plug-in modules and low-level libraries. These 5plug-in modules include support for hardware acceleration engines such as 6the Intel(R) I/O Acceleration Technology (IOAT) engine and the Intel(R) Data 7Streaming Accelerator (DSA) engine. Additionally, a software plug-in module 8exists to enable use of the framework in environments without hardware 9acceleration capabilities. ISA/L is used for optimized CRC32C calculation within 10the software module. 11 12## Acceleration Framework Functions {#accel_functions} 13 14Functions implemented via the framework can be found in the DoxyGen documentation of the 15framework public header file here [accel.h](https://spdk.io/doc/accel_8h.html) 16 17## Acceleration Framework Design Considerations {#accel_dc} 18 19The general interface is defined by `/include/spdk/accel.h` and implemented 20in `/lib/accel`. These functions may be called by an SPDK application and in 21most cases, except where otherwise documented, are asynchronous and follow the 22standard SPDK model for callbacks with a callback argument. 23 24If the acceleration framework is started without initializing a hardware module, 25optimized software implementations of the operations will back the public API. All 26operations supported by the framework have a backing software implementation in 27the event that no hardware accelerators have been enabled for that operation. 28 29When multiple hardware modules are enabled the framework will assign each operation to 30a module based on the order in which it was initialized. So, for example if two modules are 31enabled, IOAT and software, the software module will be used for every operation except those 32supported by IOAT. 33 34## Acceleration Low Level Libraries {#accel_libs} 35 36Low level libraries provide only the most basic functions that are specific to 37the hardware. Low level libraries are located in the '/lib' directory with the 38exception of the software implementation which is implemented as part of the 39framework itself. The software low level library does not expose a public API. 40Applications may choose to interact directly with a low level library if there are 41specific needs/considerations not met via accessing the library through the 42framework/module. Note that when using the low level libraries directly, the 43framework abstracted interface is bypassed as the application will call the public 44functions exposed by the individual low level libraries. Thus, code written this 45way needs to be certain that the underlying hardware exists everywhere that it runs. 46 47The low level library for IOAT is located in `/lib/ioat`. The low level library 48for DSA and IAA is in `/lib/idxd` (IDXD stands for Intel(R) Data Acceleration Driver and 49supports both DSA and IAA hardware accelerators). In `/lib/idxd` folder, SPDK supports the ability 50to use either user space and kernel space drivers. The following describes each usage scenario: 51 52Leveraging user space idxd driver: The DSA devices are managed by the SPDK user space 53driver in a dedicated SPDK process, then the device cannot be shared by another 54process. The benefit of this usage is no kernel dependency. 55 56Leveraging kernel space driver: The DSA devices are managed by kernel 57space drivers. And the Work queues inside the DSA device can be shared among 58different processes. Naturally, it can be used in cloud native scenario. The drawback of 59this usage is the kernel dependency, i.e., idxd kernel driver must be supported and loaded 60in the kernel. 61 62## Acceleration Plug-In Modules {#accel_modules} 63 64Plug-in modules depend on low level libraries to interact with the hardware and 65add additional functionality such as queueing during busy conditions or flow 66control in some cases. The framework in turn depends on the modules to provide 67the complete implementation of the acceleration component. A module must be 68selected via startup RPC when the application is started. Otherwise, if no startup 69RPC is provided, the framework is available and will use the software plug-in module. 70 71### IOAT Module {#accel_ioat} 72 73To use the IOAT module, use the RPC [`ioat_scan_accel_module`](https://spdk.io/doc/jsonrpc.html) before starting the application. 74 75### DSA Module {#accel_dsa} 76 77The DSA module supports the DSA hardware and relies on the low level IDXD library. 78 79To use the DSA module, use the RPC 80[`dsa_scan_accel_module`](https://spdk.io/doc/jsonrpc.html). By default, this 81will attempt to load the SPDK user-space idxd driver. To use the built-in 82kernel driver on Linux, add the `-k` parameter. See the next section for 83details on using the kernel driver. 84 85The DSA hardware supports a limited queue depth and channels. This means that 86only a limited number of `spdk_thread`s will be able to acquire a channel. 87Design software to deal with the inability to get a channel. 88 89### How to use kernel idxd driver {#accel_idxd_kernel} 90 91There are several dependencies to leverage the Linux idxd driver for driving DSA devices. 92 931 Linux kernel support: You need to have a Linux kernel with the `idxd` driver 94loaded. Further, add the following command line options to the kernel boot 95commands: 96 97```bash 98intel_iommu=on,sm_on 99``` 100 1012 User library dependency: Users need to install the developer version of the 102`accel-config` library. This is often packaged, but the source is available on 103[GitHub](https://github.com/intel/idxd-config). After the library is installed, 104users can use the `accel-config` command to configure the work queues(WQs) of 105the idxd devices managed by the kernel with the following steps: 106 107Note: this library must be installed before you run `configure` 108 109```bash 110accel-config disable-wq dsa0/wq0.1 111accel-config disable-device dsa0 112accel-config config-wq --group-id=0 --mode=dedicated --wq-size=128 --type=user --name="MyApp1" 113 --priority=10 --block-on-fault=1 dsa0/wq0.1 114accel-config config-engine dsa0/engine0.0 --group-id=0 115accel-config config-engine dsa0/engine0.1 --group-id=0 116accel-config config-engine dsa0/engine0.2 --group-id=0 117accel-config config-engine dsa0/engine0.3 --group-id=0 118accel-config enable-device dsa0 119accel-config enable-wq dsa0/wq0.1 120``` 121 122DSA can be configured in many ways, but the above configuration is needed for use with SPDK. 123Before you can run using the kernel driver you need to make sure that the hardware is bound 124to the kernel driver and not VFIO. By default when you run `setup.sh` DSA devices will be 125bound to VFIO. To exclude DSA devices, pass a whitespace separated list of DSA devices BDF 126using the PCI_BLOCKED parameter as shown below. 127 128```bash 129sudo PCI_BLOCKED="0000:04:00.0 0000:05:00.0" ./setup.sh 130``` 131 132Note: you might need to run `sudo ./setup.sh reset` to unbind all drivers before performing 133the step above. 134 135### Software Module {#accel_sw} 136 137The software module is enabled by default. If no hardware module is explicitly 138enabled via startup RPC as discussed earlier, the software module will use ISA-L 139if available for functions such as CRC32C. Otherwise, standard glibc calls are 140used to back the framework API. 141 142### Module to Operation Code Assignment {#accel_assignments} 143 144When multiple modules are initialized, the accel framework will assign op codes to 145modules by first assigning all op codes to the Software Module and then overriding 146op code assignments to Hardware Modules in the order in which they were initialized. 147The RPC `accel_get_opc_assignments` can be used at any time to see the current 148assignment map including the names of valid operations. The RPC `accel_assign_opc` 149can be used after initializing the desired Hardware Modules but before starting the 150framework in the event that a specific override is desired. Note that to start an 151application and send startup RPC's use the `--wait-for-rpc` parameter and then use the 152`framework_start_init` RPC to continue. For example, assume the DSA Module is initialized 153but for some reason the desire is to have the Software Module handle copies instead. 154The following RPCs would accomplish the copy override: 155 156```bash 157./scripts/rpc.py dsa_scan_accel_module 158./scripts/rpc.py accel_assign_opc -o copy -m software 159./scripts/rpc.py framework_start_init 160./scripts/rpc.py accel_get_opc_assignments 161{ 162 "copy": "software", 163 "fill": "dsa", 164 "dualcast": "dsa", 165 "compare": "dsa", 166 "crc32c": "dsa", 167 "copy_crc32c": "dsa", 168 "compress": "software", 169 "decompress": "software" 170} 171``` 172 173To determine the name of available modules and their supported operations use the 174RPC `accel_get_module_info`. 175