1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2018-2022 Intel Corporation. 3 4IFPGA Rawdev Driver 5====================== 6 7FPGA is used more and more widely in Cloud and NFV, one primary reason is 8that FPGA not only provides ASIC performance but also it's more flexible 9than ASIC. 10 11FPGA uses Partial Reconfigure (PR) Parts of Bit Stream to achieve its 12flexibility. That means one FPGA Device Bit Stream is divided into many Parts 13of Bit Stream(each Part of Bit Stream is defined as AFU-Accelerated Function 14Unit), and each AFU is a hardware acceleration unit which can be dynamically 15reloaded respectively. 16 17By PR (Partial Reconfiguration) AFUs, one FPGA resources can be time-shared by 18different users. FPGA hot upgrade and fault tolerance can be provided easily. 19 20The SW IFPGA Rawdev Driver (**ifpga_rawdev**) provides a Rawdev driver 21that utilizes Intel FPGA Software Stack OPAE(Open Programmable Acceleration 22Engine) for FPGA management. 23 24Implementation details 25---------------------- 26 27Each instance of IFPGA Rawdev Driver is probed by Intel FpgaDev. In coordination 28with OPAE share code IFPGA Rawdev Driver provides common FPGA management ops 29for FPGA operation, OPAE provides all following operations: 30- FPGA PR (Partial Reconfiguration) management 31- FPGA AFUs Identifying 32- FPGA Thermal Management 33- FPGA Power Management 34- FPGA Performance reporting 35- FPGA Remote Debug 36 37All configuration parameters are taken by vdev_ifpga_cfg driver. Besides 38configuration, vdev_ifpga_cfg driver also hot plugs in IFPGA Bus. 39 40All of the AFUs of one FPGA may share same PCI BDF and AFUs scan depend on 41IFPGA Rawdev Driver so IFPGA Bus takes AFU device scan and AFU drivers probe. 42All AFU device driver bind to AFU device by its UUID (Universally Unique 43Identifier). 44 45To avoid unnecessary code duplication and ensure maximum performance, 46handling of AFU devices is left to different PMDs; all the design as 47summarized by the following block diagram:: 48 49 +---------------------------------------------------------------+ 50 | Application(s) | 51 +----------------------------.----------------------------------+ 52 | 53 | 54 +----------------------------'----------------------------------+ 55 | DPDK Framework (APIs) | 56 +----------|------------|--------.---------------------|--------+ 57 / \ | 58 / \ | 59 +-------'-------+ +-------'-------+ +--------'--------+ 60 | Eth PMD | | Crypto PMD | | | 61 +-------.-------+ +-------.-------+ | | 62 | | | | 63 | | | | 64 +-------'-------+ +-------'-------+ | IFPGA | 65 | Eth AFU Dev | |Crypto AFU Dev | | Rawdev Driver | 66 +-------.-------+ +-------.-------+ |(OPAE Share Code)| 67 | | | | 68 | | Rawdev | | 69 +-------'------------------'-------+ Ops | | 70 | IFPGA Bus | -------->| | 71 +-----------------.----------------+ +--------.--------+ 72 | | 73 Hot-plugin -->| | 74 | | 75 +-----------------'------------------+ +--------'--------+ 76 | vdev_ifpga_cfg driver | | Intel FpgaDev | 77 +------------------------------------+ +-----------------+ 78 79 80Run-time parameters 81------------------- 82 83This driver is invoked automatically in systems added with Intel FPGA, 84but PR and IFPGA Bus scan is triggered by command line using 85``--vdev 'ifpga_rawdev_cfg`` EAL option. 86 87The following device parameters are supported: 88 89- ``ifpga`` [string] 90 91 Provide a specific Intel FPGA device PCI BDF. Can be provided multiple 92 times for additional instances. 93 94- ``port`` [int] 95 96 Each FPGA can provide many channels to PR AFU by software, each channels 97 is identified by this parameter. 98 99- ``afu_bts`` [string] 100 101 If null, the AFU Bit Stream has been PR in FPGA, if not forces PR and 102 identifies AFU Bit Stream file. 103 104 105IFPGA AFU Driver 106================ 107 108AFU (Acceleration Function Unit) is a function or set of functions 109that perform various acceleration task on FPGA platform. 110The image of AFU is called as GBS (Green Bit Stream) 111which can be used by PR (Partial Reconfigure) tool to load into the FPGA, 112different AFUs can be dynamically reloaded respectively. 113 114AFU has two main communication paths between the host: 115 116- FPGA to host transactions 117 118 The FPGA accesses host memory using a 512 bits data path. 119 This data path has separate channels for read and write traffic 120 allowing for simultaneous read and write to occur. 121 The read and write channels support bursts of 1, 2, and 4 cache lines. 122 123- Host to FPGA (MMIO) transactions 124 125 The host can access a 256 KB address space within the FPGA. 126 This address space contains Device Feature Header (DFHs) 127 and the control and status registers of the AFU hardware. 128 129AFU must implement the following registers: 130 131- AFU DFH - a 64-bit header at MMIO address offset 0x0 132 133- AFU ID - a 128-bit UUID at MMIO address offset 0x2 134 135The AFU is enumerated and recorded by IFPGA Rawdev Driver. 136Then AFU devices are created with the help of IFPGA Bus Driver, 137AFU driver probe these AFU devices and expose them 138as standard raw devices for application to access. 139 140Implementation details 141---------------------- 142 143IFPGA Rawdev Driver identifies AFU in FPGA, AFU location (PF/VF address) 144and UUID are taken by ``ifpga_rawdev_cfg`` vdev driver 145which hot plug AFU into IFPGA Bus. 146 147IFPGA Bus takes AFU device scan and AFU driver probe. 148All AFU device driver bind to AFU device by its dedicated UUID. 149To avoid unnecessary code duplication and ensure maximum performance, 150AFU driver implements the common part of raw device driver. 151Several specific AFU drivers are provided for reference. 152The design is summarized by the following block diagram:: 153 154 +---------------------------------------------------------------+ 155 | Application(s) | 156 +----------------------------.----------------------------------+ 157 | 158 +----------------------------'----------------------------------+ 159 | DPDK Framework (Rawdev APIs) | 160 +-----------------+------------------------------------+--------+ 161 | | 162 +-----------------'----------------+ | 163 | IFPGA AFU Driver | +--------'--------+ 164 | | | | 165 |+---------------+ +--------------+| | | 166 || AFU Dev1 PMD | | AFU Dev2 PMD || | | 167 |+-------+-------+ +-------+------+| | | 168 +--------|-----------------|-------+ | | 169 | | | | 170 +--------'------+ +-------'-------+ | IFPGA | 171 | AFU Dev1 | | AFU Dev2 | | Rawdev Driver | 172 +-------.-------+ +-------.-------+ | | 173 | | Rawdev | | 174 +-------'------------------'-------+ Ops | | 175 | IFPGA Bus |--------->| | 176 +-----------------.----------------+ +--------.--------+ 177 | | 178 Hot-plugin -->| | 179 | | 180 +-----------------'------------------+ +--------'--------+ 181 | ifpga_rawdev_cfg vdev driver | | Intel FpgaDev | 182 +------------------------------------+ +-----------------+ 183 184How to test AFU function 185------------------------ 186 187Suppose AFU is found in FPGA at PCI address 31:00.0, 188then you can create and test a AFU device by following steps in application. 189 190#. rte_vdev_init("ifpga_rawdev_cfg0", "ifpga=31:00.0,port=0") 191 192#. rawdev = rte_rawdev_pmd_get_named_dev("afu_0|31:00.0") 193 194#. rte_rawdev_configure(rawdev->dev_id, &cfg, sizeof(cfg)) 195 196#. rte_rawdev_selftest(rawdev->dev_id) 197 198#. rte_vdev_uninit("ifpga_rawdev_cfg0") 199 200AFU device name format used in ``rte_rawdev_pmd_get_named_dev`` is ``afu_[port]|[BDF]``. 201Please refer to OPAE documentation for the meaning of port. 202Each AFU device has specific configuration data, they are defined in ``rte_pmd_afu.h``. 203 204 205Open FPGA Stack 206===================== 207 208Open FPGA Stack (OFS) is a collection of RTL and open source software providing 209interfaces to access the instantiated RTL easily in an FPGA. OFS leverages the 210DFL for the implementation of the FPGA RTL design. 211 212OFS designs allow for the arrangement of software interfaces across multiple 213PCIe endpoints. Some of these interfaces may be PFs defined in the static region 214that connect to interfaces in an IP that is loaded via Partial Reconfiguration (PR). 215And some of these interfaces may be VFs defined in the PR region that can be 216reconfigured by the end-user. Furthermore, these PFs/VFs may use DFLs such that 217features may be discovered and accessed in user space with the aid of a generic 218kernel driver like vfio-pci. The diagram below depicts an example design with one 219PF and two VFs. In this example, it will export the management functions via PF0 220and acceleration functions via VF0 and VF1, leverage VFIO to export the MMIO space 221to an application.:: 222 223 +-----------------+ +-------------+ +------------+ 224 | FPGA Management | | DPDK App | | User App | 225 | App | | | | | 226 +--------+--------+ +------+------+ +-----+------+ 227 | | | 228 +--------+--------+ +------+------+ | 229 | IFPGA PMD | | AFU PMD | | 230 +--------+--------+ +------+------+ | 231 | | | 232 +--------+------------------+---------------+------+ 233 | VFIO-PCI | 234 +--------+------------------+---------------+------+ 235 | | | 236 +--------+--------+ +------+------+ +-----+------+ 237 | PF0 | | PF0_VF0 | | PF0_VF1 | 238 +-----------------+ +-------------+ +------------+ 239 240As accelerators are specialized hardware, they are typically limited in the 241number installed in a given system. Many use cases require them to be shared 242across multiple software contexts or threads of software execution, either 243through partitioning of individual dedicated resources, or virtualization of 244shared resources. OFS provides several models to share the AFU resources via 245PR mechanism and hardware-based virtualization schemes. 246 247#. Legacy model. 248 With legacy model FPGA cards like Intel PAC N3000 or N5000, there is 249 a notion that the boundary between the AFU and the shell is also the unit of 250 PR for those FPGA platforms. This model is only able to handle a 251 single context, because it only has one PR engine, and one PR region which 252 has an associated Port device. 253 254#. Multiple VFs per PR slot. 255 In this model, available AFU resources may allow instantiation of many VFs 256 which have a dedicated PCIe function with their own dedicated MMIO space, or 257 partition a region of MMIO space on a single PCIe function. Intel PAC N6000 258 card has implemented this model. 259 In this model, the AFU/PR slot was not connected to port device. For DFL's view, 260 the Next_AFU pointer in FIU feature header of port device points to NULL in this 261 model. On the other hand, each VF can start with an AFU feature header without 262 being connected to a FIU Port feature header. 263 264The VFs are created through the Linux kernel driver before we use them in DPDK. 265 266OFS provides the diversity for accessing the AFU resource to RTL developer. 267An IP designer may choose to add more than one PF for interfacing with IP 268on the FPGA and choose different model to access the AFU resource. 269 270There is one reference architecture design using the "Multiple VFs per PR slot" 271model for OFS as illustrated below. In this reference design, it exports the 272FPGA management functions via PF0. PF1 will bind with DPDK virtio driver 273presenting itself as a network interface to the application. PF2 will bind to the 274vfio-pci driver allowing the user space software to discover and interface 275with the specific workload like diagnostic test. It leverages AFU PMD driver to 276access the AFU resources in DPDK.:: 277 278 +----------------------+ 279 | PF/VF mux/demux | 280 +--+--+-----+------+-+-+ 281 | | | | | 282 +------------------------+ | | | | 283 PF0 | +---------+ +-+ | | 284 +---+---+ | +---+----+ | | 285 | DFH | | | DFH | | | 286 +-------+ +-----+----+ +--------+ | | 287 | FME | | VirtIO | | Test | | | 288 +---+---+ +----------+ +--------+ | | 289 | PF1 PF2 | | 290 | | | 291 | +----------+ | 292 | | ++ 293 | | | 294 | | PF0_VF0 | PF0_VF1 295 | +-----------------+-----------+------------+ 296 | | +-----+-----------+--------+ | 297 | | | | | | | 298 | | +------+ | +--+ -+ +--+---+ | | 299 | | | Port | | | DFH | | DFH | | | 300 +-----------+ +------+ | +-----+ +------+ | | 301 | | | DEV | | DEV | | | 302 | | +-----+ +------+ | | 303 | | PR Slot | | 304 | +--------------------------+ | 305 | Port Gasket | 306 +------------------------------------------+ 307