1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2019 Intel Corporation 3 4Intel(R) FPGA 5GNR FEC Poll Mode Driver 5======================================= 6 7The BBDEV FPGA 5GNR FEC poll mode driver (PMD) supports an FPGA implementation of a VRAN 8LDPC Encode / Decode 5GNR wireless acceleration function, using Intel's PCI-e and FPGA 9based Vista Creek device. 10 11Features 12-------- 13 14FPGA 5GNR FEC PMD supports the following features: 15 16- LDPC Encode in the DL 17- LDPC Decode in the UL 18- 8 VFs per PF (physical device) 19- Maximum of 32 UL queues per VF 20- Maximum of 32 DL queues per VF 21- PCIe Gen-3 x8 Interface 22- MSI-X 23- SR-IOV 24 25FPGA 5GNR FEC PMD supports the following BBDEV capabilities: 26 27* For the LDPC encode operation: 28 - ``RTE_BBDEV_LDPC_CRC_24B_ATTACH`` : set to attach CRC24B to CB(s) 29 - ``RTE_BBDEV_LDPC_RATE_MATCH`` : if set then do not do Rate Match bypass 30 31* For the LDPC decode operation: 32 - ``RTE_BBDEV_LDPC_CRC_TYPE_24B_CHECK`` : check CRC24B from CB(s) 33 - ``RTE_BBDEV_LDPC_ITERATION_STOP_ENABLE`` : disable early termination 34 - ``RTE_BBDEV_LDPC_CRC_TYPE_24B_DROP`` : drops CRC24B bits appended while decoding 35 - ``RTE_BBDEV_LDPC_HQ_COMBINE_IN_ENABLE`` : provides an input for HARQ combining 36 - ``RTE_BBDEV_LDPC_HQ_COMBINE_OUT_ENABLE`` : provides an input for HARQ combining 37 - ``RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_IN_ENABLE`` : HARQ memory input is internal 38 - ``RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_OUT_ENABLE`` : HARQ memory output is internal 39 - ``RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_LOOPBACK`` : loopback data to/from HARQ memory 40 - ``RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_FILLERS`` : HARQ memory includes the fillers bits 41 42 43Limitations 44----------- 45 46FPGA 5GNR FEC does not support the following: 47 48- Scatter-Gather function 49 50 51Installation 52------------ 53 54Section 3 of the DPDK manual provides instructions on installing and compiling DPDK. 55 56DPDK requires hugepages to be configured as detailed in section 2 of the DPDK manual. 57The bbdev test application has been tested with a configuration 40 x 1GB hugepages. The 58hugepage configuration of a server may be examined using: 59 60.. code-block:: console 61 62 grep Huge* /proc/meminfo 63 64 65Initialization 66-------------- 67 68When the device first powers up, its PCI Physical Functions (PF) can be listed through this command: 69 70.. code-block:: console 71 72 sudo lspci -vd8086:0d8f 73 74The physical and virtual functions are compatible with Linux UIO drivers: 75``vfio`` and ``igb_uio``. However, in order to work the FPGA 5GNR FEC device firstly needs 76to be bound to one of these linux drivers through DPDK. 77 78 79Bind PF UIO driver(s) 80~~~~~~~~~~~~~~~~~~~~~ 81 82Install the DPDK igb_uio driver, bind it with the PF PCI device ID and use 83``lspci`` to confirm the PF device is under use by ``igb_uio`` DPDK UIO driver. 84 85The igb_uio driver may be bound to the PF PCI device using one of three methods: 86 87 881. PCI functions (physical or virtual, depending on the use case) can be bound to 89the UIO driver by repeating this command for every function. 90 91.. code-block:: console 92 93 insmod igb_uio.ko 94 echo "8086 0d8f" > /sys/bus/pci/drivers/igb_uio/new_id 95 lspci -vd8086:0d8f 96 97 982. Another way to bind PF with DPDK UIO driver is by using the ``dpdk-devbind.py`` tool 99 100.. code-block:: console 101 102 cd <dpdk-top-level-directory> 103 ./usertools/dpdk-devbind.py -b igb_uio 0000:06:00.0 104 105where the PCI device ID (example: 0000:06:00.0) is obtained using lspci -vd8086:0d8f 106 107 1083. A third way to bind is to use ``dpdk-setup.sh`` tool 109 110.. code-block:: console 111 112 cd <dpdk-top-level-directory> 113 ./usertools/dpdk-setup.sh 114 115 select 'Bind Ethernet/Crypto/Baseband device to IGB UIO module' 116 or 117 select 'Bind Ethernet/Crypto/Baseband device to VFIO module' depending on driver required 118 enter PCI device ID 119 select 'Display current Ethernet/Crypto/Baseband device settings' to confirm binding 120 121 122In the same way the FPGA 5GNR FEC PF can be bound with vfio, but vfio driver does not 123support SR-IOV configuration right out of the box, so it will need to be patched. 124 125 126Enable Virtual Functions 127~~~~~~~~~~~~~~~~~~~~~~~~ 128 129Now, it should be visible in the printouts that PCI PF is under igb_uio control 130"``Kernel driver in use: igb_uio``" 131 132To show the number of available VFs on the device, read ``sriov_totalvfs`` file.. 133 134.. code-block:: console 135 136 cat /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_totalvfs 137 138 where 0000\:<b>\:<d>.<f> is the PCI device ID 139 140 141To enable VFs via igb_uio, echo the number of virtual functions intended to 142enable to ``max_vfs`` file.. 143 144.. code-block:: console 145 146 echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/max_vfs 147 148 149Afterwards, all VFs must be bound to appropriate UIO drivers as required, same 150way it was done with the physical function previously. 151 152Enabling SR-IOV via vfio driver is pretty much the same, except that the file 153name is different: 154 155.. code-block:: console 156 157 echo <num-of-vfs> > /sys/bus/pci/devices/0000\:<b>\:<d>.<f>/sriov_numvfs 158 159 160Configure the VFs through PF 161~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 162 163The PCI virtual functions must be configured before working or getting assigned 164to VMs/Containers. The configuration involves allocating the number of hardware 165queues, priorities, load balance, bandwidth and other settings necessary for the 166device to perform FEC functions. 167 168This configuration needs to be executed at least once after reboot or PCI FLR and can 169be achieved by using the function ``rte_fpga_5gnr_fec_configure()``, which sets up the 170parameters defined in ``rte_fpga_5gnr_fec_conf`` structure: 171 172.. code-block:: c 173 174 struct rte_fpga_5gnr_fec_conf { 175 bool pf_mode_en; 176 uint8_t vf_ul_queues_number[FPGA_5GNR_FEC_NUM_VFS]; 177 uint8_t vf_dl_queues_number[FPGA_5GNR_FEC_NUM_VFS]; 178 uint8_t ul_bandwidth; 179 uint8_t dl_bandwidth; 180 uint8_t ul_load_balance; 181 uint8_t dl_load_balance; 182 uint16_t flr_time_out; 183 }; 184 185- ``pf_mode_en``: identifies whether only PF is to be used, or the VFs. PF and 186 VFs are mutually exclusive and cannot run simultaneously. 187 Set to 1 for PF mode enabled. 188 If PF mode is enabled all queues available in the device are assigned 189 exclusively to PF and 0 queues given to VFs. 190 191- ``vf_*l_queues_number``: defines the hardware queue mapping for every VF. 192 193- ``*l_bandwidth``: in case of congestion on PCIe interface. The device 194 allocates different bandwidth to UL and DL. The weight is configured by this 195 setting. The unit of weight is 3 code blocks. For example, if the code block 196 cbps (code block per second) ratio between UL and DL is 12:1, then the 197 configuration value should be set to 36:3. The schedule algorithm is based 198 on code block regardless the length of each block. 199 200- ``*l_load_balance``: hardware queues are load-balanced in a round-robin 201 fashion. Queues get filled first-in first-out until they reach a pre-defined 202 watermark level, if exceeded, they won't get assigned new code blocks.. 203 This watermark is defined by this setting. 204 205 If all hardware queues exceeds the watermark, no code blocks will be 206 streamed in from UL/DL code block FIFO. 207 208- ``flr_time_out``: specifies how many 16.384us to be FLR time out. The 209 time_out = flr_time_out x 16.384us. For instance, if you want to set 10ms for 210 the FLR time out then set this setting to 0x262=610. 211 212 213An example configuration code calling the function ``rte_fpga_5gnr_fec_configure()`` is shown 214below: 215 216.. code-block:: c 217 218 struct rte_fpga_5gnr_fec_conf conf; 219 unsigned int i; 220 221 memset(&conf, 0, sizeof(struct rte_fpga_5gnr_fec_conf)); 222 conf.pf_mode_en = 1; 223 224 for (i = 0; i < FPGA_5GNR_FEC_NUM_VFS; ++i) { 225 conf.vf_ul_queues_number[i] = 4; 226 conf.vf_dl_queues_number[i] = 4; 227 } 228 conf.ul_bandwidth = 12; 229 conf.dl_bandwidth = 5; 230 conf.dl_load_balance = 64; 231 conf.ul_load_balance = 64; 232 233 /* setup FPGA PF */ 234 ret = rte_fpga_5gnr_fec_configure(info->dev_name, &conf); 235 TEST_ASSERT_SUCCESS(ret, 236 "Failed to configure 4G FPGA PF for bbdev %s", 237 info->dev_name); 238 239 240Test Application 241---------------- 242 243BBDEV provides a test application, ``test-bbdev.py`` and range of test data for testing 244the functionality of FPGA 5GNR FEC encode and decode, depending on the device's 245capabilities. The test application is located under app->test-bbdev folder and has the 246following options: 247 248.. code-block:: console 249 250 "-p", "--testapp-path": specifies path to the bbdev test app. 251 "-e", "--eal-params" : EAL arguments which are passed to the test app. 252 "-t", "--timeout" : Timeout in seconds (default=300). 253 "-c", "--test-cases" : Defines test cases to run. Run all if not specified. 254 "-v", "--test-vector" : Test vector path (default=dpdk_path+/app/test-bbdev/test_vectors/bbdev_null.data). 255 "-n", "--num-ops" : Number of operations to process on device (default=32). 256 "-b", "--burst-size" : Operations enqueue/dequeue burst size (default=32). 257 "-l", "--num-lcores" : Number of lcores to run (default=16). 258 "-i", "--init-device" : Initialise PF device with default values. 259 260 261To execute the test application tool using simple decode or encode data, 262type one of the following: 263 264.. code-block:: console 265 266 ./test-bbdev.py -c validation -n 64 -b 1 -v ./ldpc_dec_default.data 267 ./test-bbdev.py -c validation -n 64 -b 1 -v ./ldpc_enc_default.data 268 269 270The test application ``test-bbdev.py``, supports the ability to configure the PF device with 271a default set of values, if the "-i" or "- -init-device" option is included. The default values 272are defined in test_bbdev_perf.c as: 273 274- VF_UL_QUEUE_VALUE 4 275- VF_DL_QUEUE_VALUE 4 276- UL_BANDWIDTH 3 277- DL_BANDWIDTH 3 278- UL_LOAD_BALANCE 128 279- DL_LOAD_BALANCE 128 280- FLR_TIMEOUT 610 281 282 283Test Vectors 284~~~~~~~~~~~~ 285 286In addition to the simple LDPC decoder and LDPC encoder tests, bbdev also provides 287a range of additional tests under the test_vectors folder, which may be useful. The results 288of these tests will depend on the FPGA 5GNR FEC capabilities. 289 290 291Alternate Baseband Device configuration tool 292~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 293 294On top of the embedded configuration feature supported in test-bbdev using "- -init-device" 295option, there is also a tool available to perform that device configuration using a companion 296application. 297The ``pf_bb_config`` application notably enables then to run bbdev-test from the VF 298and not only limited to the PF as captured above. 299 300See for more details: https://github.com/intel/pf-bb-config 301 302Specifically for the BBDEV FPGA 5GNR FEC PMD, the command below can be used: 303 304.. code-block:: console 305 306 ./pf_bb_config FPGA_5GNR -c fpga_5gnr/fpga_5gnr_config_vf.cfg 307 ./test-bbdev.py -e="-c 0xff0 -w${VF_PCI_ADDR}" -c validation -n 64 -b 32 -l 1 -v ./ldpc_dec_default.data 308