1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2015-2017 Netronome Systems, Inc. All rights reserved. 3 Copyright(c) 2021 Corigine, Inc. All rights reserved. 4 5NFP poll mode driver library 6============================ 7 8Netronome and Corigine's sixth generation of flow processors pack 216 9programmable cores and over 100 hardware accelerators that uniquely combine 10packet, flow, security and content processing in a single device that scales 11up to 400-Gb/s. 12 13This document explains how to use DPDK with the Network Flow Processor (NFP) 14Poll Mode Driver (PMD) supporting Netronome and Corigine's NFP-6xxx, NFP-4xxx 15and NFP-38xx product lines. 16 17NFP is a SR-IOV capable device and the PMD supports the physical 18function (PF) and the virtual functions (VFs). 19 20Dependencies 21------------ 22 23Before using the NFP DPDK PMD some NFP configuration, 24which is not related to DPDK, is required. The system requires 25installation of the **nfp-bsp (Board Support Package)** along 26with a specific NFP firmware application. The NSP ABI 27version should be 0.20 or higher. 28 29If you have a NFP device you should already have the documentation to perform 30this configuration. Contact **support@netronome.com** (for Netronome products) 31or **smartnic-support@corigine.com** (for Corigine products) to obtain the 32latest available firmware. 33 34The NFP Linux netdev kernel driver for VFs has been a part of the 35vanilla kernel since kernel version 4.5, and support for the PF 36since kernel version 4.11. Support for older kernels can be obtained 37on Github at 38**https://github.com/Netronome/nfp-drv-kmods** along with the build 39instructions. 40 41NFP PMD needs to be used along with UIO ``igb_uio`` or VFIO (``vfio-pci``) 42Linux kernel driver. 43 44Building the software 45--------------------- 46 47The NFP PMD code is provided in the **drivers/net/nfp** directory. Although 48NFP PMD has BSP dependencies, it is possible to compile it along with other 49DPDK PMDs even if no BSP was installed previously. 50Of course, a DPDK app will require such a BSP installed for using the 51NFP PMD, along with a specific NFP firmware application. 52 53Once the DPDK is built all the DPDK apps and examples include support for 54the NFP PMD. 55 56 57Driver compilation and testing 58------------------------------ 59 60Refer to the document 61:ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` for details. 62 63Using the PF 64------------ 65 66The PMD PF has extra work to do which will delay the DPDK app initialization 67like uploading the firmware and configure the Link state properly when starting 68or stopping a PF port. Since DPDK 18.05 the firmware upload happens when 69a PF is initialized, which was not always true with older DPDK versions. 70 71Depending on the product installed in the system, firmware files should be 72available under ``/lib/firmware/netronome``. DPDK PMD supporting the PF looks 73for a firmware file in this order: 74 75 1) First try to find a firmware image specific for this device using the 76 NFP serial number: 77 78 serial-00-15-4d-12-20-65-10-ff.nffw 79 80 2) Then try the PCI name: 81 82 pci-0000:04:00.0.nffw 83 84 3) Finally try the card type and media: 85 86 nic_AMDA0099-0001_2x25.nffw 87 88Netronome and Corigine's software packages install firmware files under 89``/lib/firmware/netronome`` to support all the Netronome and Corigine SmartNICs 90and different firmware applications. This is usually done using file names 91based on SmartNIC type and media and with a directory per firmware application. 92Options 1 and 2 for firmware filenames allow more than one SmartNIC, same type 93of SmartNIC or different ones, and to upload a different firmware to each 94SmartNIC. 95 96 .. Note:: 97 Currently the NFP PMD supports using the PF with Agilio Firmware with 98 NFD3 and Agilio Firmware with NFDk. See 99 `Netronome Support <https://help.netronome.com/support/solutions>`_. 100 for more information on the various firmwares supported by the Netronome 101 Agilio SmartNIC range, or 102 `Corigine Support <https://www.corigine.com/productsOverviewList-30.html>`_. 103 for more information about Corigine's range. 104 105PF multiport support 106-------------------- 107 108The NFP PMD can work with up to 8 ports on the same PF device. The number of 109available ports is firmware and hardware dependent, and the driver looks for a 110firmware symbol during initialization to know how many can be used. 111 112DPDK apps work with ports, and a port is usually a PF or a VF PCI device. 113However, with the NFP PF multiport there is just one PF PCI device. Supporting 114this particular configuration requires the PMD to create ports in a special 115way, although once they are created, DPDK apps should be able to use them as 116normal PCI ports. 117 118NFP ports belonging to the same PF can be seen inside PMD initialization with a 119suffix added to the PCI ID: wwww:xx:yy.z_portn. For example, a PF with PCI ID 1200000:03:00.0 and four ports is seen by the PMD code as: 121 122 .. code-block:: console 123 124 0000:03:00.0_port0 125 0000:03:00.0_port1 126 0000:03:00.0_port2 127 0000:03:00.0_port3 128 129 .. Note:: 130 131 There are some limitations with multiport support: RX interrupts and 132 device hot-plugging are not supported. 133 134PF multiprocess support 135----------------------- 136 137The NFP PMD supports the PF multiprocess. 138Having basic multiprocess support is important for allowing development 139and debugging through the PF using a secondary process, 140which will create a CPP bridge for user space tools accessing the NFP. 141 142System configuration 143-------------------- 144 145#. **Enable SR-IOV on the NFP device:** The current NFP PMD supports the PF and 146 the VFs on a NFP device. However, it is not possible to work with both at 147 the same time when using the ``nfp`` Linux netdev kernel driver. If the PF 148 is bound to the ``nfp`` kernel module, and VFs are created, the VFs may be 149 bound to the ``vfio-pci`` kernel module. It is also possible to bind the PF 150 to the ``vfio-pci`` kernel module, and create VFs afterwards. This requires 151 loading the ``vfio-pci`` module with the following parameters: 152 153 .. code-block:: console 154 155 modprobe vfio-pci enable_sriov=1 disable_idle_d3=1 156 157 VFs need to be enabled before they can be used with the PMD. Before enabling 158 the VFs it is useful to obtain information about the current NFP PCI device 159 detected by the system. This can be done on Netronome SmartNICs using: 160 161 .. code-block:: console 162 163 lspci -d 19ee: 164 165 and on Corigine SmartNICs using: 166 167 .. code-block:: console 168 169 lspci -d 1da8: 170 171 Now, for example, to configure two virtual functions on a NFP device 172 whose PCI system identity is "0000:03:00.0": 173 174 .. code-block:: console 175 176 echo 2 > /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs 177 178 The result of this command may be shown using lspci again on Netronome 179 SmartNICs: 180 181 .. code-block:: console 182 183 lspci -kd 19ee: 184 185 and on Corigine SmartNICs: 186 187 .. code-block:: console 188 189 lspci -kd 1da8: 190 191 Two new PCI devices should appear in the output of the above command. The 192 -k option shows the device driver, if any, that the devices are bound to. 193 Depending on the modules loaded, at this point the new PCI devices may be 194 bound to the ``nfp`` kernel driver or ``vfio-pci``. 195 196 197Flow offload 198------------ 199 200Using the flower firmware application, some types of Netronome or Corigine 201SmartNICs can offload the flows onto the cards. 202 203The flower firmware application requires the PMD running two services: 204 205 * PF vNIC service: handling the feedback traffic. 206 * ctrl vNIC service: communicate between PMD and firmware through 207 control messages. 208 209To achieve the offload of flow, the representor ports are exposed to OVS. 210The flower firmware application supports representor port for VF and physical 211port. There will always exist a representor port for each physical port, 212and the number of the representor port for VF is specified by the user through 213a parameter. 214 215In the Rx direction, the flower firmware application will prepend the input 216port information into metadata for each packet which can't offloaded. The PF 217vNIC service will keep polling packets from the firmware, and multiplex them 218to the corresponding representor port. 219 220In the Tx direction, the representor port will prepend the output port 221information into metadata for each packet, and then send it to the firmware 222through the PF vNIC. 223 224The ctrl vNIC service handles various control messages, for example, the 225creation and configuration of representor port, the pattern and action of flow 226rules, the statistics of flow rules, etc. 227 228NFP devargs 229----------- 230 231- ``force_reload_fw`` (default **0**) 232 233 The NFP PF PMD supports force reload the firmware 234 and ignore the firmware version. 235 For example, user can force a PF with PCI ID 0000:af:00.0 reload firmware by: 236 237 .. code-block:: console 238 239 -a af:00.0,force_reload_fw=1 -- -i 240 241 .. note:: 242 243 This parameter can be ignored in some case. 244 245 For example: when using a 2-port NFP card and both with this reload 246 firmware option, only the first one will cause the firmware reload 247 and the second one will be ignored. 248 249- ``cpp_service_enable`` (default **0**) 250 251 The NFP PF PMD supports enable CPP service. 252 For example, user let a PF with PCI ID 0000:af:00.0 enable CPP service by: 253 254 .. code-block:: console 255 256 -a af:00.0,cpp_service_enable=1 -- -i 257 258Metadata Format 259--------------- 260 261The NFP packet metadata format 262 263NFD3 264~~~~ 265 266The packet metadata starts with a field type header that can contain up-to 2678 4-bit datatype specifiers (32-bits in total). This is followed by up to 8 26832-bit words of data for each field described in the header. And directly 269following the metadata (header and data) comes the packet. 270 271The order of type is correspond with the data, but the nums of data field are 272decided by the corresponding type, if the type need N data field, it need to 273be wrote N times in the heads. 274:: 275 276 3 2 1 0 277 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 278 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 279 | Type7 | Type6 | Type5 | Type4 | Type3 | Type2 | Type1 | Type0 | 280 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 281 | Data for field 0 | 282 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 283 | Data for field 1 | 284 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 285 | Data for field 2 | 286 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 287 | Data for field 3 | 288 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 289 | Data for field 4 | 290 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 291 | Data for field 5 | 292 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 293 | Data for field 6 | 294 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 295 | Data for field 7 | 296 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 297 | Packet Data | 298 | ... | 299 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 300 301NFDk 302~~~~ 303 304The packet metadata starts with a field type header that can contain 8 bit 305metadata length and 6 4-bit datatype specifiers (32-bits in total). This is 306followed by up to 6 32-bit words of data for each field described in the 307header. And directly following the metadata (header and data) comes the 308packet. 309 310The order of type is correspond with the data, but the nums of data field are 311decided by the corresponding type, if the type need N data field, it need to 312be wrote N times in the heads. It is the same with NFD3. 313:: 314 315 3 2 1 0 316 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 317 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 318 | Type5 | Type4 | Type3 | Type2 | Type1 | Type0 |metadata length| 319 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 320 | Data for field 0 | 321 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 322 | Data for field 1 | 323 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 324 | Data for field 2 | 325 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 326 | Data for field 3 | 327 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 328 | Data for field 4 | 329 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 330 | Data for field 5 | 331 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 332 | Packet Data | 333 | ... | 334 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 335 336There are two classes of metadata one for ingress and one for egress. In each 337class the supported NFP types are: 338 339RX 340~~ 341 342NFP_NET_META_HASH 343The hash type is 4 bit which is next field type after NFP_NET_META_HASH in 344the header. The hash value is 32 bit which need 1 data field. 345:: 346 347 ----------------------------------------------------------------- 348 3 2 1 0 349 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 350 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 351 | Hash value | 352 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 353 354NFP_NET_META_VLAN 355Metadata with L2 (1W/4B) 356:: 357 358 ---------------------------------------------------------------- 359 3 2 1 0 360 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 361 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 362 |f| reserved | tpid| PCP |p| vlan outermost VID | 363 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 364 ^ ^ 365 NOTE: | TCI | 366 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 367 f 0 -> not stripping 368 1 -> stripping 369 370 tpid 0 -> RTE_ETHER_TYPE_VLAN 0x8100 IEEE 802.1Q VLAN tagging 371 1 -> RTE_ETHER_TYPE_QINQ 0x88a8 IEEE 802.1ad QINQ tagging 372 Tpid just be stored, now we don't handle it 373 374 The vlan[0] is the innermost VLAN 375 The vlan[1] is the QinQ info 376 377NFP_NET_META_IPSEC 378The IPsec type requires 4 bit. 379The SA index value is 32 bit which need 1 data field. 380:: 381 382 ---------------------------------------------------------------- 383 3 2 1 0 384 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 385 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 386 | sa_idx | 387 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 388 389TX 390~~ 391 392NFP_NET_META_VLAN 393:: 394 395 ----------------------------------------------------------------- 396 3 2 1 0 397 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 398 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 399 | TPID | PCP |p| vlan outermost VID | 400 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 401 ^ ^ 402 NOTE: | TCI | 403 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 404 405NFP_NET_META_IPSEC 406The IPsec type requires 12 bit, because it requires three data fields. 407:: 408 409 ---------------------------------------------------------------- 410 3 2 1 0 411 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 412 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 413 | sa_idx | 414 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 415 | nfp_ipsec_force_seq_low | 416 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 417 | nfp_ipsec_force_seq_hi | 418 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 419 420 The sa_idx is 32 bit which need 1 data field. 421 The nfp_ipsec_force_seq_low & nfp_ipsec_force_seq_hi is Anti-re-anti-count, 422 which is 64 bit need two data fields. 423