1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright (c) 2015-2021 Atomic Rules LLC 3 All rights reserved. 4 5ARK Poll Mode Driver 6==================== 7 8The ARK PMD is a DPDK poll-mode driver for the Atomic Rules Arkville 9(ARK) family of devices. 10 11More information can be found at the `Atomic Rules website 12<http://atomicrules.com>`_. 13 14Overview 15-------- 16 17The Atomic Rules Arkville product is DPDK and AXI compliant product 18that marshals packets across a PCIe conduit between host DPDK mbufs and 19FPGA AXI streams. 20 21The ARK PMD, and the spirit of the overall Arkville product, 22has been to take the DPDK API/ABI as a fixed specification; 23then implement much of the business logic in FPGA RTL circuits. 24The approach of *working backwards* from the DPDK API/ABI and having 25the GPP host software *dictate*, while the FPGA hardware *copes*, 26results in significant performance gains over a naive implementation. 27 28While this document describes the ARK PMD software, it is helpful to 29understand what the FPGA hardware is and is not. The Arkville RTL 30component provides a single PCIe Physical Function (PF) supporting 31some number of RX/Ingress and TX/Egress Queues. The ARK PMD controls 32the Arkville core through a dedicated opaque Core BAR (CBAR). 33To allow users full freedom for their own FPGA application IP, 34an independent FPGA Application BAR (ABAR) is provided. 35 36One popular way to imagine Arkville's FPGA hardware aspect is as the 37FPGA PCIe-facing side of a so-called Smart NIC. The Arkville core does 38not contain any MACs, and is link-speed independent, as well as 39agnostic to the number of physical ports the application chooses to 40use. The ARK driver exposes the familiar PMD interface to allow packet 41movement to and from mbufs across multiple queues. 42 43However FPGA RTL applications could contain a universe of added 44functionality that an Arkville RTL core does not provide or can 45not anticipate. To allow for this expectation of user-defined 46innovation, the ARK PMD provides a dynamic mechanism of adding 47capabilities without having to modify the ARK PMD. 48 49The ARK PMD is intended to support all instances of the Arkville 50RTL Core, regardless of configuration, FPGA vendor, or target 51board. While specific capabilities such as number of physical 52hardware queue-pairs are negotiated; the driver is designed to 53remain constant over a broad and extendable feature set. 54 55* FPGA Vendors Supported: AMD/Xilinx and Intel 56* Number of RX/TX Queue-Pairs: up to 128 57* PCIe Endpoint Technology: Gen3, Gen4, Gen5 58 59Intentionally, Arkville by itself DOES NOT provide common NIC 60capabilities such as offload or receive-side scaling (RSS). 61These capabilities would be viewed as a gate-level "tax" on 62Green-box FPGA applications that do not require such function. 63Instead, they can be added as needed with essentially no 64overhead to the FPGA Application. 65 66The ARK PMD also supports optional user extensions, through dynamic linking. 67The ARK PMD user extensions are a feature of Arkville’s DPDK 68net/ark poll mode driver, allowing users to add their 69own code to extend the net/ark functionality without 70having to make source code changes to the driver. One motivation for 71this capability is that while DPDK provides a rich set of functions 72to interact with NIC-like capabilities (e.g. MAC addresses and statistics), 73the Arkville RTL IP does not include a MAC. Users can supply their 74own MAC or custom FPGA applications, which may require control from 75the PMD. The user extension is the means providing the control 76between the user's FPGA application and the existing DPDK features via 77the PMD. 78 79Device Parameters 80----------------- 81 82The ARK PMD supports device parameters that are used for packet 83routing and for internal packet generation and packet checking. This 84section describes the supported parameters. These features are 85primarily used for diagnostics, testing, and performance verification 86under the guidance of an Arkville specialist. The nominal use of 87Arkville does not require any configuration using these parameters. 88 89"Pkt_dir" 90 91The Packet Director controls connectivity between Arkville's internal 92hardware components. The features of the Pkt_dir are only used for 93diagnostics and testing; it is not intended for nominal use. The full 94set of features are not published at this level. 95 96Format: 97Pkt_dir=0x00110F10 98 99"Pkt_gen" 100 101The packet generator parameter takes a file as its argument. The file 102contains configuration parameters used internally for regression 103testing and are not intended to be published at this level. The 104packet generator is an internal Arkville hardware component. 105 106Format: 107Pkt_gen=./config/pg.conf 108 109"Pkt_chkr" 110 111The packet checker parameter takes a file as its argument. The file 112contains configuration parameters used internally for regression 113testing and are not intended to be published at this level. The 114packet checker is an internal Arkville hardware component. 115 116Format: 117Pkt_chkr=./config/pc.conf 118 119 120Data Path Interface 121------------------- 122 123Ingress RX and Egress TX operation is by the nominal DPDK API . 124The driver supports single-port, multi-queue for both RX and TX. 125 126Configuration Information 127------------------------- 128 129**DPDK Configuration Parameter** 130 131 * **RTE_LIBRTE_ARK_MIN_TX_PKTLEN** (default 0): Sets the minimum 132 packet length for tx packets to the FPGA. Packets less than this 133 length are padded to meet the requirement. This allows padding to 134 be offloaded or remain in host software. 135 136 137Dynamic PMD Extension 138--------------------- 139 140Dynamic PMD extensions allow users to customize net/ark functionality 141using their own code. Arkville RTL and this PMD support high-throughput data 142movement, and these extensions allow PMD support for users' FPGA 143features. 144Dynamic PMD extensions operate by having users supply a shared object 145file which is loaded by Arkville PMD during initialization. The 146object file contains extension (or hook) functions that are registered 147and then called during PMD operations. 148 149The allowable set of extension functions are defined and documented in 150``ark_ext.h``, only the initialization function, 151``rte_pmd_ark_dev_init()``, is required; all others are optional. The 152following sections give a small extension example along with 153instructions for compiling and using the extension. 154 155 156Extension Example 157^^^^^^^^^^^^^^^^^ 158 159The following example shows an extension which populates mbuf fields 160during RX from user meta data coming from FPGA hardware. 161 162.. code-block:: c 163 164 #include <ark_ext.h> 165 #include <rte_mbuf.h> 166 #include <rte_ethdev.h> 167 #include <rte_malloc.h> 168 169 /* Global structure passed to extension/hook functions */ 170 struct ark_user_extension { 171 int timestamp_dynfield_offset; 172 }; 173 174 /* RX tuser field based on user's hardware */ 175 struct __rte_packed_begin user_rx_meta { 176 uint64_t timestamp; 177 uint32_t rss; 178 } __rte_packed_end; 179 180 /* Create ark_user_extension object for use in other hook functions */ 181 void *rte_pmd_ark_dev_init(struct rte_eth_dev * dev, 182 void * abar, int port_id ) 183 { 184 RTE_SET_USED(dev); 185 RTE_SET_USED(abar); 186 fprintf(stderr, "Called Arkville user extension for port %u\n", 187 port_id); 188 189 struct ark_user_extension *xdata = rte_zmalloc("macExtS", 190 sizeof(struct ark_user_extension), 64); 191 if (!xdata) 192 return NULL; 193 194 /* register dynfield for rx timestamp */ 195 rte_mbuf_dyn_rx_timestamp_register(&xdata->timestamp_dynfield_offset, 196 NULL); 197 198 fprintf(stderr, "timestamp fields offset in extension is %d\n", 199 xdata->timestamp_dynfield_offset); 200 return xdata; 201 } 202 203 /* uninitialization */ 204 void rte_pmd_ark_dev_uninit(struct rte_eth_dev * dev, void *user_data) 205 { 206 rte_free(user_data); 207 } 208 209 /* Hook function -- called for each RX packet 210 * Extract RX timestamp and RSS from meta and place in mbuf 211 */ 212 void rte_pmd_ark_rx_user_meta_hook(struct rte_mbuf *mbuf, 213 const uint32_t *meta, 214 void *user_data) 215 { 216 struct ark_user_extension *xdata = user_data; 217 struct user_rx_meta *user_rx = (struct user_rx_meta*)meta; 218 *RTE_MBUF_DYNFIELD(mbuf, xdata->timestamp_dynfield_offset, uint64_t*) = 219 user_rx->timestamp; 220 mbuf->hash.rss = user_rx->rss; 221 } 222 223 224Compiling Extension 225^^^^^^^^^^^^^^^^^^^ 226 227It is recommended to the compile the extension code with 228``-Wmissing-prototypes`` flag to insure correct function types. Typical 229DPDK options will also be needed. 230 231 232An example command line is give below 233 234.. code-block:: console 235 236 cc `pkg-config --cflags libdpdk` \ 237 -O3 -DALLOW_EXPERIMENTAL_API -fPIC -Wall -Wmissing-prototypes -c \ 238 -o pmd_net_ark_ext.o pmd_net_ark_ext.c 239 # Linking 240 cc -o libfx1_100g_ext.so.1 -shared \ 241 `pkg-config --libs libdpdk` \ 242 -Wl,--unresolved-symbols=ignore-all \ 243 -Wl,-soname,libpmd_net_ark_ext.so.1 pmd_net_ark_ext.o 244 245In a ``Makefile`` this would be 246 247.. code-block:: Makefile 248 249 CFLAGS += $(shell pkg-config --cflags libdpdk) 250 CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API -fPIC -Wall -Wmissing-prototypes 251 # Linking 252 LDFLAGS += $(shell pkg-config --libs libdpdk) 253 LDFLAGS += -Wl,--unresolved-symbols=ignore-all -Wl,-soname,libpmd_net_ark_ext.so.1 254 255The application must be linked with the ``-export-dynamic`` flags if any 256DPDK or application specific code will called from the extension. 257 258 259Enabling Extension 260^^^^^^^^^^^^^^^^^^ 261 262The extensions are enabled in the application through the use of an 263environment variable ``ARK_EXT_PATH`` This variable points to the lib 264extension file generated above. For example: 265 266.. code-block:: console 267 268 export ARK_EXT_PATH=$(PWD)/libpmd_net_ark_ext.so.1 269 testpmd ... 270 271 272Building DPDK 273------------- 274 275See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for 276instructions on how to build DPDK. 277 278By default the ARK PMD library will be built into the DPDK library. 279 280For configuring and using UIO and VFIO frameworks, please also refer :ref:`the 281documentation that comes with DPDK suite <linux_gsg>`. 282 283To build with a non-zero minimum tx packet length, set the above macro in your 284CFLAGS environment prior to the meson build step. I.e., 285 286.. code-block:: console 287 288 export CFLAGS="-DRTE_LIBRTE_ARK_MIN_TX_PKTLEN=60" 289 meson setup build 290 291 292Supported ARK RTL PCIe Instances 293-------------------------------- 294 295ARK PMD supports the following Arkville RTL PCIe instances including: 296 297* ``1d6c:100d`` - AR-ARKA-FX0 [Arkville 32B DPDK Data Mover] 298* ``1d6c:100e`` - AR-ARKA-FX1 [Arkville 64B DPDK Data Mover] 299* ``1d6c:100f`` - AR-ARKA-FX1 [Arkville 64B DPDK Data Mover for Versal] 300* ``1d6c:1010`` - AR-ARKA-FX1 [Arkville 64B DPDK Data Mover for Agilex] 301* ``1d6c:1017`` - AR-ARK-FX1 [Arkville 64B Multi-Homed Primary Endpoint] 302* ``1d6c:1018`` - AR-ARK-FX1 [Arkville 64B Multi-Homed Secondary Endpoint] 303* ``1d6c:1019`` - AR-ARK-FX1 [Arkville 64B Multi-Homed Tertiary Endpoint] 304* ``1d6c:101a`` - AR-ARK-SRIOV-FX0 [Arkville 32B Primary Physical Function] 305* ``1d6c:101b`` - AR-ARK-SRIOV-FX1 [Arkville 64B Primary Physical Function] 306* ``1d6c:101c`` - AR-ARK-SRIOV-VF [Arkville Virtual Function] 307* ``1d6c:101e`` - AR-ARKA-FX1 [Arkville 64B DPDK Data Mover for Agilex R-Tile] 308* ``1d6c:101f`` - AR-TK242 [2x100GbE Packet Capture Device] 309* ``1d6c:1022`` - AR-ARKA-FX2 [Arkville 128B DPDK Data Mover for Agilex] 310* ``1d6c:1024`` - AR-TK242 [2x100GbE Packet Capture Device] 311* ``1d6c:1025`` - AR-TK242-FX2 [2x100GbE Gen5 Packet Capture Device] 312* ``1d6c:1026`` - AR-TK242-FX2 [1x200GbE Gen5 Packet Capture Device] 313 314Arkville RTL Core Configurations 315-------------------------------- 316 317Arkville's RTL core may be configured by the user with different 318datapath widths to balance throughput against FPGA logic area. 319The ARK PMD has introspection on the RTL core configuration and acts accordingly. 320All Arkville configurations present identical RTL user-facing AXI 321stream interfaces for both AMD/Xilinx and Intel FPGAs. 322 323* ARK-FX0 - 256-bit 32B datapath (PCIe Gen3, Gen4) 324* ARK-FX1 - 512-bit 64B datapath (PCIe Gen3, Gen4, Gen5) 325* ARK-FX2 - 1024-bit 128B datapath (PCIe Gen5x16 Only) 326 327DPDK and Arkville Firmware Versioning 328------------------------------------- 329 330Arkville's firmware releases and its PMD have version dependencies which 331must be stepped together at certain releases. PMD code ensures the 332versions are compatible. The following lists shows where version 333compatible steps have occurred between DPDK releases and the corresponding 334Arkville releases. Intermediate releases not listed below remain 335compatible, e.g., DPDK releases 21.05, 21.08, and 21.11 are all compatible 336with Arkville releases 21.05, 21.08 and 21.11. LTS versions of DPDK remain 337compatible with the corresponding Arkville version. If other combinations 338are required, please contact Atomic Rules support. 339 340* DPDK 23.11 requires Arkville 23.11. 341* DPDK 22.07 requires Arkville 22.07. 342* DPDK 22.03 requires Arkville 22.03. 343* DPDK 21.05 requires Arkville 21.05. 344* DPDK 18.11 requires Arkville 18.11. 345* DPDK 17.05 requires Arkville 17.05 -- initial version. 346 347Supported Operating Systems 348--------------------------- 349 350Any Linux distribution fulfilling the conditions described in ``System Requirements`` 351section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK 352Release Notes*. ARM and PowerPC architectures are not supported at this time. 353 354 355Supported Features 356------------------ 357 358* Dynamic ARK PMD extensions 359* Dynamic per-queue MBUF (re)sizing up to 32KB 360* SR-IOV, VF-based queue-separation 361* Multiple receive and transmit queues 362* Jumbo frames up to 9K 363* Hardware Statistics 364 365Unsupported Features 366-------------------- 367 368Features that may be part of, or become part of, the Arkville RTL IP that are 369not currently supported or exposed by the ARK PMD include: 370 371* Arkville's Packet Generator Control and Status 372* Arkville's Packet Director Control and Status 373* Arkville's Packet Checker Control and Status 374* Arkville's Timebase Management 375 376Pre-Requisites 377-------------- 378 379#. Prepare the system as recommended by DPDK suite. This includes environment 380 variables, hugepages configuration, tool-chains and configuration 381 382#. Insert igb_uio kernel module using the command 'modprobe igb_uio' 383 384#. Bind the intended ARK device to igb_uio module 385 386At this point the system should be ready to run DPDK applications. Once the 387application runs to completion, the ARK PMD can be detached from igb_uio if necessary. 388 389Usage Example 390------------- 391 392Follow instructions available in the document 393:ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch 394**testpmd** with Atomic Rules ARK devices managed by librte_net_ark. 395 396Example output: 397 398.. code-block:: console 399 400 [...] 401 EAL: PCI device 0000:01:00.0 on NUMA socket -1 402 EAL: probe driver: 1d6c:100e rte_ark_pmd 403 EAL: PCI memory mapped at 0x7f9b6c400000 404 PMD: eth_ark_dev_init(): Initializing 0:2:0.1 405 ARKP PMD CommitID: 378f3a67 406 Configuring Port 0 (socket 0) 407 Port 0: DC:3C:F6:00:00:01 408 Checking link statuses... 409 Port 0 Link Up - speed 100000 Mbps - full-duplex 410 Done 411 testpmd> 412