1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright (c) 2015-2021 Atomic Rules LLC 3 All rights reserved. 4 5ARK Poll Mode Driver 6==================== 7 8The ARK PMD is a DPDK poll-mode driver for the Atomic Rules Arkville 9(ARK) family of devices. 10 11More information can be found at the `Atomic Rules website 12<http://atomicrules.com>`_. 13 14Overview 15-------- 16 17The Atomic Rules Arkville product is DPDK and AXI compliant product 18that marshals packets across a PCIe conduit between host DPDK mbufs and 19FPGA AXI streams. 20 21The ARK PMD, and the spirit of the overall Arkville product, 22has been to take the DPDK API/ABI as a fixed specification; 23then implement much of the business logic in FPGA RTL circuits. 24The approach of *working backwards* from the DPDK API/ABI and having 25the GPP host software *dictate*, while the FPGA hardware *copes*, 26results in significant performance gains over a naive implementation. 27 28While this document describes the ARK PMD software, it is helpful to 29understand what the FPGA hardware is and is not. The Arkville RTL 30component provides a single PCIe Physical Function (PF) supporting 31some number of RX/Ingress and TX/Egress Queues. The ARK PMD controls 32the Arkville core through a dedicated opaque Core BAR (CBAR). 33To allow users full freedom for their own FPGA application IP, 34an independent FPGA Application BAR (ABAR) is provided. 35 36One popular way to imagine Arkville's FPGA hardware aspect is as the 37FPGA PCIe-facing side of a so-called Smart NIC. The Arkville core does 38not contain any MACs, and is link-speed independent, as well as 39agnostic to the number of physical ports the application chooses to 40use. The ARK driver exposes the familiar PMD interface to allow packet 41movement to and from mbufs across multiple queues. 42 43However FPGA RTL applications could contain a universe of added 44functionality that an Arkville RTL core does not provide or can 45not anticipate. To allow for this expectation of user-defined 46innovation, the ARK PMD provides a dynamic mechanism of adding 47capabilities without having to modify the ARK PMD. 48 49The ARK PMD is intended to support all instances of the Arkville 50RTL Core, regardless of configuration, FPGA vendor, or target 51board. While specific capabilities such as number of physical 52hardware queue-pairs are negotiated; the driver is designed to 53remain constant over a broad and extendable feature set. 54 55Intentionally, Arkville by itself DOES NOT provide common NIC 56capabilities such as offload or receive-side scaling (RSS). 57These capabilities would be viewed as a gate-level "tax" on 58Green-box FPGA applications that do not require such function. 59Instead, they can be added as needed with essentially no 60overhead to the FPGA Application. 61 62The ARK PMD also supports optional user extensions, through dynamic linking. 63The ARK PMD user extensions are a feature of Arkville’s DPDK 64net/ark poll mode driver, allowing users to add their 65own code to extend the net/ark functionality without 66having to make source code changes to the driver. One motivation for 67this capability is that while DPDK provides a rich set of functions 68to interact with NIC-like capabilities (e.g. MAC addresses and statistics), 69the Arkville RTL IP does not include a MAC. Users can supply their 70own MAC or custom FPGA applications, which may require control from 71the PMD. The user extension is the means providing the control 72between the user's FPGA application and the existing DPDK features via 73the PMD. 74 75Device Parameters 76------------------- 77 78The ARK PMD supports device parameters that are used for packet 79routing and for internal packet generation and packet checking. This 80section describes the supported parameters. These features are 81primarily used for diagnostics, testing, and performance verification 82under the guidance of an Arkville specialist. The nominal use of 83Arkville does not require any configuration using these parameters. 84 85"Pkt_dir" 86 87The Packet Director controls connectivity between Arkville's internal 88hardware components. The features of the Pkt_dir are only used for 89diagnostics and testing; it is not intended for nominal use. The full 90set of features are not published at this level. 91 92Format: 93Pkt_dir=0x00110F10 94 95"Pkt_gen" 96 97The packet generator parameter takes a file as its argument. The file 98contains configuration parameters used internally for regression 99testing and are not intended to be published at this level. The 100packet generator is an internal Arkville hardware component. 101 102Format: 103Pkt_gen=./config/pg.conf 104 105"Pkt_chkr" 106 107The packet checker parameter takes a file as its argument. The file 108contains configuration parameters used internally for regression 109testing and are not intended to be published at this level. The 110packet checker is an internal Arkville hardware component. 111 112Format: 113Pkt_chkr=./config/pc.conf 114 115 116Data Path Interface 117------------------- 118 119Ingress RX and Egress TX operation is by the nominal DPDK API . 120The driver supports single-port, multi-queue for both RX and TX. 121 122Configuration Information 123------------------------- 124 125**DPDK Configuration Parameter** 126 127 * **RTE_LIBRTE_ARK_MIN_TX_PKTLEN** (default 0): Sets the minimum 128 packet length for tx packets to the FPGA. Packets less than this 129 length are padded to meet the requirement. This allows padding to 130 be offloaded or remain in host software. 131 132 133Dynamic PMD Extension 134--------------------- 135 136Dynamic PMD extensions allow users to customize net/ark functionality 137using their own code. Arkville RTL and this PMD support high-throughput data 138movement, and these extensions allow PMD support for users' FPGA 139features. 140Dynamic PMD extensions operate by having users supply a shared object 141file which is loaded by Arkville PMD during initialization. The 142object file contains extension (or hook) functions that are registered 143and then called during PMD operations. 144 145The allowable set of extension functions are defined and documented in 146``ark_ext.h``, only the initialization function, 147``rte_pmd_ark_dev_init()``, is required; all others are optional. The 148following sections give a small extension example along with 149instructions for compiling and using the extension. 150 151 152Extension Example 153^^^^^^^^^^^^^^^^^ 154 155The following example shows an extension which populates mbuf fields 156during RX from user meta data coming from FPGA hardware. 157 158.. code-block:: c 159 160 #include <ark_ext.h> 161 #include <rte_mbuf.h> 162 #include <rte_ethdev.h> 163 #include <rte_malloc.h> 164 165 /* Global structure passed to extension/hook functions */ 166 struct ark_user_extension { 167 int timestamp_dynfield_offset; 168 }; 169 170 /* RX tuser field based on user's hardware */ 171 struct user_rx_meta { 172 uint64_t timestamp; 173 uint32_t rss; 174 } __rte_packed; 175 176 /* Create ark_user_extension object for use in other hook functions */ 177 void *rte_pmd_ark_dev_init(struct rte_eth_dev * dev, 178 void * abar, int port_id ) 179 { 180 RTE_SET_USED(dev); 181 RTE_SET_USED(abar); 182 fprintf(stderr, "Called Arkville user extension for port %u\n", 183 port_id); 184 185 struct ark_user_extension *xdata = rte_zmalloc("macExtS", 186 sizeof(struct ark_user_extension), 64); 187 if (!xdata) 188 return NULL; 189 190 /* register dynfield for rx timestamp */ 191 rte_mbuf_dyn_rx_timestamp_register(&xdata->timestamp_dynfield_offset, 192 NULL); 193 194 fprintf(stderr, "timestamp fields offset in extension is %d\n", 195 xdata->timestamp_dynfield_offset); 196 return xdata; 197 } 198 199 /* uninitialization */ 200 void rte_pmd_ark_dev_uninit(struct rte_eth_dev * dev, void *user_data) 201 { 202 rte_free(user_data); 203 } 204 205 /* Hook function -- called for each RX packet 206 * Extract RX timestamp and RSS from meta and place in mbuf 207 */ 208 void rte_pmd_ark_rx_user_meta_hook(struct rte_mbuf *mbuf, 209 const uint32_t *meta, 210 void *user_data) 211 { 212 struct ark_user_extension *xdata = user_data; 213 struct user_rx_meta *user_rx = (struct user_rx_meta*)meta; 214 *RTE_MBUF_DYNFIELD(mbuf, xdata->timestamp_dynfield_offset, uint64_t*) = 215 user_rx->timestamp; 216 mbuf->hash.rss = user_rx->rss; 217 } 218 219 220Compiling Extension 221^^^^^^^^^^^^^^^^^^^ 222 223It is recommended to the compile the extension code with 224``-Wmissing-prototypes`` flag to insure correct function types. Typical 225DPDK options will also be needed. 226 227 228An example command line is give below 229 230.. code-block:: console 231 232 cc `pkg-config --cflags libdpdk` \ 233 -O3 -DALLOW_EXPERIMENTAL_API -fPIC -Wall -Wmissing-prototypes -c \ 234 -o pmd_net_ark_ext.o pmd_net_ark_ext.c 235 # Linking 236 cc -o libfx1_100g_ext.so.1 -shared \ 237 `pkg-config --libs libdpdk` \ 238 -Wl,--unresolved-symbols=ignore-all \ 239 -Wl,-soname,libpmd_net_ark_ext.so.1 pmd_net_ark_ext.o 240 241In a ``Makefile`` this would be 242 243.. code-block:: Makefile 244 245 CFLAGS += $(shell pkg-config --cflags libdpdk) 246 CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API -fPIC -Wall -Wmissing-prototypes 247 # Linking 248 LDFLAGS += $(shell pkg-config --libs libdpdk) 249 LDFLAGS += -Wl,--unresolved-symbols=ignore-all -Wl,-soname,libpmd_net_ark_ext.so.1 250 251The application must be linked with the ``-export-dynamic`` flags if any 252DPDK or application specific code will called from the extension. 253 254 255Enabling Extension 256^^^^^^^^^^^^^^^^^^ 257 258The extensions are enabled in the application through the use of an 259environment variable ``ARK_EXT_PATH`` This variable points to the lib 260extension file generated above. For example: 261 262.. code-block:: console 263 264 export ARK_EXT_PATH=$(PWD)/libpmd_net_ark_ext.so.1 265 testpmd ... 266 267 268Building DPDK 269------------- 270 271See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for 272instructions on how to build DPDK. 273 274By default the ARK PMD library will be built into the DPDK library. 275 276For configuring and using UIO and VFIO frameworks, please also refer :ref:`the 277documentation that comes with DPDK suite <linux_gsg>`. 278 279To build with a non-zero minimum tx packet length, set the above macro in your 280CFLAGS environment prior to the meson build step. I.e., 281 282.. code-block:: console 283 284 export CFLAGS="-DRTE_LIBRTE_ARK_MIN_TX_PKTLEN=60" 285 meson build 286 287 288Supported ARK RTL PCIe Instances 289-------------------------------- 290 291ARK PMD supports the following Arkville RTL PCIe instances including: 292 293* ``1d6c:100d`` - AR-ARKA-FX0 [Arkville 32B DPDK Data Mover] 294* ``1d6c:100e`` - AR-ARKA-FX1 [Arkville 64B DPDK Data Mover] 295* ``1d6c:100f`` - AR-ARKA-FX1 [Arkville 64B DPDK Data Mover for Versal] 296* ``1d6c:1010`` - AR-ARKA-FX1 [Arkville 64B DPDK Data Mover for Agilex] 297* ``1d6c:1017`` - AR-ARK-FX1 [Arkville 64B Multi-Homed Primary Endpoint] 298* ``1d6c:1018`` - AR-ARK-FX1 [Arkville 64B Multi-Homed Secondary Endpoint] 299* ``1d6c:1019`` - AR-ARK-FX1 [Arkville 64B Multi-Homed Tertiary Endpoint] 300 301Supported Operating Systems 302--------------------------- 303 304Any Linux distribution fulfilling the conditions described in ``System Requirements`` 305section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK 306Release Notes*. ARM and PowerPC architectures are not supported at this time. 307 308 309Supported Features 310------------------ 311 312* Dynamic ARK PMD extensions 313* Multiple receive and transmit queues 314* Jumbo frames up to 9K 315* Hardware Statistics 316 317Unsupported Features 318-------------------- 319 320Features that may be part of, or become part of, the Arkville RTL IP that are 321not currently supported or exposed by the ARK PMD include: 322 323* PCIe SR-IOV Virtual Functions (VFs) 324* Arkville's Packet Generator Control and Status 325* Arkville's Packet Director Control and Status 326* Arkville's Packet Checker Control and Status 327* Arkville's Timebase Management 328 329Pre-Requisites 330-------------- 331 332#. Prepare the system as recommended by DPDK suite. This includes environment 333 variables, hugepages configuration, tool-chains and configuration 334 335#. Insert igb_uio kernel module using the command 'modprobe igb_uio' 336 337#. Bind the intended ARK device to igb_uio module 338 339At this point the system should be ready to run DPDK applications. Once the 340application runs to completion, the ARK PMD can be detached from igb_uio if necessary. 341 342Usage Example 343------------- 344 345Follow instructions available in the document 346:ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch 347**testpmd** with Atomic Rules ARK devices managed by librte_net_ark. 348 349Example output: 350 351.. code-block:: console 352 353 [...] 354 EAL: PCI device 0000:01:00.0 on NUMA socket -1 355 EAL: probe driver: 1d6c:100e rte_ark_pmd 356 EAL: PCI memory mapped at 0x7f9b6c400000 357 PMD: eth_ark_dev_init(): Initializing 0:2:0.1 358 ARKP PMD CommitID: 378f3a67 359 Configuring Port 0 (socket 0) 360 Port 0: DC:3C:F6:00:00:01 361 Checking link statuses... 362 Port 0 Link Up - speed 100000 Mbps - full-duplex 363 Done 364 testpmd> 365