1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright (c) 2015-2020 Amazon.com, Inc. or its affiliates. 3 All rights reserved. 4 5ENA Poll Mode Driver 6==================== 7 8The ENA PMD is a DPDK poll-mode driver for the Amazon Elastic 9Network Adapter (ENA) family. 10 11Overview 12-------- 13 14The ENA driver exposes a lightweight management interface with a 15minimal set of memory mapped registers and an extendable command set 16through an Admin Queue. 17 18The driver supports a wide range of ENA adapters, is link-speed 19independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, 20etc.), and it negotiates and supports an extendable feature set. 21 22ENA adapters allow high speed and low overhead Ethernet traffic 23processing by providing a dedicated Tx/Rx queue pair per CPU core. 24 25The ENA driver supports industry standard TCP/IP offload features such 26as checksum offload and TCP transmit segmentation offload (TSO). 27 28Receive-side scaling (RSS) is supported for multi-core scaling. 29 30Some of the ENA devices support a working mode called Low-latency 31Queue (LLQ), which saves several more microseconds. 32 33Management Interface 34-------------------- 35 36ENA management interface is exposed by means of: 37 38* Device Registers 39* Admin Queue (AQ) and Admin Completion Queue (ACQ) 40 41ENA device memory-mapped PCIe space for registers (MMIO registers) 42are accessed only during driver initialization and are not involved 43in further normal device operation. 44 45AQ is used for submitting management commands, and the 46results/responses are reported asynchronously through ACQ. 47 48ENA introduces a very small set of management commands with room for 49vendor-specific extensions. Most of the management operations are 50framed in a generic Get/Set feature command. 51 52The following admin queue commands are supported: 53 54* Create I/O submission queue 55* Create I/O completion queue 56* Destroy I/O submission queue 57* Destroy I/O completion queue 58* Get feature 59* Set feature 60* Get statistics 61 62Refer to ``ena_admin_defs.h`` for the list of supported Get/Set Feature 63properties. 64 65Data Path Interface 66------------------- 67 68I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx 69SQ correspondingly). Each SQ has a completion queue (CQ) associated 70with it. 71 72The SQs and CQs are implemented as descriptor rings in contiguous 73physical memory. 74 75Refer to ``ena_eth_io_defs.h`` for the detailed structure of the descriptor 76 77The driver supports multi-queue for both Tx and Rx. 78 79Configuration information 80------------------------- 81 82**Runtime Configuration Parameters** 83 84 * **large_llq_hdr** (default 0) 85 86 Enables or disables usage of large LLQ headers. This option will have 87 effect only if the device also supports large LLQ headers. Otherwise, the 88 default value will be used. 89 90**ENA Configuration Parameters** 91 92 * **Number of Queues** 93 94 This is the requested number of queues upon initialization, however, the actual 95 number of receive and transmit queues to be created will be the minimum between 96 the maximal number supported by the device and number of queues requested. 97 98 * **Size of Queues** 99 100 This is the requested size of receive/transmit queues, while the actual size 101 will be the minimum between the requested size and the maximal receive/transmit 102 supported by the device. 103 104Building DPDK 105------------- 106 107See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for 108instructions on how to build DPDK. 109 110By default the ENA PMD library will be built into the DPDK library. 111 112For configuring and using UIO and VFIO frameworks, please also refer :ref:`the 113documentation that comes with DPDK suite <linux_gsg>`. 114 115Supported ENA adapters 116---------------------- 117 118Current ENA PMD supports the following ENA adapters including: 119 120* ``1d0f:ec20`` - ENA VF 121* ``1d0f:ec21`` - ENA VF RSERV0 122 123Supported Operating Systems 124--------------------------- 125 126Any Linux distribution fulfilling the conditions described in ``System Requirements`` 127section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK Release Notes*. 128 129Supported features 130------------------ 131 132* MTU configuration 133* Jumbo frames up to 9K 134* IPv4/TCP/UDP checksum offload 135* TSO offload 136* Multiple receive and transmit queues 137* RSS hash 138* RSS indirection table configuration 139* Low Latency Queue for Tx 140* Basic and extended statistics 141* LSC event notification 142* Watchdog (requires handling of timers in the application) 143* Device reset upon failure 144 145Prerequisites 146------------- 147 148#. Prepare the system as recommended by DPDK suite. This includes environment 149 variables, hugepages configuration, tool-chains and configuration. 150 151#. ENA PMD can operate with ``vfio-pci``(*) or ``igb_uio`` driver. 152 153 (*) ENAv2 hardware supports Low Latency Queue v2 (LLQv2). This feature 154 reduces the latency of the packets by pushing the header directly through 155 the PCI to the device, before the DMA is even triggered. For proper work 156 kernel PCI driver must support write combining (WC). 157 In DPDK ``igb_uio`` it must be enabled by loading module with 158 ``wc_activate=1`` flag (example below). However, mainline's vfio-pci 159 driver in kernel doesn't have WC support yet (planed to be added). 160 If vfio-pci used user should be either turn off ENAv2 (to avoid performance 161 impact) or recompile vfio-pci driver with patch provided in 162 `amzn-github <https://github.com/amzn/amzn-drivers/tree/master/userspace/dpdk/enav2-vfio-patch>`_. 163 164#. Insert ``vfio-pci`` or ``igb_uio`` kernel module using the command 165 ``modprobe vfio-pci`` or ``modprobe uio; insmod igb_uio.ko wc_activate=1`` 166 respectively. 167 168#. For ``vfio-pci`` users only: 169 Please make sure that ``IOMMU`` is enabled in your system, 170 or use ``vfio`` driver in ``noiommu`` mode:: 171 172 echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode 173 174 To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag 175 ``CONFIG_VFIO_NOIOMMU``. 176 177#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module. 178 179At this point the system should be ready to run DPDK applications. Once the 180application runs to completion, the ENA can be detached from attached module if 181necessary. 182 183**Note about usage on \*.metal instances** 184 185On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64 186hosts. 187 188* x86_64 (e.g. c5.metal, i3.metal): 189 IOMMU should be disabled by default. In that situation, the ``igb_uio`` can 190 be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please 191 see above). 192 193 When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this 194 feature, while ``vfio-pci`` should work without any changes. 195 To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file 196 ``/etc/default/grub`` with the below extra boot arguments:: 197 198 iommu=1 intel_iommu=on 199 200 Then, make the changes live by executing as a root:: 201 202 # grub2-mkconfig > /boot/grub2/grub.cfg 203 204 Finally, reboot should result in IOMMU being enabled. 205 206* arm64 (a1.metal): 207 IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't 208 supporting SMMU, which is implementation of IOMMU for arm64 architecture and 209 ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those 210 hosts, one must disable IOMMU. This can be done by updating 211 ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot 212 argument:: 213 214 iommu.passthrough=1 215 216 Then, make the changes live by executing as a root:: 217 218 # grub2-mkconfig > /boot/grub2/grub.cfg 219 220 Finally, reboot should result in IOMMU being disabled. 221 Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be 222 working in no-IOMMU mode (please see above). 223 224Usage example 225------------- 226 227Follow instructions available in the document 228:ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to launch 229**testpmd** with Amazon ENA devices managed by librte_net_ena. 230 231Example output: 232 233.. code-block:: console 234 235 [...] 236 EAL: PCI device 0000:00:06.0 on NUMA socket -1 237 EAL: Invalid NUMA socket, default to 0 238 EAL: probe driver: 1d0f:ec20 net_ena 239 240 Interactive-mode selected 241 testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 242 testpmd: preferred mempool ops selected: ring_mp_mc 243 Warning! port-topology=paired and odd forward ports number, the last port will pair with itself. 244 Configuring Port 0 (socket 0) 245 Port 0: 00:00:00:11:00:01 246 Checking link statuses... 247 248 Done 249 testpmd> 250