1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2010-2014 Intel Corporation. 3 4Poll Mode Driver for Paravirtual VMXNET3 NIC 5============================================ 6 7The VMXNET3 adapter is the next generation of a paravirtualized NIC, introduced by VMware* ESXi. 8It is designed for performance, offers all the features available in VMXNET2, and adds several new features such as, 9multi-queue support (also known as Receive Side Scaling, RSS), 10IPv6 offloads, and MSI/MSI-X interrupt delivery. 11One can use the same device in a DPDK application with VMXNET3 PMD introduced in DPDK API. 12 13In this chapter, two setups with the use of the VMXNET3 PMD are demonstrated: 14 15#. Vmxnet3 with a native NIC connected to a vSwitch 16 17#. Vmxnet3 chaining VMs connected to a vSwitch 18 19VMXNET3 Implementation in the DPDK 20---------------------------------- 21 22For details on the VMXNET3 device, refer to the VMXNET3 driver's vmxnet3 directory and support manual from VMware*. 23 24As a PMD, the VMXNET3 driver provides the packet reception and transmission callbacks, vmxnet3_recv_pkts and vmxnet3_xmit_pkts. 25 26The VMXNET3 PMD handles all the packet buffer memory allocation and resides in guest address space 27and it is solely responsible to free that memory when not needed. 28The packet buffers and features to be supported are made available to hypervisor via VMXNET3 PCI configuration space BARs. 29During RX/TX, the packet buffers are exchanged by their GPAs, 30and the hypervisor loads the buffers with packets in the RX case and sends packets to vSwitch in the TX case. 31 32The VMXNET3 PMD is compiled with vmxnet3 device headers. 33The interface is similar to that of the other PMDs available in the DPDK API. 34The driver pre-allocates the packet buffers and loads the command ring descriptors in advance. 35The hypervisor fills those packet buffers on packet arrival and write completion ring descriptors, 36which are eventually pulled by the PMD. 37After reception, the DPDK application frees the descriptors and loads new packet buffers for the coming packets. 38The interrupts are disabled and there is no notification required. 39This keeps performance up on the RX side, even though the device provides a notification feature. 40 41In the transmit routine, the DPDK application fills packet buffer pointers in the descriptors of the command ring 42and notifies the hypervisor. 43In response the hypervisor takes packets and passes them to the vSwitch, It writes into the completion descriptors ring. 44The rings are read by the PMD in the next transmit routine call and the buffers and descriptors are freed from memory. 45 46Features and Limitations of VMXNET3 PMD 47--------------------------------------- 48 49In release 1.6.0, the VMXNET3 PMD provides the basic functionality of packet reception and transmission. 50There are several options available for filtering packets at VMXNET3 device level including: 51 52#. MAC Address based filtering: 53 54 * Unicast, Broadcast, All Multicast modes - SUPPORTED BY DEFAULT 55 56 * Multicast with Multicast Filter table - NOT SUPPORTED 57 58 * Promiscuous mode - SUPPORTED 59 60 * RSS based load balancing between queues - SUPPORTED 61 62#. VLAN filtering: 63 64 * VLAN tag based filtering without load balancing - SUPPORTED 65 66#. Vmxnet3 versions and associated features. 67 68 Vmxnet3 version is tied to corresponding ESXi hardware version and each 69 version defines a set of compatible features. 70 71 * Vmxnet3 version 7, hw ver 19 72 This version adds support for Uniform Passthrough(UPT). 73 74 * Vmxnet3 version 6, hw ver 17 75 This version enhanced vmxnet3 to support queues up to 32 and also 76 removed power-of-two limitations on the queues. 77 78 * Vmxnet3 version 5, hw ver 15 79 Features not related to dpdk vmxnet3 PMD. 80 81 * Vmxnet3 version 4, hw ver 14 82 This version adds supoprt for UDP and ESP RSS 83 84 * Vmxnet3 version 3, hw ver 13 85 This version added performance enhancement features such as 86 configurable Tx data ring, Receive Data Ring, and the ability 87 to register memory regions. 88 89 * Vmxnet3 version 2, hw ver 11 90 This version adds support for Large Receive offload (LRO). 91 92.. note:: 93 94 95 * Release 1.6.0 does not support separate headers and body receive cmd_ring and hence, 96 multiple segment buffers are not supported. 97 Only cmd_ring_0 is used for packet buffers, one for each descriptor. 98 99 * Receive and transmit of scattered packets is not supported. 100 101 * Multicast with Multicast Filter table is not supported. 102 103 * IOVA as VA on AMD hosts is supported from ESXi 7.0 U1 onwards. 104 105Prerequisites 106------------- 107 108The following prerequisites apply: 109 110* Before starting a VM, a VMXNET3 interface to a VM through VMware vSphere Client must be assigned. 111 This is shown in the figure below. 112 113.. _figure_vmxnet3_int: 114 115.. figure:: img/vmxnet3_int.* 116 117 Assigning a VMXNET3 interface to a VM using VMware vSphere Client 118 119.. note:: 120 121 Depending on the Virtual Machine type, the VMware vSphere Client shows Ethernet adaptors while adding an Ethernet device. 122 Ensure that the VM type used offers a VMXNET3 device. Refer to the VMware documentation for a listed of VMs. 123 124.. note:: 125 126 Follow the *DPDK Getting Started Guide* to setup the basic DPDK environment. 127 128.. note:: 129 130 Follow the *DPDK Sample Application's User Guide*, L2 Forwarding/L3 Forwarding and 131 TestPMD for instructions on how to run a DPDK application using an assigned VMXNET3 device. 132 133VMXNET3 with a Native NIC Connected to a vSwitch 134------------------------------------------------ 135 136This section describes an example setup for Phy-vSwitch-VM-Phy communication. 137 138.. _figure_vswitch_vm: 139 140.. figure:: img/vswitch_vm.* 141 142 VMXNET3 with a Native NIC Connected to a vSwitch 143 144.. note:: 145 146 Other instructions on preparing to use DPDK such as, 147 hugepage enabling, UIO port binding are not listed here. 148 Please refer to *DPDK Getting Started Guide and DPDK Sample Application's User Guide* for detailed instructions. 149 150The packet reception and transmission flow path is:: 151 152 Packet generator -> 82576 153 -> VMware ESXi vSwitch 154 -> VMXNET3 device 155 -> Guest VM VMXNET3 port 0 rx burst 156 -> Guest VM 82599 VF port 0 tx burst 157 -> 82599 VF 158 -> Packet generator 159 160VMXNET3 Chaining VMs Connected to a vSwitch 161------------------------------------------- 162 163The following figure shows an example VM-to-VM communication over a Phy-VM-vSwitch-VM-Phy communication channel. 164 165.. _figure_vm_vm_comms: 166 167.. figure:: img/vm_vm_comms.* 168 169 VMXNET3 Chaining VMs Connected to a vSwitch 170 171.. note:: 172 173 When using the L2 Forwarding or L3 Forwarding applications, 174 a destination MAC address needs to be written in packets to hit the other VM's VMXNET3 interface. 175 176In this example, the packet flow path is:: 177 178 Packet generator -> 82599 VF 179 -> Guest VM 82599 port 0 rx burst 180 -> Guest VM VMXNET3 port 1 tx burst 181 -> VMXNET3 device 182 -> VMware ESXi vSwitch 183 -> VMXNET3 device 184 -> Guest VM VMXNET3 port 0 rx burst 185 -> Guest VM 82599 VF port 1 tx burst 186 -> 82599 VF 187 -> Packet generator 188