1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2015 Intel Corporation. 3 4How to get best performance with NICs on Intel platforms 5======================================================== 6 7This document is a step-by-step guide for getting high performance from DPDK applications on Intel platforms. 8 9 10Hardware and Memory Requirements 11-------------------------------- 12 13For best performance use an Intel Xeon class server system such as Ivy Bridge, Haswell or newer. 14 15Ensure that each memory channel has at least one memory DIMM inserted, and that the memory size for each is at least 4GB. 16**Note**: this has one of the most direct effects on performance. 17 18You can check the memory configuration using ``dmidecode`` as follows:: 19 20 dmidecode -t memory | grep Locator 21 22 Locator: DIMM_A1 23 Bank Locator: NODE 1 24 Locator: DIMM_A2 25 Bank Locator: NODE 1 26 Locator: DIMM_B1 27 Bank Locator: NODE 1 28 Locator: DIMM_B2 29 Bank Locator: NODE 1 30 ... 31 Locator: DIMM_G1 32 Bank Locator: NODE 2 33 Locator: DIMM_G2 34 Bank Locator: NODE 2 35 Locator: DIMM_H1 36 Bank Locator: NODE 2 37 Locator: DIMM_H2 38 Bank Locator: NODE 2 39 40The sample output above shows a total of 8 channels, from ``A`` to ``H``, where each channel has 2 DIMMs. 41 42You can also use ``dmidecode`` to determine the memory frequency:: 43 44 dmidecode -t memory | grep Speed 45 46 Speed: 2133 MHz 47 Configured Clock Speed: 2134 MHz 48 Speed: Unknown 49 Configured Clock Speed: Unknown 50 Speed: 2133 MHz 51 Configured Clock Speed: 2134 MHz 52 Speed: Unknown 53 ... 54 Speed: 2133 MHz 55 Configured Clock Speed: 2134 MHz 56 Speed: Unknown 57 Configured Clock Speed: Unknown 58 Speed: 2133 MHz 59 Configured Clock Speed: 2134 MHz 60 Speed: Unknown 61 Configured Clock Speed: Unknown 62 63The output shows a speed of 2133 MHz (DDR4) and Unknown (not existing). 64This aligns with the previous output which showed that each channel has one memory bar. 65 66 67Network Interface Card Requirements 68~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 69 70Use a `DPDK supported <https://core.dpdk.org/supported/>`_ high end NIC such as the Intel XL710 40GbE. 71 72Make sure each NIC has been flashed the latest version of NVM/firmware. 73 74Use PCIe Gen3 slots, such as Gen3 ``x8`` or Gen3 ``x16`` because PCIe Gen2 slots don't provide enough bandwidth 75for 2 x 10GbE and above. 76You can use ``lspci`` to check the speed of a PCI slot using something like the following:: 77 78 lspci -s 03:00.1 -vv | grep LnkSta 79 80 LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- ... 81 LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ ... 82 83When inserting NICs into PCI slots always check the caption, such as CPU0 or CPU1 to indicate which socket it is connected to. 84 85Care should be take with NUMA. 86If you are using 2 or more ports from different NICs, it is best to ensure that these NICs are on the same CPU socket. 87An example of how to determine this is shown further below. 88 89 90BIOS Settings 91~~~~~~~~~~~~~ 92 93The following are some recommendations on BIOS settings. Different platforms will have different BIOS naming 94so the following is mainly for reference: 95 96#. Establish the steady state for the system, consider reviewing BIOS settings desired for best performance characteristic e.g. optimize for performance or energy efficiency. 97 98#. Match the BIOS settings to the needs of the application you are testing. 99 100#. Typically, **Performance** as the CPU Power and Performance policy is a reasonable starting point. 101 102#. Consider using Turbo Boost to increase the frequency on cores. 103 104#. Disable all virtualization options when you test the physical function of the NIC, and turn on VT-d if you wants to use VFIO. 105 106 107Linux boot command line 108~~~~~~~~~~~~~~~~~~~~~~~ 109 110The following are some recommendations on GRUB boot settings: 111 112#. Use the default grub file as a starting point. 113 114#. Reserve 1G huge pages via grub configurations. For example to reserve 8 huge pages of 1G size:: 115 116 default_hugepagesz=1G hugepagesz=1G hugepages=8 117 118#. Isolate CPU cores which will be used for DPDK. For example:: 119 120 isolcpus=2,3,4,5,6,7,8 121 122#. If it wants to use VFIO, use the following additional grub parameters:: 123 124 iommu=pt intel_iommu=on 125 126 127Configurations before running DPDK 128---------------------------------- 129 130#. Reserve huge pages. 131 See the earlier section on :ref:`linux_gsg_hugepages` for more details. 132 133 .. code-block:: console 134 135 # Get the hugepage size. 136 awk '/Hugepagesize/ {print $2}' /proc/meminfo 137 138 # Get the total huge page numbers. 139 awk '/HugePages_Total/ {print $2} ' /proc/meminfo 140 141 # Unmount the hugepages. 142 umount `awk '/hugetlbfs/ {print $2}' /proc/mounts` 143 144 # Create the hugepage mount folder. 145 mkdir -p /mnt/huge 146 147 # Mount to the specific folder. 148 mount -t hugetlbfs nodev /mnt/huge 149 150#. Check the CPU layout using the DPDK ``cpu_layout`` utility: 151 152 .. code-block:: console 153 154 cd dpdk_folder 155 156 usertools/cpu_layout.py 157 158 Or run ``lscpu`` to check the cores on each socket. 159 160#. Check your NIC id and related socket id: 161 162 .. code-block:: console 163 164 # List all the NICs with PCI address and device IDs. 165 lspci -nn | grep Eth 166 167 For example suppose your output was as follows:: 168 169 82:00.0 Ethernet [0200]: Intel XL710 for 40GbE QSFP+ [8086:1583] 170 82:00.1 Ethernet [0200]: Intel XL710 for 40GbE QSFP+ [8086:1583] 171 85:00.0 Ethernet [0200]: Intel XL710 for 40GbE QSFP+ [8086:1583] 172 85:00.1 Ethernet [0200]: Intel XL710 for 40GbE QSFP+ [8086:1583] 173 174 Check the PCI device related numa node id: 175 176 .. code-block:: console 177 178 cat /sys/bus/pci/devices/0000\:xx\:00.x/numa_node 179 180 Usually ``0x:00.x`` is on socket 0 and ``8x:00.x`` is on socket 1. 181 **Note**: To get the best performance, ensure that the core and NICs are in the same socket. 182 In the example above ``85:00.0`` is on socket 1 and should be used by cores on socket 1 for the best performance. 183 184#. Check which kernel drivers needs to be loaded and whether there is a need to unbind the network ports from their kernel drivers. 185 More details about DPDK setup and Linux kernel requirements see :ref:`linux_gsg_compiling_dpdk` and :ref:`linux_gsg_linux_drivers`. 186