1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright 2016 Red Hat, Inc. 3 4 5 6PVP reference benchmark setup using testpmd 7=========================================== 8 9This guide lists the steps required to setup a PVP benchmark using testpmd as 10a simple forwarder between NICs and Vhost interfaces. The goal of this setup 11is to have a reference PVP benchmark without using external vSwitches (OVS, 12VPP, ...) to make it easier to obtain reproducible results and to facilitate 13continuous integration testing. 14 15The guide covers two ways of launching the VM, either by directly calling the 16QEMU command line, or by relying on libvirt. It has been tested with DPDK 17v16.11 using RHEL7 for both host and guest. 18 19 20Setup overview 21-------------- 22 23.. _figure_pvp_2nics: 24 25.. figure:: img/pvp_2nics.* 26 27 PVP setup using 2 NICs 28 29In this diagram, each red arrow represents one logical core. This use case 30requires 6 dedicated logical cores. A forwarding configuration with a single 31NIC is also possible, requiring 3 logical cores. 32 33 34Host setup 35---------- 36 37In this setup, we isolate 6 cores (from CPU2 to CPU7) on the same NUMA 38node. Two cores are assigned to the VM vCPUs running testpmd and four are 39assigned to testpmd on the host. 40 41 42Host tuning 43~~~~~~~~~~~ 44 45#. On BIOS, disable turbo-boost and hyper-threads. 46 47#. Append these options to Kernel command line: 48 49 .. code-block:: console 50 51 intel_pstate=disable mce=ignore_ce default_hugepagesz=1G hugepagesz=1G hugepages=6 isolcpus=2-7 nohz_full=2-7 iommu=pt intel_iommu=on 52 53#. Disable hyper-threads at runtime if necessary or if BIOS is not accessible: 54 55 .. code-block:: console 56 57 cat /sys/devices/system/cpu/cpu*[0-9]/topology/thread_siblings_list \ 58 | sort | uniq \ 59 | awk -F, '{system("echo 0 > /sys/devices/system/cpu/cpu"$2"/online")}' 60 61#. Disable NMIs: 62 63 .. code-block:: console 64 65 echo 0 > /proc/sys/kernel/nmi_watchdog 66 67#. Exclude isolated CPUs from the writeback cpumask: 68 69 .. code-block:: console 70 71 echo ffffff03 > /sys/bus/workqueue/devices/writeback/cpumask 72 73#. Isolate CPUs from IRQs: 74 75 .. code-block:: console 76 77 clear_mask=0xfc #Isolate CPU2 to CPU7 from IRQs 78 for i in /proc/irq/*/smp_affinity 79 do 80 echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i 81 done 82 83 84Qemu build 85~~~~~~~~~~ 86 87Build Qemu: 88 89 .. code-block:: console 90 91 git clone git://git.qemu.org/qemu.git 92 cd qemu 93 mkdir bin 94 cd bin 95 ../configure --target-list=x86_64-softmmu 96 make 97 98 99DPDK build 100~~~~~~~~~~ 101 102See :doc:`../linux_gsg/build_dpdk` for details. 103 104 105Testpmd launch 106~~~~~~~~~~~~~~ 107 108#. Assign NICs to DPDK: 109 110 .. code-block:: console 111 112 modprobe vfio-pci 113 usertools/dpdk-devbind -b vfio-pci 0000:11:00.0 0000:11:00.1 114 115 .. Note:: 116 117 The Sandy Bridge family seems to have some IOMMU limitations giving poor 118 performance results. To achieve good performance on these machines 119 consider using UIO instead. 120 121#. Launch the testpmd application: 122 123 .. code-block:: console 124 125 <build_dir>/app/dpdk-testpmd -l 0,2,3,4,5 --socket-mem=1024 -n 4 \ 126 --vdev 'net_vhost0,iface=/tmp/vhost-user1' \ 127 --vdev 'net_vhost1,iface=/tmp/vhost-user2' -- \ 128 --portmask=f -i --rxq=1 --txq=1 \ 129 --nb-cores=4 --forward-mode=io 130 131 With this command, isolated CPUs 2 to 5 will be used as lcores for PMD threads. 132 133#. In testpmd interactive mode, set the portlist to obtain the correct port 134 chaining: 135 136 .. code-block:: console 137 138 set portlist 0,2,1,3 139 start 140 141 142VM launch 143~~~~~~~~~ 144 145The VM may be launched either by calling QEMU directly, or by using libvirt. 146 147Qemu way 148^^^^^^^^ 149 150Launch QEMU with two Virtio-net devices paired to the vhost-user sockets 151created by testpmd. Below example uses default Virtio-net options, but options 152may be specified, for example to disable mergeable buffers or indirect 153descriptors. 154 155 .. code-block:: console 156 157 <QEMU path>/bin/x86_64-softmmu/qemu-system-x86_64 \ 158 -enable-kvm -cpu host -m 3072 -smp 3 \ 159 -chardev socket,id=char0,path=/tmp/vhost-user1 \ 160 -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ 161 -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01,addr=0x10 \ 162 -chardev socket,id=char1,path=/tmp/vhost-user2 \ 163 -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \ 164 -device virtio-net-pci,netdev=mynet2,mac=52:54:00:02:d9:02,addr=0x11 \ 165 -object memory-backend-file,id=mem,size=3072M,mem-path=/dev/hugepages,share=on \ 166 -numa node,memdev=mem -mem-prealloc \ 167 -net user,hostfwd=tcp::1002$1-:22 -net nic \ 168 -qmp unix:/tmp/qmp.socket,server,nowait \ 169 -monitor stdio <vm_image>.qcow2 170 171You can use this `qmp-vcpu-pin <https://patchwork.kernel.org/patch/9361617/>`_ 172script to pin vCPUs. 173 174It can be used as follows, for example to pin 3 vCPUs to CPUs 1, 6 and 7, 175where isolated CPUs 6 and 7 will be used as lcores for Virtio PMDs: 176 177 .. code-block:: console 178 179 export PYTHONPATH=$PYTHONPATH:<QEMU path>/scripts/qmp 180 ./qmp-vcpu-pin -s /tmp/qmp.socket 1 6 7 181 182Libvirt way 183^^^^^^^^^^^ 184 185Some initial steps are required for libvirt to be able to connect to testpmd's 186sockets. 187 188First, SELinux policy needs to be set to permissive, since testpmd is 189generally run as root (note, as reboot is required): 190 191 .. code-block:: console 192 193 cat /etc/selinux/config 194 195 # This file controls the state of SELinux on the system. 196 # SELINUX= can take one of these three values: 197 # enforcing - SELinux security policy is enforced. 198 # permissive - SELinux prints warnings instead of enforcing. 199 # disabled - No SELinux policy is loaded. 200 SELINUX=permissive 201 202 # SELINUXTYPE= can take one of three two values: 203 # targeted - Targeted processes are protected, 204 # minimum - Modification of targeted policy. 205 # Only selected processes are protected. 206 # mls - Multi Level Security protection. 207 SELINUXTYPE=targeted 208 209 210Also, Qemu needs to be run as root, which has to be specified in 211``/etc/libvirt/qemu.conf``: 212 213 .. code-block:: console 214 215 user = "root" 216 217Once the domain created, the following snippet is an extract of he most 218important information (hugepages, vCPU pinning, Virtio PCI devices): 219 220 .. code-block:: xml 221 222 <domain type='kvm'> 223 <memory unit='KiB'>3145728</memory> 224 <currentMemory unit='KiB'>3145728</currentMemory> 225 <memoryBacking> 226 <hugepages> 227 <page size='1048576' unit='KiB' nodeset='0'/> 228 </hugepages> 229 <locked/> 230 </memoryBacking> 231 <vcpu placement='static'>3</vcpu> 232 <cputune> 233 <vcpupin vcpu='0' cpuset='1'/> 234 <vcpupin vcpu='1' cpuset='6'/> 235 <vcpupin vcpu='2' cpuset='7'/> 236 <emulatorpin cpuset='0'/> 237 </cputune> 238 <numatune> 239 <memory mode='strict' nodeset='0'/> 240 </numatune> 241 <os> 242 <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> 243 <boot dev='hd'/> 244 </os> 245 <cpu mode='host-passthrough'> 246 <topology sockets='1' cores='3' threads='1'/> 247 <numa> 248 <cell id='0' cpus='0-2' memory='3145728' unit='KiB' memAccess='shared'/> 249 </numa> 250 </cpu> 251 <devices> 252 <interface type='vhostuser'> 253 <mac address='56:48:4f:53:54:01'/> 254 <source type='unix' path='/tmp/vhost-user1' mode='client'/> 255 <model type='virtio'/> 256 <driver name='vhost' rx_queue_size='256' /> 257 <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/> 258 </interface> 259 <interface type='vhostuser'> 260 <mac address='56:48:4f:53:54:02'/> 261 <source type='unix' path='/tmp/vhost-user2' mode='client'/> 262 <model type='virtio'/> 263 <driver name='vhost' rx_queue_size='256' /> 264 <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/> 265 </interface> 266 </devices> 267 </domain> 268 269 270Guest setup 271----------- 272 273 274Guest tuning 275~~~~~~~~~~~~ 276 277#. Append these options to the Kernel command line: 278 279 .. code-block:: console 280 281 default_hugepagesz=1G hugepagesz=1G hugepages=1 intel_iommu=on iommu=pt isolcpus=1,2 nohz_full=1,2 282 283#. Disable NMIs: 284 285 .. code-block:: console 286 287 echo 0 > /proc/sys/kernel/nmi_watchdog 288 289#. Exclude isolated CPU1 and CPU2 from the writeback cpumask: 290 291 .. code-block:: console 292 293 echo 1 > /sys/bus/workqueue/devices/writeback/cpumask 294 295#. Isolate CPUs from IRQs: 296 297 .. code-block:: console 298 299 clear_mask=0x6 #Isolate CPU1 and CPU2 from IRQs 300 for i in /proc/irq/*/smp_affinity 301 do 302 echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i 303 done 304 305 306DPDK build 307~~~~~~~~~~ 308 309See :doc:`../linux_gsg/build_dpdk` for details. 310 311 312Testpmd launch 313~~~~~~~~~~~~~~ 314 315Probe vfio module without iommu: 316 317 .. code-block:: console 318 319 modprobe -r vfio_iommu_type1 320 modprobe -r vfio 321 modprobe vfio enable_unsafe_noiommu_mode=1 322 cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode 323 modprobe vfio-pci 324 325Bind the virtio-net devices to DPDK: 326 327 .. code-block:: console 328 329 usertools/dpdk-devbind.py -b vfio-pci 0000:00:10.0 0000:00:11.0 330 331Start testpmd: 332 333 .. code-block:: console 334 335 <build_dir>/app/dpdk-testpmd -l 0,1,2 --socket-mem 1024 -n 4 \ 336 --proc-type auto --file-prefix pg -- \ 337 --portmask=3 --forward-mode=macswap --port-topology=chained \ 338 --disable-rss -i --rxq=1 --txq=1 \ 339 --rxd=256 --txd=256 --nb-cores=2 --auto-start 340 341Results template 342---------------- 343 344Below template should be used when sharing results: 345 346 .. code-block:: none 347 348 Traffic Generator: <Test equipment (e.g. IXIA, Moongen, ...)> 349 Acceptable Loss: <n>% 350 Validation run time: <n>min 351 Host DPDK version/commit: <version, SHA-1> 352 Guest DPDK version/commit: <version, SHA-1> 353 Patches applied: <link to patchwork> 354 QEMU version/commit: <version> 355 Virtio features: <features (e.g. mrg_rxbuf='off', leave empty if default)> 356 CPU: <CPU model>, <CPU frequency> 357 NIC: <NIC model> 358 Result: <n> Mpps 359