1.. BSD LICENSE 2 Copyright(c) 2016 Intel Corporation. All rights reserved. 3 All rights reserved. 4 5 Redistribution and use in source and binary forms, with or without 6 modification, are permitted provided that the following conditions 7 are met: 8 9 * Redistributions of source code must retain the above copyright 10 notice, this list of conditions and the following disclaimer. 11 * Redistributions in binary form must reproduce the above copyright 12 notice, this list of conditions and the following disclaimer in 13 the documentation and/or other materials provided with the 14 distribution. 15 * Neither the name of Intel Corporation nor the names of its 16 contributors may be used to endorse or promote products derived 17 from this software without specific prior written permission. 18 19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 31.. _virtio_user_for_container_networking: 32 33Virtio_user for Container Networking 34==================================== 35 36Container becomes more and more popular for strengths, like low overhead, fast 37boot-up time, and easy to deploy, etc. How to use DPDK to accelerate container 38networking becomes a common question for users. There are two use models of 39running DPDK inside containers, as shown in 40:numref:`figure_use_models_for_running_dpdk_in_containers`. 41 42.. _figure_use_models_for_running_dpdk_in_containers: 43 44.. figure:: img/use_models_for_running_dpdk_in_containers.* 45 46 Use models of running DPDK inside container 47 48This page will only cover aggregation model. 49 50Overview 51-------- 52 53The virtual device, virtio-user, with unmodified vhost-user backend, is designed 54for high performance user space container networking or inter-process 55communication (IPC). 56 57The overview of accelerating container networking by virtio-user is shown 58in :numref:`figure_virtio_user_for_container_networking`. 59 60.. _figure_virtio_user_for_container_networking: 61 62.. figure:: img/virtio_user_for_container_networking.* 63 64 Overview of accelerating container networking by virtio-user 65 66Different virtio PCI devices we usually use as a para-virtualization I/O in the 67context of QEMU/VM, the basic idea here is to present a kind of virtual devices, 68which can be attached and initialized by DPDK. The device emulation layer by 69QEMU in VM's context is saved by just registering a new kind of virtual device 70in DPDK's ether layer. And to minimize the change, we reuse already-existing 71virtio PMD code (driver/net/virtio/). 72 73Virtio, in essence, is a shm-based solution to transmit/receive packets. How is 74memory shared? In VM's case, qemu always shares the whole physical layout of VM 75to vhost backend. But it's not feasible for a container, as a process, to share 76all virtual memory regions to backend. So only those virtual memory regions 77(aka, hugepages initialized in DPDK) are sent to backend. It restricts that only 78addresses in these areas can be used to transmit or receive packets. 79 80Sample Usage 81------------ 82 83Here we use Docker as container engine. It also applies to LXC, Rocket with 84some minor changes. 85 86#. Compile DPDK. 87 88 .. code-block:: console 89 90 make install RTE_SDK=`pwd` T=x86_64-native-linuxapp-gcc 91 92#. Write a Dockerfile like below. 93 94 .. code-block:: console 95 96 cat <<EOT >> Dockerfile 97 FROM ubuntu:latest 98 WORKDIR /usr/src/dpdk 99 COPY . /usr/src/dpdk 100 ENV PATH "$PATH:/usr/src/dpdk/x86_64-native-linuxapp-gcc/app/" 101 EOT 102 103#. Build a Docker image. 104 105 .. code-block:: console 106 107 docker build -t dpdk-app-testpmd . 108 109#. Start a testpmd on the host with a vhost-user port. 110 111 .. code-block:: console 112 113 $(testpmd) -c 0x3 -n 4 --socket-mem 1024,1024 \ 114 --vdev 'eth_vhost0,iface=/tmp/sock0' --no-pci -- -i 115 116#. Start a container instance with a virtio-user port. 117 118 .. code-block:: console 119 120 docker run -i -t -v /tmp/sock0:/var/run/usvhost \ 121 -v /dev/hugepages:/dev/hugepages \ 122 dpdk-app-testpmd testpmd -c 0xc -n 4 -m 1024 --no-pci \ 123 --vdev=virtio_user0,path=/var/run/usvhost \ 124 -- -i --txqflags=0xf00 --disable-hw-vlan 125 126Note: If we run all above setup on the host, it's a shm-based IPC. 127 128Limitations 129----------- 130 131We have below limitations in this solution: 132 * Cannot work with --huge-unlink option. As we need to reopen the hugepage 133 file to share with vhost backend. 134 * Cannot work with --no-huge option. Currently, DPDK uses anonymous mapping 135 under this option which cannot be reopened to share with vhost backend. 136 * Cannot work when there are more than VHOST_MEMORY_MAX_NREGIONS(8) hugepages. 137 In another word, do not use 2MB hugepage so far. 138 * Applications should not use file name like HUGEFILE_FMT ("%smap_%d"). That 139 will bring confusion when sharing hugepage files with backend by name. 140 * Root privilege is a must. DPDK resolves physical addresses of hugepages 141 which seems not necessary, and some discussions are going on to remove this 142 restriction. 143