1.. BSD LICENSE 2 Copyright(c) 2016 Intel Corporation. All rights reserved. 3 All rights reserved. 4 5 Redistribution and use in source and binary forms, with or without 6 modification, are permitted provided that the following conditions 7 are met: 8 9 * Redistributions of source code must retain the above copyright 10 notice, this list of conditions and the following disclaimer. 11 * Redistributions in binary form must reproduce the above copyright 12 notice, this list of conditions and the following disclaimer in 13 the documentation and/or other materials provided with the 14 distribution. 15 * Neither the name of Intel Corporation nor the names of its 16 contributors may be used to endorse or promote products derived 17 from this software without specific prior written permission. 18 19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 31.. _virtio_user_for_container_networking: 32 33Virtio_user for Container Networking 34==================================== 35 36Container becomes more and more popular for strengths, like low overhead, fast 37boot-up time, and easy to deploy, etc. How to use DPDK to accelerate container 38networking becomes a common question for users. There are two use models of 39running DPDK inside containers, as shown in 40:numref:`figure_use_models_for_running_dpdk_in_containers`. 41 42.. _figure_use_models_for_running_dpdk_in_containers: 43 44.. figure:: img/use_models_for_running_dpdk_in_containers.* 45 46 Use models of running DPDK inside container 47 48This page will only cover aggregation model. 49 50Overview 51-------- 52 53The virtual device, virtio-user, with unmodified vhost-user backend, is designed 54for high performance user space container networking or inter-process 55communication (IPC). 56 57The overview of accelerating container networking by virtio-user is shown 58in :numref:`figure_virtio_user_for_container_networking`. 59 60.. _figure_virtio_user_for_container_networking: 61 62.. figure:: img/virtio_user_for_container_networking.* 63 64 Overview of accelerating container networking by virtio-user 65 66Different virtio PCI devices we usually use as a para-virtualization I/O in the 67context of QEMU/VM, the basic idea here is to present a kind of virtual devices, 68which can be attached and initialized by DPDK. The device emulation layer by 69QEMU in VM's context is saved by just registering a new kind of virtual device 70in DPDK's ether layer. And to minimize the change, we reuse already-existing 71virtio PMD code (driver/net/virtio/). 72 73Virtio, in essence, is a shm-based solution to transmit/receive packets. How is 74memory shared? In VM's case, qemu always shares the whole physical layout of VM 75to vhost backend. But it's not feasible for a container, as a process, to share 76all virtual memory regions to backend. So only those virtual memory regions 77(aka, hugepages initialized in DPDK) are sent to backend. It restricts that only 78addresses in these areas can be used to transmit or receive packets. 79 80Sample Usage 81------------ 82 83Here we use Docker as container engine. It also applies to LXC, Rocket with 84some minor changes. 85 86#. Compile DPDK. 87 88 .. code-block:: console 89 90 make install RTE_SDK=`pwd` T=x86_64-native-linuxapp-gcc 91 92#. Write a Dockerfile like below. 93 94 .. code-block:: console 95 96 cat <<EOT >> Dockerfile 97 FROM ubuntu:latest 98 WORKDIR /usr/src/dpdk 99 COPY . /usr/src/dpdk 100 ENV PATH "$PATH:/usr/src/dpdk/x86_64-native-linuxapp-gcc/app/" 101 EOT 102 103#. Build a Docker image. 104 105 .. code-block:: console 106 107 docker build -t dpdk-app-testpmd . 108 109#. Start a testpmd on the host with a vhost-user port. 110 111 .. code-block:: console 112 113 $(testpmd) -l 0-1 -n 4 --socket-mem 1024,1024 \ 114 --vdev 'eth_vhost0,iface=/tmp/sock0' \ 115 --file-prefix=host --no-pci -- -i 116 117#. Start a container instance with a virtio-user port. 118 119 .. code-block:: console 120 121 docker run -i -t -v /tmp/sock0:/var/run/usvhost \ 122 -v /dev/hugepages:/dev/hugepages \ 123 dpdk-app-testpmd testpmd -l 6-7 -n 4 -m 1024 --no-pci \ 124 --vdev=virtio_user0,path=/var/run/usvhost \ 125 --file-prefix=container \ 126 -- -i --txqflags=0xf00 --disable-hw-vlan 127 128Note: If we run all above setup on the host, it's a shm-based IPC. 129 130Limitations 131----------- 132 133We have below limitations in this solution: 134 * Cannot work with --huge-unlink option. As we need to reopen the hugepage 135 file to share with vhost backend. 136 * Cannot work with --no-huge option. Currently, DPDK uses anonymous mapping 137 under this option which cannot be reopened to share with vhost backend. 138 * Cannot work when there are more than VHOST_MEMORY_MAX_NREGIONS(8) hugepages. 139 In another word, do not use 2MB hugepage so far. 140 * Applications should not use file name like HUGEFILE_FMT ("%smap_%d"). That 141 will bring confusion when sharing hugepage files with backend by name. 142 * Root privilege is a must. DPDK resolves physical addresses of hugepages 143 which seems not necessary, and some discussions are going on to remove this 144 restriction. 145