1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2010-2014 Intel Corporation. 3 4.. _Multi-process_Support: 5 6Multi-process Support 7===================== 8 9In the DPDK, multi-process support is designed to allow a group of DPDK processes 10to work together in a simple transparent manner to perform packet processing, 11or other workloads. 12To support this functionality, 13a number of additions have been made to the core DPDK Environment Abstraction Layer (EAL). 14 15The EAL has been modified to allow different types of DPDK processes to be spawned, 16each with different permissions on the hugepage memory used by the applications. 17For now, there are two types of process specified: 18 19* primary processes, which can initialize and which have full permissions on shared memory 20 21* secondary processes, which cannot initialize shared memory, 22 but can attach to pre- initialized shared memory and create objects in it. 23 24Standalone DPDK processes are primary processes, 25while secondary processes can only run alongside a primary process or 26after a primary process has already configured the hugepage shared memory for them. 27 28.. note:: 29 30 Secondary processes should run alongside primary process with same DPDK version. 31 32 Secondary processes which requires access to physical devices in Primary process, must 33 be passed with the same whitelist and blacklist options. 34 35To support these two process types, and other multi-process setups described later, 36two additional command-line parameters are available to the EAL: 37 38* ``--proc-type:`` for specifying a given process instance as the primary or secondary DPDK instance 39 40* ``--file-prefix:`` to allow processes that do not want to co-operate to have different memory regions 41 42A number of example applications are provided that demonstrate how multiple DPDK processes can be used together. 43These are more fully documented in the "Multi- process Sample Application" chapter 44in the *DPDK Sample Application's User Guide*. 45 46Memory Sharing 47-------------- 48 49The key element in getting a multi-process application working using the DPDK is to ensure that 50memory resources are properly shared among the processes making up the multi-process application. 51Once there are blocks of shared memory available that can be accessed by multiple processes, 52then issues such as inter-process communication (IPC) becomes much simpler. 53 54On application start-up in a primary or standalone process, 55the DPDK records to memory-mapped files the details of the memory configuration it is using - hugepages in use, 56the virtual addresses they are mapped at, the number of memory channels present, etc. 57When a secondary process is started, these files are read and the EAL recreates the same memory configuration 58in the secondary process so that all memory zones are shared between processes and all pointers to that memory are valid, 59and point to the same objects, in both processes. 60 61.. note:: 62 63 Refer to `Multi-process Limitations`_ for details of 64 how Linux kernel Address-Space Layout Randomization (ASLR) can affect memory sharing. 65 66.. _figure_multi_process_memory: 67 68.. figure:: img/multi_process_memory.* 69 70 Memory Sharing in the DPDK Multi-process Sample Application 71 72 73The EAL also supports an auto-detection mode (set by EAL ``--proc-type=auto`` flag ), 74whereby an DPDK process is started as a secondary instance if a primary instance is already running. 75 76Deployment Models 77----------------- 78 79Symmetric/Peer Processes 80~~~~~~~~~~~~~~~~~~~~~~~~ 81 82DPDK multi-process support can be used to create a set of peer processes where each process performs the same workload. 83This model is equivalent to having multiple threads each running the same main-loop function, 84as is done in most of the supplied DPDK sample applications. 85In this model, the first of the processes spawned should be spawned using the ``--proc-type=primary`` EAL flag, 86while all subsequent instances should be spawned using the ``--proc-type=secondary`` flag. 87 88The simple_mp and symmetric_mp sample applications demonstrate this usage model. 89They are described in the "Multi-process Sample Application" chapter in the *DPDK Sample Application's User Guide*. 90 91Asymmetric/Non-Peer Processes 92~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 93 94An alternative deployment model that can be used for multi-process applications 95is to have a single primary process instance that acts as a load-balancer or 96server distributing received packets among worker or client threads, which are run as secondary processes. 97In this case, extensive use of rte_ring objects is made, which are located in shared hugepage memory. 98 99The client_server_mp sample application shows this usage model. 100It is described in the "Multi-process Sample Application" chapter in the *DPDK Sample Application's User Guide*. 101 102Running Multiple Independent DPDK Applications 103~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 104 105In addition to the above scenarios involving multiple DPDK processes working together, 106it is possible to run multiple DPDK processes side-by-side, 107where those processes are all working independently. 108Support for this usage scenario is provided using the ``--file-prefix`` parameter to the EAL. 109 110By default, the EAL creates hugepage files on each hugetlbfs filesystem using the rtemap_X filename, 111where X is in the range 0 to the maximum number of hugepages -1. 112Similarly, it creates shared configuration files, memory mapped in each process, using the /var/run/.rte_config filename, 113when run as root (or $HOME/.rte_config when run as a non-root user; 114if filesystem and device permissions are set up to allow this). 115The rte part of the filenames of each of the above is configurable using the file-prefix parameter. 116 117In addition to specifying the file-prefix parameter, 118any DPDK applications that are to be run side-by-side must explicitly limit their memory use. 119This is done by passing the -m flag to each process to specify how much hugepage memory, in megabytes, 120each process can use (or passing ``--socket-mem`` to specify how much hugepage memory on each socket each process can use). 121 122.. note:: 123 124 Independent DPDK instances running side-by-side on a single machine cannot share any network ports. 125 Any network ports being used by one process should be blacklisted in every other process. 126 127Running Multiple Independent Groups of DPDK Applications 128~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 129 130In the same way that it is possible to run independent DPDK applications side- by-side on a single system, 131this can be trivially extended to multi-process groups of DPDK applications running side-by-side. 132In this case, the secondary processes must use the same ``--file-prefix`` parameter 133as the primary process whose shared memory they are connecting to. 134 135.. note:: 136 137 All restrictions and issues with multiple independent DPDK processes running side-by-side 138 apply in this usage scenario also. 139 140Multi-process Limitations 141------------------------- 142 143There are a number of limitations to what can be done when running DPDK multi-process applications. 144Some of these are documented below: 145 146* The multi-process feature requires that the exact same hugepage memory mappings be present in all applications. 147 The Linux security feature - Address-Space Layout Randomization (ASLR) can interfere with this mapping, 148 so it may be necessary to disable this feature in order to reliably run multi-process applications. 149 150.. warning:: 151 152 Disabling Address-Space Layout Randomization (ASLR) may have security implications, 153 so it is recommended that it be disabled only when absolutely necessary, 154 and only when the implications of this change have been understood. 155 156* All DPDK processes running as a single application and using shared memory must have distinct coremask/corelist arguments. 157 It is not possible to have a primary and secondary instance, or two secondary instances, 158 using any of the same logical cores. 159 Attempting to do so can cause corruption of memory pool caches, among other issues. 160 161* The delivery of interrupts, such as Ethernet* device link status interrupts, do not work in secondary processes. 162 All interrupts are triggered inside the primary process only. 163 Any application needing interrupt notification in multiple processes should provide its own mechanism 164 to transfer the interrupt information from the primary process to any secondary process that needs the information. 165 166* The use of function pointers between multiple processes running based of different compiled binaries is not supported, 167 since the location of a given function in one process may be different to its location in a second. 168 This prevents the librte_hash library from behaving properly as in a multi-threaded instance, 169 since it uses a pointer to the hash function internally. 170 171To work around this issue, it is recommended that multi-process applications perform the hash calculations by directly calling 172the hashing function from the code and then using the rte_hash_add_with_hash()/rte_hash_lookup_with_hash() functions 173instead of the functions which do the hashing internally, such as rte_hash_add()/rte_hash_lookup(). 174 175* Depending upon the hardware in use, and the number of DPDK processes used, 176 it may not be possible to have HPET timers available in each DPDK instance. 177 The minimum number of HPET comparators available to Linux* userspace can be just a single comparator, 178 which means that only the first, primary DPDK process instance can open and mmap /dev/hpet. 179 If the number of required DPDK processes exceeds that of the number of available HPET comparators, 180 the TSC (which is the default timer in this release) must be used as a time source across all processes instead of the HPET. 181