1.. BSD LICENSE 2 Copyright(c) 2016 Intel Corporation. All rights reserved. 3 All rights reserved. 4 5 Redistribution and use in source and binary forms, with or without 6 modification, are permitted provided that the following conditions 7 are met: 8 9 * Redistributions of source code must retain the above copyright 10 notice, this list of conditions and the following disclaimer. 11 * Redistributions in binary form must reproduce the above copyright 12 notice, this list of conditions and the following disclaimer in 13 the documentation and/or other materials provided with the 14 distribution. 15 * Neither the name of Intel Corporation nor the names of its 16 contributors may be used to endorse or promote products derived 17 from this software without specific prior written permission. 18 19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 22 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 23 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 24 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 25 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 31 32L2 Forwarding Sample Application with Cache Allocation Technology (CAT) 33======================================================================= 34 35Basic Forwarding sample application is a simple *skeleton* example of 36a forwarding application. It has been extended to make use of CAT via extended 37command line options and linking against the libpqos library. 38 39It is intended as a demonstration of the basic components of a DPDK forwarding 40application and use of the libpqos library to program CAT. 41For more detailed implementations see the L2 and L3 forwarding 42sample applications. 43 44CAT and Code Data Prioritization (CDP) features allow management of the CPU's 45last level cache. CAT introduces classes of service (COS) that are essentially 46bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to 47one cache way in last level cache. 48A CPU core is always assigned to one of the CAT classes. 49By programming CPU core assignment and COS bitmasks, applications can be given 50exclusive, shared, or mixed access to the CPU's last level cache. 51CDP extends CAT so that there are two bitmasks per COS, 52one for data and one for code. 53The number of classes and number of valid bits in a COS bitmask is CPU model 54specific and COS bitmasks need to be contiguous. Sample code calls this bitmask 55``cbm`` or capacity bitmask. 56By default, after reset, all CPU cores are assigned to COS 0 and all classes 57are programmed to allow fill into all cache ways. 58CDP is off by default. 59 60For more information about CAT please see: 61 62* https://github.com/01org/intel-cmt-cat 63 64White paper demonstrating example use case: 65 66* `Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit <http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html>`_ 67 68Compiling the Application 69------------------------- 70 71Requires ``libpqos`` from Intel's 72`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_ 73hosted on GitHub repository. For installation notes, please see ``README`` file. 74 75GIT: 76 77* https://github.com/01org/intel-cmt-cat 78 79To compile the application export the path to PQoS lib 80and the DPDK source tree and go to the example directory: 81 82.. code-block:: console 83 84 export PQOS_INSTALL_PATH=/path/to/libpqos 85 export RTE_SDK=/path/to/rte_sdk 86 87 cd ${RTE_SDK}/examples/l2fwd-cat 88 89Set the target, for example: 90 91.. code-block:: console 92 93 export RTE_TARGET=x86_64-native-linuxapp-gcc 94 95See the *DPDK Getting Started* Guide for possible ``RTE_TARGET`` values. 96 97Build the application as follows: 98 99.. code-block:: console 100 101 make 102 103 104Running the Application 105----------------------- 106 107To run the example in a ``linuxapp`` environment and enable CAT on cpus 0-2: 108 109.. code-block:: console 110 111 ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="0x3@(0-2)" 112 113or to enable CAT and CDP on cpus 1,3: 114 115.. code-block:: console 116 117 ./build/l2fwd-cat -c 2 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)" 118 119If CDP is not supported it will fail with following error message: 120 121.. code-block:: console 122 123 PQOS: CDP requested but not supported. 124 PQOS: Requested CAT configuration is not valid! 125 PQOS: Shutting down PQoS library... 126 EAL: Error - exiting with code: 1 127 Cause: PQOS: L3CA init failed! 128 129The option to enable CAT is: 130 131* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``: 132 133 where ``cbm`` stands for capacity bitmask and must be expressed in 134 hexadecimal form. 135 136 ``common_cbm`` is a single mask, for a CDP enabled system, a group of two 137 masks (``code_cbm`` and ``data_cbm``) is used. 138 139 ``(`` and ``)`` are necessary if it's a group. 140 141 ``cpus`` could be a single digit/range or a group and must be expressed in 142 decimal form. 143 144 ``(`` and ``)`` are necessary if it's a group. 145 146 e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'`` 147 148 * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6; 149 150 * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3; 151 152 * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways; 153 154 * cpu 7 has exclusive access to all of its 4 ways; 155 156 e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system 157 158 * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and 159 data ways are not overlapping. 160 161 162Refer to *DPDK Getting Started Guide* for general information on running 163applications and the Environment Abstraction Layer (EAL) options. 164 165 166To reset or list CAT configuration and control CDP please use ``pqos`` tool 167from Intel's 168`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_. 169 170To enabled or disable CDP: 171 172.. code-block:: console 173 174 sudo ./pqos -S cdp-on 175 176 sudo ./pqos -S cdp-off 177 178to reset CAT configuration: 179 180.. code-block:: console 181 182 sudo ./pqos -R 183 184to list CAT config: 185 186.. code-block:: console 187 188 sudo ./pqos -s 189 190For more info about ``pqos`` tool please see its man page or 191`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_. 192 193 194Explanation 195----------- 196 197The following sections provide an explanation of the main components of the 198code. 199 200All DPDK library functions used in the sample code are prefixed with ``rte_`` 201and are explained in detail in the *DPDK API Documentation*. 202 203 204The Main Function 205~~~~~~~~~~~~~~~~~ 206 207The ``main()`` function performs the initialization and calls the execution 208threads for each lcore. 209 210The first task is to initialize the Environment Abstraction Layer (EAL). The 211``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` 212function. The value returned is the number of parsed arguments: 213 214.. code-block:: c 215 216 int ret = rte_eal_init(argc, argv); 217 if (ret < 0) 218 rte_exit(EXIT_FAILURE, "Error with EAL initialization\n"); 219 220The next task is to initialize the PQoS library and configure CAT. The 221``argc`` and ``argv`` arguments are provided to the ``cat_init()`` 222function. The value returned is the number of parsed arguments: 223 224.. code-block:: c 225 226 int ret = cat_init(argc, argv); 227 if (ret < 0) 228 rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n"); 229 230``cat_init()`` is a wrapper function which parses the command, validates 231the requested parameters and configures CAT accordingly. 232 233Parsing of command line arguments is done in ``parse_args(...)``. 234libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is 235queried for system CPU information and L3CA capabilities via 236``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)`` 237calls. When all capability and topology information is collected, the requested 238CAT configuration is validated. A check is then performed (on per socket basis) 239for a sufficient number of unassociated COS. COS are selected and 240configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to 241relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls. 242 243``atexit(...)`` is used to register ``cat_exit(...)`` to be called on 244a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating 245COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls.