1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2016 Intel Corporation. 3 4L2 Forwarding Sample Application with Cache Allocation Technology (CAT) 5======================================================================= 6 7Basic Forwarding sample application is a simple *skeleton* example of 8a forwarding application. It has been extended to make use of CAT via extended 9command line options and linking against the libpqos library. 10 11It is intended as a demonstration of the basic components of a DPDK forwarding 12application and use of the libpqos library to program CAT. 13For more detailed implementations see the L2 and L3 forwarding 14sample applications. 15 16CAT and Code Data Prioritization (CDP) features allow management of the CPU's 17last level cache. CAT introduces classes of service (COS) that are essentially 18bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to 19one cache way in last level cache. 20A CPU core is always assigned to one of the CAT classes. 21By programming CPU core assignment and COS bitmasks, applications can be given 22exclusive, shared, or mixed access to the CPU's last level cache. 23CDP extends CAT so that there are two bitmasks per COS, 24one for data and one for code. 25The number of classes and number of valid bits in a COS bitmask is CPU model 26specific and COS bitmasks need to be contiguous. Sample code calls this bitmask 27``cbm`` or capacity bitmask. 28By default, after reset, all CPU cores are assigned to COS 0 and all classes 29are programmed to allow fill into all cache ways. 30CDP is off by default. 31 32For more information about CAT please see: 33 34* https://github.com/01org/intel-cmt-cat 35 36White paper demonstrating example use case: 37 38* `Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit <http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html>`_ 39 40Compiling the Application 41------------------------- 42.. note:: 43 44 Requires ``libpqos`` from Intel's 45 `intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_ 46 hosted on GitHub repository. For installation notes, please see ``README`` file. 47 48 GIT: 49 50 * https://github.com/01org/intel-cmt-cat 51 52 53#. To compile the application export the path to PQoS lib 54 and the DPDK source tree and go to the example directory: 55 56 .. code-block:: console 57 58 export PQOS_INSTALL_PATH=/path/to/libpqos 59 60 61To compile the sample application see :doc:`compiling`. 62 63The application is located in the ``l2fwd-cat`` sub-directory. 64 65 66Running the Application 67----------------------- 68 69To run the example in a ``linux`` environment and enable CAT on cpus 0-2: 70 71.. code-block:: console 72 73 ./<build_dir>/examples/dpdk-l2fwd-cat -l 1 -n 4 -- --l3ca="0x3@(0-2)" 74 75or to enable CAT and CDP on cpus 1,3: 76 77.. code-block:: console 78 79 ./<build_dir>/examples/dpdk-l2fwd-cat -l 1 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)" 80 81If CDP is not supported it will fail with following error message: 82 83.. code-block:: console 84 85 PQOS: CDP requested but not supported. 86 PQOS: Requested CAT configuration is not valid! 87 PQOS: Shutting down PQoS library... 88 EAL: Error - exiting with code: 1 89 Cause: PQOS: L3CA init failed! 90 91The option to enable CAT is: 92 93* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``: 94 95 where ``cbm`` stands for capacity bitmask and must be expressed in 96 hexadecimal form. 97 98 ``common_cbm`` is a single mask, for a CDP enabled system, a group of two 99 masks (``code_cbm`` and ``data_cbm``) is used. 100 101 ``(`` and ``)`` are necessary if it's a group. 102 103 ``cpus`` could be a single digit/range or a group and must be expressed in 104 decimal form. 105 106 ``(`` and ``)`` are necessary if it's a group. 107 108 e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'`` 109 110 * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6; 111 112 * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3; 113 114 * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways; 115 116 * cpu 7 has exclusive access to all of its 4 ways; 117 118 e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system 119 120 * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and 121 data ways are not overlapping. 122 123 124Refer to *DPDK Getting Started Guide* for general information on running 125applications and the Environment Abstraction Layer (EAL) options. 126 127 128To reset or list CAT configuration and control CDP please use ``pqos`` tool 129from Intel's 130`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_. 131 132To enabled or disable CDP: 133 134.. code-block:: console 135 136 sudo ./pqos -S cdp-on 137 138 sudo ./pqos -S cdp-off 139 140to reset CAT configuration: 141 142.. code-block:: console 143 144 sudo ./pqos -R 145 146to list CAT config: 147 148.. code-block:: console 149 150 sudo ./pqos -s 151 152For more info about ``pqos`` tool please see its man page or 153`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_. 154 155 156Explanation 157----------- 158 159The following sections provide an explanation of the main components of the 160code. 161 162All DPDK library functions used in the sample code are prefixed with ``rte_`` 163and are explained in detail in the *DPDK API Documentation*. 164 165 166The Main Function 167~~~~~~~~~~~~~~~~~ 168 169The ``main()`` function performs the initialization and calls the execution 170threads for each lcore. 171 172The first task is to initialize the Environment Abstraction Layer (EAL). The 173``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` 174function. The value returned is the number of parsed arguments: 175 176.. code-block:: c 177 178 int ret = rte_eal_init(argc, argv); 179 if (ret < 0) 180 rte_exit(EXIT_FAILURE, "Error with EAL initialization\n"); 181 182The next task is to initialize the PQoS library and configure CAT. The 183``argc`` and ``argv`` arguments are provided to the ``cat_init()`` 184function. The value returned is the number of parsed arguments: 185 186.. code-block:: c 187 188 int ret = cat_init(argc, argv); 189 if (ret < 0) 190 rte_exit(EXIT_FAILURE, "PQOS: L3CA init failed!\n"); 191 192``cat_init()`` is a wrapper function which parses the command, validates 193the requested parameters and configures CAT accordingly. 194 195Parsing of command line arguments is done in ``parse_args(...)``. 196libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is 197queried for system CPU information and L3CA capabilities via 198``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)`` 199calls. When all capability and topology information is collected, the requested 200CAT configuration is validated. A check is then performed (on per socket basis) 201for a sufficient number of un-associated COS. COS are selected and 202configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to 203relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls. 204 205``atexit(...)`` is used to register ``cat_exit(...)`` to be called on 206a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating 207COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls. 208