1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2016 Intel Corporation. 3 4L2 Forwarding Sample Application with Cache Allocation Technology (CAT) 5======================================================================= 6 7The basic forwarding sample application is a *skeleton* example 8of a forwarding application. 9It has been extended to make use of CAT 10via extended command line options and linking against the libpqos library. 11 12Overview 13-------- 14 15This app is intended as a demonstration of the basic components 16of a DPDK forwarding application 17and use of the libpqos library to the program CAT. 18For more detailed implementations, see the L2 and L3 forwarding sample applications. 19 20CAT and Code Data Prioritization (CDP) features allow management of the CPU's 21last level cache. CAT introduces classes of service (COS) that are essentially 22bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to 23one cache way in last level cache. 24 25A CPU core is always assigned to one of the CAT classes. 26By programming CPU core assignment and COS bitmasks, applications can be given 27exclusive, shared, or mixed access to the CPU's last level cache. 28CDP extends CAT so that there are two bitmasks per COS, 29one for data and one for code. 30The number of classes and number of valid bits in a COS bitmask is CPU model 31specific and COS bitmasks need to be contiguous. Sample code calls this bitmask 32``cbm`` or capacity bitmask. 33By default, after reset, all CPU cores are assigned to COS 0 and all classes 34are programmed to allow fill into all cache ways. 35CDP is off by default. 36 37For more information about CAT, please see: 38 39* https://github.com/01org/intel-cmt-cat 40 41White paper demonstrating example use case: 42 43* `Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit <http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html>`_ 44 45Compiling the Application 46------------------------- 47 48.. note:: 49 50 Requires ``libpqos`` from Intel's 51 `intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_ 52 hosted on GitHub repository. For installation notes, please see ``README`` file. 53 54 GIT: 55 56 * https://github.com/01org/intel-cmt-cat 57 58To compile the application, export the path to PQoS lib: 59 60.. code-block:: console 61 62 export CFLAGS=-I/path/to/intel-cmt-cat/include 63 export LDFLAGS=-L/path/to/intel-cmt-cat/lib 64 65To compile the sample application, see :doc:`compiling`. 66 67The application is located in the ``l2fwd-cat`` sub-directory. 68 69 70Running the Application 71----------------------- 72 73To run the example in a ``linux`` environment and enable CAT on cpus 0-2: 74 75.. code-block:: console 76 77 ./<build_dir>/examples/dpdk-l2fwd-cat -l 1 -n 4 -- --l3ca="0x3@(0-2)" 78 79Or to enable CAT and CDP on cpus 1,3: 80 81.. code-block:: console 82 83 ./<build_dir>/examples/dpdk-l2fwd-cat -l 1 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)" 84 85If CDP is not supported, it will fail with following error message: 86 87.. code-block:: console 88 89 PQOS: CDP requested but not supported. 90 PQOS: Requested CAT configuration is not valid! 91 PQOS: Shutting down PQoS library... 92 EAL: Error - exiting with code: 1 93 Cause: PQOS: L3CA init failed! 94 95The option to enable CAT is: 96 97* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``: 98 99 where ``cbm`` stands for capacity bitmask and must be expressed in 100 hexadecimal form. 101 102 ``common_cbm`` is a single mask, for a CDP enabled system, a group of two 103 masks (``code_cbm`` and ``data_cbm``) is used. 104 105 ``(`` and ``)`` are necessary if it's a group. 106 107 ``cpus`` could be a single digit/range or a group and must be expressed in 108 decimal form. 109 110 ``(`` and ``)`` are necessary if it's a group. 111 112 e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'`` 113 114 * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6; 115 116 * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3; 117 118 * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways; 119 120 * cpu 7 has exclusive access to all of its 4 ways; 121 122 e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system 123 124 * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and 125 data ways are not overlapping. 126 127 128Refer to *DPDK Getting Started Guide* for general information on running 129applications and the Environment Abstraction Layer (EAL) options. 130 131 132To reset or list CAT configuration and control CDP please use ``pqos`` tool 133from Intel's 134`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_. 135 136To enabled or disable CDP: 137 138.. code-block:: console 139 140 sudo ./pqos -S cdp-on 141 142 sudo ./pqos -S cdp-off 143 144to reset CAT configuration: 145 146.. code-block:: console 147 148 sudo ./pqos -R 149 150to list CAT config: 151 152.. code-block:: console 153 154 sudo ./pqos -s 155 156For more info about ``pqos`` tool please see its man page or 157`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_. 158 159 160Explanation 161----------- 162 163The following sections provide an explanation of the main components of the 164code. 165 166All DPDK library functions used in the sample code are prefixed with ``rte_`` 167and are explained in detail in the *DPDK API Documentation*. 168 169 170The Main Function 171~~~~~~~~~~~~~~~~~ 172 173The ``main()`` function performs the initialization and calls the execution 174threads for each lcore. 175 176The first task is to initialize the Environment Abstraction Layer (EAL). The 177``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` 178function. The value returned is the number of parsed arguments: 179 180.. literalinclude:: ../../../examples/l2fwd-cat/l2fwd-cat.c 181 :language: c 182 :start-after: Initialize the Environment Abstraction Layer (EAL). 8< 183 :end-before: >8 End of initialization the Environment Abstraction Layer (EAL). 184 :dedent: 1 185 186The next task is to initialize the PQoS library and configure CAT. The 187``argc`` and ``argv`` arguments are provided to the ``cat_init()`` 188function. The value returned is the number of parsed arguments: 189 190.. literalinclude:: ../../../examples/l2fwd-cat/l2fwd-cat.c 191 :language: c 192 :start-after: Initialize the PQoS. 8< 193 :end-before: >8 End of initialization of PQoS. 194 :dedent: 1 195 196``cat_init()`` is a wrapper function which parses the command, validates 197the requested parameters and configures CAT accordingly. 198 199The parsing of command line arguments is done in ``parse_args(...)``. 200Libpqos is then initialized with the ``pqos_init(...)`` call. 201Next, libpqos is 202queried for system CPU information and L3CA capabilities via 203``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)`` 204calls. When all capability and topology information is collected, the requested 205CAT configuration is validated. A check is then performed (on per socket basis) 206for a sufficient number of un-associated COS. COS are selected and 207configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to 208relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls. 209 210``atexit(...)`` is used to register ``cat_exit(...)`` to be called on 211a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating 212COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls. 213