1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2016 Intel Corporation. 3 4L2 Forwarding Sample Application with Cache Allocation Technology (CAT) 5======================================================================= 6 7Basic Forwarding sample application is a simple *skeleton* example of 8a forwarding application. It has been extended to make use of CAT via extended 9command line options and linking against the libpqos library. 10 11It is intended as a demonstration of the basic components of a DPDK forwarding 12application and use of the libpqos library to program CAT. 13For more detailed implementations see the L2 and L3 forwarding 14sample applications. 15 16CAT and Code Data Prioritization (CDP) features allow management of the CPU's 17last level cache. CAT introduces classes of service (COS) that are essentially 18bitmasks. In current CAT implementations, a bit in a COS bitmask corresponds to 19one cache way in last level cache. 20A CPU core is always assigned to one of the CAT classes. 21By programming CPU core assignment and COS bitmasks, applications can be given 22exclusive, shared, or mixed access to the CPU's last level cache. 23CDP extends CAT so that there are two bitmasks per COS, 24one for data and one for code. 25The number of classes and number of valid bits in a COS bitmask is CPU model 26specific and COS bitmasks need to be contiguous. Sample code calls this bitmask 27``cbm`` or capacity bitmask. 28By default, after reset, all CPU cores are assigned to COS 0 and all classes 29are programmed to allow fill into all cache ways. 30CDP is off by default. 31 32For more information about CAT please see: 33 34* https://github.com/01org/intel-cmt-cat 35 36White paper demonstrating example use case: 37 38* `Increasing Platform Determinism with Platform Quality of Service for the Data Plane Development Kit <http://www.intel.com/content/www/us/en/communications/increasing-platform-determinism-pqos-dpdk-white-paper.html>`_ 39 40Compiling the Application 41------------------------- 42.. note:: 43 44 Requires ``libpqos`` from Intel's 45 `intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_ 46 hosted on GitHub repository. For installation notes, please see ``README`` file. 47 48 GIT: 49 50 * https://github.com/01org/intel-cmt-cat 51 52 53To compile the application, export the path to PQoS lib: 54 55.. code-block:: console 56 57 export CFLAGS=-I/path/to/intel-cmt-cat/include 58 export LDFLAGS=-L/path/to/intel-cmt-cat/lib 59 60To compile the sample application see :doc:`compiling`. 61 62The application is located in the ``l2fwd-cat`` sub-directory. 63 64 65Running the Application 66----------------------- 67 68To run the example in a ``linux`` environment and enable CAT on cpus 0-2: 69 70.. code-block:: console 71 72 ./<build_dir>/examples/dpdk-l2fwd-cat -l 1 -n 4 -- --l3ca="0x3@(0-2)" 73 74or to enable CAT and CDP on cpus 1,3: 75 76.. code-block:: console 77 78 ./<build_dir>/examples/dpdk-l2fwd-cat -l 1 -n 4 -- --l3ca="(0x00C00,0x00300)@(1,3)" 79 80If CDP is not supported it will fail with following error message: 81 82.. code-block:: console 83 84 PQOS: CDP requested but not supported. 85 PQOS: Requested CAT configuration is not valid! 86 PQOS: Shutting down PQoS library... 87 EAL: Error - exiting with code: 1 88 Cause: PQOS: L3CA init failed! 89 90The option to enable CAT is: 91 92* ``--l3ca='<common_cbm@cpus>[,<(code_cbm,data_cbm)@cpus>...]'``: 93 94 where ``cbm`` stands for capacity bitmask and must be expressed in 95 hexadecimal form. 96 97 ``common_cbm`` is a single mask, for a CDP enabled system, a group of two 98 masks (``code_cbm`` and ``data_cbm``) is used. 99 100 ``(`` and ``)`` are necessary if it's a group. 101 102 ``cpus`` could be a single digit/range or a group and must be expressed in 103 decimal form. 104 105 ``(`` and ``)`` are necessary if it's a group. 106 107 e.g. ``--l3ca='0x00F00@(1,3),0x0FF00@(4-6),0xF0000@7'`` 108 109 * cpus 1 and 3 share its 4 ways with cpus 4, 5 and 6; 110 111 * cpus 4, 5 and 6 share half (4 out of 8 ways) of its L3 with cpus 1 and 3; 112 113 * cpus 4, 5 and 6 have exclusive access to 4 out of 8 ways; 114 115 * cpu 7 has exclusive access to all of its 4 ways; 116 117 e.g. ``--l3ca='(0x00C00,0x00300)@(1,3)'`` for CDP enabled system 118 119 * cpus 1 and 3 have access to 2 ways for code and 2 ways for data, code and 120 data ways are not overlapping. 121 122 123Refer to *DPDK Getting Started Guide* for general information on running 124applications and the Environment Abstraction Layer (EAL) options. 125 126 127To reset or list CAT configuration and control CDP please use ``pqos`` tool 128from Intel's 129`intel-cmt-cat software package <https://github.com/01org/intel-cmt-cat>`_. 130 131To enabled or disable CDP: 132 133.. code-block:: console 134 135 sudo ./pqos -S cdp-on 136 137 sudo ./pqos -S cdp-off 138 139to reset CAT configuration: 140 141.. code-block:: console 142 143 sudo ./pqos -R 144 145to list CAT config: 146 147.. code-block:: console 148 149 sudo ./pqos -s 150 151For more info about ``pqos`` tool please see its man page or 152`intel-cmt-cat wiki <https://github.com/01org/intel-cmt-cat/wiki>`_. 153 154 155Explanation 156----------- 157 158The following sections provide an explanation of the main components of the 159code. 160 161All DPDK library functions used in the sample code are prefixed with ``rte_`` 162and are explained in detail in the *DPDK API Documentation*. 163 164 165The Main Function 166~~~~~~~~~~~~~~~~~ 167 168The ``main()`` function performs the initialization and calls the execution 169threads for each lcore. 170 171The first task is to initialize the Environment Abstraction Layer (EAL). The 172``argc`` and ``argv`` arguments are provided to the ``rte_eal_init()`` 173function. The value returned is the number of parsed arguments: 174 175.. literalinclude:: ../../../examples/l2fwd-cat/l2fwd-cat.c 176 :language: c 177 :start-after: Initialize the Environment Abstraction Layer (EAL). 8< 178 :end-before: >8 End of initialization the Environment Abstraction Layer (EAL). 179 :dedent: 1 180 181The next task is to initialize the PQoS library and configure CAT. The 182``argc`` and ``argv`` arguments are provided to the ``cat_init()`` 183function. The value returned is the number of parsed arguments: 184 185.. literalinclude:: ../../../examples/l2fwd-cat/l2fwd-cat.c 186 :language: c 187 :start-after: Initialize the PQoS. 8< 188 :end-before: >8 End of initialization of PQoS. 189 :dedent: 1 190 191``cat_init()`` is a wrapper function which parses the command, validates 192the requested parameters and configures CAT accordingly. 193 194Parsing of command line arguments is done in ``parse_args(...)``. 195libpqos is then initialized with the ``pqos_init(...)`` call. Next, libpqos is 196queried for system CPU information and L3CA capabilities via 197``pqos_cap_get(...)`` and ``pqos_cap_get_type(..., PQOS_CAP_TYPE_L3CA, ...)`` 198calls. When all capability and topology information is collected, the requested 199CAT configuration is validated. A check is then performed (on per socket basis) 200for a sufficient number of un-associated COS. COS are selected and 201configured via the ``pqos_l3ca_set(...)`` call. Finally, COS are associated to 202relevant CPUs via ``pqos_l3ca_assoc_set(...)`` calls. 203 204``atexit(...)`` is used to register ``cat_exit(...)`` to be called on 205a clean exit. ``cat_exit(...)`` performs a simple CAT clean-up, by associating 206COS 0 to all involved CPUs via ``pqos_l3ca_assoc_set(...)`` calls. 207