1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(C) 2020 Marvell International Ltd. 3 4L3 Forwarding Graph Sample Application 5====================================== 6 7The L3 forwarding graph application is an example of packet processing 8using the DPDK Graph framework. 9The application performs L3 forwarding using graph framework, 10and nodes written for graph framework. 11 12Overview 13-------- 14 15The application demonstrates the use of the graph framework and graph nodes 16``ethdev_rx``, ``pkt_cls``, ``ip4_lookup``/``ip6_lookup``, 17``ip4_rewrite``/``ip6_rewrite``, ``ethdev_tx`` and ``pkt_drop`` in DPDK to 18implement packet forwarding. 19 20The initialization is very similar to those of the :doc:`l3_forward`. 21There is also additional initialization of graph for graph object creation 22and configuration per lcore. 23Run-time path is main thing that differs from L3 forwarding sample application. 24Difference is that forwarding logic starting from Rx, followed by LPM lookup, 25TTL update and finally Tx is implemented inside graph nodes. These nodes are 26interconnected in graph framework. The application main loop needs to walk over 27graph using ``rte_graph_walk()`` with graph objects created one per worker lcore. 28 29The lookup method is as per implementation of ``ip4_lookup``/``ip6_lookup`` graph node. 30The ID of the output interface for the input packet is the next hop returned by 31the LPM lookup. The set of LPM rules used by the application is statically 32configured and provided to ``ip4_lookup``/``ip6_lookup`` graph node and 33``ip4_rewrite``/``ip6_rewrite`` graph node using node control API 34``rte_node_ip4_route_add()``/``rte_node_ip6_route_add`` and 35``rte_node_ip4_rewrite_add()``/``rte_node_ip6_rewrite_add``. 36 37In the sample application, IPv4 and IPv6 forwarding is supported. 38 39Compiling the Application 40------------------------- 41 42To compile the sample application, see :doc:`compiling`. 43 44The application is located in the ``l3fwd-graph`` sub-directory. 45 46Running the Application 47----------------------- 48 49The application has a number of command line options similar to l3fwd:: 50 51 ./dpdk-l3fwd-graph [EAL options] -- -p PORTMASK 52 [-P] 53 --config(port,queue,lcore)[,(port,queue,lcore)] 54 [--eth-dest=X,MM:MM:MM:MM:MM:MM] 55 [--max-pkt-len PKTLEN] 56 [--no-numa] 57 [--per-port-pool] 58 [--pcap-enable] 59 [--pcap-num-cap] 60 [--pcap-file-name] 61 [--model] 62 63Where, 64 65* ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure 66 67* ``-P:`` Optional, sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address. 68 Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted. 69 70* ``--config (port,queue,lcore)[,(port,queue,lcore)]:`` Determines which queues from which ports are mapped to which cores. 71 72* ``--eth-dest=X,MM:MM:MM:MM:MM:MM:`` Optional, ethernet destination for port X. 73 74* ``--max-pkt-len:`` Optional, maximum packet length in decimal (64-9600). 75 76* ``--no-numa:`` Optional, disables numa awareness. 77 78* ``--per-port-pool:`` Optional, set to use independent buffer pools per port. Without this option, single buffer pool is used for all ports. 79 80* ``--pcap-enable:`` Optional, Enables packet capture in pcap format on each node with mbuf and node metadata. 81 82* ``--pcap-num-cap:`` Optional, Number of packets to be captured per core. 83 84* ``--pcap-file-name:`` Optional, Pcap filename to capture packets in. 85 86* ``--model:`` Optional, select graph walking model. 87 88For example, consider a dual processor socket platform with 8 physical cores, where cores 0-7 and 16-23 appear on socket 0, 89while cores 8-15 and 24-31 appear on socket 1. 90 91To enable L3 forwarding between two ports, assuming that both ports are in the same socket, using two cores, cores 1 and 2, 92(which are in the same socket too), use the following command: 93 94.. code-block:: console 95 96 ./<build_dir>/examples/dpdk-l3fwd-graph -l 1,2 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2)" 97 98In this command: 99 100* The -l option enables cores 1, 2 101 102* The -p option enables ports 0 and 1 103 104* The --config option enables one queue on each port and maps each (port,queue) pair to a specific core. 105 The following table shows the mapping in this example: 106 107+----------+-----------+-----------+-------------------------------------+ 108| **Port** | **Queue** | **lcore** | **Description** | 109| | | | | 110+----------+-----------+-----------+-------------------------------------+ 111| 0 | 0 | 1 | Map queue 0 from port 0 to lcore 1. | 112| | | | | 113+----------+-----------+-----------+-------------------------------------+ 114| 1 | 0 | 2 | Map queue 0 from port 1 to lcore 2. | 115| | | | | 116+----------+-----------+-----------+-------------------------------------+ 117 118To enable pcap trace on each graph, use following command: 119 120.. code-block:: console 121 122 ./<build_dir>/examples/dpdk-l3fwd-graph -l 1,2 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2)" --pcap-enable --pcap-num-cap=<number of packets> --pcap-file-name "</path/to/file>" 123 124In this command: 125 126* The --pcap-enable option enables pcap trace on graph nodes. 127 128* The --pcap-num-cap option enables user to configure number packets to be captured per graph. Default 1024 packets per graph are captured. 129 130* The --pcap-file-name option enables user to give filename in which packets are to be captured. 131 132To enable mcore dispatch model, the application need change RTE_GRAPH_MODEL_SELECT to ``#define RTE_GRAPH_MODEL_SELECT RTE_GRAPH_MODEL_MCORE_DISPATCH`` 133before including rte_graph_worker.h. Recompile and use following command: 134 135.. code-block:: console 136 137 ./<build_dir>/examples/dpdk-l3fwd-graph -l 1,2,3,4 -n 4 -- -p 0x1 --config="(0,0,1)" -P --model="dispatch" 138 139To enable graph walking model selection in run-time, remove the define of ``RTE_GRAPH_MODEL_SELECT``. Recompile and use the same command. 140 141In this command: 142 143* The --model option enables user to select ``rtc`` or ``dispatch`` model. 144 145Refer to the *DPDK Getting Started Guide* for general information on running applications and 146the Environment Abstraction Layer (EAL) options. 147 148.. _l3_fwd_graph_explanation: 149 150Explanation 151----------- 152 153The following sections provide some explanation of the sample application code. 154As mentioned in the overview section, the initialization is similar to that of 155the :doc:`l3_forward`. Run-time path though similar in functionality to that of 156:doc:`l3_forward`, major part of the implementation is in graph nodes via used 157via ``librte_node`` library. 158The following sections describe aspects that are specific to the L3 Forwarding 159Graph sample application. 160 161Graph Node Pre-Init Configuration 162~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 163 164After device configuration and device Rx, Tx queue setup is complete, 165a minimal config of port id, num_rx_queues, num_tx_queues, mempools etc will 166be passed to *ethdev_** node ctrl API ``rte_node_eth_config()``. This will be 167lead to the clone of ``ethdev_rx`` and ``ethdev_tx`` nodes as ``ethdev_rx-X-Y`` and 168``ethdev_tx-X`` where X, Y represent port id and queue id associated with them. 169In case of ``ethdev_tx-X`` nodes, tx queue id assigned per instance of the node 170is same as graph id. 171 172These cloned nodes along with existing static nodes such as ``ip4_lookup``/``ip6_lookup`` 173and ``ip4_rewrite``/``ip6_rewrite`` will be used in graph creation to associate node's to lcore 174specific graph object. 175 176.. literalinclude:: ../../../examples/l3fwd-graph/main.c 177 :language: c 178 :start-after: Initialize all ports. 8< 179 :end-before: >8 End of graph creation. 180 :dedent: 1 181 182Graph Initialization 183~~~~~~~~~~~~~~~~~~~~ 184 185Now, a graph needs to be created with a specific set of nodes for every lcore. 186A graph object returned after graph creation is a per lcore object and 187cannot be shared between lcores. Since ``ethdev_tx-X`` node is per port node, 188it can be associated with all the graphs created as all the lcores should have 189Tx capability for every port. But ``ethdev_rx-X-Y`` node is created per 190(port, rx_queue_id), so they should be associated with a graph based on 191the application argument ``--config`` specifying rx queue mapping to lcore. 192 193.. note:: 194 195 The Graph creation will fail if the passed set of shell node pattern's 196 are not sufficient to meet their inter-dependency or even one node is not 197 found with a given regex node pattern. 198 199.. literalinclude:: ../../../examples/l3fwd-graph/main.c 200 :language: c 201 :start-after: Graph initialization. 8< 202 :end-before: >8 End of graph initialization. 203 :dedent: 1 204 205Forwarding data(Route, Next-Hop) addition 206~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 207 208Once graph objects are created, node specific info like routes and rewrite 209headers will be provided run-time using ``rte_node_ip4_route_add()``/ 210``rte_node_ip6_route_add`` and ``rte_node_ip4_rewrite_add()``/``rte_node_ip6_rewrite_add`` 211API. 212 213.. note:: 214 215 Since currently ``ip4_lookup``/``ip6_lookup`` and ``ip4_rewrite``/``ip6_rewrite`` 216 nodes don't support lock-less mechanisms(RCU, etc) to add run-time forwarding 217 data like route and rewrite data, forwarding data is added before packet 218 processing loop is launched on worker lcore. 219 220.. literalinclude:: ../../../examples/l3fwd-graph/main.c 221 :language: c 222 :start-after: Add routes and rewrite data to graph infra. 8< 223 :end-before: >8 End of adding routes and rewrite data to graph infa. 224 :dedent: 1 225 226Packet Forwarding using Graph Walk 227~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 228 229Now that all the device configurations are done, graph creations are done and 230forwarding data is updated with nodes, worker lcores will be launched with the graph 231main loop.The graph main loop is very simple in the sense that it needs to 232continuously call a non-blocking API ``rte_graph_walk()`` with it's lcore 233specific graph object that was already created. 234 235.. note:: 236 237 rte_graph_walk() will walk over all the sources nodes i.e ``ethdev_rx-X-Y`` 238 associated with a given graph and Rx the available packets and enqueue them 239 to the following node ``pkt_cls`` which based on the packet type will enqueue 240 them to ``ip4_lookup``/``ip6_lookup`` which then will enqueue them to 241 ``ip4_rewrite``/``ip6_rewrite`` node if LPM lookup succeeds. 242 ``ip4_rewrite``/``ip6_rewrite`` node then will update Ethernet header 243 as per next-hop data and transmit the packet via port 'Z' by enqueuing 244 to ``ethdev_tx-Z`` node instance in its graph object. 245 246.. literalinclude:: ../../../examples/l3fwd-graph/main.c 247 :language: c 248 :start-after: Main processing loop. 8< 249 :end-before: >8 End of main processing loop. 250