1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(C) 2020 Marvell International Ltd. 3 4L3 Forwarding Graph Sample Application 5====================================== 6 7The L3 Forwarding Graph application is a simple example of packet processing 8using the DPDK Graph framework. The application performs L3 forwarding using 9Graph framework and nodes written for graph framework. 10 11Overview 12-------- 13 14The application demonstrates the use of the graph framework and graph nodes 15``ethdev_rx``, ``pkt_cls``, ``ip4_lookup``/``ip6_lookup``, 16``ip4_rewrite``/``ip6_rewrite``, ``ethdev_tx`` and ``pkt_drop`` in DPDK to 17implement packet forwarding. 18 19The initialization is very similar to those of the :doc:`l3_forward`. 20There is also additional initialization of graph for graph object creation 21and configuration per lcore. 22Run-time path is main thing that differs from L3 forwarding sample application. 23Difference is that forwarding logic starting from Rx, followed by LPM lookup, 24TTL update and finally Tx is implemented inside graph nodes. These nodes are 25interconnected in graph framework. Application main loop needs to walk over 26graph using ``rte_graph_walk()`` with graph objects created one per worker lcore. 27 28The lookup method is as per implementation of ``ip4_lookup``/``ip6_lookup`` graph node. 29The ID of the output interface for the input packet is the next hop returned by 30the LPM lookup. The set of LPM rules used by the application is statically 31configured and provided to ``ip4_lookup``/``ip6_lookup`` graph node and 32``ip4_rewrite``/``ip6_rewrite`` graph node using node control API 33``rte_node_ip4_route_add()``/``rte_node_ip6_route_add`` and 34``rte_node_ip4_rewrite_add()``/``rte_node_ip6_rewrite_add``. 35 36In the sample application, IPv4 and IPv6 forwarding is supported. 37 38Compiling the Application 39------------------------- 40 41To compile the sample application see :doc:`compiling`. 42 43The application is located in the ``l3fwd-graph`` sub-directory. 44 45Running the Application 46----------------------- 47 48The application has a number of command line options similar to l3fwd:: 49 50 ./dpdk-l3fwd-graph [EAL options] -- -p PORTMASK 51 [-P] 52 --config(port,queue,lcore)[,(port,queue,lcore)] 53 [--eth-dest=X,MM:MM:MM:MM:MM:MM] 54 [--max-pkt-len PKTLEN] 55 [--no-numa] 56 [--per-port-pool] 57 [--pcap-enable] 58 [--pcap-num-cap] 59 [--pcap-file-name] 60 [--model] 61 62Where, 63 64* ``-p PORTMASK:`` Hexadecimal bitmask of ports to configure 65 66* ``-P:`` Optional, sets all ports to promiscuous mode so that packets are accepted regardless of the packet's Ethernet MAC destination address. 67 Without this option, only packets with the Ethernet MAC destination address set to the Ethernet address of the port are accepted. 68 69* ``--config (port,queue,lcore)[,(port,queue,lcore)]:`` Determines which queues from which ports are mapped to which cores. 70 71* ``--eth-dest=X,MM:MM:MM:MM:MM:MM:`` Optional, ethernet destination for port X. 72 73* ``--max-pkt-len:`` Optional, maximum packet length in decimal (64-9600). 74 75* ``--no-numa:`` Optional, disables numa awareness. 76 77* ``--per-port-pool:`` Optional, set to use independent buffer pools per port. Without this option, single buffer pool is used for all ports. 78 79* ``--pcap-enable:`` Optional, Enables packet capture in pcap format on each node with mbuf and node metadata. 80 81* ``--pcap-num-cap:`` Optional, Number of packets to be captured per core. 82 83* ``--pcap-file-name:`` Optional, Pcap filename to capture packets in. 84 85* ``--model:`` Optional, select graph walking model. 86 87For example, consider a dual processor socket platform with 8 physical cores, where cores 0-7 and 16-23 appear on socket 0, 88while cores 8-15 and 24-31 appear on socket 1. 89 90To enable L3 forwarding between two ports, assuming that both ports are in the same socket, using two cores, cores 1 and 2, 91(which are in the same socket too), use the following command: 92 93.. code-block:: console 94 95 ./<build_dir>/examples/dpdk-l3fwd-graph -l 1,2 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2)" 96 97In this command: 98 99* The -l option enables cores 1, 2 100 101* The -p option enables ports 0 and 1 102 103* The --config option enables one queue on each port and maps each (port,queue) pair to a specific core. 104 The following table shows the mapping in this example: 105 106+----------+-----------+-----------+-------------------------------------+ 107| **Port** | **Queue** | **lcore** | **Description** | 108| | | | | 109+----------+-----------+-----------+-------------------------------------+ 110| 0 | 0 | 1 | Map queue 0 from port 0 to lcore 1. | 111| | | | | 112+----------+-----------+-----------+-------------------------------------+ 113| 1 | 0 | 2 | Map queue 0 from port 1 to lcore 2. | 114| | | | | 115+----------+-----------+-----------+-------------------------------------+ 116 117To enable pcap trace on each graph, use following command: 118 119.. code-block:: console 120 121 ./<build_dir>/examples/dpdk-l3fwd-graph -l 1,2 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2)" --pcap-enable --pcap-num-cap=<number of packets> --pcap-file-name "</path/to/file>" 122 123In this command: 124 125* The --pcap-enable option enables pcap trace on graph nodes. 126 127* The --pcap-num-cap option enables user to configure number packets to be captured per graph. Default 1024 packets per graph are captured. 128 129* The --pcap-file-name option enables user to give filename in which packets are to be captured. 130 131To enable mcore dispatch model, the application need change RTE_GRAPH_MODEL_SELECT to ``#define RTE_GRAPH_MODEL_SELECT RTE_GRAPH_MODEL_MCORE_DISPATCH`` 132before including rte_graph_worker.h. Recompile and use following command: 133 134.. code-block:: console 135 136 ./<build_dir>/examples/dpdk-l3fwd-graph -l 1,2,3,4 -n 4 -- -p 0x1 --config="(0,0,1)" -P --model="dispatch" 137 138To enable graph walking model selection in run-time, remove the define of ``RTE_GRAPH_MODEL_SELECT``. Recompile and use the same command. 139 140In this command: 141 142* The --model option enables user to select ``rtc`` or ``dispatch`` model. 143 144Refer to the *DPDK Getting Started Guide* for general information on running applications and 145the Environment Abstraction Layer (EAL) options. 146 147.. _l3_fwd_graph_explanation: 148 149Explanation 150----------- 151 152The following sections provide some explanation of the sample application code. 153As mentioned in the overview section, the initialization is similar to that of 154the :doc:`l3_forward`. Run-time path though similar in functionality to that of 155:doc:`l3_forward`, major part of the implementation is in graph nodes via used 156via ``librte_node`` library. 157The following sections describe aspects that are specific to the L3 Forwarding 158Graph sample application. 159 160Graph Node Pre-Init Configuration 161~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 162 163After device configuration and device Rx, Tx queue setup is complete, 164a minimal config of port id, num_rx_queues, num_tx_queues, mempools etc will 165be passed to *ethdev_** node ctrl API ``rte_node_eth_config()``. This will be 166lead to the clone of ``ethdev_rx`` and ``ethdev_tx`` nodes as ``ethdev_rx-X-Y`` and 167``ethdev_tx-X`` where X, Y represent port id and queue id associated with them. 168In case of ``ethdev_tx-X`` nodes, tx queue id assigned per instance of the node 169is same as graph id. 170 171These cloned nodes along with existing static nodes such as ``ip4_lookup``/``ip6_lookup`` 172and ``ip4_rewrite``/``ip6_rewrite`` will be used in graph creation to associate node's to lcore 173specific graph object. 174 175.. literalinclude:: ../../../examples/l3fwd-graph/main.c 176 :language: c 177 :start-after: Initialize all ports. 8< 178 :end-before: >8 End of graph creation. 179 :dedent: 1 180 181Graph Initialization 182~~~~~~~~~~~~~~~~~~~~ 183 184Now a graph needs to be created with a specific set of nodes for every lcore. 185A graph object returned after graph creation is a per lcore object and 186cannot be shared between lcores. Since ``ethdev_tx-X`` node is per port node, 187it can be associated with all the graphs created as all the lcores should have 188Tx capability for every port. But ``ethdev_rx-X-Y`` node is created per 189(port, rx_queue_id), so they should be associated with a graph based on 190the application argument ``--config`` specifying rx queue mapping to lcore. 191 192.. note:: 193 194 The Graph creation will fail if the passed set of shell node pattern's 195 are not sufficient to meet their inter-dependency or even one node is not 196 found with a given regex node pattern. 197 198.. literalinclude:: ../../../examples/l3fwd-graph/main.c 199 :language: c 200 :start-after: Graph initialization. 8< 201 :end-before: >8 End of graph initialization. 202 :dedent: 1 203 204Forwarding data(Route, Next-Hop) addition 205~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 206 207Once graph objects are created, node specific info like routes and rewrite 208headers will be provided run-time using ``rte_node_ip4_route_add()``/ 209``rte_node_ip6_route_add`` and ``rte_node_ip4_rewrite_add()``/``rte_node_ip6_rewrite_add`` 210API. 211 212.. note:: 213 214 Since currently ``ip4_lookup``/``ip6_lookup`` and ``ip4_rewrite``/``ip6_rewrite`` 215 nodes don't support lock-less mechanisms(RCU, etc) to add run-time forwarding 216 data like route and rewrite data, forwarding data is added before packet 217 processing loop is launched on worker lcore. 218 219.. literalinclude:: ../../../examples/l3fwd-graph/main.c 220 :language: c 221 :start-after: Add routes and rewrite data to graph infra. 8< 222 :end-before: >8 End of adding routes and rewrite data to graph infa. 223 :dedent: 1 224 225Packet Forwarding using Graph Walk 226~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 227 228Now that all the device configurations are done, graph creations are done and 229forwarding data is updated with nodes, worker lcores will be launched with graph 230main loop. Graph main loop is very simple in the sense that it needs to 231continuously call a non-blocking API ``rte_graph_walk()`` with it's lcore 232specific graph object that was already created. 233 234.. note:: 235 236 rte_graph_walk() will walk over all the sources nodes i.e ``ethdev_rx-X-Y`` 237 associated with a given graph and Rx the available packets and enqueue them 238 to the following node ``pkt_cls`` which based on the packet type will enqueue 239 them to ``ip4_lookup``/``ip6_lookup`` which then will enqueue them to 240 ``ip4_rewrite``/``ip6_rewrite`` node if LPM lookup succeeds. 241 ``ip4_rewrite``/``ip6_rewrite`` node then will update Ethernet header 242 as per next-hop data and transmit the packet via port 'Z' by enqueuing 243 to ``ethdev_tx-Z`` node instance in its graph object. 244 245.. literalinclude:: ../../../examples/l3fwd-graph/main.c 246 :language: c 247 :start-after: Main processing loop. 8< 248 :end-before: >8 End of main processing loop. 249