1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2010-2014 Intel Corporation. 3 4L2 Forwarding Eventdev Sample Application 5========================================= 6 7The L2 Forwarding eventdev sample application is an example of 8packet processing using the Data Plane Development Kit (DPDK) 9to demonstrate the usage of the poll and event mode packet I/O mechanism. 10 11Overview 12-------- 13 14The L2 forwarding eventdev sample application performs 15L2 forwarding for each packet that is received on an Rx port. 16The destination port is the adjacent port from the enabled portmask. 17If the first four ports are enabled (portmask=0x0f), 18ports 1 and 2 forward into each other, 19and ports 3 and 4 forward into each other. 20Also, if MAC address updating is enabled, 21the MAC addresses are affected as follows: 22 23* The source MAC address is replaced by the Tx port MAC address 24* The destination MAC address is replaced by 02:00:00:00:00:<Tx port> 25 26Application receives packets from Rx port using these methods: 27 28* Poll mode 29* Eventdev mode (default) 30 31This application can be used to benchmark performance using a traffic-generator, 32as shown in the :numref:`figure_l2fwd_event_benchmark_setup`. 33 34.. _figure_l2fwd_event_benchmark_setup: 35 36.. figure:: img/l2_fwd_benchmark_setup.* 37 38 Performance Benchmark Setup (Basic Environment) 39 40Compiling the Application 41------------------------- 42 43To compile the sample application, see :doc:`compiling`. 44 45The application is located in the ``l2fwd-event`` sub-directory. 46 47Running the Application 48----------------------- 49 50The application requires a number of command line options: 51 52.. code-block:: console 53 54 ./<build_dir>/examples/dpdk-l2fwd-event [EAL options] -- -p PORTMASK 55 [-q NQ] 56 [--[no-]mac-updating] 57 [--mode=MODE] 58 [--eventq-sched=SCHED_MODE] 59 [--event-vector [--event-vector-size SIZE] [--event-vector-tmo NS]] 60 61where, 62 63* p PORTMASK: A hexadecimal bitmask of the ports to configure 64 65* q NQ: Maximum number of queues per lcore (default is 1) 66 67* --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default). 68 69* --mode=MODE: Packet transfer mode for I/O, poll or eventdev. Eventdev by default. 70 71* --eventq-sched=SCHED_MODE: Event queue schedule mode, Ordered, Atomic or Parallel. Atomic by default. 72 73* --config: Configure forwarding port pair mapping. Alternate port pairs by default. 74 75* --event-vector: Enable event vectorization. Only valid if --mode=eventdev. 76 77* --event-vector-size: Max vector size if event vectorization is enabled. 78 79* --event-vector-tmo: Max timeout to form vector in nanoseconds if event vectorization is enabled. 80 81Sample usage commands are given below to run the application into different modes: 82 83Poll mode with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address updating enabled, 84issue the command: 85 86.. code-block:: console 87 88 ./<build_dir>/examples/dpdk-l2fwd-event -l 0-3 -n 4 -- -q 8 -p ffff --mode=poll 89 90Eventdev mode with 4 lcores, 16 ports, schedule method ordered and MAC address updating enabled, 91issue the command: 92 93.. code-block:: console 94 95 ./<build_dir>/examples/dpdk-l2fwd-event -l 0-3 -n 4 -- -p ffff --eventq-sched=ordered 96 97or 98 99.. code-block:: console 100 101 ./<build_dir>/examples/dpdk-l2fwd-event -l 0-3 -n 4 -- -q 8 -p ffff --mode=eventdev --eventq-sched=ordered 102 103Refer to the *DPDK Getting Started Guide* for general information on running 104applications and Environment Abstraction Layer (EAL) options. 105 106To run the application with S/W scheduler, it uses following DPDK services: 107 108* Software scheduler 109* Rx adapter service function 110* Tx adapter service function 111 112The application needs service cores to run the above mentioned services. 113Service cores must be provided as EAL parameters 114along with the ``--vdev=event_sw0`` to enable S/W scheduler. 115The following is the sample command: 116 117.. code-block:: console 118 119 ./<build_dir>/examples/dpdk-l2fwd-event -l 0-7 -s 0-3 -n 4 --vdev event_sw0 -- -q 8 -p ffff --mode=eventdev --eventq-sched=ordered 120 121Explanation 122----------- 123 124The following sections provide an explanation of the code. 125 126.. _l2_fwd_event_app_cmd_arguments: 127 128Command Line Arguments 129~~~~~~~~~~~~~~~~~~~~~~ 130 131The L2 forwarding eventdev sample application takes specific parameters 132and Environment Abstraction Layer (EAL) arguments. 133The preferred way to parse parameters is to use the ``getopt()`` function 134since it is part of a well-defined and portable library. 135 136The parsing of arguments is done in the ``l2fwd_parse_args()`` function 137for non-eventdev parameters 138and in ``parse_eventdev_args()`` for eventdev parameters. 139This method of argument parsing is not described here. 140Refer to the *glibc getopt(3)* main page for details. 141 142EAL arguments are parsed first, then application-specific arguments follow. 143This is done at the beginning of the ``main()`` function and eventdev parameters 144are parsed in ``eventdev_resource_setup()`` function during eventdev setup: 145 146.. literalinclude:: ../../../examples/l2fwd-event/main.c 147 :language: c 148 :start-after: Init EAL. 8< 149 :end-before: >8 End of init EAL. 150 :dedent: 1 151 152Mbuf Pool Initialization 153~~~~~~~~~~~~~~~~~~~~~~~~ 154 155Once the arguments are parsed, the mbuf pool is created. 156The mbuf pool contains a set of mbuf objects that will be used by the driver 157and the application to store network packet data: 158 159.. literalinclude:: ../../../examples/l2fwd-event/main.c 160 :language: c 161 :start-after: Create the mbuf pool. 8< 162 :end-before: >8 End of creation of mbuf pool. 163 :dedent: 1 164 165The rte_mempool is a generic structure used to handle pools of objects. 166In this case, it is necessary to create a pool that will be used by the driver. 167The number of allocated mbufs is ``NB_MBUF``, 168with a data room size of ``RTE_MBUF_DEFAULT_BUF_SIZE`` each. 169A per-lcore cache of 32 mbufs is kept. 170The memory is allocated in NUMA socket 0, 171but it is possible to extend this code to allocate one mbuf pool per socket. 172 173The ``rte_pktmbuf_pool_create()`` function uses the default mbuf pool 174and mbuf initializers, respectively ``rte_pktmbuf_pool_init()`` and ``rte_pktmbuf_init()``. 175An advanced application may want to use the mempool API to create the 176mbuf pool with more control. 177 178.. _l2_fwd_event_app_drv_init: 179 180Driver Initialization 181~~~~~~~~~~~~~~~~~~~~~ 182 183The main part of the code in the ``main()`` function relates to the initialization 184of the driver. To fully understand this code, it is recommended to study the 185chapters related to the Poll Mode and Event mode Driver in the 186*DPDK Programmer's Guide* - Rel 1.4 EAR and the *DPDK API Reference*. 187 188.. literalinclude:: ../../../examples/l2fwd-event/main.c 189 :language: c 190 :start-after: Reset l2fwd_dst_ports. 8< 191 :end-before: >8 End of reset l2fwd_dst_ports. 192 :dedent: 1 193 194The next step is to configure the RX and TX queues. For each port, there is only 195one RX queue (only one lcore is able to poll a given port). The number of TX 196queues depends on the number of available lcores. The ``rte_eth_dev_configure()`` 197function is used to configure the number of queues for a port: 198 199.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_common.c 200 :language: c 201 :start-after: Configure RX and TX queue. 8< 202 :end-before: >8 End of configuration RX and TX queue. 203 :dedent: 2 204 205RX Queue Initialization 206~~~~~~~~~~~~~~~~~~~~~~~ 207 208The application uses one lcore to poll one or several ports, depending on the -q 209option, which specifies the number of queues per lcore. 210 211For example, if the user specifies -q 4, the application is able to poll four 212ports with one lcore. If there are 16 ports on the target (and if the portmask 213argument is -p ffff ), the application will need four lcores to poll all the 214ports. 215 216.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_common.c 217 :language: c 218 :start-after: Using lcore to poll one or several ports. 8< 219 :end-before: >8 End of using lcore to poll one or several ports. 220 :dedent: 2 221 222The list of queues that must be polled for a given lcore is stored in a private 223structure called struct lcore_queue_conf. 224 225.. literalinclude:: ../../../examples/l2fwd/main.c 226 :language: c 227 :start-after: List of queues to be polled for a given lcore. 8< 228 :end-before: >8 End of list of queues to be polled for a given lcore. 229 230The values n_rx_port and rx_port_list[] are used in the main packet processing 231loop (see :ref:`l2_fwd_event_app_rx_tx_packets`). 232 233.. _l2_fwd_event_app_tx_init: 234 235TX Queue Initialization 236~~~~~~~~~~~~~~~~~~~~~~~ 237 238Each lcore should be able to transmit on any port. For each port, a single TX 239queue is initialized. 240 241.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_common.c 242 :language: c 243 :start-after: Init one TX queue on each port. 8< 244 :end-before: >8 End of init one TX queue on each port. 245 :dedent: 2 246 247To configure eventdev support, the application sets up following components: 248 249* Event dev 250* Event queue 251* Event Port 252* Rx/Tx adapters 253* Ethernet ports 254 255.. _l2_fwd_event_app_event_dev_init: 256 257Event device Initialization 258~~~~~~~~~~~~~~~~~~~~~~~~~~~ 259 260Application can use either H/W or S/W based event device scheduler 261implementation and supports single instance of event device. It configures event 262device as per the following configuration: 263 264.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_event_generic.c 265 :language: c 266 :start-after: Configures event device as per below configuration. 8< 267 :end-before: >8 End of configuration event device as per below configuration. 268 :dedent: 1 269 270In case of S/W scheduler, the application runs eventdev 271scheduler service on the service core. 272The application retrieves the service id 273and finds the best possible service core to run S/W scheduler. 274 275.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_event.c 276 :language: c 277 :start-after: Running eventdev scheduler service on service core. 8< 278 :end-before: >8 End of running eventdev scheduler service on service core. 279 :dedent: 1 280 281Event queue Initialization 282~~~~~~~~~~~~~~~~~~~~~~~~~~ 283 284Each Ethernet device is assigned a dedicated event queue which will be linked 285to all available event ports, i.e. each lcore can dequeue packets from any of the 286Ethernet ports. 287 288.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_event_generic.c 289 :language: c 290 :start-after: Event queue initialization. 8< 291 :end-before: >8 End of event queue initialization. 292 :dedent: 1 293 294In case of S/W scheduler, an extra event queue is created which will be used for 295Tx adapter service function for enqueue operation. 296 297.. _l2_fwd_app_event_port_init: 298 299Event port Initialization 300~~~~~~~~~~~~~~~~~~~~~~~~~ 301 302Each worker thread is assigned a dedicated event port for enq/deq operations 303to/from an event device. 304All event ports are linked to all available event queues. 305 306.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_event_generic.c 307 :language: c 308 :start-after: Event port initialization. 8< 309 :end-before: >8 End of event port initialization. 310 :dedent: 1 311 312In case of S/W scheduler, an extra event port is created by the DPDK library 313which is retrieved by the application. 314It will be used by Tx adapter service. 315 316.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_event_generic.c 317 :language: c 318 :start-after: Extra port created. 8< 319 :end-before: >8 End of extra port created. 320 :dedent: 1 321 322Rx/Tx adapter Initialization 323~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 324 325Each Ethernet port is assigned a dedicated Rx/Tx adapter for H/W scheduler. Each 326Ethernet port's Rx queues are connected to its respective event queue at 327priority 0 via Rx adapter configuration and Ethernet port's tx queues are 328connected via Tx adapter. 329 330.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_event_internal_port.c 331 :language: c 332 :start-after: Assigned ethernet port. 8< 333 :end-before: >8 End of assigned ethernet port. 334 :dedent: 1 335 336For S/W scheduler instead of dedicated adapters, common Rx/Tx adapters are 337configured which will be shared among all the Ethernet ports. 338Also, the DPDK library needs service cores to run internal services for Rx/Tx adapters. 339The application gets a service id for Rx/Tx adapters. 340After a successful setup, it runs the services on dedicated service cores. 341 342.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_event.c 343 :language: c 344 :start-after: Gets service ID for RX/TX adapters. 8< 345 :end-before: >8 End of get service ID for RX/TX adapters. 346 :dedent: 1 347 348.. _l2_fwd_event_app_rx_tx_packets: 349 350Receive, Process and Transmit Packets 351~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 352 353In the ``l2fwd_main_loop()`` function, 354the main task is to read ingress packets from the Rx queues. 355This is done using the following code: 356 357.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_poll.c 358 :language: c 359 :start-after: Reading ingress packets. 8< 360 :end-before: >8 End of reading ingress packets. 361 :dedent: 2 362 363Packets are read in a burst of size ``MAX_PKT_BURST``. 364The ``rte_eth_rx_burst()`` function writes the mbuf pointers 365in a local table and returns the number of available mbufs in the table. 366 367Then, each mbuf in the table is processed by the ``l2fwd_simple_forward()`` function. 368The processing is very simple: process the Tx port from the Rx port, 369then replace the source and destination MAC addresses if MAC address updating is enabled. 370 371During the initialization process, a static array of destination ports 372(``l2fwd_dst_ports[]``) is filled so that for each source port, a destination port 373is assigned that is either the next or previous enabled port from the portmask. 374If the number of ports are odd in portmask, then the packet from the last port will be 375forwarded to first port i.e. if portmask=0x07, then forwarding will take place 376like p0--->p1, p1--->p2, p2--->p0. 377 378Also, to optimize enqueue operation, ``l2fwd_simple_forward()`` stores incoming mbufs 379up to ``MAX_PKT_BURST``. 380Once it reaches the limit, all packets are transmitted to destination ports. 381 382.. literalinclude:: ../../../examples/l2fwd/main.c 383 :language: c 384 :start-after: Simple forward. 8< 385 :end-before: >8 End of simple forward. 386 387For this test application, the processing is exactly the same for all packets 388arriving on the same RX port. Therefore, it would have been possible to call 389the ``rte_eth_tx_buffer()`` function directly from the main loop to send all the 390received packets on the same TX port, using the burst-oriented send function, 391which is more efficient. 392 393However, in real-life applications (such as L3 routing), 394packet N is not necessarily forwarded on the same port as packet N-1. 395The application is implemented to illustrate so the same approach can be 396reused in a more complex application. 397 398To ensure that no packets remain in the tables, each lcore does a draining of TX 399queue in its main loop. This technique introduces some latency when there are 400not many packets to send. However, it improves performance: 401 402.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_poll.c 403 :language: c 404 :start-after: Draining TX queue in main loop. 8< 405 :end-before: >8 End of draining TX queue in main loop. 406 :dedent: 2 407 408In the ``l2fwd_event_loop()`` function, 409the main task is to read ingress packets from the event ports. 410This is done using the following code: 411 412.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_event.c 413 :language: c 414 :start-after: Read packet from eventdev. 8< 415 :end-before: >8 End of reading packets from eventdev. 416 :dedent: 2 417 418Before reading packets, ``deq_len`` is fetched 419to ensure the correct allowed deq length by the eventdev. 420The ``rte_event_dequeue_burst()`` function writes the mbuf pointers in a local table 421and returns the number of available mbufs in the table. 422 423Then, each mbuf in the table is processed by the ``l2fwd_eventdev_forward()`` 424function. The processing is very simple: process the TX port from the RX port, 425then replace the source and destination MAC addresses if MAC address updating 426is enabled. 427 428During the initialization process, a static array of destination ports 429(``l2fwd_dst_ports[]``) is filled so that for each source port, a destination port 430is assigned that is either the next or previous enabled port from the portmask. 431If number of ports are odd in portmask then packet from last port will be 432forwarded to first port i.e. if portmask=0x07, then forwarding will take place 433like p0--->p1, p1--->p2, p2--->p0. 434 435``l2fwd_eventdev_forward()`` does not stores incoming mbufs. 436The packet will forwarded to destination ports 437via Tx adapter or generic event dev enqueue API 438depending H/W or S/W scheduler is used. 439 440.. literalinclude:: ../../../examples/l2fwd-event/l2fwd_event.c 441 :language: c 442 :start-after: Read packet from eventdev. 8< 443 :end-before: >8 End of reading packets from eventdev. 444 :dedent: 2 445