xref: /dpdk/doc/guides/sample_app_ug/l2_forward_crypto.rst (revision 8750576fb2a9a067ffbcce4bab6481f3bfa47097)
15630257fSFerruh Yigit..  SPDX-License-Identifier: BSD-3-Clause
25630257fSFerruh Yigit    Copyright(c) 2016-2017 Intel Corporation.
3ba7b86b1SPablo de Lara
4ba7b86b1SPablo de LaraL2 Forwarding with Crypto Sample Application
5ba7b86b1SPablo de Lara============================================
6ba7b86b1SPablo de Lara
7*8750576fSNandini PersadThe L2 Forwarding with Crypto (l2fwd-crypto) sample application
8*8750576fSNandini Persadis a simple example of packet processing
9*8750576fSNandini Persadusing the Data Plane Development Kit (DPDK)
10*8750576fSNandini Persadin conjunction with the cryptodev library.
11ba7b86b1SPablo de Lara
12ba7b86b1SPablo de LaraOverview
13ba7b86b1SPablo de Lara--------
14ba7b86b1SPablo de Lara
158d257235SJohn McNamaraThe L2 Forwarding with Crypto sample application performs a crypto operation (cipher/hash)
16*8750576fSNandini Persadspecified by the user from the command line (or using the default values),
17ba7b86b1SPablo de Larawith a crypto device capable of doing that operation,
187fe84b0dSJakub Poczatekfor each packet that is received on an RX_PORT and performs L2 forwarding.
19*8750576fSNandini Persad
20*8750576fSNandini PersadThe destination port is the adjacent port from the enabled portmask.
21*8750576fSNandini PersadIf the first four ports are enabled (portmask 0xf),
22ba7b86b1SPablo de Laraports 0 and 1 forward into each other, and ports 2 and 3 forward into each other.
23*8750576fSNandini PersadIf the MAC addresses updating is enabled, the MAC addresses are affected as follows:
24ba7b86b1SPablo de Lara
25ba7b86b1SPablo de Lara*   The source MAC address is replaced by the TX_PORT MAC address
26ba7b86b1SPablo de Lara
27ba7b86b1SPablo de Lara*   The destination MAC address is replaced by  02:00:00:00:00:TX_PORT_ID
28ba7b86b1SPablo de Lara
29ba7b86b1SPablo de LaraCompiling the Application
30ba7b86b1SPablo de Lara-------------------------
31ba7b86b1SPablo de Lara
32*8750576fSNandini PersadTo compile the sample application, see :doc:`compiling`.
33ba7b86b1SPablo de Lara
34cf0556f4SJoel KavanaghThe application is located in the ``l2fwd-crypto`` sub-directory.
35ba7b86b1SPablo de Lara
36ba7b86b1SPablo de LaraRunning the Application
37ba7b86b1SPablo de Lara-----------------------
38ba7b86b1SPablo de Lara
39ba7b86b1SPablo de LaraThe application requires a number of command line options:
40ba7b86b1SPablo de Lara
41ba7b86b1SPablo de Lara.. code-block:: console
42ba7b86b1SPablo de Lara
43e2a94f9aSCiara Power    ./<build_dir>/examples/dpdk-l2fwd-crypto [EAL options] -- [-p PORTMASK] [-q NQ] [-s] [-T PERIOD] /
449959fbd5SPablo de Lara    [--cdev_type HW/SW/ANY] [--chain HASH_CIPHER/CIPHER_HASH/CIPHER_ONLY/HASH_ONLY/AEAD] /
45dcf384b4SMatan Azrad    [--cipher_algo ALGO] [--cipher_op ENCRYPT/DECRYPT] [--cipher_dataunit_len SIZE] /
46dcf384b4SMatan Azrad    [--cipher_key KEY] [--cipher_key_random_size SIZE] [--cipher_iv IV] /
47dcf384b4SMatan Azrad    [--cipher_iv_random_size SIZE] /
48ba7b86b1SPablo de Lara    [--auth_algo ALGO] [--auth_op GENERATE/VERIFY] [--auth_key KEY] /
49acf86169SPablo de Lara    [--auth_key_random_size SIZE] [--auth_iv IV] [--auth_iv_random_size SIZE] /
509959fbd5SPablo de Lara    [--aead_algo ALGO] [--aead_op ENCRYPT/DECRYPT] [--aead_key KEY] /
519959fbd5SPablo de Lara    [--aead_key_random_size SIZE] [--aead_iv] [--aead_iv_random_size SIZE] /
52acf86169SPablo de Lara    [--aad AAD] [--aad_random_size SIZE] /
53acdfecbaSKuba Kozak    [--digest size SIZE] [--sessionless] [--cryptodev_mask MASK] /
54acdfecbaSKuba Kozak    [--mac-updating] [--no-mac-updating]
55ba7b86b1SPablo de Lara
56ba7b86b1SPablo de Larawhere,
57ba7b86b1SPablo de Lara
587fe84b0dSJakub Poczatek*   p PORTMASK: A hexadecimal bitmask of the ports to configure. (Default is all the ports.)
59ba7b86b1SPablo de Lara
60885807aeSStephen Hemminger*   q NQ: Maximum number of queues per lcore (default is 1)
61ba7b86b1SPablo de Lara
627fe84b0dSJakub Poczatek*   s: manage all ports from a single core.
63ba7b86b1SPablo de Lara
647fe84b0dSJakub Poczatek*   T PERIOD: statistics will be refreshed each PERIOD seconds.
65ba7b86b1SPablo de Lara
667fe84b0dSJakub Poczatek    (0 to disable, 10 default, 86400 maximum.)
67ba7b86b1SPablo de Lara
687fe84b0dSJakub Poczatek*   cdev_type: select preferred crypto device type: HW, SW or anything (ANY).
69ba7b86b1SPablo de Lara
707fe84b0dSJakub Poczatek    (Default is ANY.)
71ba7b86b1SPablo de Lara
72ba7b86b1SPablo de Lara*   chain: select the operation chaining to perform: Cipher->Hash (CIPHER_HASH),
73ba7b86b1SPablo de Lara
74ba7b86b1SPablo de Lara    Hash->Cipher (HASH_CIPHER), Cipher (CIPHER_ONLY), Hash (HASH_ONLY)
75ba7b86b1SPablo de Lara
767fe84b0dSJakub Poczatek    or AEAD (AEAD).
772661f4fbSPablo de Lara
787fe84b0dSJakub Poczatek    (Default is Cipher->Hash.)
79ba7b86b1SPablo de Lara
807fe84b0dSJakub Poczatek*   cipher_algo: select the ciphering algorithm. (Default is aes-cbc.)
81ba7b86b1SPablo de Lara
827fe84b0dSJakub Poczatek*   cipher_op: select the ciphering operation to perform: ENCRYPT or DECRYPT.
83ba7b86b1SPablo de Lara
847fe84b0dSJakub Poczatek    (Default is ENCRYPT.)
85ba7b86b1SPablo de Lara
86dcf384b4SMatan Azrad*   cipher_dataunit_len: set the length of the cipher data-unit.
87dcf384b4SMatan Azrad
887fe84b0dSJakub Poczatek*   cipher_key: set the ciphering key to be used. Bytes have to be separated with ":".
89ba7b86b1SPablo de Lara
90ba7b86b1SPablo de Lara*   cipher_key_random_size: set the size of the ciphering key,
91ba7b86b1SPablo de Lara
92ba7b86b1SPablo de Lara    which will be generated randomly.
93ba7b86b1SPablo de Lara
94ba7b86b1SPablo de Lara    Note that if --cipher_key is used, this will be ignored.
95ba7b86b1SPablo de Lara
967fe84b0dSJakub Poczatek*   cipher_iv: set the cipher IV to be used. Bytes have to be separated with ":".
97ba7b86b1SPablo de Lara
98acf86169SPablo de Lara*   cipher_iv_random_size: set the size of the cipher IV, which will be generated randomly.
99ba7b86b1SPablo de Lara
100acf86169SPablo de Lara    Note that if --cipher_iv is used, this will be ignored.
101ba7b86b1SPablo de Lara
1027fe84b0dSJakub Poczatek*   auth_algo: select the authentication algorithm. (Default is sha1-hmac.)
103ba7b86b1SPablo de Lara
1047fe84b0dSJakub Poczatek*   auth_op: select the authentication operation to perform: GENERATE or VERIFY.
105ba7b86b1SPablo de Lara
1067fe84b0dSJakub Poczatek    (Default is GENERATE.)
107ba7b86b1SPablo de Lara
1087fe84b0dSJakub Poczatek*   auth_key: set the authentication key to be used. Bytes have to be separated with ":".
109ba7b86b1SPablo de Lara
110ba7b86b1SPablo de Lara*   auth_key_random_size: set the size of the authentication key,
111ba7b86b1SPablo de Lara
112ba7b86b1SPablo de Lara    which will be generated randomly.
113ba7b86b1SPablo de Lara
114ba7b86b1SPablo de Lara    Note that if --auth_key is used, this will be ignored.
115ba7b86b1SPablo de Lara
1167fe84b0dSJakub Poczatek*   auth_iv: set the auth IV to be used. Bytes have to be separated with ":".
117acf86169SPablo de Lara
118acf86169SPablo de Lara*   auth_iv_random_size: set the size of the auth IV, which will be generated randomly.
119acf86169SPablo de Lara
120acf86169SPablo de Lara    Note that if --auth_iv is used, this will be ignored.
121acf86169SPablo de Lara
1227fe84b0dSJakub Poczatek*   aead_algo: select the AEAD algorithm. (Default is aes-gcm.)
1232661f4fbSPablo de Lara
1247fe84b0dSJakub Poczatek*   aead_op: select the AEAD operation to perform: ENCRYPT or DECRYPT.
1252661f4fbSPablo de Lara
1267fe84b0dSJakub Poczatek    (Default is ENCRYPT.)
1272661f4fbSPablo de Lara
1287fe84b0dSJakub Poczatek*   aead_key: set the AEAD key to be used. Bytes have to be separated with ":".
1292661f4fbSPablo de Lara
1302661f4fbSPablo de Lara*   aead_key_random_size: set the size of the AEAD key,
1312661f4fbSPablo de Lara
1322661f4fbSPablo de Lara    which will be generated randomly.
1332661f4fbSPablo de Lara
1342661f4fbSPablo de Lara    Note that if --aead_key is used, this will be ignored.
1352661f4fbSPablo de Lara
1367fe84b0dSJakub Poczatek*   aead_iv: set the AEAD IV to be used. Bytes have to be separated with ":".
1372661f4fbSPablo de Lara
1382661f4fbSPablo de Lara*   aead_iv_random_size: set the size of the AEAD IV, which will be generated randomly.
1392661f4fbSPablo de Lara
1402661f4fbSPablo de Lara    Note that if --aead_iv is used, this will be ignored.
1412661f4fbSPablo de Lara
1427fe84b0dSJakub Poczatek*   aad: set the AAD to be used. Bytes have to be separated with ":".
143ba7b86b1SPablo de Lara
144ba7b86b1SPablo de Lara*   aad_random_size: set the size of the AAD, which will be generated randomly.
145ba7b86b1SPablo de Lara
146ba7b86b1SPablo de Lara    Note that if --aad is used, this will be ignored.
147ba7b86b1SPablo de Lara
148ba7b86b1SPablo de Lara*   digest_size: set the size of the digest to be generated/verified.
149ba7b86b1SPablo de Lara
150ba7b86b1SPablo de Lara*   sessionless: no crypto session will be created.
151ba7b86b1SPablo de Lara
152d2797f51SFan Zhang*   cryptodev_mask: A hexadecimal bitmask of the cryptodevs to be used by the
153d2797f51SFan Zhang    application.
154d2797f51SFan Zhang
1557fe84b0dSJakub Poczatek    (Default is all cryptodevs.)
156d2797f51SFan Zhang
1577fe84b0dSJakub Poczatek*   [no-]mac-updating: Enable or disable MAC addresses updating. (Enabled by default.)
158acdfecbaSKuba Kozak
159ba7b86b1SPablo de Lara
160ba7b86b1SPablo de LaraThe application requires that crypto devices capable of performing
161ba7b86b1SPablo de Larathe specified crypto operation are available on application initialization.
162ba7b86b1SPablo de LaraThis means that HW crypto device/s must be bound to a DPDK driver or
163ba7b86b1SPablo de Laraa SW crypto device/s (virtual crypto PMD) must be created (using --vdev).
164ba7b86b1SPablo de Lara
165*8750576fSNandini PersadTo run the application in Linux environment with 2 lcores, 2 ports and 2 crypto devices, issue the command:
166ba7b86b1SPablo de Lara
167ba7b86b1SPablo de Lara.. code-block:: console
168ba7b86b1SPablo de Lara
1697fe84b0dSJakub Poczatek    $ ./<build_dir>/examples/dpdk-l2fwd-crypto -l 0-1 --vdev "crypto_aesni_mb0" \
17030883f3eSPablo de Lara    --vdev "crypto_aesni_mb1" -- -p 0x3 --chain CIPHER_HASH \
1714790f99dSPablo de Lara    --cipher_op ENCRYPT --cipher_algo aes-cbc \
172ba7b86b1SPablo de Lara    --cipher_key 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f \
1734790f99dSPablo de Lara    --auth_op GENERATE --auth_algo aes-xcbc-mac \
174ba7b86b1SPablo de Lara    --auth_key 10:11:12:13:14:15:16:17:18:19:1a:1b:1c:1d:1e:1f
175ba7b86b1SPablo de Lara
176ba7b86b1SPablo de LaraRefer to the *DPDK Getting Started Guide* for general information on running applications
177ba7b86b1SPablo de Laraand the Environment Abstraction Layer (EAL) options.
178ba7b86b1SPablo de Lara
17910b9d471SVipin Varghese.. Note::
18010b9d471SVipin Varghese
18110b9d471SVipin Varghese    * The ``l2fwd-crypto`` sample application requires IPv4 packets for crypto operation.
18210b9d471SVipin Varghese
1837fe84b0dSJakub Poczatek    * If multiple Ethernet ports are passed, then equal number of crypto devices are to be passed.
18410b9d471SVipin Varghese
18510b9d471SVipin Varghese    * All crypto devices shall use the same session.
18610b9d471SVipin Varghese
187ba7b86b1SPablo de LaraExplanation
188ba7b86b1SPablo de Lara-----------
189ba7b86b1SPablo de Lara
190ba7b86b1SPablo de LaraThe L2 forward with Crypto application demonstrates the performance of a crypto operation
1917fe84b0dSJakub Poczatekon a packet received on an RX PORT before forwarding it to a TX PORT.
192ba7b86b1SPablo de Lara
193ba7b86b1SPablo de LaraThe following figure illustrates a sample flow of a packet in the application,
194ba7b86b1SPablo de Larafrom reception until transmission.
195ba7b86b1SPablo de Lara
196ba7b86b1SPablo de Lara.. _figure_l2_fwd_encrypt_flow:
197ba7b86b1SPablo de Lara
198ba7b86b1SPablo de Lara.. figure:: img/l2_fwd_encrypt_flow.*
199ba7b86b1SPablo de Lara
2007fe84b0dSJakub Poczatek   Encryption flow through the L2 Forwarding with Crypto Application
201ba7b86b1SPablo de Lara
202ba7b86b1SPablo de Lara
203*8750576fSNandini PersadThe following sections provide explanation of the application.
204ba7b86b1SPablo de Lara
205ba7b86b1SPablo de LaraCrypto operation specification
206ba7b86b1SPablo de Lara~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
207ba7b86b1SPablo de Lara
208*8750576fSNandini PersadAll the packets received in all the ports get transformed by the crypto devices
209ba7b86b1SPablo de Lara(ciphering and/or authentication).
2107fe84b0dSJakub PoczatekThe crypto operation to be performed on the packet is parsed from the command line.
2117fe84b0dSJakub Poczatek(Go to "Running the Application" section for all the options.)
212ba7b86b1SPablo de Lara
213ba7b86b1SPablo de LaraIf no parameter is passed, the default crypto operation is:
214ba7b86b1SPablo de Lara
215ba7b86b1SPablo de Lara* Encryption with AES-CBC with 128 bit key.
216ba7b86b1SPablo de Lara
217ba7b86b1SPablo de Lara* Authentication with SHA1-HMAC (generation).
218ba7b86b1SPablo de Lara
219ba7b86b1SPablo de Lara* Keys, IV and AAD are generated randomly.
220ba7b86b1SPablo de Lara
221ba7b86b1SPablo de LaraThere are two methods to pass keys, IV and ADD from the command line:
222ba7b86b1SPablo de Lara
223ba7b86b1SPablo de Lara* Passing the full key, separated bytes by ":"::
224ba7b86b1SPablo de Lara
225ba7b86b1SPablo de Lara   --cipher_key 00:11:22:33:44
226ba7b86b1SPablo de Lara
227ba7b86b1SPablo de Lara* Passing the size, so key is generated randomly::
228ba7b86b1SPablo de Lara
229ba7b86b1SPablo de Lara   --cipher_key_random_size 16
230ba7b86b1SPablo de Lara
231ba7b86b1SPablo de Lara**Note**:
232*8750576fSNandini Persad   If the full key is passed (first method) and the size is passed as well (second method),
233ba7b86b1SPablo de Lara   the latter will be ignored.
234ba7b86b1SPablo de Lara
235*8750576fSNandini PersadThe size of these keys are checked (regardless of the method) before starting the app,
236ba7b86b1SPablo de Larato make sure that it is supported by the crypto devices.
237ba7b86b1SPablo de Lara
238ba7b86b1SPablo de LaraCrypto device initialization
239ba7b86b1SPablo de Lara~~~~~~~~~~~~~~~~~~~~~~~~~~~~
240ba7b86b1SPablo de Lara
241*8750576fSNandini PersadOnce the encryption operation is defined, the crypto devices are initialized.
242ba7b86b1SPablo de LaraThe crypto devices must be either bound to a DPDK driver (if they are physical devices)
243ba7b86b1SPablo de Laraor created using the EAL option --vdev (if they are virtual devices),
244ba7b86b1SPablo de Larawhen running the application.
245ba7b86b1SPablo de Lara
246ba7b86b1SPablo de LaraThe initialize_cryptodevs() function performs the device initialization.
247ba7b86b1SPablo de LaraIt iterates through the list of the available crypto devices and
2487fe84b0dSJakub Poczatekchecks which ones are capable of performing the operation.
249ba7b86b1SPablo de LaraEach device has a set of capabilities associated with it,
250ba7b86b1SPablo de Larawhich are stored in the device info structure, so the function checks if the operation
251ba7b86b1SPablo de Larais within the structure of each device.
252ba7b86b1SPablo de Lara
253ba7b86b1SPablo de LaraThe following code checks if the device supports the specified cipher algorithm
254ba7b86b1SPablo de Lara(similar for the authentication algorithm):
255ba7b86b1SPablo de Lara
2569a212dc0SConor Fogarty.. literalinclude:: ../../../examples/l2fwd-crypto/main.c
2579a212dc0SConor Fogarty    :language: c
2589a212dc0SConor Fogarty    :start-after: Check if device supports cipher algo. 8<
2599a212dc0SConor Fogarty    :end-before: >8 End of check if device supports cipher algo.
2609a212dc0SConor Fogarty    :dedent: 2
261ba7b86b1SPablo de Lara
262ba7b86b1SPablo de LaraIf a capable crypto device is found, key sizes are checked to see if they are supported
263ba7b86b1SPablo de Lara(cipher key and IV for the ciphering):
264ba7b86b1SPablo de Lara
2659a212dc0SConor Fogarty.. literalinclude:: ../../../examples/l2fwd-crypto/main.c
2669a212dc0SConor Fogarty    :language: c
2679a212dc0SConor Fogarty    :start-after: Check if capable cipher is supported. 8<
2689a212dc0SConor Fogarty    :end-before: >8 End of checking if cipher is supported.
2699a212dc0SConor Fogarty    :dedent: 2
270ba7b86b1SPablo de Lara
271ba7b86b1SPablo de LaraAfter all the checks, the device is configured and it is added to the
272ba7b86b1SPablo de Laracrypto device list.
273ba7b86b1SPablo de Lara
274ba7b86b1SPablo de Lara**Note**:
275ba7b86b1SPablo de Lara   The number of crypto devices that supports the specified crypto operation
276ba7b86b1SPablo de Lara   must be at least the number of ports to be used.
277ba7b86b1SPablo de Lara
278ba7b86b1SPablo de LaraSession creation
279ba7b86b1SPablo de Lara~~~~~~~~~~~~~~~~
280ba7b86b1SPablo de Lara
281ba7b86b1SPablo de LaraThe crypto operation has a crypto session associated to it, which contains
282*8750576fSNandini Persadinformation such as the transform chain to perform (e.g. ciphering then hashing)
283ba7b86b1SPablo de Larapointers to the keys, lengths... etc.
284ba7b86b1SPablo de Lara
285ba7b86b1SPablo de LaraThis session is created and is later attached to the crypto operation:
286ba7b86b1SPablo de Lara
2879a212dc0SConor Fogarty.. literalinclude:: ../../../examples/l2fwd-crypto/main.c
2889a212dc0SConor Fogarty    :language: c
2899a212dc0SConor Fogarty    :start-after: Session is created and is later attached to the crypto operation. 8<
2909a212dc0SConor Fogarty    :end-before: >8 End of creation of session.
291ba7b86b1SPablo de Lara
292ba7b86b1SPablo de LaraCrypto operation creation
293ba7b86b1SPablo de Lara~~~~~~~~~~~~~~~~~~~~~~~~~
294ba7b86b1SPablo de Lara
2957fe84b0dSJakub PoczatekGiven N packets received from an RX PORT, N crypto operations are allocated
296ba7b86b1SPablo de Laraand filled:
297ba7b86b1SPablo de Lara
2989a212dc0SConor Fogarty.. literalinclude:: ../../../examples/l2fwd-crypto/main.c
2999a212dc0SConor Fogarty    :language: c
3009a212dc0SConor Fogarty    :start-after: Allocate and fillcrypto operations. 8<
3019a212dc0SConor Fogarty    :end-before: >8 End of crypto operation allocated and filled.
3029a212dc0SConor Fogarty    :dedent: 3
303ba7b86b1SPablo de Lara
304ba7b86b1SPablo de LaraAfter filling the crypto operation (including session attachment),
305ba7b86b1SPablo de Larathe mbuf which will be transformed is attached to it::
306ba7b86b1SPablo de Lara
307ba7b86b1SPablo de Lara   op->sym->m_src = m;
308ba7b86b1SPablo de Lara
309ba7b86b1SPablo de LaraSince no destination mbuf is set, the source mbuf will be overwritten
310ba7b86b1SPablo de Laraafter the operation is done (in-place).
311ba7b86b1SPablo de Lara
3128d257235SJohn McNamaraCrypto operation enqueuing/dequeuing
3138d257235SJohn McNamara~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
314ba7b86b1SPablo de Lara
315ba7b86b1SPablo de LaraOnce the operation has been created, it has to be enqueued in one of the crypto devices.
316ba7b86b1SPablo de LaraBefore doing so, for performance reasons, the operation stays in a buffer.
317ba7b86b1SPablo de LaraWhen the buffer has enough operations (MAX_PKT_BURST), they are enqueued in the device,
318ba7b86b1SPablo de Larawhich will perform the operation at that moment:
319ba7b86b1SPablo de Lara
3209a212dc0SConor Fogarty.. literalinclude:: ../../../examples/l2fwd-crypto/main.c
3219a212dc0SConor Fogarty    :language: c
3229a212dc0SConor Fogarty    :start-after: Crypto enqueue. 8<
3239a212dc0SConor Fogarty    :end-before: >8 End of crypto enqueue.
324ba7b86b1SPablo de Lara
3259a212dc0SConor Fogarty.. literalinclude:: ../../../examples/l2fwd-crypto/main.c
3269a212dc0SConor Fogarty    :language: c
3279a212dc0SConor Fogarty    :start-after: l2fwd_crypto_send_burst 8<
3289a212dc0SConor Fogarty    :end-before: >8 End of l2fwd_crypto_send_burst.
329ba7b86b1SPablo de Lara
330ba7b86b1SPablo de LaraAfter this, the operations are dequeued from the device, and the transformed mbuf
331ba7b86b1SPablo de Larais extracted from the operation. Then, the operation is freed and the mbuf is
332ba7b86b1SPablo de Laraforwarded as it is done in the L2 forwarding application.
333ba7b86b1SPablo de Lara
3349a212dc0SConor Fogarty.. literalinclude:: ../../../examples/l2fwd-crypto/main.c
3359a212dc0SConor Fogarty    :language: c
3369a212dc0SConor Fogarty    :start-after: Dequeue packets from Crypto device. 8<
3379a212dc0SConor Fogarty    :end-before: >8 End of dequeue packets from crypto device.
3389a212dc0SConor Fogarty    :dedent: 3
339