1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(C) 2021 Marvell. 3 4Marvell cnxk platform guide 5=========================== 6 7This document gives an overview of **Marvell OCTEON CN9K and CN10K** RVU H/W block, 8packet flow and procedure to build DPDK on OCTEON cnxk platform. 9 10More information about CN9K and CN10K SoC can be found at `Marvell Official Website 11<https://www.marvell.com/embedded-processors/infrastructure-processors/>`_. 12 13Supported OCTEON cnxk SoCs 14-------------------------- 15 16- CN93xx 17- CN96xx 18- CN98xx 19- CN106xx 20- CNF105xx 21- CN103XX 22 23Resource Virtualization Unit architecture 24----------------------------------------- 25 26The :numref:`figure_cnxk_resource_virtualization` diagram depicts the 27RVU architecture and a resource provisioning example. 28 29.. _figure_cnxk_resource_virtualization: 30 31.. figure:: img/cnxk_resource_virtualization.* 32 33 cnxk Resource virtualization architecture and provisioning example 34 35 36Resource Virtualization Unit (RVU) on Marvell's OCTEON CN9K/CN10K SoC maps HW 37resources belonging to the network, crypto and other functional blocks onto 38PCI-compatible physical and virtual functions. 39 40Each functional block has multiple local functions (LFs) for 41provisioning to different PCIe devices. RVU supports multiple PCIe SRIOV 42physical functions (PFs) and virtual functions (VFs). 43 44The :numref:`table_cnxk_rvu_dpdk_mapping` shows the various local 45functions (LFs) provided by the RVU and its functional mapping to 46DPDK subsystem. 47 48.. _table_cnxk_rvu_dpdk_mapping: 49 50.. table:: RVU managed functional blocks and its mapping to DPDK subsystem 51 52 +---+-----+--------------------------------------------------------------+ 53 | # | LF | DPDK subsystem mapping | 54 +===+=====+==============================================================+ 55 | 1 | NIX | rte_ethdev, rte_tm, rte_event_eth_[rt]x_adapter, rte_security| 56 +---+-----+--------------------------------------------------------------+ 57 | 2 | NPA | rte_mempool | 58 +---+-----+--------------------------------------------------------------+ 59 | 3 | NPC | rte_flow | 60 +---+-----+--------------------------------------------------------------+ 61 | 4 | CPT | rte_cryptodev, rte_event_crypto_adapter | 62 +---+-----+--------------------------------------------------------------+ 63 | 5 | SSO | rte_eventdev | 64 +---+-----+--------------------------------------------------------------+ 65 | 6 | TIM | rte_event_timer_adapter | 66 +---+-----+--------------------------------------------------------------+ 67 | 7 | LBK | rte_ethdev | 68 +---+-----+--------------------------------------------------------------+ 69 | 8 | DPI | rte_dmadev | 70 +---+-----+--------------------------------------------------------------+ 71 | 9 | SDP | rte_ethdev | 72 +---+-----+--------------------------------------------------------------+ 73 | 10| REE | rte_regexdev | 74 +---+-----+--------------------------------------------------------------+ 75 | 11| BPHY| rte_rawdev | 76 +---+-----+--------------------------------------------------------------+ 77 | 12| GPIO| rte_rawdev | 78 +---+-----+--------------------------------------------------------------+ 79 | 13| ML | rte_mldev | 80 +---+-----+--------------------------------------------------------------+ 81 | 14| RVU | rte_rawdev | 82 +---+-----+--------------------------------------------------------------+ 83 84PF0 is called the administrative / admin function (AF) and has exclusive 85privileges to provision RVU functional block's LFs to each of the PF/VF. 86 87PF/VFs communicates with AF via a shared memory region (mailbox).Upon receiving 88requests from PF/VF, AF does resource provisioning and other HW configuration. 89 90AF is always attached to host, but PF/VFs may be used by host kernel itself, 91or attached to VMs or to userspace applications like DPDK, etc. So, AF has to 92handle provisioning/configuration requests sent by any device from any domain. 93 94The AF driver does not receive or process any data. 95It is only a configuration driver used in control path. 96 97The :numref:`figure_cnxk_resource_virtualization` diagram also shows a 98resource provisioning example where, 99 100#. PFx and PFx-VF0 bound to Linux netdev driver. 101 102#. PFx-VF1 ethdev driver bound to the first DPDK application. 103 104#. PFy ethdev driver, PFy-VF0 ethdev driver, PFz eventdev driver, PFm-VF0 cryptodev driver bound to the second DPDK application. 105 106LBK HW Access 107------------- 108 109Loopback HW Unit (LBK) receives packets from NIX-RX and sends packets back to NIX-TX. 110The loopback block has N channels and contains data buffering that is shared across 111all channels. The LBK HW Unit is abstracted using ethdev subsystem, Where PF0's 112VFs are exposed as ethdev device and odd-even pairs of VFs are tied together, 113that is, packets sent on odd VF end up received on even VF and vice versa. 114This would enable HW accelerated means of communication between two domains 115where even VF bound to the first domain and odd VF bound to the second domain. 116 117Typical application usage models are, 118 119#. Communication between the Linux kernel and DPDK application. 120 121#. Exception path to Linux kernel from DPDK application as SW ``KNI`` replacement. 122 123#. Communication between two different DPDK applications. 124 125SDP interface 126------------- 127 128System DPI Packet Interface unit(SDP) provides PCIe endpoint support for remote host 129to DMA packets into and out of cnxk SoC. SDP interface comes in to live only when 130cnxk SoC is connected in PCIe endpoint mode. It can be used to send/receive 131packets to/from remote host machine using input/output queue pairs exposed to it. 132SDP interface receives input packets from remote host from NIX-RX and sends packets 133to remote host using NIX-TX. Remote host machine need to use corresponding driver 134(kernel/user mode) to communicate with SDP interface on cnxk SoC. SDP supports 135single PCIe SRIOV physical function(PF) and multiple virtual functions(VF's). Users 136can bind PF or VF to use SDP interface and it will be enumerated as ethdev ports. 137 138The primary use case for SDP is to enable the smart NIC use case. Typical usage models are, 139 140#. Communication channel between remote host and cnxk SoC over PCIe. 141 142#. Transfer packets received from network interface to remote host over PCIe and 143 vice-versa. 144 145cnxk packet flow 146---------------------- 147 148The :numref:`figure_cnxk_packet_flow_hw_accelerators` diagram depicts 149the packet flow on cnxk SoC in conjunction with use of various HW accelerators. 150 151.. _figure_cnxk_packet_flow_hw_accelerators: 152 153.. figure:: img/cnxk_packet_flow_hw_accelerators.* 154 155 cnxk packet flow in conjunction with use of HW accelerators 156 157HW Offload Drivers 158------------------ 159 160This section lists dataplane H/W block(s) available in cnxk SoC. 161 162#. **Ethdev Driver** 163 See :doc:`../nics/cnxk` for NIX Ethdev driver information. 164 165#. **Mempool Driver** 166 See :doc:`../mempool/cnxk` for NPA mempool driver information. 167 168#. **Baseband PHY Driver** 169 See :doc:`../rawdevs/cnxk_bphy` for Baseband PHY driver information. 170 171#. **Dmadev Driver** 172 See :doc:`../dmadevs/cnxk` for DPI Dmadev driver information. 173 174#. **Regex Device Driver** 175 See :doc:`../regexdevs/cn9k` for REE Regex device driver information. 176 177#. **ML Device Driver** 178 See :doc:`../mldevs/cnxk` for Machine Learning device driver information. 179 180#. **RVU LF Driver** 181 See :doc:`../rawdevs/cnxk_rvu_lf` for RVU LF driver information. 182 183Procedure to Setup Platform 184--------------------------- 185 186There are three main prerequisites for setting up DPDK on cnxk 187compatible board: 188 189#. **RVU AF Linux kernel driver** 190 191 The dependent kernel drivers can be obtained from the 192 `kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/marvell/octeontx2>`_. 193 194 Alternatively, the Marvell SDK also provides the required kernel drivers. 195 196 Linux kernel should be configured with the following features enabled: 197 198 .. code-block:: console 199 200 # 64K pages enabled for better performance 201 CONFIG_ARM64_64K_PAGES=y 202 CONFIG_ARM64_VA_BITS_48=y 203 # huge pages support enabled 204 CONFIG_HUGETLBFS=y 205 CONFIG_HUGETLB_PAGE=y 206 # VFIO enabled with TYPE1 IOMMU at minimum 207 CONFIG_VFIO_IOMMU_TYPE1=y 208 CONFIG_VFIO_VIRQFD=y 209 CONFIG_VFIO=y 210 CONFIG_VFIO_NOIOMMU=y 211 CONFIG_VFIO_PCI=y 212 CONFIG_VFIO_PCI_MMAP=y 213 # SMMUv3 driver 214 CONFIG_ARM_SMMU_V3=y 215 # ARMv8.1 LSE atomics 216 CONFIG_ARM64_LSE_ATOMICS=y 217 # OCTEONTX2 drivers 218 CONFIG_OCTEONTX2_MBOX=y 219 CONFIG_OCTEONTX2_AF=y 220 # Enable if netdev PF driver required 221 CONFIG_OCTEONTX2_PF=y 222 # Enable if netdev VF driver required 223 CONFIG_OCTEONTX2_VF=y 224 CONFIG_CRYPTO_DEV_OCTEONTX2_CPT=y 225 # Enable if OCTEONTX2 DMA PF driver required 226 CONFIG_OCTEONTX2_DPI_PF=n 227 228#. **ARM64 Linux Tool Chain** 229 230 For example, the *aarch64* Linaro Toolchain, which can be obtained from 231 `here <https://releases.linaro.org/components/toolchain/binaries/7.4-2019.02/aarch64-linux-gnu/>`_. 232 233 Alternatively, the Marvell SDK also provides GNU GCC toolchain, which is 234 optimized for cnxk CPU. 235 236#. **Rootfile system** 237 238 Any *aarch64* supporting filesystem may be used. For example, 239 Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained 240 from `<http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_. 241 242 Alternatively, the Marvell SDK provides the buildroot based root filesystem. 243 The SDK includes all the above prerequisites necessary to bring up the cnxk board. 244 245- Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment. 246 247 248Debugging Options 249----------------- 250 251.. _table_cnxk_common_debug_options: 252 253.. table:: cnxk common debug options 254 255 +---+------------+-------------------------------------------------------+ 256 | # | Component | EAL log command | 257 +===+============+=======================================================+ 258 | 1 | Common | --log-level='pmd\.common\.cnxk\.base,8' | 259 +---+------------+-------------------------------------------------------+ 260 | 2 | Mailbox | --log-level='pmd\.common\.cnxk\.mbox,8' | 261 +---+------------+-------------------------------------------------------+ 262 263Debugfs support 264~~~~~~~~~~~~~~~ 265 266The **RVU AF Linux kernel driver** provides support to dump RVU blocks 267context or stats using debugfs. 268 269Enable ``debugfs`` by: 270 271#. Compile kernel with debugfs enabled, i.e ``CONFIG_DEBUG_FS=y``. 272 273#. Boot OCTEON CN9K/CN10K with debugfs supported kernel. 274 275#. Verify ``debugfs`` mounted by default "mount | grep -i debugfs" or mount it manually by using. 276 277 .. code-block:: console 278 279 # mount -t debugfs none /sys/kernel/debug 280 281Currently ``debugfs`` supports the following RVU blocks NIX, NPA, NPC, NDC, 282SSO & RPM. 283 284The file structure under ``/sys/kernel/debug`` is as follows 285 286.. code-block:: console 287 288 octeontx2/ 289 | 290 cn10k/ 291 |-- rpm 292 | |-- rpm0 293 | | '-- lmac0 294 | | '-- stats 295 | |-- rpm1 296 | | |-- lmac0 297 | | | '-- stats 298 | | '-- lmac1 299 | | '-- stats 300 | '-- rpm2 301 | '-- lmac0 302 | '-- stats 303 |-- cpt 304 | |-- cpt_engines_info 305 | |-- cpt_engines_sts 306 | |-- cpt_err_info 307 | |-- cpt_lfs_info 308 | '-- cpt_pc 309 |---- nix 310 | |-- cq_ctx 311 | |-- ndc_rx_cache 312 | |-- ndc_rx_hits_miss 313 | |-- ndc_tx_cache 314 | |-- ndc_tx_hits_miss 315 | |-- qsize 316 | |-- rq_ctx 317 | '-- sq_ctx 318 |-- npa 319 | |-- aura_ctx 320 | |-- ndc_cache 321 | |-- ndc_hits_miss 322 | |-- pool_ctx 323 | '-- qsize 324 |-- npc 325 | |-- mcam_info 326 | |-- mcam_rules 327 | '-- rx_miss_act_stats 328 |-- rsrc_alloc 329 '-- sso 330 |-- hws 331 | '-- sso_hws_info 332 '-- hwgrp 333 |-- sso_hwgrp_aq_thresh 334 |-- sso_hwgrp_iaq_walk 335 |-- sso_hwgrp_pc 336 |-- sso_hwgrp_free_list_walk 337 |-- sso_hwgrp_ient_walk 338 '-- sso_hwgrp_taq_walk 339 340RVU block LF allocation: 341 342.. code-block:: console 343 344 cat /sys/kernel/debug/cn10k/rsrc_alloc 345 346 pcifunc NPA NIX SSO GROUP SSOWS TIM CPT 347 PF1 0 0 348 PF4 1 349 PF13 0, 1 0, 1 0 350 351RPM example usage: 352 353.. code-block:: console 354 355 cat /sys/kernel/debug/cn10k/rpm/rpm0/lmac0/stats 356 357 =======Link Status====== 358 359 Link is UP 25000 Mbps 360 361 =======NIX RX_STATS(rpm port level)====== 362 363 rx_ucast_frames: 0 364 rx_mcast_frames: 0 365 rx_bcast_frames: 0 366 rx_frames: 0 367 rx_bytes: 0 368 rx_drops: 0 369 rx_errors: 0 370 371 =======NIX TX_STATS(rpm port level)====== 372 373 tx_ucast_frames: 0 374 tx_mcast_frames: 0 375 tx_bcast_frames: 0 376 tx_frames: 0 377 tx_bytes: 0 378 tx_drops: 0 379 380 =======rpm RX_STATS====== 381 382 Octets of received packets: 0 383 Octets of received packets with out error: 0 384 Received packets with alignment errors: 0 385 Control/PAUSE packets received: 0 386 Packets received with Frame too long Errors: 0 387 Packets received with a1nrange length Errors: 0 388 Received packets: 0 389 Packets received with FrameCheckSequenceErrors: 0 390 Packets received with VLAN header: 0 391 Error packets: 0 392 Packets received with unicast DMAC: 0 393 Packets received with multicast DMAC: 0 394 Packets received with broadcast DMAC: 0 395 Dropped packets: 0 396 Total frames received on interface: 0 397 Packets received with an octet count < 64: 0 398 Packets received with an octet count == 64: 0 399 Packets received with an octet count of 65–127: 0 400 Packets received with an octet count of 128-255: 0 401 Packets received with an octet count of 256-511: 0 402 Packets received with an octet count of 512-1023: 0 403 Packets received with an octet count of 1024-1518: 0 404 Packets received with an octet count of > 1518: 0 405 Oversized Packets: 0 406 Jabber Packets: 0 407 Fragmented Packets: 0 408 CBFC(class based flow control) pause frames received for class 0: 0 409 CBFC pause frames received for class 1: 0 410 CBFC pause frames received for class 2: 0 411 CBFC pause frames received for class 3: 0 412 CBFC pause frames received for class 4: 0 413 CBFC pause frames received for class 5: 0 414 CBFC pause frames received for class 6: 0 415 CBFC pause frames received for class 7: 0 416 CBFC pause frames received for class 8: 0 417 CBFC pause frames received for class 9: 0 418 CBFC pause frames received for class 10: 0 419 CBFC pause frames received for class 11: 0 420 CBFC pause frames received for class 12: 0 421 CBFC pause frames received for class 13: 0 422 CBFC pause frames received for class 14: 0 423 CBFC pause frames received for class 15: 0 424 MAC control packets received: 0 425 426 =======rpm TX_STATS====== 427 428 Total octets sent on the interface: 0 429 Total octets transmitted OK: 0 430 Control/Pause frames sent: 0 431 Total frames transmitted OK: 0 432 Total frames sent with VLAN header: 0 433 Error Packets: 0 434 Packets sent to unicast DMAC: 0 435 Packets sent to the multicast DMAC: 0 436 Packets sent to a broadcast DMAC: 0 437 Packets sent with an octet count == 64: 0 438 Packets sent with an octet count of 65–127: 0 439 Packets sent with an octet count of 128-255: 0 440 Packets sent with an octet count of 256-511: 0 441 Packets sent with an octet count of 512-1023: 0 442 Packets sent with an octet count of 1024-1518: 0 443 Packets sent with an octet count of > 1518: 0 444 CBFC(class based flow control) pause frames transmitted for class 0: 0 445 CBFC pause frames transmitted for class 1: 0 446 CBFC pause frames transmitted for class 2: 0 447 CBFC pause frames transmitted for class 3: 0 448 CBFC pause frames transmitted for class 4: 0 449 CBFC pause frames transmitted for class 5: 0 450 CBFC pause frames transmitted for class 6: 0 451 CBFC pause frames transmitted for class 7: 0 452 CBFC pause frames transmitted for class 8: 0 453 CBFC pause frames transmitted for class 9: 0 454 CBFC pause frames transmitted for class 10: 0 455 CBFC pause frames transmitted for class 11: 0 456 CBFC pause frames transmitted for class 12: 0 457 CBFC pause frames transmitted for class 13: 0 458 CBFC pause frames transmitted for class 14: 0 459 CBFC pause frames transmitted for class 15: 0 460 MAC control packets sent: 0 461 Total frames sent on the interface: 0 462 463CPT example usage: 464 465.. code-block:: console 466 467 cat /sys/kernel/debug/cn10k/cpt/cpt_pc 468 469 CPT instruction requests 0 470 CPT instruction latency 0 471 CPT NCB read requests 0 472 CPT NCB read latency 0 473 CPT read requests caused by UC fills 0 474 CPT active cycles pc 1395642 475 CPT clock count pc 5579867595493 476 477NIX example usage: 478 479.. code-block:: console 480 481 Usage: echo <nixlf> [cq number/all] > /sys/kernel/debug/cn10k/nix/cq_ctx 482 cat /sys/kernel/debug/cn10k/nix/cq_ctx 483 echo 0 0 > /sys/kernel/debug/cn10k/nix/cq_ctx 484 cat /sys/kernel/debug/cn10k/nix/cq_ctx 485 486 =====cq_ctx for nixlf:0 and qidx:0 is===== 487 W0: base 158ef1a00 488 489 W1: wrptr 0 490 W1: avg_con 0 491 W1: cint_idx 0 492 W1: cq_err 0 493 W1: qint_idx 0 494 W1: bpid 0 495 W1: bp_ena 0 496 497 W2: update_time 31043 498 W2:avg_level 255 499 W2: head 0 500 W2:tail 0 501 502 W3: cq_err_int_ena 5 503 W3:cq_err_int 0 504 W3: qsize 4 505 W3:caching 1 506 W3: substream 0x000 507 W3: ena 1 508 W3: drop_ena 1 509 W3: drop 64 510 W3: bp 0 511 512NPA example usage: 513 514.. code-block:: console 515 516 Usage: echo <npalf> [pool number/all] > /sys/kernel/debug/cn10k/npa/pool_ctx 517 cat /sys/kernel/debug/cn10k/npa/pool_ctx 518 echo 0 0 > /sys/kernel/debug/cn10k/npa/pool_ctx 519 cat /sys/kernel/debug/cn10k/npa/pool_ctx 520 521 ======POOL : 0======= 522 W0: Stack base 1375bff00 523 W1: ena 1 524 W1: nat_align 1 525 W1: stack_caching 1 526 W1: stack_way_mask 0 527 W1: buf_offset 1 528 W1: buf_size 19 529 W2: stack_max_pages 24315 530 W2: stack_pages 24314 531 W3: op_pc 267456 532 W4: stack_offset 2 533 W4: shift 5 534 W4: avg_level 255 535 W4: avg_con 0 536 W4: fc_ena 0 537 W4: fc_stype 0 538 W4: fc_hyst_bits 0 539 W4: fc_up_crossing 0 540 W4: update_time 62993 541 W5: fc_addr 0 542 W6: ptr_start 1593adf00 543 W7: ptr_end 180000000 544 W8: err_int 0 545 W8: err_int_ena 7 546 W8: thresh_int 0 547 W8: thresh_int_ena 0 548 W8: thresh_up 0 549 W8: thresh_qint_idx 0 550 W8: err_qint_idx 0 551 552NPC example usage: 553 554.. code-block:: console 555 556 cat /sys/kernel/debug/cn10k/npc/mcam_info 557 558 NPC MCAM info: 559 RX keywidth : 224bits 560 TX keywidth : 224bits 561 562 MCAM entries : 2048 563 Reserved : 158 564 Available : 1890 565 566 MCAM counters : 512 567 Reserved : 1 568 Available : 511 569 570SSO example usage: 571 572.. code-block:: console 573 574 Usage: echo [<hws>/all] > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info 575 echo 0 > /sys/kernel/debug/cn10k/sso/hws/sso_hws_info 576 577 ================================================== 578 SSOW HWS[0] Arbitration State 0x0 579 SSOW HWS[0] Guest Machine Control 0x0 580 SSOW HWS[0] SET[0] Group Mask[0] 0xffffffffffffffff 581 SSOW HWS[0] SET[0] Group Mask[1] 0xffffffffffffffff 582 SSOW HWS[0] SET[0] Group Mask[2] 0xffffffffffffffff 583 SSOW HWS[0] SET[0] Group Mask[3] 0xffffffffffffffff 584 SSOW HWS[0] SET[1] Group Mask[0] 0xffffffffffffffff 585 SSOW HWS[0] SET[1] Group Mask[1] 0xffffffffffffffff 586 SSOW HWS[0] SET[1] Group Mask[2] 0xffffffffffffffff 587 SSOW HWS[0] SET[1] Group Mask[3] 0xffffffffffffffff 588 ================================================== 589 590Compile DPDK 591------------ 592 593DPDK may be compiled either natively on OCTEON CN9K/CN10K platform or cross-compiled on 594an x86 based platform. 595The Meson build option ``enable_iova_as_pa`` should be set to false 596because on CNXK platforms, IOVA is same as the virtual address. 597Disabling the iova field in the mbuf frees it up to be used as a dynamic field. 598 599Native Compilation 600~~~~~~~~~~~~~~~~~~ 601 602Refer to :doc:`../linux_gsg/build_dpdk` for generic native builds. 603 604CN9K: 605 606.. code-block:: console 607 608 meson setup -Dplatform=cn9k -Denable_iova_as_pa=false build 609 ninja -C build 610 611CN10K: 612 613.. code-block:: console 614 615 meson setup -Dplatform=cn10k -Denable_iova_as_pa=false build 616 ninja -C build 617 618Cross Compilation 619~~~~~~~~~~~~~~~~~ 620 621Refer to :doc:`../linux_gsg/cross_build_dpdk_for_arm64` for generic arm64 details. 622 623CN9K: 624 625.. code-block:: console 626 627 meson setup -Denable_iova_as_pa=false build --cross-file config/arm/arm64_cn9k_linux_gcc 628 ninja -C build 629 630CN10K: 631 632.. code-block:: console 633 634 meson setup -Denable_iova_as_pa=false build --cross-file config/arm/arm64_cn10k_linux_gcc 635 ninja -C build 636 637.. note:: 638 639 By default, meson cross compilation uses ``aarch64-linux-gnu-gcc`` toolchain, 640 if Marvell toolchain is available then it can be used by overriding the 641 c, cpp, ar, strip ``binaries`` attributes to respective Marvell 642 toolchain binaries in ``config/arm/arm64_cn10k_linux_gcc`` file. 643 644Environment Variables 645~~~~~~~~~~~~~~~~~~~~~ 646 647* ``BPHY_INTR_MLOCK_DISABLE`` 648 When defined disables memory locking in 649 BPHY environment. 650 651* ``ROC_CN10K_MBOX_TIMEOUT``, ``ROC_MBOX_TIMEOUT`` 652 When set, overrides MBOX timeout by value in milli seconds. 653 654* ``ETH_SEC_IV_OVR`` 655 When set, overrides outbound inline SA IV. By default IV is generated 656 by HW. Format of variable is string of comma separated one byte values as 657 for ex: "0x0, 0x10, 0x20, ..." 658