1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright(c) 2022-2023 PANTHEON.tech s.r.o. 3 4DPDK Test Suite 5=============== 6 7The DPDK Test Suite, abbreviated DTS, is a Python test framework with test suites 8implementing functional and performance tests used to test DPDK. 9 10 11DTS Terminology 12--------------- 13 14DTS node 15 A generic description of any host/server DTS connects to. 16 17DTS runtime environment 18 An environment containing Python with packages needed to run DTS. 19 20DTS runtime environment node 21 A node where at least one DTS runtime environment is present. 22 This is the node where we run DTS and from which DTS connects to other nodes. 23 24System under test 25 An SUT is the combination of DPDK and the hardware we're testing 26 in conjunction with DPDK (NICs, crypto and other devices). 27 28System under test node 29 A node where at least one SUT is present. 30 31Traffic generator 32 A TG is either software or hardware capable of sending packets. 33 34Traffic generator node 35 A node where at least one TG is present. 36 In case of hardware traffic generators, the TG and the node are literally the same. 37 38 39In most cases, interchangeably referring to a runtime environment, SUT, TG or the node 40they're running on (e.g. using SUT and SUT node interchangeably) doesn't cause confusion. 41There could theoretically be more than of these running on the same node and in that case 42it's useful to have stricter definitions. 43An example would be two different traffic generators (such as Trex and Scapy) 44running on the same node. 45A different example would be a node containing both a DTS runtime environment 46and a traffic generator, in which case it's both a DTS runtime environment node and a TG node. 47 48 49DTS Environment 50--------------- 51 52DTS is written entirely in Python using a variety of dependencies. 53DTS uses Poetry as its Python dependency management. 54Python build/development and runtime environments are the same and DTS development environment, 55DTS runtime environment or just plain DTS environment are used interchangeably. 56 57 58Setting up DTS environment 59~~~~~~~~~~~~~~~~~~~~~~~~~~ 60 61#. **Python Version** 62 63 The Python Version required by DTS is specified in ``dts/pyproject.toml`` in the 64 **[tool.poetry.dependencies]** section: 65 66 .. literalinclude:: ../../../dts/pyproject.toml 67 :language: cfg 68 :start-at: [tool.poetry.dependencies] 69 :end-at: python 70 71 The Python dependency manager DTS uses, Poetry, doesn't install Python, so you may need 72 to satisfy this requirement by other means if your Python is not up-to-date. 73 A tool such as `Pyenv <https://github.com/pyenv/pyenv>`_ is a good way to get Python, 74 though not the only one. 75 76#. **Poetry** 77 78 The typical style of python dependency management, pip with ``requirements.txt``, 79 has a few issues. 80 The advantages of Poetry include specifying what Python version is required and forcing you 81 to specify versions, enforced by a lockfile, both of which help prevent broken dependencies. 82 Another benefit is the usage of ``pyproject.toml``, which has become the standard config file 83 for python projects, improving project organization. 84 To install Poetry, visit their `doc pages <https://python-poetry.org/docs/>`_. 85 The recommended Poetry version is at least 1.5.1. 86 87#. **Getting a Poetry shell** 88 89 Once you have Poetry along with the proper Python version all set up, it's just a matter 90 of installing dependencies via Poetry and using the virtual environment Poetry provides: 91 92 .. code-block:: console 93 94 poetry install --no-root 95 poetry shell 96 97#. **SSH Connection** 98 99 DTS uses the Fabric Python library for SSH connections between DTS environment 100 and the other hosts. 101 The authentication method used is pubkey authentication. 102 Fabric tries to use a passed key/certificate, 103 then any key it can with through an SSH agent, 104 then any "id_rsa", "id_dsa" or "id_ecdsa" key discoverable in ``~/.ssh/`` 105 (with any matching OpenSSH-style certificates). 106 DTS doesn't pass any keys, so Fabric tries to use the other two methods. 107 108 109Setting up System Under Test 110---------------------------- 111 112There are two areas that need to be set up on a System Under Test: 113 114#. **DPDK dependencies** 115 116 DPDK will be built and run on the SUT. 117 Consult the Getting Started guides for the list of dependencies for each distribution. 118 119#. **Hardware dependencies** 120 121 Any hardware DPDK uses needs a proper driver 122 and most OS distributions provide those, but the version may not be satisfactory. 123 It's up to each user to install the driver they're interested in testing. 124 The hardware also may also need firmware upgrades, which is also left at user discretion. 125 126#. **Hugepages** 127 128 There are two ways to configure hugepages: 129 130 * DTS configuration 131 132 You may specify the optional hugepage configuration in the DTS config file. 133 If you do, DTS will take care of configuring hugepages, 134 overwriting your current SUT hugepage configuration. 135 136 * System under test configuration 137 138 It's possible to use the hugepage configuration already present on the SUT. 139 If you wish to do so, don't specify the hugepage configuration in the DTS config file. 140 141#. **User with administrator privileges** 142 143.. _sut_admin_user: 144 145 DTS needs administrator privileges to run DPDK applications (such as testpmd) on the SUT. 146 The SUT user must be able run commands in privileged mode without asking for password. 147 On most Linux distributions, it's a matter of setting up passwordless sudo: 148 149 #. Run ``sudo visudo`` and check that it contains ``%sudo ALL=(ALL:ALL) NOPASSWD:ALL``. 150 151 #. Add the SUT user to the sudo group with: 152 153 .. code-block:: console 154 155 sudo usermod -aG sudo <sut_user> 156 157 158Setting up Traffic Generator Node 159--------------------------------- 160 161These need to be set up on a Traffic Generator Node: 162 163#. **Traffic generator dependencies** 164 165 The traffic generator running on the traffic generator node must be installed beforehand. 166 For Scapy traffic generator, only a few Python libraries need to be installed: 167 168 .. code-block:: console 169 170 sudo apt install python3-pip 171 sudo pip install --upgrade pip 172 sudo pip install scapy==2.5.0 173 174#. **Hardware dependencies** 175 176 The traffic generators, like DPDK, need a proper driver and firmware. 177 The Scapy traffic generator doesn't have strict requirements - the drivers that come 178 with most OS distributions will be satisfactory. 179 180 181#. **User with administrator privileges** 182 183 Similarly to the System Under Test, traffic generators need administrator privileges 184 to be able to use the devices. 185 Refer to the `System Under Test section <sut_admin_user>` for details. 186 187 188Running DTS 189----------- 190 191DTS needs to know which nodes to connect to and what hardware to use on those nodes. 192Once that's configured, either a DPDK source code tarball or a Git revision ID 193of choice needs to be supplied. 194DTS will use this to compile DPDK on the SUT node 195and then run the tests with the newly built binaries. 196 197 198Configuring DTS 199~~~~~~~~~~~~~~~ 200 201DTS configuration is split into nodes and executions and build targets within executions, 202and follows a defined schema as described in `Configuration Schema`_. 203By default, DTS will try to use the ``dts/conf.yaml`` :ref:`config file <configuration_schema_example>`, 204which is a template that illustrates what can be configured in DTS. 205 206The user must have :ref:`administrator privileges <sut_admin_user>` 207which don't require password authentication. 208 209 210DTS Execution 211~~~~~~~~~~~~~ 212 213DTS is run with ``main.py`` located in the ``dts`` directory after entering Poetry shell: 214 215.. code-block:: console 216 217 (dts-py3.10) $ ./main.py --help 218 usage: main.py [-h] [--config-file CONFIG_FILE] [--output-dir OUTPUT_DIR] [-t TIMEOUT] [-v] [-s] [--tarball TARBALL] [--compile-timeout COMPILE_TIMEOUT] [--test-cases TEST_CASES] [--re-run RE_RUN] 219 220 Run DPDK test suites. All options may be specified with the environment variables provided in brackets. Command line arguments have higher priority. 221 222 options: 223 -h, --help show this help message and exit 224 --config-file CONFIG_FILE 225 [DTS_CFG_FILE] configuration file that describes the test cases, SUTs and targets. (default: ./conf.yaml) 226 --output-dir OUTPUT_DIR, --output OUTPUT_DIR 227 [DTS_OUTPUT_DIR] Output directory where DTS logs and results are saved. (default: output) 228 -t TIMEOUT, --timeout TIMEOUT 229 [DTS_TIMEOUT] The default timeout for all DTS operations except for compiling DPDK. (default: 15) 230 -v, --verbose [DTS_VERBOSE] Specify to enable verbose output, logging all messages to the console. (default: False) 231 -s, --skip-setup [DTS_SKIP_SETUP] Specify to skip all setup steps on SUT and TG nodes. (default: None) 232 --tarball TARBALL, --snapshot TARBALL, --git-ref TARBALL 233 [DTS_DPDK_TARBALL] Path to DPDK source code tarball or a git commit ID, tag ID or tree ID to test. To test local changes, first commit them, then use the commit ID with this option. (default: dpdk.tar.xz) 234 --compile-timeout COMPILE_TIMEOUT 235 [DTS_COMPILE_TIMEOUT] The timeout for compiling DPDK. (default: 1200) 236 --test-cases TEST_CASES 237 [DTS_TESTCASES] Comma-separated list of test cases to execute. Unknown test cases will be silently ignored. (default: ) 238 --re-run RE_RUN, --re_run RE_RUN 239 [DTS_RERUN] Re-run each test case the specified number of times if a test failure occurs (default: 0) 240 241 242The brackets contain the names of environment variables that set the same thing. 243The minimum DTS needs is a config file and a DPDK tarball or git ref ID. 244You may pass those to DTS using the command line arguments or use the default paths. 245 246Example command for running DTS with the template configuration and DPDK tag v23.11: 247 248.. code-block:: console 249 250 (dts-py3.10) $ ./main.py --git-ref v23.11 251 252 253DTS Results 254~~~~~~~~~~~ 255 256Results are stored in the output dir by default 257which be changed with the ``--output-dir`` command line argument. 258The results contain basic statistics of passed/failed test cases and DPDK version. 259 260 261Contributing to DTS 262------------------- 263 264There are two areas of contribution: The DTS framework and DTS test suites. 265 266The framework contains the logic needed to run test cases, such as connecting to nodes, 267running DPDK applications and collecting results. 268 269The test cases call APIs from the framework to test their scenarios. 270Adding test cases may require adding code to the framework as well. 271 272 273Framework Coding Guidelines 274~~~~~~~~~~~~~~~~~~~~~~~~~~~ 275 276When adding code to the DTS framework, pay attention to the rest of the code 277and try not to divert much from it. 278The :ref:`DTS developer tools <dts_dev_tools>` will issue warnings 279when some of the basics are not met. 280 281The code must be properly documented with docstrings. 282The style must conform to the `Google style 283<https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings>`_. 284See an example of the style `here 285<https://www.sphinx-doc.org/en/master/usage/extensions/example_google.html>`_. 286For cases which are not covered by the Google style, refer to `PEP 257 287<https://peps.python.org/pep-0257/>`_. 288There are some cases which are not covered by the two style guides, 289where we deviate or where some additional clarification is helpful: 290 291 * The ``__init__()`` methods of classes are documented separately 292 from the docstring of the class itself. 293 * The docstrings of implemented abstract methods should refer to the superclass's definition 294 if there's no deviation. 295 * Instance variables/attributes should be documented in the docstring of the class 296 in the ``Attributes:`` section. 297 * The ``dataclass.dataclass`` decorator changes how the attributes are processed. 298 The dataclass attributes which result in instance variables/attributes 299 should also be recorded in the ``Attributes:`` section. 300 * Class variables/attributes, on the other hand, should be documented with ``#:`` 301 above the type annotated line. 302 The description may be omitted if the meaning is obvious. 303 * The ``Enum`` and ``TypedDict`` also process the attributes in particular ways 304 and should be documented with ``#:`` as well. 305 This is mainly so that the autogenerated documentation contains the assigned value. 306 * When referencing a parameter of a function or a method in their docstring, 307 don't use any articles and put the parameter into single backticks. 308 This mimics the style of `Python's documentation <https://docs.python.org/3/index.html>`_. 309 * When specifying a value, use double backticks:: 310 311 def foo(greet: bool) -> None: 312 """Demonstration of single and double backticks. 313 314 `greet` controls whether ``Hello World`` is printed. 315 316 Args: 317 greet: Whether to print the ``Hello World`` message. 318 """ 319 if greet: 320 print(f"Hello World") 321 322 * The docstring maximum line length is the same as the code maximum line length. 323 324 325How To Write a Test Suite 326------------------------- 327 328All test suites inherit from ``TestSuite`` defined in ``dts/framework/test_suite.py``. 329There are four types of methods that comprise a test suite: 330 331#. **Test cases** 332 333 | Test cases are methods that start with a particular prefix. 334 | Functional test cases start with ``test_``, e.g. ``test_hello_world_single_core``. 335 | Performance test cases start with ``test_perf_``, e.g. ``test_perf_nic_single_core``. 336 | A test suite may have any number of functional and/or performance test cases. 337 However, these test cases must test the same feature, 338 following the rule of one feature = one test suite. 339 Test cases for one feature don't need to be grouped in just one test suite, though. 340 If the feature requires many testing scenarios to cover, 341 the test cases would be better off spread over multiple test suites 342 so that each test suite doesn't take too long to execute. 343 344#. **Setup and Teardown methods** 345 346 | There are setup and teardown methods for the whole test suite and each individual test case. 347 | Methods ``set_up_suite`` and ``tear_down_suite`` will be executed 348 before any and after all test cases have been executed, respectively. 349 | Methods ``set_up_test_case`` and ``tear_down_test_case`` will be executed 350 before and after each test case, respectively. 351 | These methods don't need to be implemented if there's no need for them in a test suite. 352 In that case, nothing will happen when they are executed. 353 354#. **Configuration, traffic and other logic** 355 356 The ``TestSuite`` class contains a variety of methods for anything that 357 a test suite setup, a teardown, or a test case may need to do. 358 359 The test suites also frequently use a DPDK app, such as testpmd, in interactive mode 360 and use the interactive shell instances directly. 361 362 These are the two main ways to call the framework logic in test suites. 363 If there's any functionality or logic missing from the framework, 364 it should be implemented so that the test suites can use one of these two ways. 365 366#. **Test case verification** 367 368 Test case verification should be done with the ``verify`` method, which records the result. 369 The method should be called at the end of each test case. 370 371#. **Other methods** 372 373 Of course, all test suite code should adhere to coding standards. 374 Only the above methods will be treated specially and any other methods may be defined 375 (which should be mostly private methods needed by each particular test suite). 376 Any specific features (such as NIC configuration) required by a test suite 377 should be implemented in the ``SutNode`` class (and the underlying classes that ``SutNode`` uses) 378 and used by the test suite via the ``sut_node`` field. 379 380 381.. _dts_dev_tools: 382 383DTS Developer Tools 384------------------- 385 386There are three tools used in DTS to help with code checking, style and formatting: 387 388* `isort <https://pycqa.github.io/isort/>`_ 389 390 Alphabetically sorts python imports within blocks. 391 392* `black <https://github.com/psf/black>`_ 393 394 Does most of the actual formatting (whitespaces, comments, line length etc.) 395 and works similarly to clang-format. 396 397* `pylama <https://github.com/klen/pylama>`_ 398 399 Runs a collection of python linters and aggregates output. 400 It will run these tools over the repository: 401 402 .. literalinclude:: ../../../dts/pyproject.toml 403 :language: cfg 404 :start-after: [tool.pylama] 405 :end-at: linters 406 407* `mypy <https://github.com/python/mypy>`_ 408 409 Enables static typing for Python, exploiting the type hints in the source code. 410 411These three tools are all used in ``devtools/dts-check-format.sh``, 412the DTS code check and format script. 413Refer to the script for usage: ``devtools/dts-check-format.sh -h``. 414 415 416Configuration Schema 417-------------------- 418 419Definitions 420~~~~~~~~~~~ 421 422_`Node name` 423 *string* – A unique identifier for a node. 424 **Examples**: ``SUT1``, ``TG1``. 425 426_`ARCH` 427 *string* – The CPU architecture. 428 **Supported values**: ``x86_64``, ``arm64``, ``ppc64le``. 429 430_`CPU` 431 *string* – The CPU microarchitecture. Use ``native`` for x86. 432 **Supported values**: ``native``, ``armv8a``, ``dpaa2``, ``thunderx``, ``xgene1``. 433 434_`OS` 435 *string* – The operating system. **Supported values**: ``linux``. 436 437_`Compiler` 438 *string* – The compiler used for building DPDK. 439 **Supported values**: ``gcc``, ``clang``, ``icc``, ``mscv``. 440 441_`Build target` 442 *mapping* – Build targets supported by DTS for building DPDK, described as: 443 444 ==================== ================================================================= 445 ``arch`` See `ARCH`_ 446 ``os`` See `OS`_ 447 ``cpu`` See `CPU`_ 448 ``compiler`` See `Compiler`_ 449 ``compiler_wrapper`` *string* – Value prepended to the CC variable for the DPDK build. 450 451 **Example**: ``ccache`` 452 ==================== ================================================================= 453 454_`hugepages` 455 *mapping* – hugepages described as: 456 457 ==================== ================================================================ 458 ``amount`` *integer* – The amount of hugepages to configure. 459 460 Hugepage size will be the system default. 461 ``force_first_numa`` (*optional*, defaults to ``false``) – If ``true``, it forces the 462 463 configuration of hugepages on the first NUMA node. 464 ==================== ================================================================ 465 466_`Network port` 467 *mapping* – the NIC port described as: 468 469 ====================== ================================================================================= 470 ``pci`` *string* – the local PCI address of the port. **Example**: ``0000:00:08.0`` 471 ``os_driver_for_dpdk`` | *string* – this port's device driver when using with DPDK 472 | When setting up the SUT, DTS will bind the network device to this driver 473 | for compatibility with DPDK. 474 475 **Examples**: ``vfio-pci``, ``mlx5_core`` 476 ``os_driver`` | *string* – this port's device driver when **not** using with DPDK 477 | When tearing down the tests on the SUT, DTS will bind the network device 478 | *back* to this driver. This driver is meant to be the one that the SUT would 479 | normally use for this device, or whichever driver it is preferred to leave the 480 | device bound to after testing. 481 | This also represents the driver that is used in conjunction with the traffic 482 | generator software. 483 484 **Examples**: ``i40e``, ``mlx5_core`` 485 ``peer_node`` *string* – the name of the peer node connected to this port. 486 ``peer_pci`` *string* – the PCI address of the peer node port. **Example**: ``000a:01:00.1`` 487 ====================== ================================================================================= 488 489_`Test suite` 490 *string* – name of the test suite to run. **Examples**: ``hello_world``, ``os_udp`` 491 492_`Test target` 493 *mapping* – selects specific test cases to run from a test suite. Mapping is described as follows: 494 495 ========= =============================================================================================== 496 ``suite`` See `Test suite`_ 497 ``cases`` (*optional*) *sequence* of *string* – list of the selected test cases in the test suite to run. 498 499 Unknown test cases will be silently ignored. 500 ========= =============================================================================================== 501 502 503Properties 504~~~~~~~~~~ 505 506The configuration requires listing all the execution environments and nodes 507involved in the testing. These can be defined with the following mappings: 508 509``executions`` 510 `sequence <https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range>`_ listing 511 the execution environments. Each entry is described as per the following 512 `mapping <https://docs.python.org/3/library/stdtypes.html#mapping-types-dict>`_: 513 514 +----------------------------+-------------------------------------------------------------------+ 515 | ``build_targets`` | *sequence* of `Build target`_ | 516 +----------------------------+-------------------------------------------------------------------+ 517 | ``perf`` | *boolean* – Enable performance testing. | 518 +----------------------------+-------------------------------------------------------------------+ 519 | ``func`` | *boolean* – Enable functional testing. | 520 +----------------------------+-------------------------------------------------------------------+ 521 | ``test_suites`` | *sequence* of **one of** `Test suite`_ **or** `Test target`_ | 522 +----------------------------+-------------------------------------------------------------------+ 523 | ``skip_smoke_tests`` | (*optional*) *boolean* – Allows you to skip smoke testing | 524 | | if ``true``. | 525 +----------------------------+-------------------------------------------------------------------+ 526 | ``system_under_test_node`` | System under test node specified with: | 527 | +---------------+---------------------------------------------------+ 528 | | ``node_name`` | See `Node name`_ | 529 | +---------------+---------------------------------------------------+ 530 | | ``vdevs`` | (*optional*) *sequence* of *string* | 531 | | | | 532 | | | List of virtual devices passed with the ``--vdev``| 533 | | | argument to DPDK. **Example**: ``crypto_openssl`` | 534 +----------------------------+---------------+---------------------------------------------------+ 535 | ``traffic_generator_node`` | Node name for the traffic generator node. | 536 +----------------------------+-------------------------------------------------------------------+ 537 538``nodes`` 539 `sequence <https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range>`_ listing 540 the nodes. Each entry is described as per the following 541 `mapping <https://docs.python.org/3/library/stdtypes.html#mapping-types-dict>`_: 542 543 +-----------------------+---------------------------------------------------------------------------------------+ 544 | ``name`` | See `Node name`_ | 545 +-----------------------+---------------------------------------------------------------------------------------+ 546 | ``hostname`` | *string* – The network hostname or IP address of this node. | 547 +-----------------------+---------------------------------------------------------------------------------------+ 548 | ``user`` | *string* – The SSH user credential to use to login to this node. | 549 +-----------------------+---------------------------------------------------------------------------------------+ 550 | ``password`` | (*optional*) *string* – The SSH password credential for this node. | 551 | | | 552 | | **NB**: Use only as last resort. SSH keys are **strongly** preferred. | 553 +-----------------------+---------------------------------------------------------------------------------------+ 554 | ``arch`` | The architecture of this node. See `ARCH`_ for supported values. | 555 +-----------------------+---------------------------------------------------------------------------------------+ 556 | ``os`` | The operating system of this node. See `OS`_ for supported values. | 557 +-----------------------+---------------------------------------------------------------------------------------+ 558 | ``lcores`` | | (*optional*, defaults to 1) *string* – Comma-separated list of logical | 559 | | | cores to use. An empty string means use all lcores. | 560 | | | 561 | | **Example**: ``1,2,3,4,5,18-22`` | 562 +-----------------------+---------------------------------------------------------------------------------------+ 563 | ``use_first_core`` | (*optional*, defaults to ``false``) *boolean* | 564 | | | 565 | | Indicates whether DPDK should use only the first physical core or not. | 566 +-----------------------+---------------------------------------------------------------------------------------+ 567 | ``memory_channels`` | (*optional*, defaults to 1) *integer* | 568 | | | 569 | | The number of the memory channels to use. | 570 +-----------------------+---------------------------------------------------------------------------------------+ 571 | ``hugepages`` | (*optional*) See `hugepages`_. If unset, hugepages won't be configured | 572 | | | 573 | | in favour of the system configuration. | 574 +-----------------------+---------------------------------------------------------------------------------------+ 575 | ``ports`` | | *sequence* of `Network port`_ – Describe ports that are **directly** paired with | 576 | | | other nodes used in conjunction with this one. Both ends of the links must be | 577 | | | described. If there any inconsistencies DTS won't run. | 578 | | | 579 | | **Example**: port 1 of node ``SUT1`` is connected to port 1 of node ``TG1`` etc. | 580 +-----------------------+---------------------------------------------------------------------------------------+ 581 | ``traffic_generator`` | (*optional*) Traffic generator, if any, setup on this node described as: | 582 | +----------+----------------------------------------------------------------------------+ 583 | | ``type`` | *string* – **Supported values**: *SCAPY* | 584 +-----------------------+----------+----------------------------------------------------------------------------+ 585 586 587.. _configuration_schema_example: 588 589Example 590~~~~~~~ 591 592The following example (which can be found in ``dts/conf.yaml``) sets up two nodes: 593 594* ``SUT1`` which is already setup with the DPDK build requirements and any other 595 required for execution; 596* ``TG1`` which already has Scapy installed in the system. 597 598And they both have two network ports which are physically connected to each other. 599 600.. note:: 601 This example assumes that you have setup SSH keys in both the system under test 602 and traffic generator nodes. 603 604.. literalinclude:: ../../../dts/conf.yaml 605 :language: yaml 606 :start-at: executions: 607