dts: remove redundant test suite

The test suite served as a demonstration of the Scapy traffic generator
implementation. Now that we have a test suite that uses DPDK code (via
testpmd), there is no

dts: remove redundant test suite

The test suite served as a demonstration of the Scapy traffic generator
implementation. Now that we have a test suite that uses DPDK code (via
testpmd), there is no reason to keep the test suite, as there's no
expectation it'll be actually used in any setup.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: allow to get multiple expected packets

At the moment there is only one facility that allows to prepare a packet
as it is expected to be received on the traffic generator end. Extend
this to all

dts: allow to get multiple expected packets

At the moment there is only one facility that allows to prepare a packet
as it is expected to be received on the traffic generator end. Extend
this to allow to prepare multiple packets at the same time.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>
Tested-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: fix adjusting L2/L3 addresses

The _adjust_addresses function has been updated to both modify the
original packets and return new updated copies of them. Having a double
behavior even if documen

dts: fix adjusting L2/L3 addresses

The _adjust_addresses function has been updated to both modify the
original packets and return new updated copies of them. Having a double
behavior even if documented is not intuitive and can lead to bugs.

This changes the behavior to solely act on copies, leaving the original
packet untouched.

Fixes: 1a1825962777 ("dts: rework packet addressing")

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>
Tested-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: add test suite specification and discovery

Currently there is a lack of a definition which identifies all the test
suites available to test. This change intends to simplify the process to
disco

dts: add test suite specification and discovery

Currently there is a lack of a definition which identifies all the test
suites available to test. This change intends to simplify the process to
discover all the test suites and identify them.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: enforce one DPDK build per test run

Given a pre-built DPDK repository can be supplied to DTS, there is no
need to define multiple build targets. To simplify the process this
change makes each t

dts: enforce one DPDK build per test run

Given a pre-built DPDK repository can be supplied to DTS, there is no
need to define multiple build targets. To simplify the process this
change makes each test run use only one DPDK build whether it's
pre-built or it needs to be built by DTS.

Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: rename build target to DPDK build

Since the DPDK may already be built, some more general name
is needed that includes both the DPDK location and the build
config (if we are going to build).

Si

dts: rename build target to DPDK build

Since the DPDK may already be built, some more general name
is needed that includes both the DPDK location and the build
config (if we are going to build).

Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: add topology capability

Add support for marking test cases as requiring a certain topology. The
default topology is a two link topology and the other supported
topologies are one link and no li

dts: add topology capability

Add support for marking test cases as requiring a certain topology. The
default topology is a two link topology and the other supported
topologies are one link and no link topologies.

The TestProtocol of test suites and cases is extended with the topology
type each test suite or case requires. Each test case starts out as
requiring a two link topology and can be marked as requiring as
topology directly (by decorating the test case) or through its test
suite. If a test suite is decorated as requiring a certain topology, all
its test cases are marked as such. If both test suite and a test case
are decorated as requiring a topology, the test case cannot require a
more complex topology than the whole suite (but it can require a less
complex one). If a test suite is not decorated, this has no effect on
required test case topology.

Since the default topology is defined as a reference to one of the
actual topologies, the NoAliasEnum from the aenum package is utilized,
which removes the aliasing of Enums so that TopologyType.two_links and
TopologyType.default are distinct. This is needed to distinguish between
a user passed value and the default value being used (which is used when
a test suite is or isn't decorated).

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: add basic capability support

A test case or suite may require certain capabilities to be present in
the tested environment. Add the basic infrastructure for checking the
support status of capab

dts: add basic capability support

A test case or suite may require certain capabilities to be present in
the tested environment. Add the basic infrastructure for checking the
support status of capabilities:
* The Capability ABC defining the common capability API
* Extension of the TestProtocol with required capabilities (each test
suite or case stores the capabilities it requires)
* Integration with the runner which calls the new APIs to get which
capabilities are supported.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: support simpler topologies

We currently assume there are two links between the SUT and TG nodes,
but that's too strict, even for some of the already existing test cases.
Add support for topolog

dts: support simpler topologies

We currently assume there are two links between the SUT and TG nodes,
but that's too strict, even for some of the already existing test cases.
Add support for topologies with less than two links.

For topologies with no links, dummy ports are used. The expectation is
that test suites or cases that don't require any links won't be using
methods that use ports. Any test suites or cases requiring links will be
skipped in topologies with no links, but this feature is not implemented
in this commit.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: add mechanism to skip test cases or suites

If a test case is not relevant to the testing environment (such as when
a NIC doesn't support a tested feature), the framework should skip it.
The mec

dts: add mechanism to skip test cases or suites

If a test case is not relevant to the testing environment (such as when
a NIC doesn't support a tested feature), the framework should skip it.
The mechanism is a skeleton without actual logic that would set a test
case or suite to be skipped.

The mechanism uses a protocol to extend test suites and test cases with
additional attributes that track whether the test case or suite should
be skipped the reason for skipping it.

Also update the results module with the new SKIP result.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: add test case decorators

Add decorators for functional and performance test cases. These
decorators add attributes to the decorated test cases.

With the addition of decorators, we change the t

dts: add test case decorators

Add decorators for functional and performance test cases. These
decorators add attributes to the decorated test cases.

With the addition of decorators, we change the test case discovery
mechanism from looking at test case names according to a regex to simply
checking an attribute of the function added with one of the decorators.

The decorators allow us to add further variables to test cases.

Also move the test case filtering to TestSuite while changing the
mechanism to separate the logic in a more sensible manner.

Bugzilla ID: 1460

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: send packets without capture

Currently the only methods provided in the test suite class for sending
packets capture the resulting received traffic after sending. There is,
in some cases, a nee

dts: send packets without capture

Currently the only methods provided in the test suite class for sending
packets capture the resulting received traffic after sending. There is,
in some cases, a need to send multiple packets at once while not really
needing to capture any of said received traffic. It is favorable to
avoid capturing received traffic when you don't need it since not all
traffic generators will necessarily be capturing traffic generators.
The method to fulfill this need exists in the traffic generator
already, but this patch exposes the method to test suites.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: rework packet addressing

This patch updates the _adjust_addresses method of test suites so
that addresses of packets are only modified if the developer did not
configure them beforehand. This a

dts: rework packet addressing

This patch updates the _adjust_addresses method of test suites so
that addresses of packets are only modified if the developer did not
configure them beforehand. This allows for developers to have more
control over the content of their packets when sending them through the
framework.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: add ability to send/receive multiple packets

The framework allows only to send one packet at once via Scapy. This
change adds the ability to send multiple packets, and also introduces a
new fas

dts: add ability to send/receive multiple packets

The framework allows only to send one packet at once via Scapy. This
change adds the ability to send multiple packets, and also introduces a
new fast way to verify if we received several expected packets.

Moreover, it reduces code duplication by keeping a single packet sending
method only at the test suite level.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Alex Chapman <alex.chapman@arm.com>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

show more ...

dts: remove module-wide imports

Remove the imports in the testbed_model and remote_session modules init
file, to avoid the initialisation of unneeded modules, thus removing or
limiting the risk of c

dts: remove module-wide imports

Remove the imports in the testbed_model and remote_session modules init
file, to avoid the initialisation of unneeded modules, thus removing or
limiting the risk of circular dependencies.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>

show more ...

dts: unify class inheritance from object

There are two ways we specify that a class inherits from object -
implicitly and explicitly. There's no need to explicitly specify that a
class inherits from

dts: unify class inheritance from object

There are two ways we specify that a class inherits from object -
implicitly and explicitly. There's no need to explicitly specify that a
class inherits from object and is in fact mostly a remnant from Python2.
Leaving it implicit is the standard in Python3 and offers a small bonus
in cases where something would assign something else to the builtin
object variable.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

show more ...

dts: update mypy static checker

Update the mypy static checker to the latest version and fix all the
reported errors.

Bump up supported Poetry shell version.

Bugzilla ID: 1433

Signed-off-by: Luca

dts: update mypy static checker

Update the mypy static checker to the latest version and fix all the
reported errors.

Bump up supported Poetry shell version.

Bugzilla ID: 1433

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
Tested-by: Nicholas Pratte <npratte@iol.unh.edu>

show more ...

dts: add packet filtering to Scapy sniffer

Added the options to filter out LLDP and ARP packets when
sniffing for packets with scapy. This was done using BPF filters to
ensure that the noise these p

dts: add packet filtering to Scapy sniffer

Added the options to filter out LLDP and ARP packets when
sniffing for packets with scapy. This was done using BPF filters to
ensure that the noise these packets provide does not interfere with test
cases.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Tested-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: improve test suite and case filtering

The two places where we specify which test suite and test cases to run
are complimentary and not that intuitive to use. A unified way provides
a better use

dts: improve test suite and case filtering

The two places where we specify which test suite and test cases to run
are complimentary and not that intuitive to use. A unified way provides
a better user experience.

The syntax in test run configuration file has not changed, but the
environment variable and the command line arguments was changed to match
the config file syntax. This required changes in the settings module
which greatly simplified the parsing of the environment variables while
retaining the same functionality.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

show more ...

dts: refactor logging configuration

Remove unused parts of the code and add useful features:
1. Add DTS execution stages such as execution and test suite to better
identify where in the DTS lifec

dts: refactor logging configuration

Remove unused parts of the code and add useful features:
1. Add DTS execution stages such as execution and test suite to better
identify where in the DTS lifecycle we are when investigating logs,
2. Logging to separate files in specific stages, which is mainly useful
for having test suite logs in additional separate files.
3. Remove the dependence on the settings module which enhances the
usefulness of the logger module, as it can now be imported in more
modules.

The execution stages and the files to log to are the same for all DTS
loggers. To achieve this, we have one DTS root logger which should be
used for handling stage switching and all other loggers are children of
this DTS root logger. The DTS root logger is the one where we change the
behavior of all loggers (the stage and which files to log to) and the
child loggers just log messages under a different name.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

show more ...

dts: filter test suites in executions

We're currently filtering which test cases to run after some setup
steps, such as DPDK build, have already been taken. This prohibits us to
mark the test suites

dts: filter test suites in executions

We're currently filtering which test cases to run after some setup
steps, such as DPDK build, have already been taken. This prohibits us to
mark the test suites and cases that were supposed to be run as blocked
when an earlier setup fails, as that information is not available at
that time.

To remedy this, move the filtering to the beginning of each execution.
This is the first action taken in each execution and if we can't filter
the test cases, such as due to invalid inputs, we abort the whole
execution. No test suites nor cases will be marked as blocked as we
don't know which were supposed to be run.

On top of that, the filtering takes place in the TestSuite class, which
should only concern itself with test suite and test case logic, not the
processing behind the scenes. The logic has been moved to DTSRunner
which should do all the processing needed to run test suites.

The filtering itself introduces a few changes/assumptions which are more
sensible than before:
1. Assumption: There is just one TestSuite child class in each test
suite module. This was an implicit assumption before as we couldn't
specify the TestSuite classes in the test run configuration, just the
modules. The name of the TestSuite child class starts with "Test" and
then corresponds to the name of the module with CamelCase naming.
2. Unknown test cases specified both in the test run configuration and
the environment variable/command line argument are no longer silently
ignored. This is a quality of life improvement for users, as they
could easily be not aware of the silent ignoration.

Also, a change in the code results in pycodestyle warning and error:
[E] E203 whitespace before ':'
[W] W503 line break before binary operator

These two are not PEP8 compliant, so they're disabled.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

show more ...

dts: move test suite execution to runner class

Move the code responsible for running the test suite from the
TestSuite class to the DTSRunner class. This restructuring decision
was made to consolida

dts: move test suite execution to runner class

Move the code responsible for running the test suite from the
TestSuite class to the DTSRunner class. This restructuring decision
was made to consolidate and unify the related logic into a single unit.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

show more ...

dts: update docstrings

Format according to the Google format and PEP257, with slight
deviations.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

dts: adjust code for doc generation

The standard Python tool for generating API documentation, Sphinx,
imports modules one-by-one when generating the documentation. This
requires code changes:
* pro

dts: adjust code for doc generation

The standard Python tool for generating API documentation, Sphinx,
imports modules one-by-one when generating the documentation. This
requires code changes:
* properly guarding argument parsing in the if __name__ == '__main__'
block,
* the logger used by DTS runner underwent the same treatment so that it
doesn't create log files outside of a DTS run,
* however, DTS uses the arguments to construct an object holding global
variables. The defaults for the global variables needed to be moved
from argument parsing elsewhere,
* importing the remote_session module from framework resulted in
circular imports because of one module trying to import another
module. This is fixed by reorganizing the code,
* some code reorganization was done because the resulting structure
makes more sense, improving documentation clarity.

The are some other changes which are documentation related:
* added missing type annotation so they appear in the generated docs,
* reordered arguments in some methods,
* removed superfluous arguments and attributes,
* change private functions/methods/attributes to private and vice-versa.

The above all appear in the generated documentation and the with them,
the documentation is improved.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

show more ...

dts: reformat to 100 line length

Reformat to 100 from the previous 88 to unify with C recommendations.

The C recommendation is the maximum with the ideal being 80.
The Python tools are not suitable

dts: reformat to 100 line length

Reformat to 100 from the previous 88 to unify with C recommendations.

The C recommendation is the maximum with the ideal being 80.
The Python tools are not suitable for this flexibility.

We require all patches with DTS code to be validated with the
devtools/dts-check-format.sh script, part of which is the black
formatting tool. We've set up black to format all of the codebase and
the reformat is needed so that future submitters are not affected.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Acked-by: Jeremy Spewock <jspewock@iol.unh.edu>

show more ...