dts: use Pydantic in the configuration

This change brings in pydantic in place of warlock. Pydantic offers
a built-in model validation system in the classes, which allows for
a more resilient and si

dts: use Pydantic in the configuration

This change brings in pydantic in place of warlock. Pydantic offers
a built-in model validation system in the classes, which allows for
a more resilient and simpler code. As a consequence of this change:

- most validation is now built-in
- further validation is added to verify:
- cross referencing of node names and ports
- test suite and test cases names
- dictionaries representing the config schema are removed
- the config schema is no longer used and therefore dropped
- the TrafficGeneratorType enum has been changed from inheriting
StrEnum to the native str and Enum. This change was necessary to
enable the discriminator for object unions
- the structure of the classes has been slightly changed to perfectly
match the structure of the configuration files
- the test suite argument catches the ValidationError that
TestSuiteConfig can now raise
- the DPDK location has been wrapped under another configuration
mapping `dpdk_location`
- the DPDK locations are now structured and enforced by classes,
further simplifying the validation and handling thanks to
pattern matching

Bugzilla ID: 1508

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: enforce one DPDK build per test run

Given a pre-built DPDK repository can be supplied to DTS, there is no
need to define multiple build targets. To simplify the process this
change makes each t

dts: enforce one DPDK build per test run

Given a pre-built DPDK repository can be supplied to DTS, there is no
need to define multiple build targets. To simplify the process this
change makes each test run use only one DPDK build whether it's
pre-built or it needs to be built by DTS.

Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: rename build target to DPDK build

Since the DPDK may already be built, some more general name
is needed that includes both the DPDK location and the build
config (if we are going to build).

Si

dts: rename build target to DPDK build

Since the DPDK may already be built, some more general name
is needed that includes both the DPDK location and the build
config (if we are going to build).

Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: add topology capability

Add support for marking test cases as requiring a certain topology. The
default topology is a two link topology and the other supported
topologies are one link and no li

dts: add topology capability

Add support for marking test cases as requiring a certain topology. The
default topology is a two link topology and the other supported
topologies are one link and no link topologies.

The TestProtocol of test suites and cases is extended with the topology
type each test suite or case requires. Each test case starts out as
requiring a two link topology and can be marked as requiring as
topology directly (by decorating the test case) or through its test
suite. If a test suite is decorated as requiring a certain topology, all
its test cases are marked as such. If both test suite and a test case
are decorated as requiring a topology, the test case cannot require a
more complex topology than the whole suite (but it can require a less
complex one). If a test suite is not decorated, this has no effect on
required test case topology.

Since the default topology is defined as a reference to one of the
actual topologies, the NoAliasEnum from the aenum package is utilized,
which removes the aliasing of Enums so that TopologyType.two_links and
TopologyType.default are distinct. This is needed to distinguish between
a user passed value and the default value being used (which is used when
a test suite is or isn't decorated).

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: add test case decorators

Add decorators for functional and performance test cases. These
decorators add attributes to the decorated test cases.

With the addition of decorators, we change the t

dts: add test case decorators

Add decorators for functional and performance test cases. These
decorators add attributes to the decorated test cases.

With the addition of decorators, we change the test case discovery
mechanism from looking at test case names according to a regex to simply
checking an attribute of the function added with one of the decorators.

The decorators allow us to add further variables to test cases.

Also move the test case filtering to TestSuite while changing the
mechanism to separate the logic in a more sensible manner.

Bugzilla ID: 1460

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: add context manager for interactive shells

Interactive shells are managed in a way currently where they are closed
and cleaned up at the time of garbage collection. Due to there being no
guaran

dts: add context manager for interactive shells

Interactive shells are managed in a way currently where they are closed
and cleaned up at the time of garbage collection. Due to there being no
guarantee of when this garbage collection happens in Python, there is no
way to consistently know when an application will be closed without
manually closing the application yourself when you are done with it.
This doesn't cause a problem in cases where you can start another
instance of the same application multiple times on a server, but this
isn't the case for primary applications in DPDK. The introduction of
primary applications, such as testpmd, adds a need for knowing previous
instances of the application have been stopped and cleaned up before
starting a new one, which the garbage collector does not provide.

To solve this problem, a new class is added which acts as a base class
for interactive shells that enforces that instances of the
application be managed using a context manager. Using a context manager
guarantees that once you leave the scope of the block where the
application is being used for any reason, the application will be closed
immediately. This avoids the possibility of the shell not being closed
due to an exception being raised or user error. The interactive shell
class then becomes shells that can be started/stopped manually or at the
time of garbage collection rather than through a context manager.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: rework interactive shells

The way nodes and interactive shells interact makes it difficult to
develop for static type checking and hinting. The current system relies
on a top-down approach, att

dts: rework interactive shells

The way nodes and interactive shells interact makes it difficult to
develop for static type checking and hinting. The current system relies
on a top-down approach, attempting to give a generic interface to the
test developer, hiding the interaction of concrete shell classes as much
as possible. When working with strong typing this approach is not ideal,
as Python's implementation of generics is still rudimentary.

This rework reverses the tests interaction to a bottom-up approach,
allowing the test developer to call concrete shell classes directly,
and let them ingest nodes independently. While also re-enforcing type
checking and making the code easier to read.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

show more ...

dts: remove module-wide imports

Remove the imports in the testbed_model and remote_session modules init
file, to avoid the initialisation of unneeded modules, thus removing or
limiting the risk of c

dts: remove module-wide imports

Remove the imports in the testbed_model and remote_session modules init
file, to avoid the initialisation of unneeded modules, thus removing or
limiting the risk of circular dependencies.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>

show more ...

dts: filter test suites in executions

We're currently filtering which test cases to run after some setup
steps, such as DPDK build, have already been taken. This prohibits us to
mark the test suites

dts: filter test suites in executions

We're currently filtering which test cases to run after some setup
steps, such as DPDK build, have already been taken. This prohibits us to
mark the test suites and cases that were supposed to be run as blocked
when an earlier setup fails, as that information is not available at
that time.

To remedy this, move the filtering to the beginning of each execution.
This is the first action taken in each execution and if we can't filter
the test cases, such as due to invalid inputs, we abort the whole
execution. No test suites nor cases will be marked as blocked as we
don't know which were supposed to be run.

On top of that, the filtering takes place in the TestSuite class, which
should only concern itself with test suite and test case logic, not the
processing behind the scenes. The logic has been moved to DTSRunner
which should do all the processing needed to run test suites.

The filtering itself introduces a few changes/assumptions which are more
sensible than before:
1. Assumption: There is just one TestSuite child class in each test
suite module. This was an implicit assumption before as we couldn't
specify the TestSuite classes in the test run configuration, just the
modules. The name of the TestSuite child class starts with "Test" and
then corresponds to the name of the module with CamelCase naming.
2. Unknown test cases specified both in the test run configuration and
the environment variable/command line argument are no longer silently
ignored. This is a quality of life improvement for users, as they
could easily be not aware of the silent ignoration.

Also, a change in the code results in pycodestyle warning and error:
[E] E203 whitespace before ':'
[W] W503 line break before binary operator

These two are not PEP8 compliant, so they're disabled.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

show more ...

dts: fix smoke tests driver regex

Add hyphen to the regex, which is needed for drivers such as vfio-pci.

Fixes: 88489c0501af ("dts: add smoke tests")
Cc: stable@dpdk.org

Signed-off-by: Juraj Linke

dts: fix smoke tests driver regex

Add hyphen to the regex, which is needed for drivers such as vfio-pci.

Fixes: 88489c0501af ("dts: add smoke tests")
Cc: stable@dpdk.org

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Tested-by: Nicholas Pratte <npratte@iol.unh.edu>

show more ...

dts: update docstrings

Format according to the Google format and PEP257, with slight
deviations.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

dts: reformat to 100 line length

Reformat to 100 from the previous 88 to unify with C recommendations.

The C recommendation is the maximum with the ideal being 80.
The Python tools are not suitable

dts: reformat to 100 line length

Reformat to 100 from the previous 88 to unify with C recommendations.

The C recommendation is the maximum with the ideal being 80.
The Python tools are not suitable for this flexibility.

We require all patches with DTS code to be validated with the
devtools/dts-check-format.sh script, part of which is the black
formatting tool. We've set up black to format all of the codebase and
the reformat is needed so that future submitters are not affected.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Acked-by: Jeremy Spewock <jspewock@iol.unh.edu>

show more ...

dts: bind to DPDK driver before running tests

Modifies the current process so that we bind to os_driver_for_dpdk from
the configuration file before running test suites and bind back to the
os_driver

dts: bind to DPDK driver before running tests

Modifies the current process so that we bind to os_driver_for_dpdk from
the configuration file before running test suites and bind back to the
os_driver afterwards. This allows test suites to assume that the ports
are bound to a DPDK supported driver or bind to either driver as needed.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
Acked-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: add smoke tests

Adds a new test suite for running smoke tests that verify general
configuration aspects of the system under test. If any of these tests
fail, the DTS execution terminates as par

dts: add smoke tests

Adds a new test suite for running smoke tests that verify general
configuration aspects of the system under test. If any of these tests
fail, the DTS execution terminates as part of a "fail-fast" model.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

show more ...