dts: remove redundant test suite

The test suite served as a demonstration of the Scapy traffic generator
implementation. Now that we have a test suite that uses DPDK code (via
testpmd), there is no

dts: remove redundant test suite

The test suite served as a demonstration of the Scapy traffic generator
implementation. Now that we have a test suite that uses DPDK code (via
testpmd), there is no reason to keep the test suite, as there's no
expectation it'll be actually used in any setup.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: use Pydantic in the configuration

This change brings in pydantic in place of warlock. Pydantic offers
a built-in model validation system in the classes, which allows for
a more resilient and si

dts: use Pydantic in the configuration

This change brings in pydantic in place of warlock. Pydantic offers
a built-in model validation system in the classes, which allows for
a more resilient and simpler code. As a consequence of this change:

- most validation is now built-in
- further validation is added to verify:
- cross referencing of node names and ports
- test suite and test cases names
- dictionaries representing the config schema are removed
- the config schema is no longer used and therefore dropped
- the TrafficGeneratorType enum has been changed from inheriting
StrEnum to the native str and Enum. This change was necessary to
enable the discriminator for object unions
- the structure of the classes has been slightly changed to perfectly
match the structure of the configuration files
- the test suite argument catches the ValidationError that
TestSuiteConfig can now raise
- the DPDK location has been wrapped under another configuration
mapping `dpdk_location`
- the DPDK locations are now structured and enforced by classes,
further simplifying the validation and handling thanks to
pattern matching

Bugzilla ID: 1508

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: refactor build and node info classes

The DPDKBuildInfo and NodeInfo classes, representing information
gathered in runtime, were erroneously placed in the configuration
package. This moves them

dts: refactor build and node info classes

The DPDKBuildInfo and NodeInfo classes, representing information
gathered in runtime, were erroneously placed in the configuration
package. This moves them in more appropriate modules.

NodeInfo, specifically, is moved to os_session instead of node mostly
as a consequence of circular dependencies. And given os_session is the
top-most module to reference it, it appears to be the most suitable
place outside of node.

Finally NodeInfo, is better renamed to OSSessionInfo as it represents
the information on the target OS session.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>
Reviewed-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: support externally compiled DPDK

Enable the user to use either a DPDK source tree directory or a
tarball, with and without a pre-built build directory. These can be
stored on either SUT node or

dts: support externally compiled DPDK

Enable the user to use either a DPDK source tree directory or a
tarball, with and without a pre-built build directory. These can be
stored on either SUT node or the DTS host. The DPDK build setup or the
pre-built binaries can be specified through the configuration file,
the command line arguments or environment variables.

Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: enable copying directories to and from nodes

Currently there is no support to transfer whole directories between the
DTS host and the nodes. This change adds this new feature.

Signed-off-by: T

dts: enable copying directories to and from nodes

Currently there is no support to transfer whole directories between the
DTS host and the nodes. This change adds this new feature.

Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: change remote and local paths objects

The OSSession (and its subclasses) should accept PurePaths
for remote paths to translate from OS-unaware (PurePath)
to OS-aware (Path) only on the remote s

dts: change remote and local paths objects

The OSSession (and its subclasses) should accept PurePaths
for remote paths to translate from OS-unaware (PurePath)
to OS-aware (Path) only on the remote side. For local paths,
they should accept Paths, as Python is OS-aware locally.

Signed-off-by: Tomáš Ďurovec <tomas.durovec@pantheon.tech>
Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: rework interactive shells

The way nodes and interactive shells interact makes it difficult to
develop for static type checking and hinting. The current system relies
on a top-down approach, att

dts: rework interactive shells

The way nodes and interactive shells interact makes it difficult to
develop for static type checking and hinting. The current system relies
on a top-down approach, attempting to give a generic interface to the
test developer, hiding the interaction of concrete shell classes as much
as possible. When working with strong typing this approach is not ideal,
as Python's implementation of generics is still rudimentary.

This rework reverses the tests interaction to a bottom-up approach,
allowing the test developer to call concrete shell classes directly,
and let them ingest nodes independently. While also re-enforcing type
checking and making the code easier to read.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

show more ...

dts: remove module-wide imports

Remove the imports in the testbed_model and remote_session modules init
file, to avoid the initialisation of unneeded modules, thus removing or
limiting the risk of c

dts: remove module-wide imports

Remove the imports in the testbed_model and remote_session modules init
file, to avoid the initialisation of unneeded modules, thus removing or
limiting the risk of circular dependencies.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>

show more ...

dts: use params module for interactive shells

Make it so that interactive shells accept an implementation of `Params`
for app arguments. Convert EalParameters to use `Params` instead.

String comman

dts: use params module for interactive shells

Make it so that interactive shells accept an implementation of `Params`
for app arguments. Convert EalParameters to use `Params` instead.

String command line parameters can still be supplied by using the
`Params.from_str()` method.

Signed-off-by: Luca Vizzarro <luca.vizzarro@arm.com>
Reviewed-by: Paul Szczepanek <paul.szczepanek@arm.com>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Nicholas Pratte <npratte@iol.unh.edu>

show more ...

dts: clean up close in remote session

The close method was split into two methods, one with common code and
one that's subclass specific. There wasn't any common code so the split
doesn't make sense

dts: clean up close in remote session

The close method was split into two methods, one with common code and
one that's subclass specific. There wasn't any common code so the split
doesn't make sense. And if we ever need such a split, we can use super()
in the future.
There was also an unused argument, force and the order of methods was
cleaned up a little (close is usually the last thing we call in the
lifecycle of a session object, so it was moved to the last place among
public methods).

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: rename hugepage config parameter

The term 'amount' is used for uncountable nouns. Since total hugepages
is a discrete value (i.e. countable), the declaration of the 'amount'
key value pair shou

dts: rename hugepage config parameter

The term 'amount' is used for uncountable nouns. Since total hugepages
is a discrete value (i.e. countable), the declaration of the 'amount'
key value pair should be changes to a different term in both the config
and the rest of the code.

Bugzilla ID: 1370

Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: use exclusively 2MB hugepage in config

The previous implementation configures and allocates hugepage sizes
based on a system default. This can lead to two problems: overallocation of
hugepages

dts: use exclusively 2MB hugepage in config

The previous implementation configures and allocates hugepage sizes
based on a system default. This can lead to two problems: overallocation of
hugepages (which may crash the remote host), and configuration of hugepage
sizes that are not recommended during runtime. This new implementation
allows only 2MB hugepage allocation during runtime; any other unique
hugepage size must be configured by the end-user for initializing DTS.

If the amount of 2MB hugepages requested exceeds the amount of 2MB
hugepages already configured on the system, then the system will remount
hugepages to cover the difference. If the amount of hugepages requested is
either less than or equal to the amount already configured on the system,
then nothing is done.

Bugzilla ID: 1370

Signed-off-by: Nicholas Pratte <npratte@iol.unh.edu>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>

show more ...

dts: allow configuring MTU

Adds methods in both os_session and linux session to allow for setting
MTU of port interfaces so that suites that require the sending and
receiving of packets of a specifi

dts: allow configuring MTU

Adds methods in both os_session and linux session to allow for setting
MTU of port interfaces so that suites that require the sending and
receiving of packets of a specific size, or the rejection of packets
over a certain size, can configure this maximum as needed.

Signed-off-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Tested-by: Patrick Robb <probb@iol.unh.edu>

show more ...

dts: refactor logging configuration

Remove unused parts of the code and add useful features:
1. Add DTS execution stages such as execution and test suite to better
identify where in the DTS lifec

dts: refactor logging configuration

Remove unused parts of the code and add useful features:
1. Add DTS execution stages such as execution and test suite to better
identify where in the DTS lifecycle we are when investigating logs,
2. Logging to separate files in specific stages, which is mainly useful
for having test suite logs in additional separate files.
3. Remove the dependence on the settings module which enhances the
usefulness of the logger module, as it can now be imported in more
modules.

The execution stages and the files to log to are the same for all DTS
loggers. To achieve this, we have one DTS root logger which should be
used for handling stage switching and all other loggers are children of
this DTS root logger. The DTS root logger is the one where we change the
behavior of all loggers (the stage and which files to log to) and the
child loggers just log messages under a different name.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>

show more ...

dts: update docstrings

Format according to the Google format and PEP257, with slight
deviations.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

dts: adjust code for doc generation

The standard Python tool for generating API documentation, Sphinx,
imports modules one-by-one when generating the documentation. This
requires code changes:
* pro

dts: adjust code for doc generation

The standard Python tool for generating API documentation, Sphinx,
imports modules one-by-one when generating the documentation. This
requires code changes:
* properly guarding argument parsing in the if __name__ == '__main__'
block,
* the logger used by DTS runner underwent the same treatment so that it
doesn't create log files outside of a DTS run,
* however, DTS uses the arguments to construct an object holding global
variables. The defaults for the global variables needed to be moved
from argument parsing elsewhere,
* importing the remote_session module from framework resulted in
circular imports because of one module trying to import another
module. This is fixed by reorganizing the code,
* some code reorganization was done because the resulting structure
makes more sense, improving documentation clarity.

The are some other changes which are documentation related:
* added missing type annotation so they appear in the generated docs,
* reordered arguments in some methods,
* removed superfluous arguments and attributes,
* change private functions/methods/attributes to private and vice-versa.

The above all appear in the generated documentation and the with them,
the documentation is improved.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

show more ...