doc: add lcore variables guide

Add lcore variables programmer's guide. This guide gives both an
overview of the API, its implementation, and alternatives to the use
of lcore variables for maintainin

doc: add lcore variables guide

Add lcore variables programmer's guide. This guide gives both an
overview of the API, its implementation, and alternatives to the use
of lcore variables for maintaining per-lcore id data.

It has pictures, too.

Signed-off-by: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
Reviewed-by: Luka Jankovic <luka.jankovic@ericsson.com>

show more ...

doc: reorganize prog guide

Create categories in the index of the programmer's guide,
sort chapters and rewrite some titles for consistency.

Subdirectories are created for ethdev and eventdev
for gr

doc: reorganize prog guide

Create categories in the index of the programmer's guide,
sort chapters and rewrite some titles for consistency.

Subdirectories are created for ethdev and eventdev
for grouping the files together.

Useless link anchors at the beginning of files are removed,
the corresponding :ref: are replaced with :doc: links.

Signed-off-by: David Young <dave@youngcopy.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

show more ...

ptr_compress: add pointer compression library

Add a new utility header for compressing pointers. The provided
functions can store pointers as 32-bit or 16-bit offsets.

The compression takes advanta

ptr_compress: add pointer compression library

Add a new utility header for compressing pointers. The provided
functions can store pointers as 32-bit or 16-bit offsets.

The compression takes advantage of the fact that pointers are
usually located in a limited memory region (like a mempool).
We can compress them by converting them to offsets from a base
memory address. Offsets can be stored in fewer bytes (dictated
by the memory region size and alignment of the pointer).
For example: an 8 byte aligned pointer which is part of a 32GB
memory pool can be stored in 4 bytes.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Signed-off-by: Paul Szczepanek <paul.szczepanek@arm.com>
Signed-off-by: Kamalakshitha Aligeri <kamalakshitha.aligeri@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Nathan Brown <nathan.brown@arm.com>
Reviewed-by: Jack Bond-Preston <jack.bond-preston@foss.arm.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>

show more ...

argparse: introduce argparse library

Introduce argparse library (which was inspired by the thread [1]).
This commit provides public API and doc.

[1] https://patchwork.dpdk.org/project/dpdk/patch/20

argparse: introduce argparse library

Introduce argparse library (which was inspired by the thread [1]).
This commit provides public API and doc.

[1] https://patchwork.dpdk.org/project/dpdk/patch/20231105054539.22303-2-fengchengwen@huawei.com/

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>

show more ...

doc: add cmdline programmer guide

The cmdline library was not documented in our programmers guide, so add
a new chapter on it. This chapter covers step-by-step how to use the
library, rather than fo

doc: add cmdline programmer guide

The cmdline library was not documented in our programmers guide, so add
a new chapter on it. This chapter covers step-by-step how to use the
library, rather than focusing on the library internals. This complements
the existing cmdline example app document, providing more details on the
process of using the library.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: David Marchand <david.marchand@redhat.com>

show more ...

lib: introduce dispatcher library

The purpose of the dispatcher library is to help reduce coupling in an
Eventdev-based DPDK application.

In addition, the dispatcher also provides a convenient and

lib: introduce dispatcher library

The purpose of the dispatcher library is to help reduce coupling in an
Eventdev-based DPDK application.

In addition, the dispatcher also provides a convenient and flexible
way for the application to use service cores for application-level
processing.

Signed-off-by: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
Tested-by: Peter Nilsson <peter.j.nilsson@ericsson.com>
Reviewed-by: Heng Wang <heng.wang@ericsson.com>

show more ...

eventdev/dma: introduce DMA adapter

Introduce event dma adapter interface to transfer packets between
dma device and event device.

Signed-off-by: Amit Prakash Shukla <amitprakashs@marvell.com>
Acke

eventdev/dma: introduce DMA adapter

Introduce event dma adapter interface to transfer packets between
dma device and event device.

Signed-off-by: Amit Prakash Shukla <amitprakashs@marvell.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>

show more ...

log: separate logging functions out of EAL

Move the logging capability to a separate library, free from EAL. Rename
files as appropriate, and use meson.build to select the correct file to
be built f

log: separate logging functions out of EAL

Move the logging capability to a separate library, free from EAL. Rename
files as appropriate, and use meson.build to select the correct file to
be built for each operating system, rather than having a subdir per-os.
Add new documentation section in programmers guide to cover logging in
more detail.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

show more ...

remove KNI

The KNI driver had design flaws such as calling userspace with kernel
mutex held that made it prone to deadlock. The design also introduced
security risks because the kernel driver truste

remove KNI

The KNI driver had design flaws such as calling userspace with kernel
mutex held that made it prone to deadlock. The design also introduced
security risks because the kernel driver trusted the userspace
(DPDK) kni interface. The kernel driver was never reviewed by
the upstream kernel community and would never have been accepted.

And since the Linux kernel API is not stable, it was a continual
source of maintenance issues especially with distribution kernels.

There are better ways to inject packets into the kernel such as
virtio_user, tap and XDP drivers. All of these do not need out of
tree kernel drivers.

The deprecation was announced in 22.11 release (see links for the
the techboard discussions); and users were directed to alternatives
there.

Link: https://mails.dpdk.org/archives/dev/2021-January/197077.html
Link: https://mails.dpdk.org/archives/dev/2022-June/243596.html

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

show more ...

remove flow classify library

The flow_classify library was marked for removal last year
because there was no maintainer and the functionality is
limited.

Signed-off-by: Stephen Hemminger <stephen@n

remove flow classify library

The flow_classify library was marked for removal last year
because there was no maintainer and the functionality is
limited.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

show more ...

doc: add PDCP library guide

Add guide for PDCP library.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Volodymyr Fialko <vfia

doc: add PDCP library guide

Add guide for PDCP library.

Signed-off-by: Anoob Joseph <anoobj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Volodymyr Fialko <vfialko@marvell.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>

show more ...

mldev: introduce machine learning device API

Add mldev API specification to standardize and use the machine learning
device and inference operations in vendor neutral way.

Following operations are

mldev: introduce machine learning device API

Add mldev API specification to standardize and use the machine learning
device and inference operations in vendor neutral way.

Following operations are abstracted through APIs

- ML device capability probe
- ML device configuration
- ML device queue pair configuration
- ML device state management
- ML device stat/xstat operations
- ML model load/unload/start/stop operations
- ML model information probe
- ML IO operations to find size for input and output buffers
- ML quantize and dequantize operations
- ML ops pool creation and free operations
- ML device enqueue/dequeue fastpath interference operations

Also added programming guide.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Srikanth Yalavarthi <syalavarthi@marvell.com>

show more ...

doc: improve ordering and remove old titles in prog guide

Move the "source_org" page to after overview, where it fits
better to explain the source-code layout of DPDK, before getting
into details of

doc: improve ordering and remove old titles in prog guide

Move the "source_org" page to after overview, where it fits
better to explain the source-code layout of DPDK, before getting
into details of specific libraries such as EAL.

Also removes the older titles from the 3 documents which still had them.

Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

show more ...

doc: add RIB and FIB programmer guides

Currently, programmer's guide for the RIB and FIB libraries are missing.
This commit adds them.

Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.co

doc: add RIB and FIB programmer guides

Currently, programmer's guide for the RIB and FIB libraries are missing.
This commit adds them.

Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
Reviewed-by: Conor Walsh <conor.walsh@intel.com>

show more ...

gpudev: introduce GPU device class library

In heterogeneous computing system, processing is not only in the CPU.
Some tasks can be delegated to devices working in parallel.

The new library gpudev i

gpudev: introduce GPU device class library

In heterogeneous computing system, processing is not only in the CPU.
Some tasks can be delegated to devices working in parallel.

The new library gpudev is for dealing with GPGPU computing devices
from a DPDK application running on the CPU.

The infrastructure is prepared to welcome drivers in drivers/gpu/.

Signed-off-by: Elena Agostini <eagostini@nvidia.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

show more ...

build: enable AddressSanitizer

AddressSanitizer [1] a.k.a. ASan is a widely-used debugging tool to
detect memory access errors.
It helps to detect issues like use-after-free, various kinds of buffer

build: enable AddressSanitizer

AddressSanitizer [1] a.k.a. ASan is a widely-used debugging tool to
detect memory access errors.
It helps to detect issues like use-after-free, various kinds of buffer
overruns in C/C++ programs, and other similar errors, as well as
printing out detailed debug information whenever an error is detected.

ASan is integrated with gcc and clang and can be enabled via a meson
option: -Db_sanitize=address
See the documentation for details (especially regarding clang).

Enabling ASan has an impact on performance since additional checks are
added to generated binaries.

Enabling ASan with Windows is currently not supported in DPDK.

1: https://github.com/google/sanitizers/wiki/AddressSanitizer

Signed-off-by: Xueqin Lin <xueqin.lin@intel.com>
Signed-off-by: Zhihong Peng <zhihongx.peng@intel.com>
Acked-by: John McNamara <john.mcnamara@intel.com>

show more ...

pcapng: add new library for writing pcapng files

This is utility library for writing pcapng format files
used by Wireshark family of utilities. Older tcpdump
also knows how to read (but not write) t

pcapng: add new library for writing pcapng files

This is utility library for writing pcapng format files
used by Wireshark family of utilities. Older tcpdump
also knows how to read (but not write) this format.

See
https://github.com/pcapng/pcapng/

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Acked-by: Reshma Pattan <reshma.pattan@intel.com>
Acked-by: Ray Kinsella <mdr@ashroe.eu>

show more ...

dmadev: introduce DMA device library

The 'dmadev' is a generic type of DMA device.

This patch introduce the 'dmadev' device allocation functions.

The infrastructure is prepared to welcome drivers

dmadev: introduce DMA device library

The 'dmadev' is a generic type of DMA device.

This patch introduce the 'dmadev' device allocation functions.

The infrastructure is prepared to welcome drivers in drivers/dma/

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Jerin Jacob <jerinj@marvell.com>
Reviewed-by: Kevin Laatz <kevin.laatz@intel.com>
Reviewed-by: Conor Walsh <conor.walsh@intel.com>

show more ...

doc: add Toeplitz hash guide

Add documentation for the Toeplitz hash library.

Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
Reviewed-by: Konstantin Ananyev <konstantin.ananyev@in

doc: add Toeplitz hash guide

Add documentation for the Toeplitz hash library.

Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
Reviewed-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Reviewed-by: John McNamara <john.mcnamara@intel.com>

show more ...

build: remove makefiles

A decision was made [1] to no longer support Make in DPDK, this patch
removes all Makefiles that do not make use of pkg-config, along with
the mk directory previously used by

build: remove makefiles

A decision was made [1] to no longer support Make in DPDK, this patch
removes all Makefiles that do not make use of pkg-config, along with
the mk directory previously used by make.

[1] https://mails.dpdk.org/archives/dev/2020-April/162839.html

Signed-off-by: Ciara Power <ciara.power@intel.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

show more ...

regexdev: introduce API

As RegEx usage become more used by DPDK applications, for example:
* Next Generation Firewalls (NGFW)
* Deep Packet and Flow Inspection (DPI)
* Intrusion Prevention Systems (

regexdev: introduce API

As RegEx usage become more used by DPDK applications, for example:
* Next Generation Firewalls (NGFW)
* Deep Packet and Flow Inspection (DPI)
* Intrusion Prevention Systems (IPS)
* DDoS Mitigation
* Network Monitoring
* Data Loss Prevention (DLP)
* Smart NICs
* Grammar based content processing
* URL, spam and adware filtering
* Advanced auditing and policing of user/application security policies
* Financial data mining - parsing of streamed financial feeds
* Application recognition.
* Dmemory introspection.
* Natural Language Processing (NLP)
* Sentiment Analysis.
* Big data database acceleration.
* Computational storage.

Number of PMD providers started to work on HW implementation,
along side with SW implementations.

This lib adds the support for those kind of devices.

The RegEx Device API is composed of two parts:
- The application-oriented RegEx API that includes functions to setup
a RegEx device (configure it, setup its queue pairs and start it),
update the rule database and so on.

- The driver-oriented RegEx API that exports a function allowing
a RegEx poll Mode Driver (PMD) to simultaneously register itself as
a RegEx device driver.

RegEx device components and definitions:

+-----------------+
| |
| o---------+ rte_regexdev_[en|de]queue_burst()
| PCRE based o------+ | |
| RegEx pattern | | | +--------+ |
| matching engine o------+--+--o | | +------+
| | | | | queue |<==o===>|Core 0|
| o----+ | | | pair 0 | | |
| | | | | +--------+ +------+
+-----------------+ | | |
^ | | | +--------+
| | | | | | +------+
| | +--+--o queue |<======>|Core 1|
Rule|Database | | | pair 1 | | |
+------+----------+ | | +--------+ +------+
| Group 0 | | |
| +-------------+ | | | +--------+ +------+
| | Rules 0..n | | | | | | |Core 2|
| +-------------+ | | +--o queue |<======>| |
| Group 1 | | | pair 2 | +------+
| +-------------+ | | +--------+
| | Rules 0..n | | |
| +-------------+ | | +--------+
| Group 2 | | | | +------+
| +-------------+ | | | queue |<======>|Core n|
| | Rules 0..n | | +-------o pair n | | |
| +-------------+ | +--------+ +------+
| Group n |
| +-------------+ |<-------rte_regexdev_rule_db_update()
| | | |<-------rte_regexdev_rule_db_compile_activate()
| | Rules 0..n | |<-------rte_regexdev_rule_db_import()
| +-------------+ |------->rte_regexdev_rule_db_export()
+-----------------+

RegEx: A regular expression is a concise and flexible means for matching
strings of text, such as particular characters, words, or patterns of
characters. A common abbreviation for this is â~@~\RegExâ~@~].

RegEx device: A hardware or software-based implementation of RegEx
device API for PCRE based pattern matching syntax and semantics.

PCRE RegEx syntax and semantics specification:
http://regexkit.sourceforge.net/Documentation/pcre/pcrepattern.html

RegEx queue pair: Each RegEx device should have one or more queue pair to
transmit a burst of pattern matching request and receive a burst of
receive the pattern matching response. The pattern matching
request/response embedded in *rte_regex_ops* structure.

Rule: A pattern matching rule expressed in PCRE RegEx syntax along with
Match ID and Group ID to identify the rule upon the match.

Rule database: The RegEx device accepts regular expressions and converts
them into a compiled rule database that can then be used to scan data.
Compilation allows the device to analyze the given pattern(s) and
pre-determine how to scan for these patterns in an optimized fashion that
would be far too expensive to compute at run-time. A rule database
contains a set of rules that compiled in device specific binary form.

Match ID or Rule ID: A unique identifier provided at the time of rule
creation for the application to identify the rule upon match.

Group ID: Group of rules can be grouped under one group ID to enable
rule isolation and effective pattern matching. A unique group identifier
provided at the time of rule creation for the application to identify
the rule upon match.

Scan: A pattern matching request through *enqueue* API.

It may possible that a given RegEx device may not support all the
features
of PCRE. The application may probe unsupported features through
struct rte_regexdev_info::pcre_unsup_flags

By default, all the functions of the RegEx Device API exported by a PMD
are lock-free functions which assume to not be invoked in parallel on
different logical cores to work on the same target object. For instance,
the dequeue function of a PMD cannot be invoked in parallel on two logical
cores to operates on same RegEx queue pair. Of course, this function
can be invoked in parallel by different logical core on different queue
pair. It is the responsibility of the upper level application to
enforce this rule.

In all functions of the RegEx API, the RegEx device is
designated by an integer >= 0 named the device identifier *dev_id*

At the RegEx driver level, RegEx devices are represented by a generic
data structure of type *rte_regexdev*.
RegEx devices are dynamically registered during the PCI/SoC device
probing phase performed at EAL initialization time.
When a RegEx device is being probed, a *rte_regexdev* structure and
a new device identifier are allocated for that device. Then, the
regexdev_init() function supplied by the RegEx driver matching the
probed device is invoked to properly initialize the device.

The role of the device init function consists of resetting the hardware
or software RegEx driver implementations.

If the device init operation is successful, the correspondence between
the device identifier assigned to the new device and its associated
*rte_regexdev* structure is effectively registered.
Otherwise, both the *rte_regexdev* structure and the device identifier
are freed.

The functions exported by the application RegEx API to setup a device
designated by its device identifier must be invoked in the following
order:
- rte_regexdev_configure()
- rte_regexdev_queue_pair_setup()
- rte_regexdev_start()

Then, the application can invoke, in any order, the functions
exported by the RegEx API to enqueue pattern matching job, dequeue
pattern matching response, get the stats, update the rule database,
get/set device attributes and so on

If the application wants to change the configuration (i.e. call
rte_regexdev_configure() or rte_regexdev_queue_pair_setup()), it must
call rte_regexdev_stop() first to stop the device and then do the
reconfiguration before calling rte_regexdev_start() again. The enqueue and
dequeue functions should not be invoked when the device is stopped.

Finally, an application can close a RegEx device by invoking the
rte_regexdev_close() function.

Each function of the application RegEx API invokes a specific function
of the PMD that controls the target device designated by its device
identifier.

For this purpose, all device-specific functions of a RegEx driver are
supplied through a set of pointers contained in a generic structure of
type *regexdev_ops*.
The address of the *regexdev_ops* structure is stored in the
*rte_regexdev* structure by the device init function of the RegEx driver,
which is invoked during the PCI/SoC device probing phase, as explained
earlier.

In other words, each function of the RegEx API simply retrieves the
*rte_regexdev* structure associated with the device identifier and
performs an indirect invocation of the corresponding driver function
supplied in the *regexdev_ops* structure of the *rte_regexdev*
structure.

For performance reasons, the address of the fast-path functions of the
RegEx driver is not contained in the *regexdev_ops* structure.
Instead, they are directly stored at the beginning of the *rte_regexdev*
structure to avoid an extra indirect memory access during their
invocation.

RTE RegEx device drivers do not use interrupts for enqueue or dequeue
operation. Instead, RegEx drivers export Poll-Mode enqueue and dequeue
functions to applications.

The *enqueue* operation submits a burst of RegEx pattern matching
request to the RegEx device and the *dequeue* operation gets a burst of
pattern matching response for the ones submitted through *enqueue*
operation.

Typical application utilisation of the RegEx device API will follow the
following programming flow.

- rte_regexdev_configure()
- rte_regexdev_queue_pair_setup()
- rte_regexdev_rule_db_update() Needs to invoke if precompiled rule
database not
provided in rte_regexdev_config::rule_db for rte_regexdev_configure()
and/or application needs to update rule database.
- rte_regexdev_rule_db_compile_activate() Needs to invoke if
rte_regexdev_rule_db_update function was used.
- Create or reuse exiting mempool for *rte_regex_ops* objects.
- rte_regexdev_start()
- rte_regexdev_enqueue_burst()
- rte_regexdev_dequeue_burst()

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Ori Kam <orika@mellanox.com>

show more ...

doc: remove outdated guidelines for library addition

There was a doc about how to extend DPDK by adding a library.
It could have been useful but was never updated,
so it is lacking a lot of explanat

doc: remove outdated guidelines for library addition

There was a doc about how to extend DPDK by adding a library.
It could have been useful but was never updated,
so it is lacking a lot of explanations about doxygen,
meson, versioning, maintainership, etc.

Anyway such guidelines should fit in the contributors guide.
Better to completely remove this obsolete document.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: David Marchand <david.marchand@redhat.com>

show more ...

doc: update telemetry guides

The existing documentation for Telemetry is updated, and further
documentation is added.

Signed-off-by: Ciara Power <ciara.power@intel.com>
Reviewed-by: Keith Wiles <ke

doc: update telemetry guides

The existing documentation for Telemetry is updated, and further
documentation is added.

Signed-off-by: Ciara Power <ciara.power@intel.com>
Reviewed-by: Keith Wiles <keith.wiles@intel.com>

show more ...

doc: add graph library guide

Adding programmer's guide for Graph library and the inbuilt nodes.
This patch also updates the release note for the new libraries.

Signed-off-by: Jerin Jacob <jerinj@ma

doc: add graph library guide

Adding programmer's guide for Graph library and the inbuilt nodes.
This patch also updates the release note for the new libraries.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>

show more ...

doc: add trace library guide

Add programmer's guide for trace library support.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
Acked-by: David Ma

doc: add trace library guide

Add programmer's guide for trace library support.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Sunil Kumar Kori <skori@marvell.com>
Acked-by: David Marchand <david.marchand@redhat.com>

show more ...