ethdev: optimize activation of fast path tracepoints

Split the tracepoints rte_ethdev_trace_rx_burst and
rte_eth_trace_call_rx_callbacks into two separate ones
for empty and non-empty calls to avoid

ethdev: optimize activation of fast path tracepoints

Split the tracepoints rte_ethdev_trace_rx_burst and
rte_eth_trace_call_rx_callbacks into two separate ones
for empty and non-empty calls to avoid saturating
quickly the trace buffer.

Signed-off-by: Adel Belkhiri <adel.belkhiri@polymtl.ca>
Acked-by: Jerin Jacob <jerinj@marvell.com>

show more ...

ethdev: add Tx queue used count query

Introduce a new API to retrieve the number of used descriptors
in a Tx queue. Applications can leverage this API in the fast path to
inspect the Tx queue occupa

ethdev: add Tx queue used count query

Introduce a new API to retrieve the number of used descriptors
in a Tx queue. Applications can leverage this API in the fast path to
inspect the Tx queue occupancy and take appropriate actions based on the
available free descriptors.

A notable use case could be implementing Random Early Discard (RED)
in software based on Tx queue occupancy.

Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@amd.com>

show more ...

lib: replace logging helpers

This is a preparation step before the next change.

Many libraries have their own logging helpers that do not add a newline
in their format string.
Some previous changes

lib: replace logging helpers

This is a preparation step before the next change.

Many libraries have their own logging helpers that do not add a newline
in their format string.
Some previous changes fixed places where some of those helpers are
called without a trailing newline.
Using RTE_LOG_LINE in the existing helpers will ensure we don't
introduce new issues in the future.

The problem is that if we simply convert to the RTE_LOG_LINE helper,
a future fix may introduce a regression since the logging helper
change won't be backported.

To address this concern, rename existing helpers: backporting a call to
them will trigger some conflict or build issue in LTS branches.

Note:
- bpf and vhost that still has some debug multilines messages, a direct
call to RTE_LOG/RTE_LOG_DP is used: this will make it easier to notice
such special cases,
- about previously publicly exposed logging helpers, when such helper is
not publicly used (iow in public inline API), it is removed from the
public API (this is the case for the member library),

Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

show more ...

lib: use dedicated logtypes and macros

No printf!
When a dedicated log helper exists, use it.
And no usurpation please: a library should log under its logtype
(see the eventdev rx adapter update for

lib: use dedicated logtypes and macros

No printf!
When a dedicated log helper exists, use it.
And no usurpation please: a library should log under its logtype
(see the eventdev rx adapter update for example).

Note: the RTE_ETH_VALID_PORTID_OR_GOTO_ERR_RET macro is renamed for
consistency with the rest of eventdev (private) macros.

Cc: stable@dpdk.org

Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

show more ...

ethdev: refresh shared memory reference in secondary process

In the process of releasing ethdev port in the primary process,
secondary processes are asked to stop referencing such a port.
Doing so,

ethdev: refresh shared memory reference in secondary process

In the process of releasing ethdev port in the primary process,
secondary processes are asked to stop referencing such a port.
Doing so, those processes can't predict whether the primary process
will later invalidate/free the shared memory. So they may live with a
reference to an old shared memory memzone and location in memory.

Refresh the shared memory pointer in secondary process when requesting
access to the shared memory: this is the best moment ethdev can check
it because the primary process will (re-)create a shared memory only if
no secondary process is referencing a ethdev port.

Bugzilla ID: 1303
Fixes: 36c46e738120 ("ethdev: cleanup shared data with the last port")

Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@amd.com>

show more ...

ethdev: use stdatomic API

Replace the use of gcc builtin __atomic_xxx intrinsics with
corresponding rte_atomic_xxx optional stdatomic API

Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com

ethdev: use stdatomic API

Replace the use of gcc builtin __atomic_xxx intrinsics with
corresponding rte_atomic_xxx optional stdatomic API

Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>
Acked-by: David Marchand <david.marchand@redhat.com>

show more ...

ethdev: cleanup shared data with the last port

If no port is allocated and no port owner is still registered,
ethdev from a primary process may release the memzone used to store
port data.
This make

ethdev: cleanup shared data with the last port

If no port is allocated and no port owner is still registered,
ethdev from a primary process may release the memzone used to store
port data.
This makes it possible for the DPDK memory allocator to release
associated resources back to the OS.

Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>

show more ...

ethdev: avoid panicking in absence of ethdev shared data

This is a preparation step before freeing the ethdev shared data
memzone.

Previously, because the primary process never freed the memzone, a

ethdev: avoid panicking in absence of ethdev shared data

This is a preparation step before freeing the ethdev shared data
memzone.

Previously, because the primary process never freed the memzone, a
secondary process could assume this memzone was present.
But in the next commit, this will change.

Make eth_dev_shared_data_prepare() report whether the memzone is
available so that upper level API can react accordingly.

Signed-off-by: David Marchand <david.marchand@redhat.com>

show more ...

ethdev: protect shared memory accesses under one lock

ethdev currently uses two locks to protect access around
eth_dev_shared_data:
- one (process local) for avoiding multiple threads to reserve/loo

ethdev: protect shared memory accesses under one lock

ethdev currently uses two locks to protect access around
eth_dev_shared_data:
- one (process local) for avoiding multiple threads to reserve/lookup
the eth_dev_shared_data memzone,
- one (shared with other processes) for protecting port
allocation/destruction,

A next change will make it possible for eth_dev_shared_data memzone to
be freed during a DPDK application lifetime. Accessing its content must
then be protected for concurrent accesses and with this new protection,
existing locks become unneeded.

Simplify the logic and put everything under a single lock in DPDK shared
memory config (which cannot be freed during a DPDK application lifetime).

Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>

show more ...

ethdev: add API for mbuf recycle mode

Add 'rte_eth_recycle_rx_queue_info_get' and 'rte_eth_recycle_mbufs'
APIs to recycle used mbufs from a transmit queue of an Ethernet device,
and move these mbufs

ethdev: add API for mbuf recycle mode

Add 'rte_eth_recycle_rx_queue_info_get' and 'rte_eth_recycle_mbufs'
APIs to recycle used mbufs from a transmit queue of an Ethernet device,
and move these mbufs into a mbuf ring for a receive queue of an Ethernet
device. This can bypass mempool 'put/get' operations hence saving CPU
cycles.

For each recycling mbufs, the rte_eth_recycle_mbufs() function performs
the following operations:
- Copy used *rte_mbuf* buffer pointers from Tx mbuf ring into Rx mbuf
ring.
- Replenish the Rx descriptors with the recycling *rte_mbuf* mbufs freed
from the Tx mbuf ring.

Suggested-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Suggested-by: Ruifeng Wang <ruifeng.wang@arm.com>
Signed-off-by: Feifei Wang <feifei.wang2@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>

show more ...

ethdev: add trace points

Adds trace points for ethdev functions.

The rte_ethdev_trace.h is removed. The file ethdev_trace.h is added as
an internal header. ethdev_trace.h contains internal slow pat

ethdev: add trace points

Adds trace points for ethdev functions.

The rte_ethdev_trace.h is removed. The file ethdev_trace.h is added as
an internal header. ethdev_trace.h contains internal slow path and
fast path tracepoints. The public fast path tracepoints are present in
rte_ethdev_trace_fp.h header.

Signed-off-by: Ankur Dwivedi <adwivedi@marvell.com>
Acked-by: Sunil Kumar Kori <skori@marvell.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@amd.com>

show more ...

remove extra blank line at EOF

These source files all had unnecessary blank line at end of file.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

ethdev: move driver interface functions to its own file

ethdev has two interfaces, one interface between applications and
library, these APIs are declared in the rte_ethdev.h public header.
Other in

ethdev: move driver interface functions to its own file

ethdev has two interfaces, one interface between applications and
library, these APIs are declared in the rte_ethdev.h public header.
Other interface is between drivers and library, these functions are
declared in ethdev_driver.h and marked as internal.

But all functions are defined in rte_ethdev.c file. This patch moves
functions for drivers to its own file, ethdev_driver.c for cleanup, no
functional change in functions.

Some public APIs and driver helpers call common internal functions,
which were mostly static since both were in same file. To be able to
move driver helpers, common functions are moved to ethdev_private.c.
(ethdev_private.c is used for functions that are internal to the library
and shared by multiple .c files in the ethdev library.)

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>

show more ...

ethdev: warn once when using port not ready

Warning continuously is a pain when developping or if a unit test
is/gets broken.

It could also be a problem if application behaves badly only in some
co

ethdev: warn once when using port not ready

Warning continuously is a pain when developping or if a unit test
is/gets broken.

It could also be a problem if application behaves badly only in some
corner cases and a DoS results of those logs being continuously displayed.

Let's warn once per port and per rx/tx.

Getting such a log is scary, but let's make it more eye catching by
dumping a backtrace with it.

Tested by introducing a bug in testpmd:
static int
eth_dev_start_mp(uint16_t port_id)
{
- if (is_proc_primary())
+ if (!is_proc_primary())
return rte_eth_dev_start(port_id);

return 0;

Then, running a basic null test:
$ ./devtools/test-null.sh
...
Start automatic packet forwarding
io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support
enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 2 streams:
RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

lcore 0 called rx_pkt_burst for not ready port 0
8: [build/app/dpdk-testpmd() [0x59e839]]
7: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff481b69555]]
6: [build/app/dpdk-testpmd(main+0x54b) [0x662d24]]
5: [build/app/dpdk-testpmd(start_packet_forwarding+0x263) [0x65e795]]
4: [build/app/dpdk-testpmd() [0x65e1be]]
3: [build/app/dpdk-testpmd() [0x65a996]]
2: [build/app/dpdk-testpmd() [0xa6cbc7]]
1: [build/app/dpdk-testpmd(rte_dump_stack+0x27) [0xaee796]]
lcore 0 called rx_pkt_burst for not ready port 1
8: [build/app/dpdk-testpmd() [0x59e839]]
7: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff481b69555]]
6: [build/app/dpdk-testpmd(main+0x54b) [0x662d24]]
5: [build/app/dpdk-testpmd(start_packet_forwarding+0x263) [0x65e795]]
4: [build/app/dpdk-testpmd() [0x65e1be]]
3: [build/app/dpdk-testpmd() [0x65a996]]
2: [build/app/dpdk-testpmd() [0xa6cbc7]]
1: [build/app/dpdk-testpmd(rte_dump_stack+0x27) [0xaee796]]
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=2
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0

Fixes: c87d435a4d79 ("ethdev: copy fast-path API into separate structure")

Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Tested-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>

show more ...

ethdev: make fast-path functions to use new flat array

Rework fast-path ethdev functions to use rte_eth_fp_ops[].
While it is an API/ABI breakage, this change is intended to be
transparent for both

ethdev: make fast-path functions to use new flat array

Rework fast-path ethdev functions to use rte_eth_fp_ops[].
While it is an API/ABI breakage, this change is intended to be
transparent for both users (no changes in user app is required) and
PMD developers (no changes in PMD is required).
One extra thing to note - RX/TX callback invocation will cause extra
function call with these changes. That might cause some insignificant
slowdown for code-path where RX/TX callbacks are heavily involved.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
Tested-by: Feifei Wang <feifei.wang2@arm.com>

show more ...

ethdev: copy fast-path API into separate structure

Copy public function pointers (rx_pkt_burst(), etc.) and related
pointers to internal data from rte_eth_dev structure into a
separate flat array. T

ethdev: copy fast-path API into separate structure

Copy public function pointers (rx_pkt_burst(), etc.) and related
pointers to internal data from rte_eth_dev structure into a
separate flat array. That array will remain in a public header.
The intention here is to make rte_eth_dev and related structures internal.
That should allow future possible changes to core eth_dev structures
to be transparent to the user and help to avoid ABI/API breakages.
The plan is to keep minimal part of data from rte_eth_dev public,
so we still can use inline functions for fast-path calls
(like rte_eth_rx_burst(), etc.) to avoid/minimize slowdown.
The whole idea beyond this new schema:
1. PMDs keep to setup fast-path function pointers and related data
inside rte_eth_dev struct in the same way they did it before.
2. Inside rte_eth_dev_start() and inside rte_eth_dev_probing_finish()
(for secondary process) we call eth_dev_fp_ops_setup, which
copies these function and data pointers into rte_eth_fp_ops[port_id].
3. Inside rte_eth_dev_stop() and inside rte_eth_dev_release_port()
we call eth_dev_fp_ops_reset(), which resets rte_eth_fp_ops[port_id]
into some dummy values.
4. fast-path ethdev API (rte_eth_rx_burst(), etc.) will use that new
flat array to call PMD specific functions.
That approach should allow us to make rte_eth_devices[] private
without introducing regression and help to avoid changes in drivers code.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
Tested-by: Feifei Wang <feifei.wang2@arm.com>

show more ...

lib: remove librte_ prefix from directory names

There is no reason for the DPDK libraries to all have 'librte_' prefix on
the directory names. This prefix makes the directory names longer and also
m

lib: remove librte_ prefix from directory names

There is no reason for the DPDK libraries to all have 'librte_' prefix on
the directory names. This prefix makes the directory names longer and also
makes it awkward to add features referring to individual libraries in the
build - should the lib names be specified with or without the prefix.
Therefore, we can just remove the library prefix and use the library's
unique name as the directory name, i.e. 'eal' rather than 'librte_eal'

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

show more ...