net/mlx5: fix real time counter reading from PCI BAR

There is the mlx5_txpp_read_clock() routine reading
the 64-bit real time counter from the device PCI BAR.
It introduced two issues:

- it check

net/mlx5: fix real time counter reading from PCI BAR

There is the mlx5_txpp_read_clock() routine reading
the 64-bit real time counter from the device PCI BAR.
It introduced two issues:

- it checks the PCI BAR mapping into process address
space and tries to map this on demand. This might be
problematic if something goes wrong and mapping fails.
It happens on every read_clock API call, invokes kernel
taking a long time and causing application malfunction.

- the 64-bit counter should be read in single atomic
transaction

Fixes: 9b31fc9007f9 ("net/mlx5: fix read device clock in real time mode")
Cc: stable@dpdk.org

Signed-off-by: Tim Martin <timothym@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

drivers: use stdatomic API

Replace the use of gcc builtin __atomic_xxx intrinsics with
corresponding rte_atomic_xxx optional rte stdatomic API.

Signed-off-by: Tyler Retzlaff <roretzla@linux.microso

drivers: use stdatomic API

Replace the use of gcc builtin __atomic_xxx intrinsics with
corresponding rte_atomic_xxx optional rte stdatomic API.

Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: Mattias Rönnblom <mattias.ronnblom@ericsson.com>

show more ...

net/mlx5: add timestamp order error statistics

The ConnectX NICs support packet send scheduling on specified
moment of time. Application can set the desired timestamp value
in dynamic mbuf field and

net/mlx5: add timestamp order error statistics

The ConnectX NICs support packet send scheduling on specified
moment of time. Application can set the desired timestamp value
in dynamic mbuf field and driver will push the special WAIT WQE
to the hardware queue in order to suspend the entire queue
operations till the specified time moment, then PMD pushes the
regular WQE for packet sending.

In the following packets the scheduling can be requested again,
with different timestamps, and driver pushes WAIT WQE accordingly.
The timestamps should be provided by application in ascending
order as packets are queued to the hardware queue, otherwise
hardware would not be able to perform scheduling correctly -
it discovers the WAIT WQEs in order as they were pushed, there is
no any reordering - neither in PMD, not in the NIC, and, obviously,
the regular hardware can't work as time machine and wait for some
elapsed moment in the past.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix build with GCC 12 and ASan

Building with gcc 12 and ASan raises this warning:

../drivers/net/mlx5/mlx5_txpp.c: In function ‘mlx5_txpp_xstats_get_names’:
../drivers/net/mlx5/mlx5_txpp.

net/mlx5: fix build with GCC 12 and ASan

Building with gcc 12 and ASan raises this warning:

../drivers/net/mlx5/mlx5_txpp.c: In function ‘mlx5_txpp_xstats_get_names’:
../drivers/net/mlx5/mlx5_txpp.c:1066:25: error: ‘strncpy’ specified bound
64 equals destination size [-Werror=stringop-truncation]
1066 | strncpy(xstats_names[i + n_used].name,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1067 | mlx5_txpp_stat_names[i],
| ~~~~~~~~~~~~~~~~~~~~~~~~
1068 | RTE_ETH_XSTATS_NAME_SIZE);
| ~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors

Prefer strlcpy for xstats.

Fixes: 3b025c0ca425 ("net/mlx5: provide send scheduling error statistics")
Cc: stable@dpdk.org

Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Raslan Darawsheh <rasland@nvidia.com>

show more ...

net/mlx5: fix read device clock in real time mode

Since ConnectX-6DX the real time timestamp mode is supported.
The rte_eth_read_clock() routine queries current timestamp
value from the PMD.

The ml

net/mlx5: fix read device clock in real time mode

Since ConnectX-6DX the real time timestamp mode is supported.
The rte_eth_read_clock() routine queries current timestamp
value from the PMD.

The mlx5 PMD has special infrastructure to schedule packet
sending in real time mode which can be engaged with tx_pp devarg.
This infrastructure provides the timestamp reading from the special
queue CEQs directly from the host memory in user space, without
involving kernel calls.

The ConnectX-7 NIC has hardware capability to schedule packet
sending without special infrastructure and tx_pp devarg can be
omitted. If there is no tx_pp devarg specified the mlx5 uses kernel
calls to query current timestamp value. The kernel can be completely
unaware about engaged real time mode, also kernel might use its
internal queue CQEs to get timestamps, that is neither precise nor
reliable, inconsistent values might be returned, causing send
scheduling malfunction.

The HCA PCI BAR provides the real time direct reading from hardware.
This patch maps PCI resource to the process address space on demand
and allows reading the real time timestamp values from the NIC
directly.

Fixes: b94d93ca73803 ("net/mlx5: support reading device clock")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

common/mlx5: share interrupt management

There are many duplicate code of creating and initializing rte_intr_handle.
Add a new mlx5_os API to do this, replace all PMD related code with this
API.

Sig

common/mlx5: share interrupt management

There are many duplicate code of creating and initializing rte_intr_handle.
Add a new mlx5_os API to do this, replace all PMD related code with this
API.

Signed-off-by: Spike Du <spiked@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net/mlx5: add shared device context config structure

Add configuration structure for shared device context. This structure
contains all configurations coming from devargs which oriented to
device. I

net/mlx5: add shared device context config structure

Add configuration structure for shared device context. This structure
contains all configurations coming from devargs which oriented to
device. It is a field of shared device context (SH) structure, and is
updated once in mlx5_alloc_shared_dev_ctx() function.
This structure cannot be changed when probing again, so add function to
prevent it. The mlx5_probe_again_args_validate() function creates a
temporary IB context configure structure according to new devargs
attached in probing again, then checks the match between the temporary
structure and the existing IB context configure structure.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net/mlx5: remove HCA attribute structure duplication

The HCA attribute structure is field of net configure structure.
It is also field of common configure structure.

There is no need for this dupli

net/mlx5: remove HCA attribute structure duplication

The HCA attribute structure is field of net configure structure.
It is also field of common configure structure.

There is no need for this duplication, because there is a reference to
the common structure from within the net structures.

This patch removes it from net configure structure and uses the common
config structure instead.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net/mlx5: fix mutex unlock in Tx packet pacing cleanup

The lock sh->txpp.mutex was not correctly released on one path
of cleanup function return, potentially causing the deadlock.

Fixes: d133f4cdb7

net/mlx5: fix mutex unlock in Tx packet pacing cleanup

The lock sh->txpp.mutex was not correctly released on one path
of cleanup function return, potentially causing the deadlock.

Fixes: d133f4cdb706 ("net/mlx5: create clock queue for packet pacing")
Cc: stable@dpdk.org

Signed-off-by: Chengfeng Ye <cyeaa@connect.ust.hk>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

common/mlx5: fix post doorbell barrier

The rdma-core library can map doorbell register in two ways, depending
on the environment variable "MLX5_SHUT_UP_BF":

- as regular cached memory, the variab

common/mlx5: fix post doorbell barrier

The rdma-core library can map doorbell register in two ways, depending
on the environment variable "MLX5_SHUT_UP_BF":

- as regular cached memory, the variable is either missing or set to
zero. This type of mapping may cause the significant doorbell
register writing latency and requires an explicit memory write
barrier to mitigate this issue and prevent write combining.

- as non-cached memory, the variable is present and set to not "0"
value. This type of mapping may cause performance impact under
heavy loading conditions but the explicit write memory barrier is
not required and it may improve core performance.

The UAR creation function maps a doorbell in one of the above ways
according to the system. In run time, it always adds an explicit memory
barrier after writing to.
In cases where the doorbell was mapped as non-cached memory, the
explicit memory barrier is unnecessary and may impair performance.

The commit [1] solved this problem for a Tx queue. In run time, it
checks the mapping type and provides the memory barrier after writing to
a Tx doorbell register if it is needed. The mapping type is extracted
directly from the uar_mmap_offset field in the queue properties.

This patch shares this code between the drivers and extends the above
solution for each of them.

[1] commit 8409a28573d3
("net/mlx5: control transmit doorbell register mapping")

Fixes: f8c97babc9f4 ("compress/mlx5: add data-path functions")
Fixes: 8e196c08ab53 ("crypto/mlx5: support enqueue/dequeue operations")
Fixes: 4d4e245ad637 ("regex/mlx5: support enqueue")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

drivers: remove direct access to interrupt handle

Removing direct access to interrupt handle structure fields,
rather use respective get set APIs for the same.
Making changes to all the drivers acce

drivers: remove direct access to interrupt handle

Removing direct access to interrupt handle structure fields,
rather use respective get set APIs for the same.
Making changes to all the drivers access the interrupt handle fields.

Signed-off-by: Harman Kalra <hkalra@marvell.com>
Acked-by: Hyong Youb Kim <hyonkim@cisco.com>
Signed-off-by: David Marchand <david.marchand@redhat.com>
Tested-by: Raslan Darawsheh <rasland@nvidia.com>

show more ...

net/mlx5: set Tx queue affinity in round-robin

Previously, we set txq affinity to 0 and let firmware
to perform round-robin when bonding. Firmware uses a
global counter to assign txq affinity to dif

net/mlx5: set Tx queue affinity in round-robin

Previously, we set txq affinity to 0 and let firmware
to perform round-robin when bonding. Firmware uses a
global counter to assign txq affinity to different
physical ports accord to remainder after division.

There are three dis-advantages:
1. The global counter is shared between kernel and dpdk.
2. After restarting pmd or port, the previous counter value
is reused, so the new affinity is unpredictable.
3. There is no way to get what affinity is set by firmware.

In this update, we will create several TISs up to the
number of bonding ports and bind each TIS to one PF port.

For each port, it will start to pick up TIS using its port
index. Upper layer application can quickly calculate each txq's
affinity without querying.

At DPDK layer, when creating txq with 2 bonding ports, the
affinity is set like:
port 0: 1-->2-->1-->2
port 1: 2-->1-->2-->1
port 2: 1-->2-->1-->2

Note: Only applicable to DevX api.
This affinity subjects to HW hash.

Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: share HCA capabilities handle

Add HCA attributes structure as a field of device config structure.
It query in common probing, and updates the timestamp format fields.

Each driver use H

common/mlx5: share HCA capabilities handle

Add HCA attributes structure as a field of device config structure.
It query in common probing, and updates the timestamp format fields.

Each driver use HCA attributes from common device config structure,
instead of query it for itself.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: share protection domain object

Create shared Protection Domain in common area and add it and its PDN as
fields of common device structure.

Use this Protection Domain in all drivers and

common/mlx5: share protection domain object

Create shared Protection Domain in common area and add it and its PDN as
fields of common device structure.

Use this Protection Domain in all drivers and remove the PD and PDN
fields from their private structure.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: share device context object

Create shared context device in common area and add it as a field of
common device.
Use this context device in all drivers and remove the ctx field from
thei

common/mlx5: share device context object

Create shared context device in common area and add it as a field of
common device.
Use this context device in all drivers and remove the ctx field from
their private structure.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net: rename Ethernet header fields

Definition of `rte_ether_addr` structure used a workaround allowing DPDK
and Windows SDK headers to be used in the same file, because Windows SDK
defines `s_addr`

net: rename Ethernet header fields

Definition of `rte_ether_addr` structure used a workaround allowing DPDK
and Windows SDK headers to be used in the same file, because Windows SDK
defines `s_addr` as a macro. Rename `s_addr` to `src_addr` and `d_addr`
to `dst_addr` to avoid the conflict and remove the workaround.
Deprecation notice:
https://mails.dpdk.org/archives/dev/2021-July/215270.html

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>

show more ...

net/mlx5: fix timestamp initialization on empty clock queue

The committing completions by clock queue might be delayed
after queue initialization is done and the only Clock Queue
completion entry (C

net/mlx5: fix timestamp initialization on empty clock queue

The committing completions by clock queue might be delayed
after queue initialization is done and the only Clock Queue
completion entry (CQE) might keep the invalid status till
the CQE first update happens.

The mlx5_txpp_update_timestamp() wrongly recognized invalid
status as error and reported about lost synchronization.

The patch recognizes the invalid status as "not updated yet"
and accurate scheduling initialization routine waits till
CQE first update happens.

Some collateral typos in comment are fixed as well.

Fixes: 77522be0a56d ("net/mlx5: introduce clock queue service routine")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: separate Tx function declarations to another file

This patch separates Tx function declarations to different header file
in preparation for removing their implementation from the source fi

net/mlx5: separate Tx function declarations to another file

This patch separates Tx function declarations to different header file
in preparation for removing their implementation from the source file
and as an optional preparation for Tx cleanup.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: separate Rx function declarations to another file

The mlx5_rxtx.c file contains a lot of Tx burst functions, each of those
is performance-optimized for the specific set of requested offloa

net/mlx5: separate Rx function declarations to another file

The mlx5_rxtx.c file contains a lot of Tx burst functions, each of those
is performance-optimized for the specific set of requested offloads.
These ones are generated on the basis of the template function and it
takes significant time to compile, just due to a large number of giant
functions generated in the same file and this compilation is not being
done in parallel with using multithreading.

Therefore we can split the mlx5_rxtx.c file into several separate files
to allow different functions to be compiled simultaneously.
In this patch, we separate Rx function declarations to different header
file in preparation for removing them from the source file and as an
optional preparation step for further consolidation of Rx burst
functions.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: support timestamp format

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC time

net/mlx5: support timestamp format

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 86fc67fc9315 ("net/mlx5: create advanced RxQ object via DevX")
Fixes: ae18a1ae9692 ("net/mlx5: support Tx hairpin queues")
Fixes: 15c3807e86ab ("common/mlx5: support DevX QP operations")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>

show more ...

ethdev: make driver-only headers private

The rte_ethdev_driver.h, rte_ethdev_vdev.h and rte_ethdev_pci.h files are
for drivers only and should be a private to DPDK and not installed.

Signed-off-by:

ethdev: make driver-only headers private

The rte_ethdev_driver.h, rte_ethdev_vdev.h and rte_ethdev_pci.h files are
for drivers only and should be a private to DPDK and not installed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Steven Webster <steven.webster@windriver.com>

show more ...

net/mlx5: move rearm and clock queue SQ creation to common

Using common function for DevX SQ creation for rearm and clock queue.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan A

net/mlx5: move rearm and clock queue SQ creation to common

Using common function for DevX SQ creation for rearm and clock queue.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net/mlx5: move rearm and clock queue CQ creation to common

Using common function for CQ creation at rearm queue and clock queue.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan A

net/mlx5: move rearm and clock queue CQ creation to common

Using common function for CQ creation at rearm queue and clock queue.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: fix completion queue entry size configuration

According to the current data-path implementation in the PMD the CQE
size must follow the cache-line size.
So, the configuration of the CQE

common/mlx5: fix completion queue entry size configuration

According to the current data-path implementation in the PMD the CQE
size must follow the cache-line size.
So, the configuration of the CQE size should be depended in
RTE_CACHE_LINE_SIZE.

Wrongly, part of the CQE creations didn't follow it exactly what caused
an incompatibility between HW and SW in the data-path when working in
128B cache-line size systems.

Adjust the rule for any CQE creation.
Remove the cqe_size attribute from the DevX CQ creation command and set
it inside the command translation according to the cache-line size.

Fixes: 79a7e409a2f6 ("common/mlx5: prepare support of packet pacing")
Fixes: 5cd0a83f413e ("common/mlx5: support more fields in DevX CQ create")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: wrap event channel functions per OS

Wrap the API to create/destroy event channel and to subscribe an event
with OS calls. In Linux those calls are implemented by glue functions
while in

common/mlx5: wrap event channel functions per OS

Wrap the API to create/destroy event channel and to subscribe an event
with OS calls. In Linux those calls are implemented by glue functions
while in Windows they are not supported.

Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...