net/mlx5: fix Tx tracing to use single clock source

A prior commit introduced tracing for mlx5, but there is a mixture of
two unrelated clocks used: the TSC for host work submission timestamps
and t

net/mlx5: fix Tx tracing to use single clock source

A prior commit introduced tracing for mlx5, but there is a mixture of
two unrelated clocks used: the TSC for host work submission timestamps
and the NIC HW clock for CQE completion times. It is necessary to have
timestamps from a single common clock, and the NIC HW clock is the
better choice since it can be used with externally synchronized clocks.

This patch adds the NIC HW clock as an additional logged parameter for
trace_tx_entry, trace_tx_exit, and trace_tx_wqe. The included trace
analysis python script is also updated to use the new clock when
it is available.

Fixes: a1e910f5b8d4 ("net/mlx5: introduce tracepoints")
Fixes: 9725191a7e14 ("net/mlx5: add Tx datapath trace analyzing script")
Cc: stable@dpdk.org

Signed-off-by: Tim Martin <timothym@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix real time counter reading from PCI BAR

There is the mlx5_txpp_read_clock() routine reading
the 64-bit real time counter from the device PCI BAR.
It introduced two issues:

- it check

net/mlx5: fix real time counter reading from PCI BAR

There is the mlx5_txpp_read_clock() routine reading
the 64-bit real time counter from the device PCI BAR.
It introduced two issues:

- it checks the PCI BAR mapping into process address
space and tries to map this on demand. This might be
problematic if something goes wrong and mapping fails.
It happens on every read_clock API call, invokes kernel
taking a long time and causing application malfunction.

- the 64-bit counter should be read in single atomic
transaction

Fixes: 9b31fc9007f9 ("net/mlx5: fix read device clock in real time mode")
Cc: stable@dpdk.org

Signed-off-by: Tim Martin <timothym@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: support flow match with external Tx queue

For using external created Tx queues in RTE_FLOW_ITEM_TX_QUEUE,
this commit provides the map and unmap functions to convert the
external created S

net/mlx5: support flow match with external Tx queue

For using external created Tx queues in RTE_FLOW_ITEM_TX_QUEUE,
this commit provides the map and unmap functions to convert the
external created SQ's devx ID to DPDK flow item Tx queue ID.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>

show more ...

drivers: use stdatomic API

Replace the use of gcc builtin __atomic_xxx intrinsics with
corresponding rte_atomic_xxx optional rte stdatomic API.

Signed-off-by: Tyler Retzlaff <roretzla@linux.microso

drivers: use stdatomic API

Replace the use of gcc builtin __atomic_xxx intrinsics with
corresponding rte_atomic_xxx optional rte stdatomic API.

Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: Mattias Rönnblom <mattias.ronnblom@ericsson.com>

show more ...

drivers: move alignment attribute on types for MSVC

Move location of __rte_aligned(a) to new conventional location. The new
placement between {struct,union} and the tag allows the desired
alignment

drivers: move alignment attribute on types for MSVC

Move location of __rte_aligned(a) to new conventional location. The new
placement between {struct,union} and the tag allows the desired
alignment to be imparted on the type regardless of the toolchain being
used for both C and C++. Additionally, it avoids confusion by Doxygen
when generating documentation.

Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>

show more ...

net/mlx5: replace zero length array with flex array

Zero length arrays are GNU extension. Replace with
standard flex array.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by

net/mlx5: replace zero length array with flex array

Zero length arrays are GNU extension. Replace with
standard flex array.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Dariusz Sosnowski <dsosnowski@nvidia.com>

show more ...

net/mlx5: fix multi-segment Tx inline data length

If packet data length exceeds the configured limit for packet
to be inlined in the queue descriptor the driver checks if hardware
requires to do min

net/mlx5: fix multi-segment Tx inline data length

If packet data length exceeds the configured limit for packet
to be inlined in the queue descriptor the driver checks if hardware
requires to do minimal data inline or the VLAN insertion offload is
requested and not supported in hardware (that means we have to do VLAN
insertion in software with inline data). Then driver scans the mbuf
chain to find the minimal segment amount to satisfy the data needed
for minimal inline.

There was incorrect first segment inline data length calculation
with missing VLAN header being inserted, that could lead to the
segmentation fault in the mbuf chain scanning, for example for
the packets:

packet:
mbuf0 pkt_len = 288, data_len = 156
mbuf1 pkt_len = 132, data_len = 132

txq->inlen_send = 290

The driver was trying to reach the inlen_send inline data length
with missing VLAN header length added and was running out of the
mbuf chain (there were just not enough data in the packet to satisfy
the criteria).

Fixes: 18a1c20044c0 ("net/mlx5: implement Tx burst template")
Fixes: ec837ad0fc7c ("net/mlx5: fix multi-segment inline for the first segments")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Suanming Mou <suanmingm@nvidia.com>

show more ...

net/mlx5: add comprehensive send completion trace

There is the demand to trace the send completions of
every WQE if time scheduling is enabled.

The patch extends the size of completion queue and
re

net/mlx5: add comprehensive send completion trace

There is the demand to trace the send completions of
every WQE if time scheduling is enabled.

The patch extends the size of completion queue and
requests completion on every issued WQE in the
send queue. As the result hardware provides CQE on
each completed WQE and driver is able to fetch
completion timestamp for dedicated operation.

The add code is under conditional compilation
RTE_ENABLE_TRACE_FP flag and does not impact the
release code.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: introduce tracepoints

There is an intention to engage DPDK tracing capabilities
for mlx5 PMDs monitoring and profiling in various modes.
The patch introduces tracepoints for the Tx datapat

net/mlx5: introduce tracepoints

There is an intention to engage DPDK tracing capabilities
for mlx5 PMDs monitoring and profiling in various modes.
The patch introduces tracepoints for the Tx datapath in
the ethernet device driver.

To engage this tracing capability the following steps
should be taken:

- meson option -Denable_trace_fp=true
- meson option -Dc_args='-DALLOW_EXPERIMENTAL_API'
- EAL command line parameter --trace=pmd.net.mlx5.tx.*

The Tx datapath tracing allows to get information how packets
are pushed into hardware descriptors, time stamping for
scheduled wait and send completions, etc.

To provide the human readable form of trace results the
dedicated post-processing script is presumed.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: add timestamp order error statistics

The ConnectX NICs support packet send scheduling on specified
moment of time. Application can set the desired timestamp value
in dynamic mbuf field and

net/mlx5: add timestamp order error statistics

The ConnectX NICs support packet send scheduling on specified
moment of time. Application can set the desired timestamp value
in dynamic mbuf field and driver will push the special WAIT WQE
to the hardware queue in order to suspend the entire queue
operations till the specified time moment, then PMD pushes the
regular WQE for packet sending.

In the following packets the scheduling can be requested again,
with different timestamps, and driver pushes WAIT WQE accordingly.
The timestamps should be provided by application in ascending
order as packets are queued to the hardware queue, otherwise
hardware would not be able to perform scheduling correctly -
it discovers the WAIT WQEs in order as they were pushed, there is
no any reordering - neither in PMD, not in the NIC, and, obviously,
the regular hardware can't work as time machine and wait for some
elapsed moment in the past.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: enhance Tx queue affinity

The rte_eth_dev_map_aggr_tx_affinity() was introduced in
ethdev lib, it was used to set the affinity value per Tx queue.

This patch adds the MLX5 PMD support for

net/mlx5: enhance Tx queue affinity

The rte_eth_dev_map_aggr_tx_affinity() was introduced in
ethdev lib, it was used to set the affinity value per Tx queue.

This patch adds the MLX5 PMD support for two device ops:
- map_aggr_tx_affinity
- count_aggr_ports

After maps a Tx queue with an aggregated port by call
map_aggr_tx_affinity() and starts sending traffic, the MLX5 PMD
updates TIS creation with tx_aggr_affinity value of Tx queue.
TIS index 1 goes to first physical port, TIS index 2 goes to second
physical port, and so on, TIS index 0 is reserved for default
HW hash mode.

Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix wait descriptor opcode for ConnectX-7

Since ConnectX-7 the special hardware capability to schedule
packet sending on specified time is provided. It uses special
WAIT WQE (hardware desc

net/mlx5: fix wait descriptor opcode for ConnectX-7

Since ConnectX-7 the special hardware capability to schedule
packet sending on specified time is provided. It uses special
WAIT WQE (hardware descriptor) with wait condition defined
by opcode. The CYCLIC_BIGGER was wrongly used instead of
CYCLIC_SMALLER. This caused arbitrary and stale time shift
in traffic scheduling.

Fixes: 49e8797619e40 ("net/mlx5: support wait on time in Tx")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix first segment inline length

Packets can be split into several mbufs with various data sizes.
There is no limitation on how small these segments can be.
But there is a limitation on Tx

net/mlx5: fix first segment inline length

Packets can be split into several mbufs with various data sizes.
There is no limitation on how small these segments can be.
But there is a limitation on Tx side for inline configuration:
send WQEs with inline headers less than the required are dropped.
The very first segment must be more than minimal inline eth segment.
Enforce this requirement by merging a few segments in this case.

Fixes: ec837ad0fc7c ("net/mlx5: fix multi-segment inline for the first segments")
Cc: stable@dpdk.org

Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: support device control for E-Switch default rule

This patch adds support for fdb_def_rule_en device argument to HW
Steering, which controls:

- the creation of the default FDB jump flow ru

net/mlx5: support device control for E-Switch default rule

This patch adds support for fdb_def_rule_en device argument to HW
Steering, which controls:

- the creation of the default FDB jump flow rule.
- the ability of the user to create transfer flow rules in the root
table.

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Signed-off-by: Xueming Li <xuemingl@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix Tx check for hardware descriptor length

If hardware descriptor (WQE) length exceeds one the HW can handle,
the Tx queue failure occurs. PMD does the length check but there was
a bug -

net/mlx5: fix Tx check for hardware descriptor length

If hardware descriptor (WQE) length exceeds one the HW can handle,
the Tx queue failure occurs. PMD does the length check but there was
a bug - the length limit was expressed in 16B units (WQEBB segments),
while the calculated WQE length and limit were in 64B units (WQEBBs).
Fix the condition to avoid subsequent Tx queue failure.

Fixes: 18a1c20 ("net/mlx5: implement Tx burst template")
Cc: stable@dpdk.org

Signed-off-by: Raja Zidane <rzidane@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix inline length exceeding descriptor limit

The hardware descriptor (WQE) length field is 6 bits wide
and we have the native limitation for the overall descriptor
length. To improve the P

net/mlx5: fix inline length exceeding descriptor limit

The hardware descriptor (WQE) length field is 6 bits wide
and we have the native limitation for the overall descriptor
length. To improve the PCIe bandwidth the packet data can be
inline into descriptor. If PMD was configured to inline large
amount of data it happened there was no enough space remaining
in the descriptor to specify all the packet data segments and
PMD rejected problematic packets.

The patch tries to adjust the inline data length conservatively
and allows to avoid error occurring.

Fixes: 18a1c20044c0 ("net/mlx5: implement Tx burst template")
Fixes: e2259f93ef45 ("net/mlx5: fix Tx when inlining is impossible")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Reviewed-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>

show more ...

net/mlx5: fix single not inline packet storing

The mlx5 PMD can inline packet data into transmitting descriptor (WQE)
and free mbuf immediately as data no longer needed, for non-inline
packets the m

net/mlx5: fix single not inline packet storing

The mlx5 PMD can inline packet data into transmitting descriptor (WQE)
and free mbuf immediately as data no longer needed, for non-inline
packets the mbuf pointer should be stored in elts array for coming
freeing on send completion. There was an optimization on storing
pointers in batch and there was missed storing mbuf for single
packet if non-inline was explicitly requested by flag.

Fixes: cacb44a09962 ("net/mlx5: add no-inline Tx flag")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix check for orphan wait descriptor

The mlx5 PMD supports send scheduling feature, it allows
to send packets at specified moment of time, to do that
PMD pushes special wait descriptor (WQ

net/mlx5: fix check for orphan wait descriptor

The mlx5 PMD supports send scheduling feature, it allows
to send packets at specified moment of time, to do that
PMD pushes special wait descriptor (WQE) to the hardware
queue and then pushes descriptor for packet data as usual.
If queue is close to be full or there is no enough elts
buffers to store mbufs being sent the data descriptors might
be not pushed and the orphan wait WQE (not followed by the
data) might reside in queue on tx_burst routine exit.

To avoid orphan wait WQEs there was the check for enough
free space in the queue WQE buffer and enough amount of the
free elts in queue mbuf storage. This check was incomplete
and did not cover all the cases for Enhanced Multi-Packet
Write descriptors.

Fixes: 2f827f5ea6e1 ("net/mlx5: support scheduling on send routine template")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix Rx/Tx stats concurrency

Queue statistics are being continuously updated in Rx/Tx burst
routines while handling traffic. In addition to that, statistics
can be reset (written with zeroe

net/mlx5: fix Rx/Tx stats concurrency

Queue statistics are being continuously updated in Rx/Tx burst
routines while handling traffic. In addition to that, statistics
can be reset (written with zeroes) on statistics reset in other
threads, causing a race condition, which in turn could result in
wrong stats.

The patch provides an approach with reference values, allowing
the actual counters to be writable within Rx/Tx burst threads
only, and updating reference values on stats reset.

Fixes: 87011737b715 ("mlx5: add software counters")
Cc: stable@dpdk.org

Signed-off-by: Raja Zidane <rzidane@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix Tx when inlining is impossible

When txq_inline_max is too large and an mbuf is multi-segment
it may be impossible to inline data and build a valid WQE,
because WQE length would be larg

net/mlx5: fix Tx when inlining is impossible

When txq_inline_max is too large and an mbuf is multi-segment
it may be impossible to inline data and build a valid WQE,
because WQE length would be larger then HW can represent.
It is impossible to detect misconfiguration at startup,
because the condition depends on the mbuf composition.
The check on the data path to prevent the error
treated the length limit as expressed in 64B units,
while the calculated length and limit are in 16B units.
Fix the condition to avoid subsequent TxQ failure and recovery.

Fixes: 18a1c20044c0 ("net/mlx5: implement Tx burst template")
Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: optimize queue type checks

The RxQ/TxQ control structure has a field named type. This type is enum
with values for standard and hairpin.
The use of this field is to check whether the queue

net/mlx5: optimize queue type checks

The RxQ/TxQ control structure has a field named type. This type is enum
with values for standard and hairpin.
The use of this field is to check whether the queue is of the hairpin
type or standard.

This patch replaces it with a boolean variable that saves whether it is
a hairpin.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net/mlx5: support wait on time in Tx

The hardware since ConnectX-7 supports waiting on
specified moment of time with new introduced wait
descriptor. A timestamp can be directly placed
into descripto

net/mlx5: support wait on time in Tx

The hardware since ConnectX-7 supports waiting on
specified moment of time with new introduced wait
descriptor. A timestamp can be directly placed
into descriptor and pushed to sending queue.
Once hardware encounter the wait descriptor the
queue operation is suspended till specified moment
of time. This patch update the Tx datapath to handle
this new hardware wait capability.

PMD documentation and release notes updated accordingly.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: configure Tx queue with send on time offload

The wait on time configuration flag is copied to the Tx queue
structure due to performance considerations. Timestamp
mask is prepared and store

net/mlx5: configure Tx queue with send on time offload

The wait on time configuration flag is copied to the Tx queue
structure due to performance considerations. Timestamp
mask is prepared and stored in queue structure as well.

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

ethdev: introduce generic dummy packet burst function

Multiple PMDs have dummy/noop Rx/Tx packet burst functions.

These dummy functions are very simple, introduce a common function in
the ethdev an

ethdev: introduce generic dummy packet burst function

Multiple PMDs have dummy/noop Rx/Tx packet burst functions.

These dummy functions are very simple, introduce a common function in
the ethdev and update drivers to use it instead of each driver having
its own functions.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>

show more ...

net/mlx5: fix inline length for multi-segment TSO

This patch removes a redundant assert in mlx5_tx_packet_multi_tso().
That assert assured that the amount of bytes requested to be inlined
is greater

net/mlx5: fix inline length for multi-segment TSO

This patch removes a redundant assert in mlx5_tx_packet_multi_tso().
That assert assured that the amount of bytes requested to be inlined
is greater than or equal to the minimum amount of bytes required
to be inlined. This requirement is either derived from the NIC
inlining mode or configured through devargs. When using TSO this
requirement can be disregarded, because on all NICs it is satisfied by
TSO inlining requirements, since TSO requires L2, L3, and L4 headers to
be inlined.

Fixes: 18a1c20044c0 ("net/mlx5: implement Tx burst template")
Cc: stable@dpdk.org

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...